V 0.5.2: object storage in the standard library

SigV4 in pure V, no dependencies

V 0.5.2 shipped on July 13th. Plenty landed in it, but the part I want to write about is one I contributed: net.s3, an S3-compatible client in the standard library, and the scheme dispatch that lets http.fetch follow an s3:// URL.

Why put it in the standard library#

Talking to object storage is a normal thing for a program to do, and until now it meant either pulling a third-party module or writing AWS Signature Version 4 by hand. Neither is appealing for a language whose selling point is a small self-contained binary with no dependencies.

V already had the pieces: crypto.hmac, crypto.sha256 and an HTTP client. net.s3 is SigV4 built on those, in pure V, with nothing else underneath. The same client speaks to AWS, to a self-hosted MinIO, or to any provider that implements the protocol, by changing the endpoint.

The shortest version#

import net.s3

fn main() {
	c := s3.new_client(s3.Credentials.from_env())

	c.put('hello.txt', 'Hi from V!'.bytes(), bucket: 'my-bucket')!
	text := c.get_string('hello.txt', bucket: 'my-bucket')!
	println(text)
}

That is the whole thing: a client, a put, a get_string. I ran exactly this against a live Clever Cloud Cellar bucket while writing this post, and it printed Hi from V! back.

Credentials nobody has to configure#

Credentials.from_env() is the part I am happiest with. Every provider names its environment variables differently, so the resolver takes the first non-empty one per field across all of them. Among them:

• key id: S3_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID, CELLAR_ADDON_KEY_ID, SCW_ACCESS_KEY, B2_APPLICATION_KEY_ID, R2_ACCESS_KEY_ID, SPACES_KEY
• secret: S3_SECRET_ACCESS_KEY, AWS_SECRET_ACCESS_KEY, CELLAR_ADDON_KEY_SECRET, SCW_SECRET_KEY, B2_APPLICATION_KEY, R2_SECRET_ACCESS_KEY, SPACES_SECRET
• endpoint: S3_ENDPOINT, AWS_ENDPOINT_URL, CELLAR_ADDON_HOST, B2_ENDPOINT, R2_ENDPOINT, SPACES_ENDPOINT

Session token and region resolve the same way, from S3_SESSION_TOKEN / AWS_SESSION_TOKEN and S3_REGION / AWS_REGION / AWS_DEFAULT_REGION / SCW_DEFAULT_REGION.

The practical effect: deploy that program on a platform that injects CELLAR_ADDON_* and it works with no configuration at all. Move it to a machine with AWS_* set and it still works. The code does not change and there is no adapter to write.

Presigned URLs, which are the useful trick#

A presigned URL grants time-limited access to one object, without handing out credentials:

url := c.presign('hello.txt', bucket: 'my-bucket', expires_in: 3600)!

The signature is worth proving. I generated one with a five minute expiry and fetched it with plain curl, no credentials anywhere:

$ curl -s "$URL"
Hi from V!

Then the same URL with the query string removed, so the signature is gone:

$ curl -s -o /dev/null -w "%{http_code}\n" "${URL%%\?*}"
403

The object is private and the signed link works, which is what I set out to check. The expiry I am taking from the protocol rather than from a test: SigV4 puts it in the signed query string, so the same URL stops verifying past expires_in. That is how you serve a user’s file without proxying it through your application.

s3:// as a URL scheme#

Importing net.s3 registers a scheme handler with net.http, so an s3:// URL becomes fetchable like any other:

resp := s3.fetch('s3://my-bucket/hello.txt')!
println(resp.body.bytestr())

Same output, same credentials resolution, and it also works through the generic http.fetch(url: 's3://...') route. Code that already takes a URL keeps taking a URL, and object storage becomes one more scheme instead of a separate API to integrate.

There is a File handle too, for call sites where repeating the bucket gets tedious:

f := c.file('hello.txt', bucket: 'my-bucket')
text := f.text()!
url := f.presign(expires_in: 3600)!

What else is in there#

stat returns size and content type, exists and size are shortcuts on it, and delete, create_bucket, delete_bucket and bucket_exists cover the management side. For large objects, upload_file picks single-shot or multipart on its own based on file size, and start_multipart returns a stateful uploader when you want to stream chunks you generate on the fly.

The unit test suite is offline and runs by default. The integration suite is gated behind S3_INTEGRATION=1 and needs a live endpoint, which is the right split: nobody’s v test should depend on the network.

The rest of 0.5.2#

The S3 client was one of eleven patches I got merged this cycle, and the rest divide into new modules and quiet fixes.

New: encoding.cbor implements RFC 8949, and the yaml module was split up with better conformance and performance.

Fixes worth knowing about if you hit them: rand now takes uuid_v4, uuid_v7, UUIDSession and ulid from the OS CSPRNG instead of a userspace generator, net canonicalises IPv6 per RFC 5952 in pure V so Ip6.str() stops emitting the deprecated ::a.b.c.d form on libc, net.http no longer hangs on HEAD, 1xx, 204 and 304 responses over Content-Length, strconv.atou64 stopped silently wrapping on overflow, and fmt keeps the enum name when it stringifies a fixed-array size expression.

The cli module also renders command groups, inherited flags, examples and a learn-more section in its --help, and mcp was aligned with the current revision of the spec. That last one deserves its own post.

If you want to look at how the S3 module is put together, it is PR #27008. And if you find a provider whose environment variables are not in the resolver yet, that is a three-line contribution and I would take the patch 😉

← → jump to the previous / next post