# davlgd tech blog --- # Projects _Open-source bits I maintain._ --- # Search _grep -r through the archive._ --- # Posts _All my posts — software/hardware experiments, Linux deep-dives, language tinkering._ --- # RFC Atlas: the whole RFC series, on a sphere _9834 documents, 65511 relationships_ The RFC series is almost ten thousand documents that constantly point at each other. One obsoletes three others, which update a fourth, which normatively references a dozen more. The index is a list, and a list is exactly the wrong shape for that. So I built [RFC Atlas](https://rfc.davlgd.com), which puts the corpus on a sphere and draws the relationships. Thus, you can see and explore the connections between Internet standards and how they evolve over time. ![RFC Atlas: 9834 RFCs on a Fibonacci sphere, with relationship arcs](/images/2026-08-rfc-atlas-sphere.webp) ## Why a sphere A citation graph laid out in two dimensions turns into a hairball at a few hundred nodes. There are 9834 RFCs. Atlas places them on a uniform Fibonacci sphere, which distributes points evenly over a surface with no clustering artefacts and no poles where everything piles up. Relationships are drawn as **geodesic arcs** that follow the surface instead of cutting through the middle, so an edge between two distant documents stays readable instead of becoming one more line through a solid ball of them. Everything renders in WebGL 2, points on the GPU, with orbit controls, inertia, zoom and raycasting to pick a document. There are 65511 edges in the published graph, and drawing that on the CPU is not a conversation worth having. ## Where the data comes from Three sources, and the split matters for trust: - **`rfc-index.xml`** from the RFC Editor, the official index, for metadata and the `updates` and `obsoletes` relationships. - **The IETF Datatracker API** for normative, informative and unclassified references, which the index does not carry. - **Stéphane Bortzmeyer's RFC index**, for a link to his analysis when he has written one. If you read RFCs in French you already know why [that link](https://www.bortzmeyer.org/rfcs.html) is worth having. A generator normalises and deduplicates all of it through a local SQLite cache, computes citation degrees, and writes a single self-contained `graph.json`. Conditional HTTP requests make refreshes cheap, and the artifact can be rebuilt offline from the cache. That last point is deliberate. The site loads one static file. There is no API behind it, nothing to keep running, and no request to a third party when you open a document. ## It is meant to be read, not only looked at A 3D toy that you cannot link to is a demo. So the parts that make it usable are the boring ones: search by number, title, author or keyword. Filters on publication range, status, stream and relationship type. Every RFC has its own route and the filters live in the URL. The pages are crawlable, shareable, with structured metadata and a sitemap, which for a WebGL application is not automatic and was an important part of the work. ## The command line half Atlas is the visual companion to [`ietf-tools`](https://github.com/davlgd/ietf-tools), a CLI I wrote in [V](/posts/2024-10-21-how-to-create-a-cli-vlang/) for the same corpus. The two answer different questions. The CLI answers one document at a time: ```bash $ rfc info 9110 RFC9110 — HTTP Semantics Authors: R. Fielding, Ed., M. Nottingham, Ed., J. Reschke, Ed. Date: June 2022 Status: INTERNET STANDARD Obsoletes: RFC2818, RFC7230, RFC7231, RFC7232, RFC7233, RFC7235, RFC7538, RFC7615, RFC7694 ``` Nine documents replaced by one. `xref` shows the graph around a document in both directions: ```bash $ rfc xref 2616 RFC2616 — Hypertext Transfer Protocol -- HTTP/1.1 Obsoletes: 2068 Hypertext Transfer Protocol -- HTTP/1.1 (January 1997) Obsoleted by: 7230 Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing (June 2014) 7231 Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content (June 2014) 7232 Hypertext Transfer Protocol (HTTP/1.1): Conditional Requests (June 2014) 7233 Hypertext Transfer Protocol (HTTP/1.1): Range Requests (June 2014) 7234 Hypertext Transfer Protocol (HTTP/1.1): Caching (June 2014) 7235 Hypertext Transfer Protocol (HTTP/1.1): Authentication (June 2014) Updated by: 2817 Upgrading to TLS Within HTTP/1.1 (May 2000) 5785 Defining Well-Known Uniform Resource Identifiers (URIs) (April 2010) 6266 Use of the Content-Disposition Header Field in the Hypertext Transfer Protocol (HTTP) (June 2011) 6585 Additional HTTP Status Codes (April 2012) ``` One document replaced by six and amended by four more, in three directions at once. There is also `search`, `track` for an Internet-Draft's Datatracker state, `errata`, `iana` for registry lookups, and `latest`. Everything caches locally and `--offline` never touches the network, which is the behaviour I want from a tool I use on a train. It installs the way I install everything now: ```bash mise use -g github:davlgd/ietf-tools ``` ## What it is for Honestly, curiosity first. Watching the HTTP lineage light up across the sphere, or seeing which documents everything points at, is the kind of thing you cannot get from an index page. The practical use is the one that started it: working out what is still current. "Is this RFC obsolete, and by what?" is a question with a graph-shaped answer, and until now the way to answer it was to open a lot of tabs. Both are open source. If a relationship type is missing or a rendering choice annoys you, [the issues](https://github.com/davlgd/rfc-atlas/issues) are open 😉 --- # uutils: what a Rust coreutils changes for you _One binary, 114 commands_ I have spent a year of posts poking at small UNIX commands on three systems: macOS, Debian and Ubuntu. Along the way Ubuntu swapped its coreutils for a Rust reimplementation, first in 25.10 and now in the 26.04 LTS. So the last post in this run is about the thing that was quietly under all the others. ## What actually changed [uutils](https://github.com/uutils/coreutils) is a from-scratch rewrite of coreutils in Rust, and Ubuntu made it the default. GNU is still installed, under a prefix: ```bash $ readlink -f /usr/bin/ls /usr/lib/cargo/bin/coreutils/ls $ /usr/bin/ls --version | head -1 ls (uutils coreutils) 0.8.0 $ gnuls --version | head -1 ls (GNU coreutils) 9.7 ``` Both are there, one command apart. That is a good decision: when something behaves oddly, you can compare against the reference implementation without installing anything. ## One binary wearing 114 hats The packaging surprised me more than the language did: ```bash $ ls -l /usr/lib/cargo/bin/coreutils/ | head -3 total 1264032 -rwxr-xr-x 115 root root 11352352 [ -rwxr-xr-x 115 root root 11352352 arch ``` Every entry is 11 MB, and the link count is 115. They are hard links to a single multi-call binary that decides what to do from `argv[0]`, the same trick BusyBox uses. That matters when you compare sizes. Line up one command against its GNU twin and the result looks damning: ```bash $ ls -l /usr/bin/gnuls /usr/lib/cargo/bin/coreutils/ls 150568 /usr/bin/gnuls 11352352 /usr/lib/cargo/bin/coreutils/ls ``` Seventy-five times bigger. Except that 11 MB is paid once for all 114 commands, so the honest comparison is the whole set: ```bash $ du -sh /usr/lib/cargo/bin/coreutils/ 11M $ du -ch /usr/bin/gnu* | tail -1 5.7M ``` Eleven megabytes against 5.7, for 114 commands against 104. Roughly double, which is a real cost and not the catastrophe the per-file number suggests. ## Does it behave the same? This is the question that matters, and I have a year of accidental test cases to answer it with. **It matched GNU everywhere I looked.** [`seq -s`](/posts/2026-03-25-seq-command/) produced byte-identical output including the trailing newline. [`factor`](/posts/2026-02-25-factor-primes/) agreed on every number I threw at it, including a 39-digit Mersenne prime. [`tr`](/posts/2026-05-06-tr-translate/) reproduced GNU's behaviour down to the surprises, including mapping `hello` to `world` as `wolld` and treating a UTF-8 `é` as two bytes. Not "close enough": the same bytes. The one difference I found was not a bug in either. Ubuntu's split between the two projects is not clean: ```bash $ readlink -f /usr/bin/true /usr/bin/gnutrue $ readlink -f /usr/bin/false /usr/lib/cargo/bin/coreutils/false ``` `true` is C, `false` is Rust, on the same machine. The two smallest programs on the system come from different projects, and their `--help` text differs accordingly. Nothing breaks, but it tells you the migration is per-command rather than wholesale. ## What you actually get The pitch is memory safety, and it is a reasonable one for code that parses arguments and walks filesystems as root. For a user, the visible benefits are smaller than the debate suggests. Better error messages in places, some commands faster and some slower, and a project that takes GNU compatibility seriously enough to run the GNU test suite against itself. The visible risks are equally modest, and they are about the long tail. The common paths are well covered. An obscure flag combination, a locale edge case, a behaviour some script has depended on for twenty years: that is where a reimplementation earns its scars, and it is why keeping GNU one prefix away was the right call. ## What to do about it Nothing, mostly. If you use these commands the way most people do, you will not notice. Two things are worth the effort. When something behaves unexpectedly on Ubuntu, run the `gnu`-prefixed version before you blame your script, because that comparison takes five seconds and settles it. And in anything portable, do not assume which implementation you are on: after a year of testing across macOS, Debian and Ubuntu, the differences that actually bit me were BSD against GNU, not GNU against Rust. That is the note to end this run of posts on. Every one of them found something I did not expect, and it was almost never in the command. It was in the gap between two implementations that both claim to do the same thing 😉 --- # MCP in V: shipping against a moving specification _Aligned to the spec, then the spec moved_ I mentioned at the end of [the `net.s3` post](/posts/2026-07-22-v-net-s3/) that another of my patches in V 0.5.2 deserved its own article. This is it: `mcp`, the [Model Context Protocol](https://modelcontextprotocol.io/) module, which I rewrote to match the 2025-11-25 revision. Three weeks before this post, the specification moved again. That turns out to be the more interesting story. ## What MCP is, briefly MCP is how an AI assistant talks to the outside world. A server exposes tools it can call, resources it can read and prompts it can use, and the assistant discovers them at runtime instead of having them hardcoded. JSON-RPC 2.0 underneath, over stdio for local processes or HTTP for remote ones. The interesting part for a standard library is that it is a *protocol*, not an API. There is a document, it has revisions, and either you match one or you do not. ## What the rewrite did A first `mcp` module had landed in V earlier in the cycle, covering the server side against an older revision. The specification had moved to 2025-11-25, and the gap was not cosmetic: transports, session handling, several new capabilities and a pile of metadata clients had started to expect. So the module targets that revision, on both sides: ```v pub const protocol_version = '2025-11-25' ``` Five thousand lines across the module. Full coverage of JSON-RPC 2.0, stdio and Streamable HTTP transports with SSE and sessions, `Origin` validation for DNS rebinding protection, tools, resources with subscriptions, prompts, completions, logging, progress notifications with cooperative cancellation, and the server-initiated calls `roots/list`, `sampling/createMessage` and `elicitation/create`. Two things are deliberately deferred and marked as such: the experimental tasks utility, and OAuth authorization, which the spec lists as a `SHOULD`. ## Seeing it work The module ships a demo server exercising every capability. Being stdio, you can drive it with `printf`, which is my favourite property of a line-oriented protocol. The whole exchange, pasteable: ```bash { printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"demo","version":"1"}}}' printf '%s\n' '{"jsonrpc":"2.0","method":"notifications/initialized"}' printf '%s\n' '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"echo","arguments":{"text":"bonjour depuis MCP"}}}' } | ./server ``` The handshake reply, in full: ```json {"jsonrpc":"2.0","result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{"listChanged":true},"resources":{"listChanged":true,"subscribe":true},"prompts":{"listChanged":true},"logging":{},"completions":{}},"serverInfo":{"name":"v.mcp.showcase","version":"1.0.0","title":"V MCP Showcase","description":"Reference server for vlib/mcp covering every capability of the 2025-11-25 spec.","websiteUrl":"https://vlang.io","icons":[{"src":"https://vlang.io/img/v-logo.png","mimeType":"image/png","sizes":["256x256"]}]},"instructions":"Demo server exercising every MCP capability shipped by vlib/mcp."},"id":1} ``` It advertises what it can do, including that its tool and resource lists can change and that resources support subscription. `serverInfo` carries the display metadata the revision added, down to an icon. Then the call: ```json {"jsonrpc":"2.0","result":{"content":[{"type":"text","text":"{\"text\":\"bonjour depuis MCP\"}"}],"isError":false},"id":2} ``` Three tools in that demo: an `echo`, a `count_to` that emits progress notifications and honours cancellation, and a `delete_record` that demonstrates destructive-operation annotations without deleting anything. ## Then the specification moved On July 28th, three weeks before this post, MCP published [the 2026-07-28 revision](https://blog.modelcontextprotocol.io/posts/2026-07-28/). It is not an increment. It turns MCP from a bidirectional stateful protocol into a stateless request/response one, so a server can sit behind a load balancer with no session affinity. The `initialize` and `initialized` exchange is retired: each request now carries its protocol version, client identity and capabilities in `_meta`, with an optional `server/discover` for capability discovery. Roots, Sampling and Logging are deprecated, with at least twelve months of support. Server-initiated requests give way to Multi Round-Trip Requests, and there is header-based routing through `Mcp-Method` and `Mcp-Name`. Read that against the section above and the awkwardness is plain. **The handshake I just showed you is the thing the current revision removes.** V's module speaks 2025-11-25, and on the day this post is dated that is one revision behind. ## What that is actually like I do not think this is a failure, and I am not going to pretend the module is current when it is not. A specification with dated revisions is a moving target by construction, and anything implementing it is a photograph. The module matched the spec when it was merged, it matches a named revision today, and it declares which one in a public constant rather than leaving you to find out from a runtime error. That last part is the property that matters: `protocol_version` is not decoration, it is the contract you can check before you deploy. The alternative model, where a client and a server negotiate a version at connection time, is exactly what 2026-07-28 removes, and there is a lesson in that too. Stateless request/response is easier to operate and harder to version, so the revision moves version identity into every request. For anyone using it now: the module is correct against 2025-11-25, which is what a large share of deployed clients still speak. Anything targeting 2026-07-28 needs work that has not been done yet, and the twelve-month deprecation window on the retired features is the budget for doing it. The module is [PR #27133](https://github.com/vlang/v/pull/27133), the revision it implements is [published in full](https://modelcontextprotocol.io/specification/2025-11-25), and the one that supersedes it is linked above. If you want to help close that gap, the issue is open 😉 --- # mktemp: stop inventing your own temporary files _Two lines that make a script safe_ Every shell script eventually needs somewhere to put something. The usual answer looks reasonable and has three separate problems: ```bash tmp="/tmp/myapp.$$" ``` The name is predictable, since `$$` is just the PID. It collides, because PIDs are reused. And it is left behind on every path out of the script that is not the happy one. That third problem is the one behind the temporary-file dance I described when [`sponge`](/posts/2025-07-16-sponge-command/) came up: write to `tmp`, then `mv` it into place, and hope nothing exits in between. ## mktemp does the whole job ```bash $ mktemp /tmp/tmp.NJNZmoAnih $ mktemp -d /tmp/tmp.M8TB94Q5qR ``` Random name, and the file or directory is **created atomically**. That is the part that matters: there is no window between "pick a name" and "create it" where somebody else can get there first, the race that turns a temporary file into a symlink attack. The permissions are right by default too: ```bash $ ls -l "$(mktemp)" -rw------- $ ls -ld "$(mktemp -d)" drwx------ ``` `0600` and `0700`. Owner only, no group, no world, without you having to remember a `chmod`. ## Cleaning up, properly Creating it is half the job. This is the pattern worth memorising: ```bash #!/bin/bash tmp=$(mktemp -d) trap 'rm -rf "$tmp"' EXIT echo "working in $tmp" touch "$tmp/data" ``` `trap ... EXIT` runs on **every** exit: success, failure, `set -e` aborting, or a signal. I checked the failing case specifically, since that is the one people assume leaks: ```bash $ bash fail.sh created /tmp/tmp.52TglmlH79 $ echo $? 1 $ ls -d /tmp/tmp.* | wc -l 0 ``` Exit code 1, directory gone. Nothing to clean up by hand tomorrow. Note the single quotes around the trap command. With double quotes, `$tmp` expands when the trap is *installed* rather than when it fires, and if it were empty at that moment you would be running `rm -rf ""`. I made exactly that mistake writing this post, and my test left two directories behind instead of removing them. ## Naming it something you recognise Random names are unhelpful when you are looking at `/tmp` wondering what left this there. A template fixes that, and the `X`es are where the randomness goes: ```bash $ mktemp /tmp/demo.XXXXXX /tmp/demo.XFmHa9 ``` At least six `X`es, and they have to be at the end. `-t` does something similar with a prefix, and it puts the result in the system temporary directory rather than a path you chose. ## Where the files actually go `mktemp` honours `TMPDIR`: ```bash $ TMPDIR=/var/tmp mktemp /var/tmp/tmp.3Kc5Y0jyWV ``` That is worth knowing because macOS sets it, to a per-user directory: ```bash $ mktemp /var/folders/f3/sjyq2d_10sl210w1x336yt080000gn/T/tmp.KAhZi1OzIv ``` Not `/tmp`. A script that assumes its temporary files are in `/tmp`, or that greps for them there, behaves differently on a Mac. Take the path `mktemp` gives you and never construct it yourself. The distinction between the two locations matters too. `/tmp` is commonly cleared on reboot, `/var/tmp` is meant to survive it. For anything that should still be there after a crash, pass `TMPDIR=/var/tmp` deliberately. ## Portability `mktemp` is on macOS, Debian and Ubuntu. The templates, `-d` and `TMPDIR` behaved identically everywhere I tested. `-t` is the exception: on GNU it is deprecated and takes a template, on BSD it takes a bare prefix. If your script needs a prefix, give a full template with `X`es instead and the ambiguity disappears. --- # env: the shebang line, and why it breaks _One argument, two behaviours_ I promised this one [while writing about `expect`](/posts/2025-05-07-expect-command/), where a script ran on my Mac and failed on Linux because of its first line. The cause is worth a post of its own, and `env` does more than that line suggests anyway. ## What the shebang actually does `#!` at the very start of an executable file tells the kernel which interpreter to run it with. The kernel reads that line, and executes the named program with your file as an argument. `#!/bin/bash script.sh`, effectively. That means the path has to be exact, because the kernel does not search `$PATH`. `#!/usr/bin/python3` fails on a machine where Python lives elsewhere, which is most machines with a virtualenv, a Homebrew install or a version manager. Hence the idiom: ```bash #!/usr/bin/env python3 ``` `env` *is* at a fixed path, and its job here is to look up `python3` in `$PATH` and exec it. One indirection, and the script becomes portable. ## Where it breaks Add an argument and the two systems disagree. Same file, same content: ```bash #!/usr/bin/env python3 -u print(1) ``` On macOS it runs and prints `1`. On Ubuntu: ```bash $ ./a.py env: 'python3 -u': No such file or directory env: use -[v]S to pass options in shebang lines ``` Linux passes everything after the interpreter path as a **single argument**, so `env` is looking for a program literally named `python3 -u`, space included. macOS splits it. Neither is wrong, the behaviour was never specified, and the error message tells you the fix: ```bash #!/usr/bin/env -S python3 -u ``` `-S` splits the string itself. I checked it on both systems and both print `1`, so that is the portable spelling. It is a GNU coreutils extension that BSD picked up, so it is safe on anything current and will confuse a very old system. The other option is to take no arguments at all, and set what you need inside the script. For Python, `-u` has an equivalent environment variable, and for most interpreters something similar exists. ## env is not only for shebangs Three flags make it useful on the command line. Setting a variable for one command, without touching your shell: ```bash $ env GREETING=bonjour sh -c 'echo $GREETING' bonjour ``` You can do that with `GREETING=bonjour cmd` in most shells, and `env` is the version that works everywhere, including where the command is not a simple name. Removing one, which the assignment syntax cannot do: ```bash $ export FOO=1 $ sh -c 'echo [$FOO]' [1] $ env -u FOO sh -c 'echo [$FOO]' [] ``` And starting from nothing at all, with `-i`: ```bash $ env -i sh -c 'env | wc -l' 1 ``` One variable left. That is the flag for reproducing "it works on my machine": run the failing command under `env -i` and add variables back one at a time until it works. Whatever you added last is your answer. Called with no arguments, `env` just prints the environment, which is the same as `printenv` and a slightly shorter thing to type. ## The takeaway `#!/usr/bin/env prog` for portability, `#!/usr/bin/env -S prog --flag` when you need an argument, and never `#!/usr/bin/env prog --flag`, however well it works on the machine in front of you. That last form is the trap, because it fails only when your script reaches someone else's system. Which is exactly when you are not there to fix it. --- # V 0.5.2: object storage in the standard library _SigV4 in pure V, no dependencies_ [V](https://vlang.io/) 0.5.2 shipped on July 13th. Plenty landed in it, but the part I want to write about is one I contributed: `net.s3`, an S3-compatible client in the standard library, and the scheme dispatch that lets `http.fetch` follow an `s3://` URL. ## Why put it in the standard library Talking to object storage is a normal thing for a program to do, and until now it meant either pulling a third-party module or writing AWS Signature Version 4 by hand. Neither is appealing for a language whose selling point is a small self-contained binary with no dependencies. V already had the pieces: `crypto.hmac`, `crypto.sha256` and an HTTP client. `net.s3` is SigV4 built on those, in pure V, with nothing else underneath. The same client speaks to AWS, to a self-hosted MinIO, or to any provider that implements the protocol, by changing the endpoint. ## The shortest version ```v import net.s3 fn main() { c := s3.new_client(s3.Credentials.from_env()) c.put('hello.txt', 'Hi from V!'.bytes(), bucket: 'my-bucket')! text := c.get_string('hello.txt', bucket: 'my-bucket')! println(text) } ``` That is the whole thing: a client, a `put`, a `get_string`. I ran exactly this against a live [Clever Cloud Cellar](https://www.clever-cloud.com/developers/doc/addons/cellar/) bucket while writing this post, and it printed `Hi from V!` back. ## Credentials nobody has to configure `Credentials.from_env()` is the part I am happiest with. Every provider names its environment variables differently, so the resolver takes the first non-empty one per field across all of them. Among them: - key id: `S3_ACCESS_KEY_ID`, `AWS_ACCESS_KEY_ID`, `CELLAR_ADDON_KEY_ID`, `SCW_ACCESS_KEY`, `B2_APPLICATION_KEY_ID`, `R2_ACCESS_KEY_ID`, `SPACES_KEY` - secret: `S3_SECRET_ACCESS_KEY`, `AWS_SECRET_ACCESS_KEY`, `CELLAR_ADDON_KEY_SECRET`, `SCW_SECRET_KEY`, `B2_APPLICATION_KEY`, `R2_SECRET_ACCESS_KEY`, `SPACES_SECRET` - endpoint: `S3_ENDPOINT`, `AWS_ENDPOINT_URL`, `CELLAR_ADDON_HOST`, `B2_ENDPOINT`, `R2_ENDPOINT`, `SPACES_ENDPOINT` Session token and region resolve the same way, from `S3_SESSION_TOKEN` / `AWS_SESSION_TOKEN` and `S3_REGION` / `AWS_REGION` / `AWS_DEFAULT_REGION` / `SCW_DEFAULT_REGION`. The practical effect: deploy that program on a platform that injects `CELLAR_ADDON_*` and it works with no configuration at all. Move it to a machine with `AWS_*` set and it still works. The code does not change and there is no adapter to write. ## Presigned URLs, which are the useful trick A presigned URL grants time-limited access to one object, without handing out credentials: ```v url := c.presign('hello.txt', bucket: 'my-bucket', expires_in: 3600)! ``` The signature is worth proving. I generated one with a five minute expiry and fetched it with plain `curl`, no credentials anywhere: ```bash $ curl -s "$URL" Hi from V! ``` Then the same URL with the query string removed, so the signature is gone: ```bash $ curl -s -o /dev/null -w "%{http_code}\n" "${URL%%\?*}" 403 ``` The object is private and the signed link works, which is what I set out to check. The expiry I am taking from the protocol rather than from a test: SigV4 puts it in the signed query string, so the same URL stops verifying past `expires_in`. That is how you serve a user's file without proxying it through your application. ## s3:// as a URL scheme Importing `net.s3` registers a scheme handler with `net.http`, so an `s3://` URL becomes fetchable like any other: ```v resp := s3.fetch('s3://my-bucket/hello.txt')! println(resp.body.bytestr()) ``` Same output, same credentials resolution, and it also works through the generic `http.fetch(url: 's3://...')` route. Code that already takes a URL keeps taking a URL, and object storage becomes one more scheme instead of a separate API to integrate. There is a `File` handle too, for call sites where repeating the bucket gets tedious: ```v f := c.file('hello.txt', bucket: 'my-bucket') text := f.text()! url := f.presign(expires_in: 3600)! ``` ## What else is in there `stat` returns size and content type, `exists` and `size` are shortcuts on it, and `delete`, `create_bucket`, `delete_bucket` and `bucket_exists` cover the management side. For large objects, `upload_file` picks single-shot or multipart on its own based on file size, and `start_multipart` returns a stateful uploader when you want to stream chunks you generate on the fly. The unit test suite is offline and runs by default. The integration suite is gated behind `S3_INTEGRATION=1` and needs a live endpoint, which is the right split: nobody's `v test` should depend on the network. ## The rest of 0.5.2 The S3 client was one of eleven patches I got merged this cycle, and the rest divide into new modules and quiet fixes. New: `encoding.cbor` implements RFC 8949, and the `yaml` module was split up with better conformance and performance. Fixes worth knowing about if you hit them: `rand` now takes `uuid_v4`, `uuid_v7`, `UUIDSession` and `ulid` from the OS CSPRNG instead of a userspace generator, `net` canonicalises IPv6 per RFC 5952 in pure V so `Ip6.str()` stops emitting the deprecated `::a.b.c.d` form on libc, `net.http` no longer hangs on HEAD, 1xx, 204 and 304 responses over `Content-Length`, `strconv.atou64` stopped silently wrapping on overflow, and `fmt` keeps the enum name when it stringifies a fixed-array size expression. The `cli` module also renders command groups, inherited flags, examples and a learn-more section in its `--help`, and `mcp` was aligned with the current revision of the spec. That last one deserves its own post. If you want to look at how the S3 module is put together, it is [PR #27008](https://github.com/vlang/v/pull/27008). And if you find a provider whose environment variables are not in the resolver yet, that is a three-line contribution and I would take the patch 😉 --- # time: two of them, and you use the wrong one _The shell keyword hides the real command_ Everyone times a command the same way, and almost nobody has run the program they think they are running. There are two `time`s on your system, they print different things, and the one you get by default is not a program at all. ## The one you type ```bash $ type time time is a shell keyword ``` Not a builtin, a **keyword**: part of the shell's grammar, like `if` or `while`. That is why `time` can measure a whole pipeline, which a normal command could never do. Its output depends on your shell. `bash` gives you three lines: ```bash $ time sleep 0.3 real 0m0.304s user 0m0.000s sys 0m0.003s ``` `zsh` puts the same information on one: ```bash $ time sleep 0.3 sleep 0.3 0,00s user 0,00s system 0% cpu 0,309 total ``` Same three numbers. `real` is wall clock, `user` is CPU spent in your code, `sys` is CPU spent in the kernel on your behalf. When `real` is much larger than `user + sys`, your program was waiting on something rather than computing. ## The one you have to ask for There is also a binary, and the shell keyword hides it. Call it by path: ```bash $ /usr/bin/time sleep 0.3 0.00user 0.00system 0:00.30elapsed 1%CPU (0avgtext+0avgdata 7504maxresident)k 0inputs+0outputs (0major+449minor)pagefaults 0swaps ``` Same three timings, plus **peak memory** (`maxresident`), page faults and I/O counts. That memory number is the reason to know this command exists: the keyword cannot tell you how much RAM a build used, and this can. GNU's version goes further with `-v`: ```bash $ /usr/bin/time -v sleep 0.1 Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.10 Maximum resident set size (kbytes): 7488 Voluntary context switches: 2 Involuntary context switches: 0 ``` Context switch counts distinguish a process that yielded politely from one the scheduler interrupted, which is a useful signal when something is slower than it should be. For scripts, `-f` gives you exactly the fields you want: ```bash $ /usr/bin/time -f "%e s, %M KB max" sleep 0.2 0.20 s, 7432 KB max ``` `%e` elapsed seconds, `%M` peak resident kilobytes. Two numbers, parseable, ready for a CSV of build measurements. ## macOS has neither of those flags The binary is there, and it is the BSD one: ```bash $ /usr/bin/time sleep 0.3 0,31 real 0,00 user 0,00 sys ``` Three numbers, no memory, no page faults. The detailed mode exists under a different letter, `-l` instead of `-v`: ```bash $ /usr/bin/time -l sleep 0.1 0,10 real 0,00 user 0,00 sys 1228800 maximum resident set size ``` Note that BSD reports the peak in **bytes** where GNU reports kilobytes, so the same process looks a thousand times bigger. And the custom format does not exist at all: ```bash $ /usr/bin/time -f "%e" /usr/bin/time: illegal option -- f usage: time [-al] [-h | -p] [-o file] utility [argument ...] ``` For a portable script, `brew install gnu-time` provides the GNU implementation alongside the BSD one. ## Which to reach for The keyword for a quick look, and for anything with a pipe in it, since it is the only one that can measure the whole thing. The binary when you care about memory, or when you want a machine-readable line. And remember it is `/usr/bin/time`, in full, every time: typing `time` gets you the keyword no matter how much you meant otherwise. A last note on measuring at all: one run tells you very little. `time` reports what happened once, on a machine that was doing other things. For anything you intend to act on, run it several times and look at the spread, or use a tool built for benchmarking. --- # script: record and replay your terminal _A session you can play back_ Copying a terminal session into a bug report loses everything: the colours, the order, the pauses, and the half of the output you did not select. `script` records the session itself, and its companion plays it back at the speed it happened. ## Recording The simplest form wraps a command and writes everything to a file: ```bash $ script -q -c "echo bonjour; date +%Y" sess.log $ cat sess.log Script started on 2026-08-31 11:53:28+00:00 [COMMAND="echo bonjour; date +%Y" ] bonjour 2026 Script done on 2026-08-31 11:53:28+00:00 [COMMAND_EXIT_CODE="0"] ``` `-q` suppresses `script`'s own chatter, `-c` gives it a command instead of an interactive shell. Note what the header and footer carry: the exact command, a timestamp, and the exit code. That is a lot of context you did not have to remember to include. Without `-c` it starts a shell and records until you exit, the form for "reproduce the bug while I watch". ## Replaying, with the original timing The interesting part needs a second file. `-T` records how long each chunk took: ```bash $ script -q -T timing.log -c "echo un; sleep 1; echo deux" sess2.log $ ls -l timing.log sess2.log 187 sess2.log 22 timing.log ``` Twenty-two bytes of timing beside the transcript. Then `scriptreplay` plays it: ```bash $ scriptreplay timing.log sess2.log un deux ``` You cannot see it in a paste, but `deux` arrives a second after `un`, exactly as it did during recording. A build that hangs for forty seconds hangs for forty seconds on replay. That is precisely the information a transcript throws away. Note the argument order: `scriptreplay `. Passing them the other way round gives `cannot open typescript`, which is how I learned it. ## What it is good for Demonstrations, mostly. A recorded session replays at real speed with no risk of a typo, and it is a text file, so it diffs and it goes in a repository. Then there is the awkward-question case: recording an incident while you work through it, so the postmortem has what you actually typed instead of what you remember typing. `script -q -T timing.log incident.log` at the start of the session costs nothing. It is also the honest way to capture output that a plain redirect mangles. `script` allocates a pseudo-terminal, so programs that check `isatty()` behave as they would for a human: they keep their colours, their progress bars and their line buffering. That makes it a general workaround for the [buffering problem](/posts/2025-08-13-stdbuf-buffering/), and it is why `script -qec` shows up in CI configs. ## macOS is a different command `script` exists there, and the similarity ends about there: ```bash $ script -h usage: script [-aeFkpqr] [-t time] [file [command ...]] script -p [-deq] [-T fmt] [file] ``` The command goes *after* the filename rather than behind a `-c`, so the Linux invocation fails outright. `scriptreplay` does not exist at all: BSD folds playback into `script -p`, which expects a file recorded with matching options, and I did not manage to get a round trip working in the time I gave it. For anything cross-platform, record with `script` for the transcript and leave the timing to a tool built for it. [asciinema](https://asciinema.org/) does the same job with one CLI on both systems, and produces something you can embed in a page. One last caveat, and it matters: `script` records **everything**, including what you type. A password entered during a recorded session is in that file in plain text. Check a transcript before you attach it to a ticket. --- # smolvm: microVMs for code you did not write _Three hypervisors behind one API_ An agent that writes code eventually wants to run it, and a container is a thin wall for that. [SmolVM](https://github.com/CelestoAI/SmolVM) puts a hardware boundary there instead: a microVM per workload, started in about a second, thrown away afterwards. ## Three backends, one API The interesting design choice is that it does not pick a hypervisor for you. It wraps three: - **Firecracker**, Linux with KVM only, a deliberately narrow set of virtual devices. - **QEMU**, Linux and macOS, broad hardware emulation, the one you need for a Windows guest. - **libkrun**, Linux and macOS, using Apple's Hypervisor.framework on the Mac and KVM on Linux, and marked experimental. Selection is explicit argument, then `SMOLVM_BACKEND`, then a platform default: QEMU on macOS, Firecracker on Linux. `smolvm doctor` tells you where you stand, and it was blunt about my Mac: ```text ╭──── SmolVM Doctor ─────╮ │ Backend: qemu │ │ Platform: Darwin arm64 │ │ Result: FAIL │ │ Failures: 2 │ ╰────────────────────────╯ ``` QEMU is not installed here, so no sandboxes for me today. A diagnostic that names the backend, the platform and the missing dependency before you waste an hour is worth more than most features. ## What it adds on top This is the part that justifies the project instead of telling you to use Firecracker directly. Running a microVM by hand means building a rootfs, setting up a TAP device, writing routes and sysctls, wiring vsock, and then driving whichever control interface your hypervisor exposes. Firecracker takes commands over an HTTP API on a Unix socket. QEMU has its monitor. libkrun has neither. SmolVM ships `smolvm-core`, a Rust helper that does the plumbing: Linux networking with TAP devices, routes and sysctls, sparse disk copy with zstd decompression so an image starts without being fully unpacked, QEMU monitor control for pause, resume and snapshots, and the Firecracker API socket. For libkrun specifically, that abstraction is doing the most work and delivering the least. libkrun has **no pause, no resume, no snapshots, and experimental vsock**, so anything checkpoint-shaped is unavailable there. The unified API does not invent those capabilities, it reports honestly that this backend lacks them, which is the right call and worth knowing before you pick it for its startup time. ## Driving it Three ways, and the third is the one that changes how you use it. The CLI, for a human: ```bash smolvm sandbox create --name my-sandbox smolvm sandbox shell my-sandbox smolvm sandbox snapshot create my-sandbox smolvm sandbox pause my-sandbox ``` The Python API, for a script: ```python from smolvm import SmolVM with SmolVM() as vm: result = vm.run("echo 'Hello from the sandbox!'") print(result.stdout.strip()) ``` And an **HTTP API**, which is how anything that is not Python talks to it: ```bash $ smolvm server start ``` That is `smolvm server`, "Run the local SmolVM HTTP API". It turns the whole thing into a service on the host: your agent, whatever it is written in, creates and destroys sandboxes over HTTP instead of shelling out. There is a local dashboard too, behind `smolvm ui`. The sandbox verbs are the same across all three: `create`, `list`, `info`, `shell`, `file` for copying in and out, `port` for forwarding, `env` for variables, `pause`, `resume`, `snapshot`, `delete`. ## Where the agent angle shows The CLI has verbs I did not expect in an infrastructure tool: `smolvm claude start`, `smolvm codex start`, `smolvm pi start`, `smolvm hermes start`, `smolvm openclaw start`. Each starts a sandbox with that agent's CLI preinstalled. The bare names are command groups and only print help, which caught me out. That tells you exactly who this is for. It is not a general VM manager that agents happen to use, it is built for the case where something autonomous needs a computer it can break. Which is also the honest reason to prefer it to a container here. A container shares your kernel, and "the agent wrote something that escaped" is a real category of problem and not a theoretical one. On cost, be precise, because the marketing and the measurements disagree. The project's own [benchmark report](https://github.com/CelestoAI/SmolVM) puts time-to-first-command at **1177 ms on QEMU and 1408 ms on Firecracker**, mean of five runs. Launching the hypervisor is the cheap part, tens to a couple of hundred milliseconds; the guest boot is what takes the second. Warm commands afterwards are around 1 ms over vsock. So a hypervisor boundary costs you roughly a second per sandbox, not the tens of milliseconds a container would. Version 0.0.24 is what I looked at, and the leading zeros are doing real work: the CLI surface has moved between releases. Pin it. --- # lsof: everything is a file, and here is the proof _Who is holding that open?_ "Everything is a file" is the line everyone quotes about UNIX and nobody demonstrates. `lsof` lists open files, and once you see what it counts as a file, the slogan stops being a slogan. ## Who is holding this open The first use is the one you reach for when a filesystem will not unmount or a file will not delete: ```bash $ sleep 300 > /tmp/held.log 2>&1 & $ lsof /tmp/held.log COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sleep 6817 root 1w REG 0,81 0 167898 /tmp/held.log sleep 6817 root 2w REG 0,81 0 167898 /tmp/held.log ``` Two lines for one file, because the process holds two descriptors on it: `1w` is stdout and `2w` is stderr, both redirected there and both open for writing. The `FD` column is the useful one, and `w` versus `r` tells you which direction. ## The trick that solves a real emergency Here is the situation. The disk is full. `du` says the directory is nearly empty. Nothing adds up. ```bash $ rm -f /tmp/held.log $ lsof | grep deleted sleep 6817 root 1w REG 0,81 0 167898 /tmp/held.log (deleted) ``` The file is gone from the directory, and it is still there on disk, because a process has it open. UNIX only frees the blocks when the last descriptor closes. `du` walks directory entries and sees nothing, `df` counts blocks and sees a full disk, and both are telling the truth. This is almost always a log file that was rotated while the writing process kept its old descriptor. `lsof | grep deleted` finds it in seconds, and restarting or `HUP`-ing that process gets the space back. Deleting harder does not. ## What a process has open Point it at a PID and you get the other view: ```bash $ lsof -p 1 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME tail 1 root cwd DIR 0,81 4096 160641 / tail 1 root rtd DIR 0,81 4096 160641 / tail 1 root txt REG 0,81 11352352 146240 /usr/lib/cargo/bin/coreutils/tail ``` Those `FD` values are not numbers. `cwd` is the working directory, `rtd` the root directory, `txt` the executable itself. A directory is a file, the running binary is a file, and every shared library it loaded will be in the rest of that list. That is the "everything is a file" claim, printed out. Sockets and [named pipes](/posts/2025-08-27-mkfifo-named-pipe/) show up in the same table, with `-i` narrowing to network connections: ```bash lsof -i -P -n # all network activity lsof -i :443 # who is on port 443 ``` `-P` keeps ports numeric and `-n` skips reverse DNS, which makes it faster and stops it hanging on a slow resolver. ## Useful combinations ```bash lsof -u alice # everything one user has open lsof +D /var/log # everything under a directory tree lsof -c nginx # everything a named command has open ``` `+D` is the one to remember when a mount point refuses to release. It recurses, so it is slow on a large tree, and `lsof /path` on the mount point itself is usually enough. ## On macOS It is there, but in `/usr/sbin` rather than `/usr/bin`, so a script hardcoding the path will miss it: ```bash $ command -v lsof /usr/sbin/lsof ``` The version is also well behind: 4.91 on my Mac against 4.99.4 on Ubuntu. The basics behave identically, and I got the same output shape for `lsof ` on both. Two caveats. Without `sudo` you only see your own processes, so an empty result means "nothing of yours" and not "nothing". And on a busy machine a bare `lsof` produces tens of thousands of lines, so always narrow it with a file, a PID, a user or `-i` before you look. --- # strings, file and xxd: looking inside a binary _What is this file, really?_ You have a file with no extension, or the wrong one, or one you do not trust. Three commands answer three different questions about it, and none of them needs a debugger or a hex editor. ## file, for what it is `file` ignores the name entirely and reads the contents: ```bash $ file -b 2026-05-verne-static-generator.webp RIFF (little-endian) data, Web/P image, VP8 encoding, 1792x1024, Scaling: [none]x[none], YUV color, decoders should clamp ``` Not just "an image": the container, the codec, the dimensions and the colour model. `-b` drops the filename from the output, what you want in a script. The proof that it reads bytes and not names is to take the extension away: ```bash $ cp 2026-05-verne-static-generator.webp noext $ file -b noext RIFF (little-endian) data, Web/P image, VP8 encoding, 1792x1024, Scaling: [none]x[none], YUV color, decoders should clamp ``` Identical. For scripting, `--mime-type` gives you something parseable: ```bash $ file -b --mime-type 2026-05-verne-static-generator.webp image/webp ``` That is the check to put in front of any upload handler, because the extension a user sends you is a suggestion and the bytes are not. ## xxd, for the actual bytes When `file` says something surprising, look: ```bash $ xxd -l 16 2026-05-verne-static-generator.webp 00000000: 5249 4646 7291 0000 5745 4250 5650 3820 RIFFr...WEBPVP8 ``` `-l 16` limits it to the first sixteen bytes, where the magic number lives. `RIFF`, a size, then `WEBP` and `VP8`. That is exactly what `file` pattern-matched on, and seeing it makes the whole mechanism concrete. The flag people miss is `-r`, which reverses the dump back into bytes: ```bash $ printf 'Hello' | xxd 00000000: 4865 6c6c 6f Hello $ printf 'Hello' | xxd | xxd -r Hello ``` A round trip. That makes `xxd` an editor: dump a file to hex, patch a byte with `sed`, reverse it back. Ugly, and occasionally the only way to fix a corrupted header. `od -c` covers the other common need, showing whitespace as escapes: ```bash $ printf 'a\tb\n' | od -c 0000000 a \t b \n 0000004 ``` That is how you find out whether a file uses tabs or spaces, and whether it ends in a newline. ## strings, for the text inside `strings` extracts printable sequences from binary data: ```bash $ strings -n 6 /usr/bin/gnutrue | head -3 /lib64/ld-linux-x86-64.so.2 __libc_start_main __cxa_finalize ``` `-n` sets the minimum run length, and raising it filters out noise: the default of 4 gives thousands of accidental matches. `strings /bin/ls` returns 59042 lines on my test box, unusable until you grep it. What it is good for is finding what a binary knows about: an embedded path, a version string, an error message you saw but cannot locate in the source, a hardcoded URL. It is the first thing to run on a binary you did not build. Two things it is not. It is not a decompiler, since it only recovers literals. And it is not a security check, because anything obfuscated or compressed will not show up at all. ## What you get where All three are on macOS and on Debian and Ubuntu. `file` gave byte-identical output on both systems in my tests, down to the "decoders should clamp". The one difference is `strings`. On Linux it is GNU binutils and takes `--version` and the long options. On macOS it comes from the Xcode toolchain, and the same call fails: ```bash $ strings --version error: /Applications/Xcode.app/.../usr/bin/strings: unknown flag: --version ``` `-n` works on both, which covers most uses. For anything more, `brew install binutils` gets you the GNU one. A closing note on `file` and containers, since it caught me while testing this: on a system where `/bin/ls` is a symlink, `file` reports the link rather than following it. `file -L` dereferences, and the difference is worth knowing before you conclude that a binary is missing. --- # Verne: I rewrote my blog's generator, in V _From Astro to Hugo to my own, in 22 hours_ This blog ran on [Astro and the AstroPaper theme](/posts/2023-12-how-this-blog-was-built/) from December 2023. It worked, I upgraded it a dozen times, and by May I was on AstroPaper 5.5.1. Then in two days it moved to Hugo, and then to a static site generator I wrote myself. Here is how that happened, and why the second step was smaller than the first. ## Why leave Astro Nothing was broken, which is the awkward part. AstroPaper is a good theme and Astro is a good framework. The friction was everything around it. A `node_modules` to install before I could render three hundred lines of Markdown. Framework upgrades that arrived faster than my posts did. And a build that depended on an ecosystem I do not otherwise use, for a site that is text, a stylesheet and some images. ## Step one: Hugo, and a theme built from the old one The first move was to Hugo, with a custom theme instead of an off-the-shelf one, because I wanted the site to look exactly as it did. I used [Claude Code](https://claude.com/claude-code) for the conversion: point it at the existing Astro templates and the rendered output, and have it produce Hugo templates that generate the same HTML from the same content files. That is a good task for a coding agent. The specification is unambiguous, since the target output already exists and can be diffed, and the work is mechanical without being mindless. The commit tells the story better than I can: ```text feat: migrate from Astro to Hugo with custom terminal-garden theme 198 files changed, 2089 insertions(+), 20567 deletions(-) ``` Twenty thousand lines removed. Almost none of it was mine. ## Step two: writing the generator Hugo lasted about twenty-two hours. Once the templates existed and the content was plain Markdown with YAML frontmatter, the shape of the problem was visible: read files, parse frontmatter, sort by date, render templates, write HTML. Hugo does an enormous amount more than that, and I needed none of it. So I wrote one, in [V](/posts/2024-02-how-own-web-server-vlang/), which is where I go for [small self-contained binaries](/posts/2024-10-21-how-to-create-a-cli-vlang/). The goal was narrow: produce byte-identical output to the Hugo build, from the same content and the same templates, and nothing else. That constraint is what made it feasible. There was no design work, because the target was "whatever Hugo just did", and a diff told me when I was done. The second migration commit is a fraction of the first: ```text feat: migrate to Verne build tool and template 53 files changed, 524 insertions(+), 778 deletions(-) hugo.yaml => verne.yaml ``` Most of it is that rename. The templates barely moved, since the syntax was deliberately close. ## What Verne is [Verne](https://github.com/davlgd/verne) is a static site generator in V, published with a site of its own at [verne-ssg.org](https://www.verne-ssg.org) that is, of course, built with it. The repository description is the whole joke: "The other static site generator named after a famous French author". There were already two. [Hugo](https://gohugo.io/) is Go and Victor Hugo, [Zola](https://www.getzola.org/) is Rust and Émile Zola, and Jules Verne was still available. If you want another Jules V. to read, try [Jules Vallès](https://en.wikipedia.org/wiki/Jules_Vall%C3%A8s). The borrowing goes past the name, too. Zola's template engine is [Tera](https://keats.github.io/tera/), itself in the Jinja2 family, and Verne's syntax is a deliberate subset of it. A template from this blog looks like this: ```html {% for s in site.footer_socials %}{% if not loop.first %} · {% endif %}{{ s.label }}{% endfor %} ``` Anyone who has written a Jinja2, Tera or Twig template can read that without documentation, which was the point. The design goals are what I wanted from Hugo without the rest: - **A single binary, small enough to not think about.** No Go toolchain, no Node, no asset pipeline. The 0.2.0 release runs 1.73 MB on macOS arm64, 2.05 MB on Linux arm64 and 2.31 MB on Linux x86_64. - **A template language small enough to read in one sitting**: `{{ value | filter }}` for output, `{% statement %}` for control flow, and 21 filters that only display things. No arithmetic, no macros. - **All the work at build time.** Aggregation, sorting and remote fetching happen during the build, so templates stay display-only. - **A dev server with live reload**, watching content, themes, static files and configuration. Themes came later, and those I generated with [Claude Design](https://claude.com/product/claude-design) before wiring them into the template system. ## Deploying it The build produces a directory of files, so the hosting question answers itself. It runs on [Clever Cloud](https://www.clever-cloud.com/) with the [static runtime](https://developers.clever-cloud.com/doc/applications/static/), which serves a folder and nothing else. The part worth stealing is how the build gets its tools. [mise](https://mise.jdx.dev/) has a [GitHub backend](https://mise.jdx.dev/dev-tools/backends/github.html) that installs release binaries straight from a repository, so the entire toolchain is two lines: ```toml [tools] "github:alecthomas/chroma" = "latest" "github:davlgd/verne" = "latest" ``` No package to publish, no registry to wait for. Tag a release on GitHub and every machine that runs `mise install` gets it, including the build container. For a tool with one user that is exactly the right amount of distribution. Scheduled publishing survived the move too, and it is still the [trick from 2024](/posts/2024-01-schedule-posts-astropaper/): a cron entry restarts the application, which rebuilds the site, and posts dated in the future appear when their time comes. ```json ["CRON_TZ=Europe/Paris 30 13 * * * clever restart --quiet --without-cache --app ${APP_ID}"] ``` And writing this post is how I found out that line is wrong. `13:30` there is Paris time, so 11:30 UTC in summer. Every post is dated `13:37:00Z`, which is UTC. The rebuild therefore runs **two hours and seven minutes before** the post is due, sees a future date, and skips it. The post appears at the next day's rebuild, a day late, every time. I had read `30 13` and `13:37` as seven minutes apart because they look it. They are in different timezones. The fix is to run the cron after the publication instant in UTC, or to stop mixing the two and date posts in Paris time as well. ## Was it worth it For most people, no. Writing a static site generator to publish a blog is not a rational use of a weekend, and Hugo would have served me for years. What I got is a build with no dependencies I do not control, a template language I can hold in my head, and a binary I can copy anywhere. What I gave up is every feature I have not written yet, and the certainty that someone else will fix the bugs. The honest summary is that the Hugo step did the hard work. Getting from Astro to plain Markdown, YAML frontmatter and templates that render it took twenty thousand lines of deletions. Once a site is in that shape, replacing the thing that renders it is a small program. Verne exists because Hugo made it a small program. --- # shuf and split: cutting and shuffling files _Sampling and slicing, without a script_ Two jobs that people write scripts for: take a random sample of a file, and cut a big file into smaller ones. Coreutils does both, and the flags are worth ten minutes of your time. ## shuf, for randomness The default is a full random permutation of the lines: ```bash $ shuf n.txt | head -4 12 5 14 8 ``` `-n` turns it into a sample, the form I use most. Twenty thousand log lines, show me three: ```bash $ shuf -n 3 n.txt 20 4 16 ``` It also generates without a file at all, with `-i`: ```bash # five distinct numbers from 1 to 10 $ shuf -i 1-10 -n 5 4 9 6 2 5 ``` That is a lottery draw in one command. For sampling *with* replacement, where a value can repeat, `-r`: ```bash $ shuf -r -n 6 -i 1-3 1 2 3 1 3 1 ``` And when you need randomness that reproduces, `--random-source` fixes the stream: ```bash $ shuf --random-source=/dev/zero -n 5 n.txt 1 2 3 4 5 ``` Same input and same source, same output every time. That is how you get a shuffled test fixture that a colleague can reproduce. ## sort -R is not the same thing macOS has no `shuf`, and the advice you will find is to use `sort -R`. It is not equivalent, and the difference is easy to miss. Take a file with repeated lines: ```bash $ cat dup.txt a b a c a b $ sort -R dup.txt b b a a a c ``` Every `a` ended up next to every other `a`. `sort -R` sorts by a hash of each line, so identical lines hash identically and stay grouped. `shuf` on the same file scatters them properly: ```bash $ shuf dup.txt | tr '\n' ' ' b c a a a b ``` On my Mac, three consecutive `sort -R` runs even returned the same order. If your data has duplicates, `sort -R` is not a shuffle, it is a grouping with the groups in random order. Install `coreutils` from Homebrew and use `gshuf`. ## split, for slicing By lines, the common case: ```bash $ split -l 7 n.txt part_ $ wc -l part_* 7 part_aa 7 part_ab 6 part_ac 20 total ``` Twenty lines into sevens, with the remainder in the last file. The suffix defaults to two letters (`-a` changes it), so `aa` through `zz` gives 676 pieces before it needs more. When you care about the *number* of pieces instead of their size, `-n`: ```bash $ split -n 3 n.txt chunk_ $ ls chunk_* chunk_aa chunk_ab chunk_ac ``` Three files, whatever their size. Useful for handing equal work to a fixed number of parallel workers. The alphabetic suffixes sort badly and confuse everything downstream, so `-d` gives you numbers, and `--additional-suffix` keeps the extension: ```bash $ split -l 7 -d --additional-suffix=.txt n.txt p_ $ ls p_* p_00.txt p_01.txt p_02.txt ``` That combination is what I actually type: numbered, extensioned, and it feeds straight into `xargs -P` for parallel processing. ## Portability `split` is on macOS and takes `-l`, `-n` and `-d`, all of which gave identical results to Linux in my tests. `--additional-suffix` is GNU only. `shuf` is Linux only, and given what `sort -R` does with duplicates, it is worth installing instead of working around. One caveat that applies to both: they hold what they need in memory. `shuf` reads the whole input before emitting anything, since it cannot know the last line might come first. On a file larger than your RAM, sample with `awk` and a probability instead. --- # Zen C, four months later: from promising to shipping _When a hyped language survives its first quarter_ Back in January, [Zen C](https://github.com/zenc-lang/zenc) was making the rounds in my feeds: a fresh systems language transpiling to readable GNU C, with [V](/posts/2024-02-how-own-web-server-vlang/) flavour and a few ideas of its own. It looked promising, but new languages get hyped every other month and most of them quietly die before a `1.0`. So I let it cook. A quarter in seems about the right time to look again, and tell you whether it actually went anywhere. ## What it is, in two paragraphs Zen C is a systems programming language that transpiles to human-readable GNU C/C11. You write modern code with type inference, pattern matching, generics, traits, async/await and string interpolation, and the compiler emits plain C that you can read, debug and drop into any existing C project. 100% C ABI compatibility is non-negotiable. Think of it as Rust's ergonomics meeting C's simplicity, without the borrow checker complexity. The flagship feature is the DSL plugin system: you embed Lisp, SQL, Forth, Befunge or Brainfuck blocks directly in your code, and a plugin emits C at compile time. You can write your own. ## Four months of git log I cloned the repo to take stock. The first commit lands on January 11th. By May 1st: - **938 commits total**, 859 of them since mid-January. - Roughly 330 in January, 245 in February, 216 in March, 143 in April. The pace settled but stayed serious. - One core maintainer ([Zuhaitz](https://github.com/Zuhaitz-dev)) doing around 80% of the work, 30+ external contributors picking up the rest. - Zero stagnation, no rebrand drama, just steady output. The version I was looking at in January was 0.1.0. The latest tag today is 0.4.4: ```text v0.1.0 2026-01-25 initial public release v0.4.0 2026-01-31 jump straight to 0.4 (regex stdlib, generic fixes, i18n) v0.4.1 2026-02-13 arena pattern, CMake support, more types v0.4.2 2026-02-26 default-init refactor, security fixes in utils.c v0.4.3 2026-03-12 std/hash.zc, ARM64 fixes, generics polish v0.4.4 2026-03-20 source-level debugging support ``` The leap from 0.1.x to 0.4.0 in six days is not a fluke: it lines up with the stdlib expansion and the moment external contributors started arriving in numbers. ## Building it, and the version trap I cloned and built it, and the build told me something the version file did not. ```bash $ git clone --depth 1 https://github.com/zenc-lang/zenc.git $ cd zenc && cat .version v0.4.4 $ make -j6 src/repl/repl_jit.h:15:10: fatal error: libtcc.h: No such file or directory ... ``` The REPL's JIT is built on LibTCC and its headers are a separate package. With that installed it compiles clean: ```bash $ sudo apt install build-essential libtcc-dev $ make -j6 $ ./zc --version zc a471a6a ``` A bare commit hash, not a version. That is the tell I should have read immediately, because `.version` says `v0.4.4` and this binary is **408 commits past that tag**. The version file on `main` lags the release, and a shallow clone of the default branch gives you neither the tag nor a warning. It matters more than a label. `src/repl/` in the v0.4.4 tarball contains `repl.c` and `repl.h` and no `repl_jit.*` at all, so **the LibTCC dependency I just tripped over does not exist in the release**. Same for the plugins: v0.4.4 ships them as `.c`, the `.zc` sources arrived later. So everything below describes `main` at a471a6a, not the 0.4.4 you would download. For a project moving at this pace that distinction is the whole point, and I nearly published it the wrong way round. ## The syntax that caught me What makes Zen C interesting day to day is how aggressively it cuts boilerplate. A bare string statement *is* a `println`, and a `!` prefix sends it to stderr: ```zc fn main() { "Hello, world!"; !"Error: file not found"; let user = "davlgd"; let tasks = 42; "User {user} has {tasks} pending tasks"; } ``` ```text Error: file not found Hello, world! User davlgd has 42 pending tasks ``` Interpolation happens in the same expression statement, with no `print` and no `format`. Note the ordering in that output: the stderr line arrived first, because it is unbuffered. Pattern matching feels Rust-like and compiles down to a plain C `switch`: ```zc enum Priority { Low, Medium, High, Critical } fn describe(p: Priority) -> string { match p { Priority::Low => return "Can wait", Priority::High => return "Do it now!", Priority::Critical => return "Do it now!", _ => return "Normal priority" } } ``` Calling it with `High` gives `Do it now!` and with `Medium` falls through to `Normal priority`, as you would expect. Enums with payloads destructure right in the arm, and generics use monomorphisation, so `Vec` and `Option` cost nothing at runtime. Memory management stays explicit, with three knobs: `defer free(buf)` for the manual case, `autofree` for scope-bound cleanup, and a `Drop` trait for RAII. No GC, no borrow checker. ## The plugin system This is what sets Zen C apart from the other modern-C attempts. You import a plugin, embed a domain-specific language as a `name! { ... }` block, and the plugin emits C at compile time: ```zc import plugin "plugins/lisp" as lisp fn main() { lisp! { (defun square (x) (* x x)) (print (square 5)) } } ``` That prints `25`. There is a Lisp interpreter inside my C binary, and it got there through a compile-time plugin. On `main` the `plugins/` directory ships `lisp`, `sql`, `forth`, `befunge` and `brainfuck`, each as a `.zc` source and a built `.so`. In the 0.4.4 release they are still `.c`. Regular expressions are deliberately not a plugin: they live in the standard library as a normal API. ## What landed since January The headline changes, roughly in shipping order: - **APE builds** via [Cosmopolitan Libc](https://github.com/jart/cosmopolitan), producing a single `zc.com` that runs on Linux, macOS, Windows and the BSDs, on `x86_64` and `aarch64`. - **Multi-backend compilation**: `zc run app.zc --cc gcc|clang|zig|tcc`, with the test suite at 100% on GCC, Clang and Zig, 98% on TCC. - **C++, CUDA and Objective-C interop** through `--cpp`, `--cuda` and `--objc`. - **C23 support** when the backend allows, including arbitrary-width integers compiling to `_BitInt(N)`. - **MISRA C:2012 verification** in the test suite, independent of any MISRA affiliation. - **REPL rewrite** with JIT compilation powered by LibTCC, which is why that header matters. - **Source-level debugging** in v0.4.4, and a stdlib grown to cover regex, hashing, networking, JSON and big numbers. ## How it compares to V The two are easy to mix up: both transpile to C, both go after C's ergonomics. The differences show once you spend a day with each. V has a few years of head start and a wider ecosystem out of the box: `vweb`, an ORM, a package manager, and several backends. Its C output is dense and machine-oriented, optimised for the compiler, not for a human reader. Zen C aims elsewhere. One backend, but paired with multiple C compilers and bridges into C++, CUDA and Objective-C. The generated C is meant to be read and dropped into existing codebases. The DSL plugin system has no equivalent in V. For application code with HTTP and a package manager already wired up, V wins today. For embedding modern code in a C-heavy project, hitting GPU kernels or shipping a single APE binary, Zen C is the more natural fit. They are not solving the same problem. ## What is still missing It is 0.4.x, and the honest list is short: no package manager, alpha IDE tooling, patchy documentation outside the [language tour](https://docs.zenc-lang.org/tour/), and a smaller community than the mainstream. The Discord is active and the contributor count keeps growing. In January the question was whether this was another shiny C-targeting language that would not survive six months. Four months later, with 938 commits, five releases, APE and CUDA interop, source-level debugging and a doubled stdlib, the question has changed. It is no longer "will it survive", it is "what will it be at 1.0". If you write embedded code, game engines or anything where C is the deployment target, give it a try. And tell me what you build with it 😉 --- # tr: the most underrated one-liner in your shell _Translate, squeeze, delete_ `sed` gets used for jobs that `tr` does in a quarter of the characters, and usually faster, because `tr` does not parse a regular expression: it maps bytes. Knowing the four things it does covers a surprising share of daily text mangling. ## Translate Two sets, same length, position by position: ```bash $ echo "Hello World" | tr a-z A-Z HELLO WORLD ``` Ranges work, and so does anything you can write out. ROT13 is famously one invocation: ```bash $ echo "Hello" | tr 'A-Za-z' 'N-ZA-Mn-za-m' Uryyb ``` ## The trap that explains everything This is where people get hurt, and understanding it makes the rest obvious: ```bash $ echo "hello" | tr "hello" "world" wolld ``` Not `world`. `tr` never saw a word, it built a character map: `h` to `w`, `e` to `o`, `l` to `r`, then `l` to `l` because the second `l` in the input set overrode the first, and `o` to `d`. Applying that map to `hello` gives `wolld`. So `tr` is not find-and-replace. It substitutes one character for another, everywhere, and duplicate characters in the first set mean the last mapping wins. When you want strings, you want `sed`. ## Delete `-d` drops the characters in the set instead of translating them, and the POSIX classes make it readable: ```bash $ echo "Hello World 123" | tr -d '[:digit:]' Hello World $ printf 'a\x01b\x02c\n' | tr -d '[:cntrl:]' abc ``` That second one is the fastest way I know to clean control characters out of a log before feeding it to something fragile. `[:alpha:]`, `[:space:]`, `[:punct:]` and friends all work. The most common use in my shell is stripping carriage returns from a file that came off Windows: ```bash $ printf 'line1\r\nline2\r\n' | tr -d '\r' | od -c | head -1 0000000 l i n e 1 \n l i n e 2 \n ``` ## Complement, where it gets powerful `-c` inverts the set: act on everything *except* these. Combined with `-d`, it becomes a whitelist: ```bash $ echo "abc-123_x" | tr -cd '[:alnum:]' abc123x ``` Keep letters and digits, delete everything else, in nine characters. That is sanitising a filename or an identifier without a regular expression and without a loop. ## Squeeze `-s` collapses runs of repeated characters down to one: ```bash $ echo "trop d espaces" | tr -s ' ' trop d espaces ``` Useful before `cut -d' '`, which counts every space as a separator and falls apart on ragged alignment. `tr -s ' '` first, and the fields line up. ## Portability Every example above produced identical output on macOS and on Ubuntu, including the `wolld` surprise and the POSIX classes. Then I tried an accented character, expecting a footnote, and got a real difference: ```bash # Linux, both GNU and the Rust build $ echo "café" | tr 'é' 'e' cafee # macOS $ echo "café" | tr 'é' 'e' cafe ``` `é` is two bytes in UTF-8. GNU and uutils both work on bytes, so each of those two bytes gets mapped to `e` and you get one letter too many. The macOS implementation handled the character as a character. I had assumed this would break on the Mac and work on Linux. It is the other way round, and the Linux behaviour is the one that follows the specification: `tr` is defined on bytes. Do not use it on non-ASCII text at all. `iconv -f UTF-8 -t ASCII//TRANSLIT` is the tool for that job. One more limit: `tr` reads standard input only, so there is no in-place mode and no filename argument. It is always `tr ... < file` or a pipe. --- # x402 v2: the payment status code nobody used _HTTP 402, finally answered_ HTTP has carried a `402 Payment Required` status code since [RFC 2068](https://datatracker.ietf.org/doc/html/rfc2068#section-10.4.3) defined HTTP/1.1 in January 1997, reserved for future use and unused ever since. It is not in HTTP/1.0 at all: [RFC 1945](https://datatracker.ietf.org/doc/html/rfc1945) goes straight from 401 to 403. [x402](https://x402.org/) is an attempt to finally define what a server should say when it returns one, and version 2 has a v2 specification out for implementation. ## The exchange, in four steps The protocol is deliberately small: 1. A client requests a resource. 2. With no valid payment attached, the server answers **402**, and the response carries what payment it would accept: which networks, which assets, how much, where to send it. 3. The client pays and retries with proof attached. 4. The server verifies and serves the resource. The point is that it is one HTTP exchange with headers, not a redirect to a checkout page and back. There is no account to create, no API key to provision, no out-of-band flow. A caller that can read a 402 and sign a payment can consume a paid endpoint it has never seen before. That is why this arrived alongside the agent tooling instead of before it. Three roles: the **client**, the **resource server** that gates its endpoints with middleware, and a **facilitator** that handles the on-chain part so the resource server does not have to. On the server side it stays declarative: ```js app.use(paymentMiddleware({ "GET /weather": { accepts: [...], description: "Weather data", }, })); ``` ## What v2 changed The v2 announcement is explicit that it comes out of production experience rather than design taste, and that shows in what it addresses. **Sessions.** In v1 every request was its own payment, which is correct and impractical. v2 adds wallet-based identity and reusable sessions, so a returning client authenticates once and stops paying per call. That is what makes subscription-shaped pricing expressible without leaving the protocol. **Headers instead of the body.** The old `X-` prefixed headers are gone, replaced by `PAYMENT-REQUIRED`, `PAYMENT-SIGNATURE` and `PAYMENT-RESPONSE`. All payment data moved into headers, which frees the response body entirely. A 402 can now carry a human-readable page *and* machine-readable payment terms in the same response. **A plugin architecture.** The specification, the SDK and the facilitators are separate concerns in v2, and chains and assets register through plugins instead of being patched into the core. The paywall moved out into its own package. That is the change that decides whether this ages well: adding a network should not require a spec revision. **Discovery.** An extension exposes service metadata for facilitators to crawl, so an agent can find an endpoint, read its price and capabilities, and decide, without a human having read the documentation first. **Dynamic recipients**, meaning a payment can split across several parties, which is what marketplace and platform-fee models need. And notably: **no breaking changes**. New behaviour arrives as extensions rather than spec edits, and the reference SDKs stay backward compatible with v1. ## The payments industry showed up first The specification would have been a footnote. What happened at the start of April was not. On April 2nd the Linux Foundation [announced the x402 Foundation](https://www.linuxfoundation.org/press/linux-foundation-is-launching-the-x402-foundation-and-welcoming-the-contribution-of-the-x402-protocol) as a neutral home for the protocol, with Coinbase contributing it. Read the membership sentence carefully, because it is more careful than the headlines were: "Membership will be comprised of participants from multiple verticals with initial intent and support being expressed by" a list that runs Adyen, Amazon Web Services, American Express, Cloudflare, Coinbase, Google, Mastercard, Microsoft, Shopify, Stripe and Visa. Intent and support, not membership. Nobody has joined anything yet. But a list containing three card networks, two cloud providers and Stripe is not a list you assemble by accident. Visa had moved a fortnight earlier, in a direction worth noting. On March 18th it [published a card specification and SDK](https://global-corporate.review.visa.com/en/sites/visa-perspectives/innovation/visa-card-specification-sdk-for-machine-payments-protocol.html) for the Machine Payments Protocol, the standard Stripe and Tempo announced for machine-to-machine payments. Its framing of x402 is one sentence and it is the interesting one: "While remaining protocol-agnostic, Visa built its card-based MPP specification to complement existing emerging agent payment standards, including protocols like x402." Complement, and protocol-agnostic. A card network is not betting on x402, it is making sure that whichever standard agents settle on, a card can sit behind it. That is what changed in six months. x402 went from a clever use of a forgotten status code to something the institutions whose business is settling payments are positioning around, which moves the question from "will anyone implement this" to "what does my API do when an agent asks for a price". ## What I think of it The honest summary is that the interesting part is not the payments, it is the shape. Machine-to-machine access control has always meant provisioning: someone creates an account, generates a key, configures a quota. x402 replaces that with an exchange the caller can complete on its own. Whether the settlement is a stablecoin, a card or an internal ledger is an implementation detail of the facilitator, and v2 explicitly leaves room for the traditional rails. The caution is equally plain. This is a protocol whose reference implementations are young, whose facilitators are third parties you are trusting, and whose economics on very small amounts depend entirely on settlement costs staying low. "No breaking changes" is a promise made at v2, not a track record. What makes it worth reading now is that it is the first serious answer to a question HTTP/1.1 left open in 1997. A status code with no semantics is a strange thing to carry for nearly thirty years, and it turns out the missing piece was not the code but a client that could act without a human. --- # comm and join: set operations without a database _Two sorted files, four questions_ "Which users are in both lists?" and "which are only in the old one?" are database questions, and people import CSVs into SQLite to answer them. Two coreutils commands answer them on sorted text files, and they have been doing it since long before SQLite existed. ## comm answers set questions Give it two sorted files and it prints three columns: lines only in the first, lines only in the second, lines in both. ```bash $ cat a.txt $ cat b.txt alice bob bob carol carol dave $ comm a.txt b.txt alice bob carol dave ``` The indentation is the answer. `alice` is in column one, so it exists only in `a.txt`. `dave` is indented once, only in `b.txt`. `bob` and `carol` are indented twice, present in both. That output is hard to read and easy to filter. That is the point. Each column has a number, and passing that number *suppresses* it: ```bash # suppress columns 1 and 2, keep only what is in both $ comm -12 a.txt b.txt bob carol # suppress 2 and 3, keep only what is unique to the first file $ comm -23 a.txt b.txt alice ``` Intersection and difference, in four characters. `-13` gives you what only the second file has, and the symmetric difference is `-3`. ## The sorted requirement is not a suggestion `comm` walks both files once, in lockstep, so it handles files far larger than memory. It also means unsorted input produces nonsense, and it tells you: ```bash $ comm u.txt a.txt comm: file 1 is not in sorted order alice b ``` A warning on stderr and wrong output on stdout. Always `sort` first, and be consistent about it, because two files sorted under different rules are not comparable even when each looks sorted on its own. ## join is the relational one Where `comm` compares whole lines, `join` matches on a key field and merges the rest, exactly a SQL inner join: ```bash $ cat l.txt $ cat r.txt 1 alice 1 admin 2 bob 3 user 3 carol $ join l.txt r.txt 1 alice admin 3 carol user ``` Key `2` had no match on the right, so it dropped out. To keep it, ask for a left join with `-a1`: ```bash $ join -a1 l.txt r.txt 1 alice admin 2 bob 3 carol user ``` The key defaults to the first field of both files. When it is not, name it: ```bash # key is field 2 on the left, field 1 on the right $ join -1 2 -2 1 l3.txt r3.txt 1 alice admin 2 bob user ``` And `-t` sets the delimiter, which turns it into a CSV tool: ```bash $ join -t, lc.txt rc.txt 1,alice,admin 2,bob,user ``` `join` is as strict about sorting as `comm`, and louder about it: ```bash $ join l2.txt r.txt join: l2.txt:2: is not sorted: 1 alice 3 carol user join: input is not in sorted order ``` Note that it still printed a partial result between the two warnings. In a script that ignores stderr, that silently becomes a wrong answer rather than a failure. ## Where they run Both are coreutils, present on macOS, Debian and Ubuntu, and every example above gave identical output on all three. The honest limit is scale in the other direction. For two sorted files of any size, `comm` and `join` beat anything you could load into memory. For a real join on unsorted data with several keys and types, sorting the input is the expensive part, the moment a database earns its keep. --- # column and numfmt: making shell output readable _Alignment, without the awk_ Every script eventually prints a table, and every one of them prints it badly. Fields do not line up, byte counts run to ten digits, and somebody writes a `printf` with hardcoded widths that break on the first long value. Two commands fix both halves of that. ## column, for the alignment Feed it ragged input, it works out the widths for you: ```bash $ printf 'name size owner\nfoo.txt 1024 alice\nverylongname.md 20 bob\n' | column -t name size owner foo.txt 1024 alice verylongname.md 20 bob ``` No widths given anywhere. `column -t` read every line first, found the longest value in each field and padded to match. That is exactly the loop nobody wants to write again. The default separator is whitespace. For anything else, `-s`: ```bash $ printf 'id,name,role\n1,alice,admin\n2,bob,user\n' | column -t -s, id name role 1 alice admin 2 bob user ``` That single flag turns any CSV into something you can read, which beats opening it in a spreadsheet to look at four lines. ## It also speaks JSON This one surprised me. The util-linux version can name the columns and emit structured output: ```bash $ printf '1 alice\n2 bob\n' | column -J -N id,name { "table": [ { "id": "1", "name": "alice" },{ "id": "2", "name": "bob" } ] } ``` `-N` gives the field names, `-J` switches to JSON. So a text-only tool becomes a bridge into `jq` without writing a parser. `-N` works with `-t` too, adding a header row to the plain table. ## numfmt, for the numbers Alignment does not help if the values are unreadable. `numfmt` converts to and from human scales: ```bash $ echo 1234567 | numfmt --to=iec 1.2M $ echo 1234567 | numfmt --to=si 1.3M ``` Two answers, both right. `iec` divides by 1024, `si` by 1000, and the fact that they disagree by a whole tenth at this size is precisely why the flag exists. The flag that makes it useful in a pipeline is `--field`, which converts one column and leaves the rest alone: ```bash $ du -sb /usr/bin | numfmt --field=1 --to=iec 58M /usr/bin ``` It goes the other way too: ```bash $ echo 2G | numfmt --from=iec 2147483648 ``` Which is how you turn a config value into bytes you can do arithmetic on. ## What you get on macOS `column` is there, and the basics work identically: `-t` and `-s` gave me byte-identical output. The rest does not exist: ```bash $ printf '1 alice\n' | column -J -N id,name column: illegal option -- J usage: column [-tx] [-c columns] [-s sep] [file ...] ``` That usage line is the whole of BSD `column`: four options against util-linux's twenty-six. And `numfmt` is GNU coreutils, absent from macOS entirely, though Homebrew's `coreutils` provides it as `gnumfmt`. So `column -t` is safe to use anywhere. `-J`, `-N` and every `numfmt` invocation are Linux-only, and a script that relies on them needs to say so. One caveat on `column -t` before you pipe something large into it: it buffers the entire input before printing anything, because it cannot know the widths until it has seen the last line. Timing the arrival of each line makes it obvious: ```bash $ ( echo "a 1"; sleep 2; echo "bb 22" ) | column -t | ts -i '%.s' 1.984377 a 1 0.000099 bb 22 ``` Both lines land after the two second pause, together. On a stream that never ends, `column -t` prints nothing at all. --- # seq: the little generator that beats your for loop _Counting, with a catch_ If [`yes`](/posts/2024-02-yes-command/) repeats, `seq` counts. It's the other half of the same idea: a tiny program whose only job is to produce a stream you pipe into something else. And like most tiny programs, it hides a few surprises once you cross an OS boundary. ## Why not brace expansion Most of us reach for `{1..10}` first: ```bash for i in {1..10}; do echo "$i"; done ``` It works in `bash` and `zsh`, and it's fast because the shell expands it without forking anything. But it's a shell extension, not POSIX, so it silently does nothing useful in `dash`, which is `/bin/sh` on Debian and Ubuntu: ```bash $ dash -c 'for i in {1..3}; do echo "$i"; done' {1..3} ``` One iteration, with the literal string. That's the kind of bug you discover in production. Brace expansion also refuses variables (`{1..$n}` does not expand), usually the moment people give up and call `seq`. ## Counting, three ways ```bash seq 5 # 1 to 5 seq 2 6 # 2 to 6 seq 1 2 9 # from 1 to 9, step 2: 1 3 5 7 9 seq 3 -1 1 # backwards: 3 2 1 ``` The middle argument is the increment, which trips people up: it's `seq first step last`, not `seq first last step`. Two useful flags: ```bash $ seq -w 8 11 # pad with zeros to equal width 08 09 10 11 $ seq -f 'n=%g' 1 2 # printf-style format n=1 n=2 ``` Both work on macOS, Debian and Ubuntu, so you can rely on them. ## The separator that bites `-s` sets what goes between the numbers, and this is where the implementations part ways. On Debian 13 with GNU coreutils 9.7: ```bash $ seq -s, 1 5 | od -c 0000000 1 , 2 , 3 , 4 , 5 \n ``` Five numbers, four commas, one newline. The same command on macOS: ```bash $ seq -s, 1 5 | od -c 0000000 1 , 2 , 3 , 4 , 5 , ``` A trailing comma, and no newline at all. BSD `seq` treats `-s` as a terminator instead of a separator, so `seq -s, 1 5` gives you `1,2,3,4,5,` and leaves your cursor mid-line. If you were building a CSV row or a comma-separated argument list, you now have an empty trailing field on one platform and not the other. The portable fix is to stop asking `seq` to do the joining: ```bash seq 1 5 | paste -sd, - ``` `paste -sd,` joins the lines with a comma and gives the same result everywhere. ## Floats, and the same story again `seq` handles decimals, genuinely useful for generating test data: ```bash $ seq 0.5 0.5 2 # Debian and Ubuntu 0.5 1.0 1.5 2.0 ``` macOS prints `0.5 1 1.5 2`: it trims the trailing zeros, where GNU keeps as many decimals as the operands had. Writing `seq 0.50 0.50 2` does not help, BSD still trims. Again, fine until you sort or compare the output as strings. Worth noting for the Ubuntu case: since 25.10 the default `seq` is the Rust [uutils](https://github.com/uutils/coreutils) reimplementation, and on every one of these tests it matched GNU exactly, down to the byte. That compatibility work shows. ## When it earns its place `seq` shines when the count is computed instead of written out: ```bash # run process.sh once per line of input.txt, four at a time n=$(wc -l < input.txt) seq 1 "$n" | xargs -P4 -I{} ./process.sh {} ``` It's also the shortest way to generate a fixed-size file for a test, or to drive a retry loop. What it does not do is iterate over anything but numbers: for files, `find` and globs are the right tools, and piping `seq` into `ls` is a sign you took a wrong turn. --- # true and false: two commands that do nothing _And a copyright notice longer than the code_ Your system ships two programs whose entire job is to fail, or to not fail. `true` exits with status 0, `false` exits with status 1. That's the whole specification. They have man pages, maintainers, and one of them has a copyright story that says a lot about the software industry. ## Three lines of nothing, copyrighted In early UNIX, `/bin/true` was an empty file marked executable. Running an empty shell script does nothing and returns 0, exactly the contract. Free implementation. Then the lawyers arrived. The [1984 AT&T version](https://trillian.mit.edu/~jc/humor/ATT_Copyright_true.html) of `true.sh` contained: ```sh # Copyright (c) 1984 AT&T # All Rights Reserved # THIS IS UNPUBLISHED PROPRIETARY SOURCE CODE OF AT&T # The copyright notice above does not evidence any # actual or intended publication of such source code. #ident "@(#)cmd/true.sh 50.1" ``` Seven lines of legal notice protecting zero lines of program. The file asserts that it is unpublished proprietary source code, while containing no source code. It remains one of my favourite artifacts in computing history. ## Where they actually live today Nobody ships an empty script anymore. On my Mac: ```bash $ ls -l /usr/bin/true -rwxr-xr-x 1 root wheel 84032 22 juil. 23:57 /usr/bin/true $ file -b /usr/bin/true Mach-O universal binary with 2 architectures ``` 84 KB, compiled for both `x86_64` and `arm64e`, to return 0. Ubuntu 25.10 is stranger. It moved most of its coreutils to the Rust [uutils](https://github.com/uutils/coreutils) implementation and kept the GNU ones under a `gnu` prefix, and the split fell between the twins: ```bash $ readlink -f /usr/bin/true /usr/bin/gnutrue $ readlink -f /usr/bin/false /usr/lib/cargo/bin/coreutils/false $ /usr/bin/true --version | head -1 true (GNU coreutils) 9.5 $ /usr/bin/false --version | head -1 /usr/bin/false (uutils coreutils) 0.2.2 ``` `true` is C, `false` is Rust, on the same machine. I did not expect to find the two smallest programs on the system shipped by two different projects. Except none of that usually runs. Your shell has its own: ```bash $ type true true is a shell builtin ``` The binary exists for the cases where a shell is not involved: a `find -exec`, an `execve` from a program, a `#!` line. The rest of the time, `true` never leaves the shell process. ## What they are for The first one is the infinite loop, where `true` is the condition that never ends: ```bash while true; do check_something sleep 60 done ``` The interesting one is neutralising a program you cannot remove. Point it at `true` and every call to it succeeds while doing nothing: ```bash sudo ln -sf /usr/bin/true /usr/local/bin/annoying-hook ``` Then there's the third member of the family, and it is not a command at all. `:` is a shell builtin that does the same as `true`, in one character: ```bash while :; do echo tick; sleep 1; done ``` It also works as a placeholder where the syntax demands a body but you have nothing to put there yet: ```bash if [[ -n "${DEBUG:-}" ]]; then : # nothing to do for now else run_quietly fi ``` And `false` earns its keep in tests. When you want to check that your error handling actually triggers, it's a guaranteed failure with no side effects, hard to get any other way. Two caveats worth knowing. The first is that your shell uses its builtins, so changing `/usr/bin/true` will not change what `while true` does. If you need the binary, call it by path. The second is that these two take their contract seriously, to the point of absurdity. Ask `false` for help, and the text warns you that asking changes nothing: "Any IO error during this operation is diagnosed, yet the program will also return 1." It means it: ```bash $ /usr/bin/false --help > /dev/null $ echo $? 1 ``` macOS prints nothing at all for the same call, and exits 1 too. Either way you never get a 0 out of `false`. --- # factor: your OS ships a prime factorization tool _In coreutils, of all places_ Coreutils is the package of file and text utilities: `ls`, `cat`, `wc`, `sort`. It also contains a program that does prime factorization, sitting between `expr` and `false` in the binary listing as though that were a normal thing for a file utility package to do. ```bash $ factor 1234567890 1234567890: 2 3 3 5 3607 3803 ``` Number in, prime factors out, repeated factors repeated. That is the entire interface. ## It is faster than it has any right to be Trial division would take approximately forever on anything interesting, and `factor` is plainly not doing that. A semiprime built from two twelve-digit primes: ```bash $ time factor 999999999950000000000429 999999999950000000000429: 999999999961 999999999989 real 0m0.017s ``` Seventeen milliseconds to split a 24-digit number into its two prime halves. Then there is the other direction, recognising that a number has no factors at all: ```bash $ factor 2147483647 2147483647: 2147483647 ``` A number that factors into only itself is prime, and this one is 2³¹ minus 1. It also handles values well past what a 64-bit integer can hold: ```bash $ factor 170141183460469231731687303715884105727 170141183460469231731687303715884105727: 170141183460469231731687303715884105727 ``` That is 2¹²⁷ minus 1, prime, answered instantly. Your shell cannot even represent that number. ## What it is actually good for Not cryptography. A key is safe precisely because `factor` would need longer than the age of the universe on it, and the numbers above are easy ones. Where it earns its place is the small stuff. Checking whether a number is prime is a grep away: ```bash # prints the number only when it is prime $ factor 97 | grep -qE ': [0-9]+$' && echo prime prime ``` Picking a good hash table size, checking that a chosen shard count divides cleanly, working out why a buffer size behaves oddly, or answering a Project Euler question without writing any code: ```bash $ factor 600851475143 600851475143: 71 839 1471 6857 ``` That is problem 3, solved by typing. ## Refusals It only accepts positive integers, and says so plainly: ```bash $ factor abc factor: 'abc' is not a valid positive integer ``` The one that catches people is negative numbers, because the argument parser sees the minus first: ```bash $ factor -1 factor: invalid option -- '1' ``` Not "negative numbers have no prime factorization", but a complaint about an option that does not exist. Use `--` if you are passing something that might start with a dash. ## Not on macOS ```bash $ command -v factor $ ``` `factor` is GNU coreutils, and macOS ships the BSD set. Homebrew's `coreutils` gives you `gfactor`, and both Debian and Ubuntu have it out of the box. Worth noting on Ubuntu since 25.10, where the default coreutils are the Rust [uutils](https://github.com/uutils/coreutils) build: every number in this post produced identical output on the Rust and GNU versions, including the 39-digit one. --- # zellij: a multiplexer that tells you the shortcuts _Layouts in a file, not a script_ Terminal multiplexers have a reputation problem. `tmux` is excellent and everyone learns exactly four of its keybindings, because finding the fifth means reading a manual. [Zellij](https://zellij.dev/) starts from the opposite assumption: the shortcuts are on screen, and the configuration is a document rather than a script. ## The status bar is the manual Start it and the bottom of the screen lists what you can press, changing as you enter each mode. That sounds cosmetic and it is the single biggest difference in practice: you discover the tool by using it, and the tutorial is the interface. If you already have `tmux` fingers, there is a `tmux` keybinding preset, so `Ctrl-b` behaves as you expect while you decide. ## Layouts are files This is where it separates itself. A tmux layout is a script: a sequence of `split-window` and `send-keys` calls that builds the arrangement step by step. In zellij it is a description, in KDL: ```kdl layout { pane split_direction="vertical" { pane pane split_direction="horizontal" { pane command="tail" { args "-f" "/var/log/system.log" } pane } } } ``` Nesting is nesting, a pane that runs a command says so, and the file reads as the shape it produces. `zellij --layout dev.kdl` opens it. The interface itself is built from the same primitives, which you can see by dumping the default: ```bash $ zellij setup --dump-layout default layout { pane size=1 borderless=true { plugin location="tab-bar" } pane pane size=1 borderless=true { plugin location="status-bar" } } ``` The tab bar and the status bar are **plugins in panes**. There is no special-cased chrome: the UI is made of the same things your layouts are made of, so replacing it is a matter of writing a layout instead of patching the program. ## It is scriptable from outside `zellij action` drives a running session from another terminal: ```bash zellij action new-pane zellij action dump-screen /tmp/out.txt zellij action edit-scrollback zellij action close-pane ``` `dump-screen` is the one I keep using: capture what a pane currently shows, into a file, from a script. And `edit-scrollback` opens the pane's history in your editor, which beats scrolling and copying. ## Sessions, as you would expect ```bash $ zellij list-sessions No active zellij sessions found. ``` Named sessions, detach and reattach, resurrection of a session whose processes have exited. The mental model matches tmux closely enough that the migration is about keys and configuration, not concepts. ## What it costs It is written in Rust and it is heavier than tmux, in memory and in startup. On a laptop this does not matter. On a box where you keep fifty sessions open it might, and tmux remains the leaner tool. It is also not everywhere. tmux is in every distribution's base repositories and frequently already installed on a server you did not set up. zellij you install deliberately, which makes it a poor fit for the "SSH into an unfamiliar machine" case that multiplexers exist for. That one is still tmux, or `screen` if the machine is old enough. Version 0.43.1 is what I am running here, and the leading zero is not decoration: configuration format and plugin API have both moved between minor releases. Pin the version in anything shared. Where it wins outright is the project you return to daily. One layout file in the repository, `zellij --layout dev.kdl`, and everyone on the team gets the same four panes with the same things running in them. Committing that file is the actual feature. --- # tsort: there is a topological sort in your coreutils _Version 7 UNIX, for building libraries_ Sorting a dependency graph is the kind of thing you write a script for, or pull in a library for. UNIX has shipped a command that does it since Version 7, and almost nobody knows it is there. ## Getting dressed `tsort` reads pairs of words. Each pair means "this one comes before that one", and it prints an order that satisfies every pair: ```bash $ printf 'shirt tie\ntie jacket\npants shoes\npants belt\nbelt jacket\nsocks shoes\n' | tsort pants shirt socks belt tie shoes jacket ``` Six constraints in, one valid dressing order out. Shirt before tie, tie before jacket, and `tsort` worked out that socks and pants can happen in either order because nothing said otherwise. That is the whole interface. No configuration, no format to learn: two words per line, in dependency order. ## The order is not unique, and that matters Run the same input on macOS and you get a different answer: ```text socks pants shirt belt shoes tie jacket ``` Both are correct. A topological sort produces *an* order consistent with the constraints, and there are usually many. GNU and the Rust reimplementation happened to agree on my test, macOS did not. So do not diff `tsort` output across machines, and do not rely on the position of anything the constraints left free. If you need a deterministic result, pipe your pairs through `sort` first, and even then treat it as an implementation detail. ## Cycles are an error, with the loop printed Circular dependencies are the reason you wanted the tool in the first place, and it names them: ```bash $ printf 'a b\nb c\nc a\n' | tsort b c a tsort: -: input contains a loop: tsort: a tsort: b tsort: c $ echo $? 1 ``` Exit status 1, the members of the cycle listed on stderr, and a best-effort ordering still on stdout. That is enough to build a real check: run it in CI over your module graph and let a non-zero status fail the build. ## Why it exists at all The man page is candid about it: the command is "primarily intended for building libraries, where optimal ordering" of the archive members is what you are after. Single-pass linkers of the era resolved symbols in the order object files appeared in an archive, so the order was not cosmetic, it decided whether your program linked. `tsort` computed it. The `HISTORY` section dates the command to Version 7 AT&T UNIX. Modern linkers stopped caring decades ago, and the command stayed. That is how you end up with graph theory in a package of file utilities. ## What to use it for now Anything shaped like "A must happen before B" and small enough to express as pairs. Ordering database migrations, checking a Makefile's prerequisites for cycles, sequencing service startup, sorting a list of pull requests that build on each other. Generating the pairs is usually the real work, and it is a one-liner away: ```bash # deps.txt: one line per item, its prerequisites after it $ cat deps.txt jacket tie belt tie shirt shoes pants socks belt pants # turn each line into pairs, then order them $ awk '{ for (i = 2; i <= NF; i++) print $i, $1 }' deps.txt | tsort | tr '\n' ' ' pants shirt socks belt tie shoes jacket ``` Two caveats. `tsort` speaks whitespace-separated tokens, so anything with a space in it needs escaping or replacing first. And an item with no dependencies at all never appears, because `tsort` only knows what the pairs tell it. Feed it as a pair with itself, `item item`, and it shows up in the ordering. It is on macOS, on Debian and on Ubuntu, in every case as part of coreutils. Nothing to install. --- # cgroups v2: the limits your tools cannot see _free says 31 GB, the kernel disagrees_ [`nice`](/posts/2026-01-14-nice-priority/) changes who the scheduler serves first. [`taskset`](/posts/2025-12-03-taskset-cpu-affinity/) changes where a process may run. Neither of them caps anything. The actual ceiling on Linux is a control group, and if you run anything in a container you are already inside one. ## Where the limits live cgroup v2 exposes everything as files under one tree: ```bash $ findmnt -no FSTYPE /sys/fs/cgroup cgroup2 $ cat /sys/fs/cgroup/cgroup.controllers cpu memory pids ``` Three controllers here: CPU time, memory, process count. Each one is a handful of files you read and write like any other: ```bash $ cd /sys/fs/cgroup $ cat cpu.max cpu.weight memory.max pids.max max 100000 100 max 2048 ``` `cpu.max` reads `max 100000`: no quota, over a 100 millisecond period. Set it to `50000 100000` and the group gets half a core. `cpu.weight` is the relative share when there is contention, the cgroup version of `nice`. And `pids.max` is capped at 2048 here, which turns out to matter. ## Your tools do not read any of this Here is the problem that bites people in production. Inside that same container: ```bash $ nproc 6 $ free -h | head -2 total used free shared buff/cache available Mem: 31Gi 10Gi 436Mi 808Mi 21Gi 20Gi ``` Six CPUs and 31 GB of RAM. Those are the **host's** numbers. `nproc` reads the CPU affinity mask, `free` reads `/proc/meminfo`, and neither of them knows a control group exists. That is how a JVM sizes its heap for a machine ten times bigger than the container it is in, and how a build system decides to run sixteen parallel jobs inside a group entitled to one core. The tool is not wrong, it is answering a different question from the one you meant. The kernel will tell you the truth if you ask it directly: ```bash # what this cgroup is actually allowed $ cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/cpu.max max max 100000 ``` `max` here means unlimited, because my test container had no memory cap set. On a real orchestrator those files hold numbers, and they are the numbers your process should be sizing itself against. ## The limit is not decorative I wanted to check that `pids.max` was enforced and not merely advisory, so I started spawning processes in a loop and watched what happened. What happened is that the container stopped being able to do anything at all: ```text Error: crun: fork: Resource temporarily unavailable ``` Not the loop failing, the container runtime failing to attach a new shell, because the group had no process slots left for it either. I had to destroy and recreate the container. That is the difference between a control group and `nice`. `nice` asks politely. A cgroup limit is a wall, and everything in the group hits it at the same time, including the things you were counting on to get you out of trouble. ## Pressure, the metric worth knowing cgroup v2 also exposes PSI, telling you how long tasks in the group spent *waiting* for a resource: ```bash $ cat /sys/fs/cgroup/cpu.pressure some avg10=0.00 avg60=0.00 avg300=0.00 total=1946 full avg10=0.00 avg60=0.00 avg300=0.00 total=1938 ``` `some` is the share of time at least one task was stalled, `full` the share where everything was. Averages over 10, 60 and 300 seconds. Load average tells you how many things are runnable, which conflates a busy machine with a starving one. This tells you directly whether anyone is waiting, and there are equivalent files for `memory` and `io`. ## Setting a limit yourself Writing to these files by hand needs a delegated, writable cgroup, which a rootless container does not get: mine was mounted read only and `mkdir /sys/fs/cgroup/demo` answered `Read-only file system`. On a normal systemd host the supported route is: ```bash systemd-run --scope -p CPUQuota=50% -p MemoryMax=1G ./job.sh ``` That creates a transient scope, applies the limits and cleans up afterwards, a great deal safer than editing the tree under `/sys/fs/cgroup` yourself. I have not pasted its output here because the container I test in has no systemd to run it under, and I would rather leave a gap than invent a transcript. --- # nice: being polite with the scheduler _Priority is not a quota_ `nice` is the command people reach for when a job is eating the machine. It rarely does what they expect, because it does not limit anything. It changes who the scheduler serves first, and that only matters when there is a queue. ## What the number means Niceness runs from -20 to 19. Higher is nicer: the process yields to others. Lower is greedier. The default is 0, and `nice` with no arguments prints the current value: ```bash $ nice 0 $ nice nice 10 ``` The second one is worth noticing. Running `nice` through `nice` shows 10, because the default increment is +10 when you do not pass `-n`. So `nice ./job.sh` is already a meaningful demotion. ## It does nothing until there is competition This is the part that surprises people, and it is easy to demonstrate. Two identical busy loops for eight seconds, one at nice 0 and one at nice 19, on a six core machine: ```text nice 0 : 4705843 nice 19 : 4757164 ``` The nice 19 process did slightly *more* work. Not a mistake: there were six cores and two processes, so nobody had to wait, and priority never came into play. Now the same two loops confined to a single core with [`taskset`](/posts/2025-12-03-taskset-cpu-affinity/), so they have to share: ```bash # count iterations for 8 seconds, at a given niceness $ cat burn.sh #!/bin/bash s=0; end=$((SECONDS + 8)) while (( SECONDS < end )); do (( s++ )); done echo "nice $(nice): $s" # both loops on core 0, forcing them to compete $ taskset -c 0 bash -c '( nice -n 0 ./burn.sh ) & ( nice -n 19 ./burn.sh ) & wait' ``` ```text nice 0 : 6232465 nice 19 : 93877 ``` A factor of 66. Same commands, same durations, and the only change is that the two processes now want the same core at the same time. That is the whole lesson. `nice` redistributes a contended resource. On an idle machine it is a no-op, and if your build is slow because the disk is busy, no amount of niceness will help. ## Going down is free, going up needs permission Raising the number is always allowed. Lowering it is not: ```bash $ nice -n -5 nice nice: warning: setpriority: Permission denied (os error 13) 0 ``` Read that output carefully. It printed a **warning**, then `0`, then exited with status 0. The command ran, at the priority it already had, and nothing told the script that the request failed. If you rely on a negative nice for a latency-sensitive job, check the result instead of trusting the exit code. I got that even as root, because the container had no `CAP_SYS_NICE`: ```bash $ id -u 0 $ ulimit -e 0 ``` `ulimit -e` is the ceiling on how far you may lower niceness, and 0 means not at all. Rootless container runtimes commonly drop that capability, so a `nice -n -10` in your Dockerfile may be silently doing nothing in production. ## Changing your mind, and the disk `renice` works on a process that is already running: ```bash $ renice -n 15 -p 4362 4362 (process ID) old priority 0, new priority 15 ``` Handy when a job you started an hour ago turns out to be in the way. For I/O there is a separate knob, because CPU priority says nothing about disk access: ```bash $ ionice none: prio 0 $ ionice -c 3 ionice idle $ ionice -c 2 -n 7 ionice best-effort: prio 7 ``` The classes are `1` realtime, `2` best-effort and `3` idle, with `-n 0..7` setting the priority inside the first two. A backup at `nice 19` can still saturate your disk queue, and `ionice -c 3 ./backup.sh` is the flag that keeps it out of the way. ## On macOS `nice` and `renice` are there and behave the same, including the refusal to go negative without `sudo`. `ionice` does not exist at all: I/O throttling on Darwin goes through `taskpolicy -d throttle`, which I covered [when comparing it to `taskset`](/posts/2025-12-17-taskpolicy-macos/). --- # jj: git without the staging area _Every change is a commit, and undo works_ I have written before about [rewriting git history](/posts/2024-01-github-desktop-rewrite-history/) and how easy it is to get wrong. [Jujutsu](https://jj-vcs.github.io/jj/) takes a different route: it keeps git's storage and replaces the model on top of it. After a few weeks with 0.36, two of its ideas have stuck with me. ## It sits on top of your git repository No migration, no second copy: ```bash $ git init $ jj git init --colocate Initialized repo in "." ``` `--colocate` means the directory is a working jj repo *and* a working git repo at the same time. Your colleagues clone it with git and never know. Your tooling keeps working. If you dislike it, `rm -rf .jj` and nothing is lost. That lowers the cost of trying it to about zero, and it is why I did. ## There is no staging area, because the working copy is a commit This is the idea everything else follows from. In jj, your uncommitted work *is* a commit, always: ```bash $ jj log -r @ @ nkzkrqnu test@example.com 2026-01-07 14:29:17 799678e8 │ (empty) (no description set) ``` An empty commit with no description, waiting. Edit a file and it is already in there: ```bash $ echo "bonjour" > a.txt $ jj status Working copy changes: A a.txt Working copy (@) : nkzkrqnu c2e51302 (no description set) Parent commit (@-): zzzzzzzz 00000000 (empty) ``` No `jj add`. There is nothing to add it to, since it is already part of the change. When you are happy, you describe it and start a new one: ```bash $ jj describe -m "add a.txt" $ jj new -m "second change" ``` `git add`, `git stash` and the entire index disappear from your vocabulary. Whether that is a loss depends on how much you liked partial staging, which jj handles differently with `jj split`. ## Rewriting history stops being frightening Here is the part that sold me. Three changes stacked, and I want to reword the first one: ```bash $ jj describe nkzkrqnu -m "add a.txt (reworded)" Rebased 2 descendant commits Working copy (@) now at: lvrrwwqy 2b5d2854 third change ``` `Rebased 2 descendant commits`. No interactive rebase, no detached HEAD, no editor listing `pick` lines. I named an old change, changed it, and everything built on top moved with it automatically. That works because a change has a stable **change ID** (`nkzkrqnu`) separate from its commit hash. Rewriting produces a new hash, and the change ID stays, so jj knows what the descendants were built on and where to put them. ## Undo, which git never really had ```bash $ jj undo Restored to operation: 2bd04c6434ee snapshot working copy ``` The reword is gone and the log reads as it did before. jj records every operation on the repository in an oplog, and `jj undo` steps back through it. Not "undo a commit": undo the last *thing you did*, whatever it was, including a rebase or an abandon. git has `reflog`, and anyone who has recovered work with it knows it is a forensic tool and not an undo button. This is an undo button. ## What it costs Version 0.36 is not 1.0, and the version number is honest about it: things move between releases and the documentation lags in places. The bigger cost is your own habits. Every git reflex has an equivalent, and none of them is spelled the same. `jj log` shows a graph of changes instead of a linear history. Branches are called bookmarks and do not move on their own. `jj git push` exists but you have to think about what it is pushing. And there is no useful GUI or IDE integration yet, so it is the terminal or nothing. I have not stopped using git, and colocation means I do not have to choose per repository, only per session. For a stack of changes I know I will reshuffle, jj wins outright. For everything else, muscle memory still wins. A fair description of where the tool is right now. --- # cal 9 1752: the eleven days that never existed _The month with a hole in it_ Every UNIX ships a calendar. Ask it for September 1752 and it gives you a month with a hole in the middle: ```bash $ cal 9 1752 September 1752 Su Mo Tu We Th Fr Sa 1 2 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 ``` The 2nd is followed by the 14th. Eleven days are missing, and this is not a bug. Your terminal is telling you about a piece of eighteenth century legislation. ## Eleven days removed by law The Julian calendar assumed a year of 365.25 days. It is about eleven minutes too long, and over sixteen centuries those minutes added up to a visible drift against the equinoxes. The Gregorian reform fixed the rule in 1582, and correcting the accumulated error meant deleting days from the calendar once. Britain and its colonies did not adopt it until 1752, by which point the gap had grown to eleven days. The Act made Wednesday 2 September 1752 be followed directly by Thursday 14 September. Nothing happened in between, because there was no in between. `cal` models this faithfully. It is not printing a generic grid, it is printing the calendar that was legally in force. ## Your terminal has an opinion about which country you are in Here is the part I did not expect. The eleven day gap is the *British* answer, and `cal` defaults to it. Other countries switched on other dates, and `ncal -p` lists them: ```bash $ ncal -p AT Austria 1583-10-05 IT Italy 1582-10-04 AU Australia 1752-09-02 JP Japan 1918-12-18 BE Belgium 1582-12-14 RU Russia 1918-01-31 GB United Kingdom 1752-09-02 TR Turkey 1926-12-18 GR Greece 1924-03-09 *US United States 1752-09-02 ``` Thirty-four countries, four centuries of disagreement, and a `*` marking the default. Russia held out until 1918, Turkey until 1926. `-s` picks one, and the hole moves. Note that `ncal` transposes the grid: weekdays run down the left, and a week is a column instead of a row. ```bash # France switched in December 1582, losing ten days $ ncal -s FR -m 12 1582 December 1582 Su 2 9 26 Mo 3 20 27 Tu 4 21 28 We 5 22 29 Th 6 23 30 Fr 7 24 31 Sa 1 8 25 ``` Read down the third column. That column is one week: Sunday the 9th, then Monday the 20th. Italy, first to adopt, lost ten days in October of that same year: ```bash $ ncal -s IT -m 10 1582 October 1582 Su 17 24 31 Mo 1 18 25 Tu 2 19 26 We 3 20 27 Th 4 21 28 Fr 15 22 29 Sa 16 23 30 ``` Thursday the 4th, then Friday the 15th. Three commands, three different histories, all correct. ## Week numbers, and a disagreement `ncal -w` adds a row of week numbers under the grid. Useful if you work anywhere that plans by week number, and a small trap if you assume both your machines agree: ```bash $ ncal -w -m 1 2026 January 2026 Su 4 11 18 25 Mo 5 12 19 26 Th 1 8 15 22 29 Sa 3 10 17 24 31 53 1 2 3 4 ``` That is Linux. The same command on macOS numbers the first column `1`, not `53`. The reason is visible one line higher: macOS starts its weeks on Monday and puts `Mo` in the first row, Linux starts on Sunday. January 1st 2026 is a Thursday, so with a Monday start its week is the first of the new year, and with a Sunday start that column began on December 28th and still belongs to the old one. ISO 8601 defines weeks as starting on Monday, so macOS is the one matching the standard here: ```bash $ date -j -f '%Y-%m-%d' 2026-01-01 '+%A, ISO week %V' Thursday, ISO week 01 ``` Note that `-w` belongs to `ncal` alone. `cal -w` prints a usage error on both systems. ## Where it lives macOS ships `cal` and `ncal` in `/usr/bin`, and both accept `-s`. I ran every example above on macOS and Ubuntu 25.10 and got identical output, byte for byte. Ubuntu does not install it by default: ```bash sudo apt install ncal ``` The package is called `ncal` and it provides both commands. Note that `cal --version` fails on both systems with `invalid option`, so do not go looking for it. Two flags worth keeping. `cal -3` prints last month, this month and next. That is the layout I actually want most of the time. And `ncal` is worth having in the muscle memory for its `-s`, even if the transposed grid takes a moment to read. One caveat if you script around this: `cal` is locale aware. My first run printed `Septembre` with French day names, and any parsing of that output breaks the moment it runs on a machine set to a different language. Prefix with `LC_ALL=C` when the output feeds anything but a human. --- # taskpolicy: the macOS answer to taskset _Ask, do not pin_ Last time I pinned processes to cores with [`taskset`](/posts/2025-12-03-taskset-cpu-affinity/). The natural next move is to do the same on my Mac, and the natural next discovery is that you cannot: ```bash $ command -v taskset numactl chrt cpuset $ ``` None of them. And this one is not a `brew install` away either. Homebrew does package util-linux for macOS, but the formula is explicit about what it cannot build there, and `taskset` is on that list: ```bash $ brew info util-linux The following tools are not supported for macOS, and are therefore not included: ... taskset ``` Which makes sense, because the tool would have nothing to call. Darwin does not expose a CPU affinity API the way Linux does. There is `thread_policy_set` with `THREAD_AFFINITY_POLICY`, but on Apple Silicon it is a hint the scheduler is free to ignore, and it does. That sounds like a gap. It is closer to a different philosophy, and macOS ships the tool that expresses it. ## taskpolicy, which nobody mentions It has been in `/usr/sbin` all along: ```bash $ taskpolicy -c background ./build.sh $ taskpolicy -b ./import.sh $ taskpolicy -d throttle ./backup.sh $ taskpolicy -p 12345 -b ``` Rather than naming cores, you declare what kind of work this is. `-c` sets a **QoS clamp** (`utility`, `background` or `maintenance`), `-b` puts the process in Darwin background priority, `-d throttle` throttles its disk I/O, and `-p` applies any of it to a process that is already running. The scheduler then decides where the work goes. On Apple Silicon that means choosing between performance and efficiency cores, and it takes power, thermals and everything else running into account, which you cannot do from a shell. ## Does the clamp actually do anything? This is the part I wanted to measure instead of assuming. My machine is an M3 Pro, and it is not homogeneous: ```bash $ sysctl -n machdep.cpu.brand_string Apple M3 Pro $ sysctl -n hw.ncpu hw.perflevel0.physicalcpu hw.perflevel1.physicalcpu 12 6 6 ``` Six performance cores, six efficiency cores. I ran the same busy loop for four seconds under each policy and counted iterations: ```text normal 59.05M iterations in 4s taskpolicy -c utility 58.76M iterations in 4s taskpolicy -c background 12.53M iterations in 4s taskpolicy -c maintenance 9.68M iterations in 4s taskpolicy -b 9.16M iterations in 4s ``` The clamp is real, and it is not subtle. `background` costs a factor of 4.7, `maintenance` 6.1, and Darwin background priority 6.4. Meanwhile `utility` is indistinguishable from normal, which tells you it still gets performance cores. That spread is the E-cores becoming visible from a shell prompt, without `sudo` and without a profiler. A tight integer loop on a P-core versus the same loop on an E-core is roughly the ratio you see above. ## Intent, not placement This is the mental shift, and it is worth stating plainly. On Linux you **name the cores** and the scheduler obeys. On macOS you **declare an intention** and the scheduler places the work. You cannot pin a process to the performance cores, and that is deliberate: the whole point of the design is that the system decides, so it can put your background indexing job on the efficiency cores while you type, and hand you the fast ones the moment you need them. Which means the two commands are not really equivalents. `taskset` is a constraint. `taskpolicy` is a request, with priorities attached. If your goal is "stop this job from making my laptop hot and loud", `taskpolicy` answers it better than `taskset` ever would. If your goal is a reproducible benchmark pinned to one core, macOS has no answer for you. ## Watching it happen Numbers are one thing, seeing it is another. [`asitop`](/posts/2024-01-apple-silicon-asitop/), which I wrote about a while back, reads the same counters as `powermetrics` and shows P-cluster and E-cluster load separately, plus package power in watts. ```bash sudo asitop ``` Start a heavy job normally, watch the P-cluster fill and the wattage climb. Run the same job with `taskpolicy -c background` and watch the load move to the E-cluster while the power draw drops. It's the clearest demonstration of the whole idea, and it takes two terminals and no instrumentation. One honest caveat: everything above is about *scheduling hints*, so results depend on what else the machine is doing. Run the measurement twice on a busy laptop and you will get two different ratios. The direction holds, the exact factor does not. --- # taskset: choosing which cores your command runs on _Pin it down_ You have a build that eats every core on the machine, and you would like to keep two of them for yourself. Or a benchmark whose numbers jump around because the scheduler keeps moving it between cores. Both are the same request: decide *where* a process runs. On Linux that is `taskset`. ## Affinity, not a quota The distinction matters, because three different tools get confused with each other. `taskset` sets the **CPU affinity**: the set of cores the scheduler is allowed to place the threads on. It does not cap how much CPU the process uses, it restricts where that usage can happen. `nice` changes priority: who goes first, not how much. And a real ceiling, "no more than 50% of the machine", is a cgroup thing. Affinity is a mask, one bit per core. ## Running something on specific cores My test box has six: ```bash $ nproc 6 $ cat /sys/devices/system/cpu/online 0-5 ``` Start a command on two of them: ```bash $ taskset -c 0,1 ./build.sh ``` `-c` takes a list: `0,1`, a range `2-5`, or a mix `0,2-4`. To read what a process has, point it at a PID: ```bash $ sleep 30 & $ taskset -cp $! pid 4238's current affinity list: 0-5 ``` `0-5` is the default: everything. And you can change it while the process runs, the flag combination worth remembering: ```bash $ taskset -cp 0 4238 pid 4238's current affinity list: 0-5 pid 4238's new affinity list: 0 ``` That process is now confined to core 0 without being restarted. ## The hex mask, and why -c exists Before `-c`, you wrote the mask yourself, in hexadecimal, one bit per core: ```bash $ taskset 0x3 ./build.sh ``` `0x3` is binary `11`, so cores 0 and 1. It is exactly equivalent to `-c 0,1`, and I checked that both produce the same affinity list. It's also how you end up computing `0x3f` in your head at 2am to mean "the first six cores". Use `-c`. ## Does it actually constrain anything? Worth proving instead of trusting. Four infinite loops, measured with `ps`, adding up their CPU shares: ```bash $ bash -c 'for i in 1 2 3 4; do (while :; do :; done) & done' 4 loops, unconstrained : 398% ``` Four cores saturated, as expected. Now the same four loops under `taskset`: ```bash $ taskset -c 0 bash -c 'for i in 1 2 3 4; do (while :; do :; done) & done' 4 loops, taskset -c 0 : 99% ``` 99%, one core, shared between four processes. Two things are proven at once: the restriction is real, and **affinity is inherited by children**. You set it once on the parent and the whole process tree obeys. That is what makes `taskset -c 0-3 make -j8` a useful thing to type. ## What it is genuinely good for Reproducible benchmarks are the honest use case. Pinning a benchmark to one core removes the migration noise and the cache invalidation that comes with it, so your numbers stop wobbling. Leaving yourself room is the other one. `taskset -c 0-3 make -j4` on a six-core box keeps two cores for your editor and your browser, and the machine stays usable. Beyond that, be careful. The Linux scheduler is good, and it has more information than you do about what the machine is doing. Pinning by hand mostly means telling it to ignore some of that. The cases where you win are narrow: NUMA locality, isolating a latency-sensitive thread, or reproducing a measurement. ## And the thing it cannot do `taskset` restricts to a set of cores. It cannot tell you *which* cores to prefer when they are not all the same, and it has no concept of a core being more or less power hungry. On a machine with performance and efficiency cores, that turns out to matter a lot, and Linux is not where you meet that problem first. My Mac is, and there `taskset` does not exist at all. What macOS offers instead is not an equivalent: you stop naming cores and start declaring intent, and the scheduler places the work. I measured what that costs on an M3 Pro in [the next post, about `taskpolicy`](/posts/2025-12-17-taskpolicy-macos/). --- # watch: the poor man's monitoring, in one command _Re-run it until something changes_ Everyone has written this loop, in some form: ```bash while true; do clear; df -h /var; sleep 2; done ``` Re-run a command, clear the screen, wait, repeat. It works, and `watch` does the same in eleven characters, plus a handful of things the loop cannot do at all. ```bash watch df -h /var ``` ## What you get for free The default interval is two seconds, and `-n` changes it: ```bash watch -n 1 df -h /var # every second watch -n 0.5 'ls -l | wc -l' # twice a second ``` Note the quotes on the second one. `watch` passes the command to `sh -c`, so an unquoted pipe would be interpreted by *your* shell and only `ls -l` would be watched. Quote anything with a pipe, a redirection or a semicolon. If you would rather skip the shell entirely, `-x` execs the command directly. The header shows the interval, the command and the host, which sounds cosmetic until you have four of these open and no idea which is which. `-t` removes it when you want the screen for the output. The flag that changes how you use the tool is `-d`, which highlights what changed since the previous run: ```bash watch -d -n 1 'ss -tn state established | wc -l' ``` Your eye stops scanning the whole screen and goes to the parts that moved. With `-d=permanent`, anything that has ever changed stays highlighted, so you can look away and still see what happened. ## The part nobody knows: it can stop on its own `watch` is usually described as an infinite loop. A shame, because three flags turn it into a condition to wait for. `-g` exits as soon as the output changes: ```bash watch -g -n 5 'systemctl is-active myservice' ``` That blocks until the service state moves, then returns. It's a wait-for-it primitive without a polling script. `-e` exits when the command itself fails: ```bash $ timeout 5 watch -n 1 -e false $ echo $? 1 ``` It stopped on the first failure instead of running for the full five seconds. Handy to babysit something that should keep succeeding, and to know the moment it does not. And `-q ` is the mirror image: exit when the output has *not* changed for that many cycles. Waiting for a queue to stabilise, or a file to stop growing, is a one-liner: ```bash watch -q 3 -n 2 'ls -l big.iso | awk "{print \$5}"' ``` Three identical readings and it returns. I keep meeting scripts that reimplement this with a counter and a comparison. ## It really does need a terminal `watch` draws with ncurses, so it wants a real terminal and says so when it does not have one: ```bash $ watch -n 1 date ncurses: cannot initialize terminal type ($TERM="unknown"); exiting ``` That is not a bug to work around, it's a design decision: the tool exists to redraw a screen. In a script or a CI job, the loop you were trying to replace is the right answer, or `-g`/`-q` if you only need the exit condition and can give it a `TERM`. ## Not on macOS ```bash $ command -v watch $ ``` `watch` comes from procps-ng, the Linux process tools, version 4.0.4 on my Ubuntu 25.10 box. macOS ships none of that family: ```bash brew install watch ``` Without Homebrew, the loop from the first paragraph is the fallback, and `clear` plus `sleep` gets you most of the way. What you lose is `-d`, the flag that actually makes `watch` worth installing. --- # systemd timers: cron, with a dry run _Check the schedule before you trust it_ `cron` has one property that has cost me more evenings than any other tool: you cannot ask it what it is going to do. You write five fields, you save, and you find out tomorrow. systemd timers replace it, and the reason to switch is not the syntax, it is that you can check your work. ## Two files instead of one line A timer is a pair. A `.service` that says what to run: ```ini # /etc/systemd/system/backup.service [Unit] Description=Nightly backup [Service] Type=oneshot ExecStart=/usr/local/bin/backup.sh ``` And a `.timer` that says when: ```ini # /etc/systemd/system/backup.timer [Unit] Description=Run the nightly backup [Timer] OnCalendar=daily Persistent=true [Install] WantedBy=timers.target ``` ```bash sudo systemctl enable --now backup.timer ``` Two files where cron had one line, which looks like a step backwards until you see what the second file buys. ## The flag cron never had `Persistent=true` means: if the machine was off when this should have run, run it as soon as it comes back. `cron` has no equivalent. A laptop asleep at 3am misses its 3am job, silently, and you find out when the backup is a week old. `anacron` exists to patch that hole, which tells you the hole is real. ## Checking the schedule before you trust it This is the feature that sold me. `systemd-analyze calendar` parses an expression and tells you what it understood: ```bash $ systemd-analyze calendar 'daily' Original form: daily Normalized form: *-*-* 00:00:00 Next elapse: [the next midnight] ``` It prints a normalized form and the next time it will fire, so a typo shows up immediately instead of on Sunday. Some more: ```bash $ systemd-analyze calendar 'weekly' Normalized form: Mon *-*-* 00:00:00 $ systemd-analyze calendar 'monthly' Normalized form: *-*-01 00:00:00 $ systemd-analyze calendar 'Mon..Fri 08:30' Normalized form: Mon..Fri *-*-* 08:30:00 $ systemd-analyze calendar '*:0/15' Normalized form: *-*-* *:00/15:00 ``` That last one is every fifteen minutes, and reading `*:0/15` back as `*-*-* *:00/15:00` is exactly the confirmation you want before enabling it. Compare with `*/15 * * * *`, which you either believe or you do not. The syntax itself is more readable too: named shortcuts, weekday ranges with `..`, and `DayOfWeek Year-Month-Day Hour:Minute:Second` in an order that matches how a date is written. ## What else you get Because the job is a unit, everything that applies to a service applies to it. Its output goes to the journal, so `journalctl -u backup.service` shows every run with timestamps and exit codes. That replaces cron's habit of emailing you output on a machine with no mail configured, or in other words discarding it. `systemctl list-timers` shows every timer, when it last ran and when it fires next, in one table. There is no `crontab` command that answers that question. Resource limits work: `MemoryMax`, `CPUQuota`, `Nice` and the rest go straight in the `.service`, so a runaway job is contained by the same mechanism as any other unit. And `RandomizedDelaySec=30m` spreads a job across a window, so a fleet of machines does not hit the same endpoint at the same second. Doing that in cron means generating a different crontab per host. ## When cron is still the answer For one line on one machine, `crontab -e` is faster and it is everywhere, including the containers and the BSDs where systemd is not. The switch pays off when a job matters: when missing a run has consequences, when you need its output afterwards, or when it runs on more than a handful of machines. That is also exactly when "I think that expression is right" stops being good enough, and being able to ask is worth two files. One caveat if you migrate: a timer's `OnCalendar` follows the system timezone unless you say otherwise, and cron's behaviour there differs by implementation. Set it explicitly, with `OnCalendar=Mon *-*-* 09:00:00 Europe/Paris`, and neither you nor the next person has to guess. --- # nohup, disown and setsid: surviving the hangup _What dies when you close the terminal_ You start a long job over SSH, the connection drops, and the job is gone. Everyone learns to prefix with `nohup` after that happens once. Fewer people know why it works, or that two other commands solve the same problem differently, and better in some cases. ## The signal behind it When a terminal goes away, the kernel sends `SIGHUP` to the processes attached to it. The name is literally "hang up", from the days when the terminal was a modem and the line dropped. The default action for that signal is to die. You can watch it happen without unplugging anything: ```bash $ sleep 30 & $ kill -HUP $! bash: line 11: 4162 Hangup sleep 30 ``` `Hangup`, and the process is gone. That is exactly what your job gets when the SSH session ends. ## nohup ignores it [`nohup`](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/nohup.html) runs a command with `SIGHUP` set to ignore. Same test: ```bash $ nohup sleep 30 >/dev/null 2>&1 & $ kill -HUP $! $ kill -0 $! && echo "still alive" still alive ``` The signal arrives and does nothing. That is the whole mechanism, and it is why `nohup` has to be there from the start: you cannot decide after the fact. The other half of what it does is about output. A detached job writing to a terminal that no longer exists is a problem, so `nohup` redirects standard output to `nohup.out` when stdout is a terminal. When it is not, it leaves your redirection alone: ```bash $ nohup echo test1 > /tmp/explicit.txt $ ls nohup.out ls: cannot access 'nohup.out': No such file or directory ``` No stray file, because I gave it somewhere to write. This is why `nohup cmd > log 2>&1 &` is the form worth memorising: you choose where the output goes instead of discovering a `nohup.out` in whatever directory you happened to be in. ## disown, for the job you already started `nohup` has to be decided in advance. `disown` is the fix when you forgot. It is a shell builtin, not a program, and it removes a job from the shell's job table: ```bash $ sleep 30 & $ jobs [1]+ Running sleep 30 & $ disown $ jobs $ pgrep -x sleep 4189 ``` The job list is empty, the process is still running. Since the shell only sends `SIGHUP` to jobs it knows about, forgetting the job is enough to protect it. Two consequences follow. You lose `fg`, `bg` and `wait` for that job, because the shell no longer tracks it. And the output still goes to the terminal, so `disown` alone will not save a job that writes to a screen that is about to disappear. Redirect first if you can. ## setsid goes further `nohup` and `disown` both leave the process attached to the same session and the same controlling terminal. `setsid` starts it in a brand new session, where it is the leader and has no terminal at all: ```bash $ setsid sleep 20 /dev/null 2>&1 & $ ps -o pid=,ppid=,sid=,pgid=,comm= -p $(pgrep -x sleep) 4170 4169 4170 4170 sleep ``` Look at the columns: PID 4170, and the session ID is also 4170. The process is its own session leader. Compare with a normal background job in the same shell: ```text 4174 4169 4169 4169 sleep ``` There the session ID is 4169, the shell's. That process belongs to the shell's session and will hear about it when the session ends. The `setsid` one is genuinely elsewhere, and it is how daemons are traditionally started. ## What you get on macOS `nohup` is there, in `/usr/bin`, and `disown` comes with your shell. `setsid` does not exist: ```bash $ ls /usr/bin/setsid ls: /usr/bin/setsid: No such file or directory ``` It comes from util-linux, which Apple does not ship, though `brew install util-linux` provides it. For a real background service on a Mac the answer is not a detached process anyway, it is a `launchd` plist, which restarts it and handles logging. And on Linux, past a certain point, the same is true of `systemd`. These three commands are for the job you are running now, not for the service you want running next month. The honest summary: `nohup` when you plan ahead, `disown` when you did not, `setsid` when you want the process to stop being yours. And `tmux` when you want to come back and look at it, and it is what I actually do most of the time. --- # flock: stopping a cron job from running twice _Overlap is the bug you never see_ You schedule a backup every five minutes. One day the backup takes six. Now two copies run at once, both writing the same file, and the result is a corrupted archive nobody notices until the day it's needed. Cron will happily start a job whose previous run has not finished, and it will never tell you. ## What overlap looks like Two instances of the same tiny job, started together, writing to the same file: ```bash $ for i in 1 2; do ( echo "start-$i" >> out.txt; sleep 1; echo "end-$i" >> out.txt ) & done; wait $ cat out.txt start-1 start-2 end-1 end-2 ``` Both started before either finished. With real work in the middle, that is two processes in the same directory, on the same temporary files, at the same time. ## flock, one word in front [`flock`](https://man7.org/linux/man-pages/man1/flock.1.html) takes a lock on a file and runs your command while holding it. Anyone else asking for the same lock waits their turn: ```bash $ for i in 1 2; do ( flock /tmp/lock -c "echo start-$i >> out.txt; sleep 1; echo end-$i >> out.txt" ) & done; wait $ cat out.txt start-1 end-1 start-2 end-2 ``` Fully serialised. The second run waited for the first to release, then went. Same script, one word added. The lock lives on a file, but it is not *in* the file: nothing is written to it, and its contents are irrelevant. It exists to be something both processes can point at. The kernel releases the lock when the process holding it exits, including when it crashes, the important part. A hand-rolled "create a PID file, delete it at the end" gets this wrong every time the job is killed. ## Wait, or give up Waiting is not always what you want. A backup that is still running probably should not queue another one behind it. `-n` fails instead of blocking: ```bash $ flock -n /tmp/lock -c "echo NEVER" $ echo $? 1 ``` Nothing ran, and you get exit code 1 to act on. When you would rather wait, but not forever, `-w` caps it: ```bash $ flock -w 3 /tmp/lock -c "echo acquired" acquired ``` That waited a second for the other job to finish, then took the lock and returned 0. When the wait runs out, you get the same code as `-n`: ```bash $ flock -w 2 /tmp/lock -c "echo NEVER" $ echo $? 1 ``` Two seconds, nothing run, exit 1. ## The line you actually write In a crontab, this is the whole point: ```cron */5 * * * * /usr/bin/flock -n /var/lock/backup.lock /usr/local/bin/backup.sh ``` If the previous backup is still going, this run exits immediately and quietly. No overlap, no PID file, no cleanup logic. Combine it with `chronic` from [moreutils](/posts/2025-07-30-moreutils/) and you have a cron job that is silent when things are fine, loud when they are not, and never runs twice. ## macOS does not have it ```bash $ ls /usr/bin/flock ls: /usr/bin/flock: No such file or directory ``` `flock` comes from util-linux, and Apple does not ship it. Homebrew does, though: ```bash brew install util-linux # provides flock ``` Worth knowing that this is not true of every util-linux tool: the formula lists the ones it cannot build on macOS, and `taskset` is on that list while `flock` is not. So you can have `flock` on a Mac, you cannot have `taskset`. Without Homebrew, macOS ships two relatives of its own, both in `/usr/bin`. `lockf` is the closest match and works the same way: ```bash $ lockf lk sh -c 'echo start >> out.txt; sleep 1; echo end >> out.txt' ``` Two of those serialise properly, as I checked. The flag for "do not wait" is `-t 0` instead of `-n`, and the exit code is different: ```bash $ lockf -t 0 lk echo NEVER lockf: lk: already locked $ echo $? 75 ``` 75, not 1. It is `EX_TEMPFAIL` from `sysexits.h`, more descriptive and completely incompatible with a script written against `flock`. If you check exit codes across both platforms, handle both values. The other one is `shlock`, older and PID-based: it writes a PID into the lock file and checks whether that process still exists. It works, and it has the failure mode `flock` was designed to avoid, since a PID can be reused. Prefer `lockf` when you have the choice. --- # at: scheduling a command to run once _Cron's forgotten cousin_ Everybody reaches for `cron` when something needs to happen later. But `cron` answers a different question: it schedules things that repeat. When you want a command to run once, at four in the afternoon, and never again, the tool is `at`, and it has been sitting in your `PATH` this whole time. ## Once, not every The interface is a pipe. You send it the command, you tell it when: ```bash $ echo "date > /tmp/at-result.txt" | at now + 1 minute warning: commands will be executed using /bin/sh job 11 at Mon Aug 31 00:47:00 2026 ``` A minute later, the file is there: ```bash $ cat /tmp/at-result.txt Mon Aug 31 00:47:00 UTC 2026 ``` It ran at 00:47:00, on the second, and the queue is empty again. No crontab entry to remember to delete afterwards, the part people always forget. Take that warning seriously: the job runs under `/bin/sh`, not your login shell. Any `bash` syntax you rely on is not available, and neither is your interactive environment. ## It reads almost like English The time parser is the reason to enjoy this command. All of these are valid, and I checked what each one resolved to: ```text now + 2 hours -> Mon Aug 31 02:45:00 10:00 -> Mon Aug 31 10:00:00 10:00 tomorrow -> Tue Sep 1 10:00:00 noon -> Mon Aug 31 12:00:00 midnight -> Tue Sep 1 00:00:00 teatime -> Mon Aug 31 16:00:00 next week -> Mon Sep 7 00:45:00 ``` Yes, `teatime`. It is 16:00, it is in the spec, and it has been there for decades. UNIX was not written by people in a hurry to be serious. Note that `midnight` means the *next* midnight, so a job sent at 00:45 lands 23 hours later, not in fifteen minutes. `now + 15 minutes` is what you meant. ## Looking at the queue Three commands, the entire interface: ```bash $ atq 11 Mon Aug 31 00:47:00 2026 a root $ at -c 11 # print the job, environment included $ atrm 11 # cancel it ``` `at -c` is the one worth knowing. It prints the whole script `at` saved, including a copy of every environment variable as it was when you submitted. That is how a job that worked in your terminal still works at 3am, and also how a stale `PATH` gets frozen into a job you wrote last week. ## The daemon nobody mentions `at` only queues. Something else has to wake up and run the jobs: `atd`. If it is not running, your job sits in the queue forever and you get no warning at all: ```bash sudo systemctl enable --now atd # Debian, Ubuntu ``` macOS is the sharper trap. All four commands are there, in `/usr/bin`, and submitting a job **succeeds**: ```bash $ echo "true" | at now + 1 minute job 2 at Mon Aug 31 02:47:00 2026 ``` Looks fine. But the daemon behind it ships switched off, and Apple says so in the file itself: ```bash $ plutil -p /System/Library/LaunchDaemons/com.apple.atrun.plist "Disabled" => true "Label" => "com.apple.atrun" ``` So nothing ever executes. You have to load it yourself: ```bash sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.atrun.plist ``` A command that accepts your job, prints a confident job number, and then silently does nothing is a special kind of unhelpful. If you script around `at`, check that the daemon is alive rather than trusting the exit code. ## batch, for when the machine is busy `batch` is `at` with a different trigger: instead of a time, it waits for the machine to be idle enough. The man page puts it as running "when the load average drops below 1.5 times number of active" CPUs, and `atd -l` lets you set your own limit. ```bash $ echo "./heavy-job.sh" | batch job 10 at Mon Aug 31 00:45:00 2026 ``` In `atq` it shows up in queue `=` instead of `a`. That is how you tell the two apart. It's a nice fit for a rebuild or a backup you want done today, without competing with whatever the machine is doing right now. Last thing: by default `at` mails you the job's output, which on a machine with no mail setup means the output disappears. Redirect to a file, as in the first example, and you keep it. --- # sleep and timeout: doing nothing, on purpose _Two ways to control the clock_ `sleep` is usually the second command people learn, right after `echo`, and then they never think about it again. Its sibling `timeout` is the one that stops a script from hanging until the end of time, and far fewer people know it. Both deserve a closer look, because both do more than their reputation suggests. ## sleep accepts more than integers The classic use is a whole number of seconds. It also takes decimals, on both macOS and Linux: ```bash sleep 0.3 ``` And suffixes, which many people believe are GNU-only. They are not: ```bash sleep 30s # seconds sleep 5m # minutes sleep 2h # hours sleep 1d # days ``` I checked on macOS, where `sleep 0.02m` really does wait 1.2 seconds. What it rejects is anything else: ```bash $ sleep 1x sleep: invalid time interval: 1x ``` The surprise is what happens with several arguments. Both implementations add them up: ```bash $ /usr/bin/time -p sleep 0.5 0.5 0.5 real 1,51 ``` 1.5 seconds, not 0.5. Handy for `sleep 1h 30m`, and a trap if you meant to pass one duration and a stray argument slipped in. Note that on macOS `sleep` lives in `/bin`, not `/usr/bin`, so a script hardcoding `/usr/bin/sleep` fails there with `no such file or directory`. Call it by name and let `PATH` do its job. ## timeout, and the exit codes that tell you what happened `timeout` runs a command and kills it if it takes too long. That single sentence hides three different outcomes, and they are distinguishable: ```bash $ timeout 5 echo ok ok $ echo $? 0 ``` The command finished in time, so you get its own status. Now the interesting one: ```bash $ timeout 1 sleep 5 $ echo $? 124 ``` **124 means "I killed it"**. It is the whole reason `timeout` is scriptable: you can tell a timeout apart from a genuine failure of the command, which no `&` plus `kill` hand-rolled version gives you for free. ```bash timeout 30 ./deploy.sh; rc=$? if (( rc != 0 )); then (( rc == 124 )) && echo "deploy timed out" || echo "deploy failed (rc=$rc)" fi ``` Capture the status into a variable on the line itself. I first wrote that block as `if ! timeout 30 ./deploy.sh; then [[ $? -eq 124 ]] ...` and it reports every timeout as a plain failure, because the `!` has already replaced `$?` with its own result by the time the branch runs. If you would rather see the signal that killed it, `--preserve-status` reports that instead: ```bash $ timeout --preserve-status 1 sleep 5 $ echo $? 143 ``` 143 is 128 + 15, meaning SIGTERM. ## When the command refuses to die `timeout` sends SIGTERM, which a program is free to ignore. Then your timeout does not time anything out, which defeats the purpose. `-k` adds a hard deadline after the polite one: ```bash $ timeout -k 1 1 bash -c 'trap "" TERM; sleep 10' $ echo $? 137 ``` That process explicitly ignores SIGTERM. After one second `timeout` asks nicely, after one more it sends SIGKILL, and 137 is 128 + 9. Nothing survives that. In any script that must not hang, `-k` is the difference between a timeout and a suggestion. ## The macOS gap `sleep` is everywhere. `timeout` is not: ```bash $ command -v timeout gtimeout $ ``` Nothing. It is a GNU coreutils tool, and macOS ships the BSD set. Homebrew's `coreutils` package installs it prefixed: ```bash brew install coreutils # then use gtimeout ``` Which means a script meant to run on both needs to pick the right name: ```bash TIMEOUT=$(command -v timeout || command -v gtimeout) "${TIMEOUT:?no timeout available}" 30 ./slow-thing ``` Not elegant, but honest, and it fails loudly on a machine where neither exists instead of silently running without a limit. One last thing worth knowing: `timeout` limits wall-clock time, not CPU time. A process stuck waiting on the network and one burning a core both hit the same deadline. If you want to cap CPU specifically, that is `ulimit -t`, and a different story. --- # pv: put a progress bar on any pipe _Watching bytes go by_ You start a `tar` over a few gigabytes, or restore a database dump, and the terminal goes quiet. Is it working? Is it stuck? Will it take a minute or an hour? The command has no idea it should tell you, and most of them never will. `pv`, for Pipe Viewer, adds the missing display to anything that moves bytes. ## A meter you drop into the pipe `pv` copies standard input to standard output, unchanged, and reports on standard error what went through: ```bash $ pv big.bin > /dev/null 2.86MiB 0:00:00 [5.38GiB/s] [================================>] 100% ``` Five things at a glance: how much passed, how long it took, the current rate, a bar, and the percentage. Because it only touches stderr, the data itself is untouched and you can put it anywhere in a pipeline: ```bash pv backup.sql | mysql mydb tar czf - /var/www | pv | ssh server 'cat > site.tgz' ``` The second one is my favourite shape: you get a live rate for a transfer that would otherwise be a black box. ## When it cannot know the size Give `pv` a filename and it stats the file, so it knows the total and can show a percentage. Put it in the middle of a pipe and it has no idea what is coming: ```bash $ cat big.bin | pv > /dev/null 2.86MiB 0:00:00 [ 476MiB/s] [<=> ] ``` Volume and rate are still there, but the bar has become a `<=>` marker bouncing left to right, and there is no percentage: it cannot compute one without knowing the total. If you know it, pass it with `-s`: ```bash $ cat big.bin | pv -s 3000000 > /dev/null 2.86MiB 0:00:00 [2.07GiB/s] [================================>] 100% ``` The percentage comes back. In a script, `-s $(stat -c%s file)` on Linux or `-s $(stat -f%z file)` on macOS gets it for you. ## Throttling, the underrated part `-L` caps the rate. This turns `pv` from a display into a traffic shaper: ```bash $ pv -L 500k small.bin > /dev/null 500KiB 0:00:01 [ 461KiB/s] [===============> ] 51% ETA 0:00:01 976KiB 0:00:01 [ 491KiB/s] [================================>] 100% ``` A megabyte held at 491 KiB/s instead of the 476 MiB/s the same machine does unthrottled. A factor of a thousand, on request. That matters when you are copying to a NAS over a link you share with other people, or restoring a dump on a live database and you would rather not saturate the disk. Slowing a job down on purpose is a real tool, and this is the shortest way to do it. ## The detail that confuses everyone Run `pv` in a script and redirect its output, and the bar vanishes. That is deliberate: `pv` checks whether stderr is a terminal, and stays quiet when it is not, so it does not pollute your logs with thousands of redraw lines. When you do want the output anyway, `-f` forces it: ```bash pv -f big.bin > /dev/null 2> progress.log ``` Every example in this post was captured that way, the only reason I can paste real bars here instead of describing them. ## Getting it `pv` is not in coreutils and not on macOS: ```bash sudo apt install pv # Debian, Ubuntu brew install pv # macOS ``` Version 1.10.3 is what shipped on my Ubuntu test box. Two caveats. `pv` measures bytes, not work: a compressed stream shows the compressed rate, so `pv` before `gzip` and `pv` after it report different numbers, and neither is wrong. And it adds a process to the pipeline, so on a very fast local copy you are measuring the pipe as much as the source. For anything network or disk bound, when you actually care, the cost disappears in the noise. --- # mkosi: one config file, twelve kinds of image _Declarative, and it burns to USB_ Building a bootable system image usually means `debootstrap` and a folder of shell scripts nobody wants to touch. [`mkosi`](https://github.com/systemd/mkosi) replaces that with an INI file, and it comes from the systemd project, which tells you something about the target audience. ## The whole thing is a config file ```ini # mkosi.conf [Distribution] Distribution=debian Release=trixie [Output] Format=directory ImageId=demo [Content] Packages=bash systemd curl ``` Three sections: which distribution, what shape of output, what goes inside. Then `mkosi build`. The verb to learn before that one is `summary`, which shows what it *would* build: ```bash $ mkosi summary DISTRIBUTION: Distribution: debian Release: trixie Output Format: directory Image ID: demo Packages: bash ``` Configuration layers across several files and drop-in directories, so being able to ask "what did all of that add up to" saves a build every time. `cat-config` goes further and prints the merged result. ## The format list is the real feature That one `Format=` line is where this stops being a container tool: ```text confext, cpio, directory, disk, esp, none, portable, sysext, tar, uki, oci, addon ``` Twelve artefact types from the same configuration and the same package list. `disk` gives a bootable GPT image. `uki` gives a Unified Kernel Image, kernel and initrd and command line in one signed PE binary. `sysext` and `confext` are systemd system and configuration extensions, layered onto a running immutable host. `portable` builds a portable service. And `oci` produces a container image, so the tool people describe as "not Docker" will hand you a Docker-compatible artefact if you ask. That is the argument for it. A Dockerfile is a script whose result depends on when you ran it, and it produces exactly one kind of thing. `mkosi.conf` is a description, and the output shape is a parameter. Bootloaders are a parameter too: `--bootloader` takes `systemd-boot`, `grub`, `uki`, and signed variants of each. `mkosi genkey` generates the keys, the part that makes Secure Boot approachable instead of a weekend. ## It is a lifecycle, not a builder The verbs after `build` are why I keep it around: ```bash mkosi shell # run a command in the image, without booting it mkosi boot # boot it under systemd-nspawn mkosi vm # boot it in a virtual machine mkosi ssh # get a shell in the running one mkosi journalctl # read the built image's journal mkosi coredumpctl # inspect crashes from it ``` `journalctl` and `coredumpctl` against a built image are the ones nobody expects. You build, it fails to boot, and you read its journal with the same command you would use on a live machine. That covers the gap I hit in [chrooting into a distribution](/posts/2024-05-chroot-to-any-linux-to-test-it/): a chroot gets you a userland and stops at anything involving init, the kernel or the boot path. Here the loop from "change a package list" to "did it boot" is two commands. ## Getting the image somewhere Three more verbs, and they cover the awkward last mile: ```bash mkosi serve # serve the output directory over HTTP mkosi burn /dev/sdX # write the image straight to a USB stick mkosi sysupdate # image-based updates, via systemd-sysupdate ``` `burn` is the one that makes it concrete: config file to bootable USB stick without `dd` and without looking up the arguments again. `sysupdate` is the other end, where the unit of deployment is a whole signed image and updating a fleet means publishing a new one. ## What it needs from you Unpacking a distribution means creating files owned by arbitrary users, so `mkosi` needs real privileges. Run it on a machine where you have root, or under its own `mkosi sandbox`. Without that it gets a fair way in and then stops at the first `chown`. That is the error you will see if you try it inside an unprivileged container. It is Linux only, since it leans on systemd tooling and Linux namespaces, so a Mac needs a VM around it. Install is `apt install mkosi` on Debian and Ubuntu, or straight from the repository. Version 26 is what I looked at here, and the configuration surface is wide enough that `summary` and `cat-config` are the two commands worth learning first. --- # mkfifo: a file that is actually a pipe _Zero bytes, and it carries everything_ The `|` you type between two commands creates a pipe that has no name and no existence outside that one command line. Both ends are born and die together. `mkfifo` gives you the same thing with a name on the filesystem, which changes what you can do with it. ## A file whose type is p ```bash $ mkfifo pipe1 $ ls -l pipe1 prw-r--r-- 1 root root 0 Aug 31 00:21 pipe1 ``` Look at the first character of the mode. Not `-` for a regular file, not `d` for a directory: `p`, for pipe. The tools agree: ```bash $ stat -c 'type=%F size=%s' pipe1 type=fifo size=0 $ file pipe1 pipe1: fifo (named pipe) ``` It behaves the same on macOS, where `stat -f '%HT'` answers `Fifo File`. This is POSIX, so you get it everywhere. ## It blocks, and that is the point Write to it with no reader, and your process stops: ```bash $ ( echo "message" > pipe1 ) & $ sleep 0.3; kill -0 $! && echo "writer still alive, blocked" writer still alive, blocked $ cat pipe1 message ``` The writer sits in `open()` until somebody opens the other end. This is not a limitation, it's the synchronisation mechanism: a FIFO connects two processes and makes them wait for each other, without a lock file or a polling loop. ## Nothing is ever stored The size stays at zero, whatever goes through: ```bash $ ( for i in 1 2 3; do echo "l$i"; done > pipe1 ) & $ stat -c 'size=%s' pipe1 size=0 $ cat pipe1 l1 l2 l3 ``` Three lines went through the file, and the file still holds nothing. The data lives in a kernel buffer between the two processes and never touches the disk. What the filesystem stores is the rendezvous point, not the content. That has a practical consequence: streaming 40 GB through a FIFO costs no disk space, where a temporary file would cost 40 GB. ## What you gain over the plain pipe The name is the whole difference. With `|`, both commands are on the same line, started by the same shell. With a FIFO, they can be anything, anywhere. Two terminals, two processes that never knew about each other: ```bash # terminal 1 mkfifo /tmp/sortme sort < /tmp/sortme # terminal 2 printf 'cerise\nbanane\npomme\n' > /tmp/sortme ``` Terminal 1 prints the sorted list. The two shells share nothing but a path. It also survives its uses, which surprises people. A FIFO is not consumed after one exchange: ```bash $ ( cat pipe1 & echo "first" > pipe1; wait ) first $ ( cat pipe1 & echo "second" > pipe1; wait ) second ``` The same FIFO handled both, and it is still there afterwards. It disappears when you `rm` it, like any other file. The classic real use is handing a stream to a program that insists on a filename instead of reading standard input. You give it the FIFO's path, and feed it from elsewhere. Process substitution in `bash` (`<(command)`) solves the same problem, and it is worth knowing that it does *not* go through a FIFO when it can avoid it: ```bash $ echo <(echo x) /dev/fd/63 $ ls -l <(echo x) lr-x------ 1 root root 64 /dev/fd/62 -> pipe:[8076713] ``` An anonymous pipe exposed under `/dev/fd`, not a named one. Bash falls back to real FIFOs only on systems without `/dev/fd`. So reach for `mkfifo` when you need the rendezvous to outlive a single command line, and for `<(...)` when you don't. ## What to watch out for Three things bite. A FIFO has no memory. If nobody is reading when you write, you block, and if nobody ever reads, you block forever. A script that opens one without a partner hangs with no error message, a miserable thing to debug. Writing from several processes at once is only safe below `PIPE_BUF` bytes per write, 4096 on Linux. Above that, two writers can interleave inside a single line. And a reader that goes away leaves the writer with `SIGPIPE`: ```bash $ ( yes > p; echo "writer exit=$?" ) & $ head -c 10 < p > /dev/null writer exit=141 ``` 141 is 128 + 13, the shell's way of saying the process died on signal 13. Worth recognising in a log, because it usually means the consumer stopped early, not that the producer failed. None of that makes FIFOs fragile, it makes them what they are: a pipe you can name, with a pipe's semantics and not a file's. --- # stdbuf: why your tail -f | grep prints nothing _The buffer nobody asked for_ You tail a log, filter it, and watch nothing happen: ```bash tail -f app.log | grep ERROR ``` The errors are in the file. `grep ERROR app.log` finds them. But live, the terminal stays empty for a long while, then dumps a block of lines at once. Nothing is broken. Your output went into a buffer, and the buffer is not full yet. ## Two modes, and the C library picks When a program writes to standard output, the C library decides how to batch those writes: - **line buffered** when stdout is a terminal, so you see each line as it is produced, - **block buffered** when stdout is anything else, and it waits until it has a full buffer's worth before writing. The choice is made on what stdout is attached to, not on what you meant. Alone in a terminal, `grep` is line buffered. Put a pipe after it and the same `grep` switches to blocks, because that is faster for the common case of piping into a file. The consequence is the one that confuses everyone: **adding a pipe changes the behaviour of the command before it.** ## Measuring it Rather than argue about it, here is the effect, timed. The producer emits one line every 0.5 s, and `ts -i` from [moreutils](/posts/2025-07-30-moreutils/) stamps how long each line took to arrive: ```bash $ (for i in 1 2 3 4; do echo "line $i"; sleep 0.5; done) | grep line | ts -i '%.s' 2.009094 line 1 0.000214 line 2 0.000040 line 3 0.000022 line 4 ``` Two seconds of silence, then all four lines within a fraction of a millisecond. `grep` held everything until its input closed. That is the bug you were chasing, and it is not in your code. ## stdbuf [`stdbuf`](https://www.gnu.org/software/coreutils/manual/html_node/stdbuf-invocation.html) runs a command with a different buffering mode, by preloading a small library that changes the defaults before `main` starts: ```bash $ (for i in 1 2 3 4; do echo "line $i"; sleep 0.5; done) | stdbuf -oL grep line | ts -i '%.s' 0.000006 line 1 0.487687 line 2 0.507754 line 3 0.507074 line 4 ``` Lines now arrive as they are produced. `-oL` means line buffered on stdout, `-o0` unbuffered, and `-eL`/`-e0` do the same for stderr. Good news for the cross-platform case: macOS ships `stdbuf` in `/usr/bin`, and it works the same way. The buffering problem itself is identical there, with the system `grep`: ```bash $ (for i in 1 2 3 4; do echo "line $i"; sleep 0.5; done) | /usr/bin/grep line | cat 46.317060000 line 1 46.325222000 line 2 46.330607000 line 3 46.334794000 line 4 ``` Four lines in 17 milliseconds, after two seconds of nothing. Same story, BSD grep 2.6.0 included. A detail that cost me a minute: if your `grep` is not the system one, the result changes. I had [ugrep](https://ugrep.com) first in my `PATH`, and it is line buffered by default, so the problem did not reproduce at all until I called `/usr/bin/grep` explicitly. ## Prefer the native flag when there is one `stdbuf` is the general answer, but several tools already know about this and their own flag is more reliable: ```bash tail -f app.log | grep --line-buffered ERROR tail -f app.log | sed -u 's/x/y/' tail -f app.log | jq --unbuffered . ``` `grep --line-buffered` gave me timings identical to `stdbuf -oL`, to the millisecond. ## Where stdbuf gives up `stdbuf` only changes what the C library does by default. A program that manages its own buffering ignores it entirely, and `awk` is the example that will bite you: ```bash $ (for i in 1 2 3; do echo "l$i"; sleep 0.4; done) | stdbuf -oL awk '{print}' | ts -i '%.s' 1.202923 l1 0.000240 l2 0.000038 l3 ``` Still batched. Worse, the usual advice of calling `fflush()` does not save you on Ubuntu, because the default `awk` there is `mawk`, and `mawk` buffers its *input*: it has not read your line yet, so there is nothing to flush. Two things actually work, and they differ per implementation: ```bash mawk -Winteractive '{print}' # mawk: line buffered i/o gawk '{print; fflush()}' # gawk: flush after each line ``` I checked all four combinations. `gawk` with `fflush()` streams, `gawk` without it does not. `mawk` with `-Winteractive` streams, `mawk` with `fflush()` does not. So the answer to "why is my pipeline silent" depends on which `awk` your distribution installed. A fine reason to name the one you mean in a script. --- # moreutils: the coreutils that never shipped _Fifteen tools you will want tomorrow_ Last week I wrote about [`sponge`](/posts/2025-07-16-sponge-command/), the tool that makes `sort file | sponge file` safe. `sponge` does not come from coreutils, it comes from [moreutils](https://joeyh.name/code/moreutils/), a package Joey Hess has been curating since 2006. Version 0.69 on Ubuntu installs fifteen commands, and several of them solve problems I used to write shell functions for. ```bash sudo apt install moreutils # Debian, Ubuntu brew install moreutils # macOS ``` Here are the ones I actually use. ## ts, for when things happened `ts` prefixes every line of a stream with a timestamp. It's the fastest way to find out which step of a slow script is the slow one: ```bash $ printf 'start\ndone\n' | ts '%H:%M:%.S' 23:40:41.898510 start 23:40:41.898560 done ``` The format is `strftime`, so you can print whatever you like. The flag that matters most is `-i`, which prints the time elapsed *since the previous line* instead of the clock: ```bash $ (echo start; sleep 1; echo end) | ts -i '%.s' 0.000007 start 0.985476 end ``` Pipe a build log through `ts -i` and the offender stands out without any instrumentation. ## chronic, the cure for cron spam Cron mails you everything a job prints. So a job that chats on stdout mails you every night, you add `> /dev/null`, and now you have silenced the error output you actually wanted. `chronic` fixes the logic: run the command, stay quiet if it succeeds, print everything if it fails. ```bash $ chronic echo "you will not see this" $ echo $? 0 $ chronic sh -c 'echo output; echo error >&2; exit 3' output error $ echo $? 3 ``` Both streams come back on failure, and the original exit code is preserved. A crontab line becomes `chronic /path/to/job`, and you hear from it when something is wrong. ## ifne, for empty streams `ifne` runs a command only if standard input is not empty: ```bash $ : | ifne echo "NOT CALLED" $ echo x | ifne echo "CALLED" CALLED ``` It pairs naturally with the [`xargs` empty-input trap](/posts/2025-07-02-xargs-command/): where `xargs -r` protects one command, `ifne` guards a whole pipeline. `find . -name '*.err' | ifne mail -s "errors found" me@example.com` sends nothing on a quiet day. ## errno, for the number in the log You have `errno = 2` in a stack trace and no idea what it means: ```bash $ errno 2 ENOENT 2 No such file or directory $ errno ENOENT ENOENT 2 No such file or directory ``` It works in both directions, and `errno -l` prints all 134 of them. Small, but it saves a search every time. ## pee, tee for commands Where `tee` sends a stream to several files, `pee` sends it to several commands: ```bash $ echo "bonjour" | pee "wc -c" "tr a-z A-Z" BONJOUR 8 ``` Note the order: the commands run in parallel, so their outputs arrive as they come, not in the order you wrote them. Fine when each command does its own thing and the order does not matter, useless when you need the outputs in a predictable one. ## vidir and vipe, editing in the middle `vidir` opens a directory listing in your editor. Rename a line, the file is renamed. Delete a line, the file is deleted: ```bash $ ls a.txt b.txt $ EDITOR='sed -i s/a.txt/renamed.txt/' vidir . $ ls b.txt renamed.txt ``` Bulk renaming with your editor's multi-cursor, and it beats any `rename` invocation I have written. `vipe` is the same idea for a pipe: it drops your editor between two commands so you can hand-edit the stream before it moves on. Worth knowing that it needs a real terminal, and fails with `reopen stdin: No such device or address` if you try it from a script. ## Two more worth a mention `mispipe` pipes two commands but returns the exit status of the *first* one. That is what you wanted every time you reached for `pipefail`. And `combine` does set algebra on files with readable keywords: ```bash $ cat x.txt $ cat y.txt a b b c c d $ combine x.txt and y.txt # b c $ combine x.txt not y.txt # a $ combine x.txt xor y.txt # a d ``` A whole class of `sort`/`comm` incantations replaced by a sentence. Watch out for `or`, which concatenates without deduplicating, so it gives `a b c b c d` rather than an union. None of these will change your life on their own. Together they cover the gaps that make you write a 20-line helper script. Exactly what a good toolbox does. --- # sponge: why sort file > file empties your file _Truncated before it even began_ Sort a file in place. It looks like it should work, and it is the fastest way to lose data I know of: ```bash $ printf 'banane\npomme\ncerise\n' > f.txt $ sort f.txt > f.txt $ wc -c < f.txt 0 ``` Three lines in, zero bytes out. No error, no warning, no file. ## The shell opens the file first The explanation is the same one behind [why `sudo command > file` fails](/posts/2025-06-18-tee-command/): redirection is the shell's job, and the shell does it before your command exists. When you type `sort f.txt > f.txt`, the shell: 1. opens `f.txt` for writing, which truncates it to zero length, 2. forks and runs `sort f.txt`, 3. `sort` opens an empty file, reads nothing, writes nothing. The truncation happens first, so `sort` never sees your data. The order is not a bug, it is how `>` has always worked: the shell sets up the file descriptors, then hands over. ## The workaround, and its cost The reflex is a temporary file: ```bash sort f.txt > tmp && mv tmp f.txt ``` It works, and it is what most scripts do. The cost shows up in the details: you need a name that will not collide, you leave `tmp` behind if the command fails halfway, and `mv` across filesystems is a copy, which changes the inode, the permissions and the ownership. ## sponge [`sponge`](https://joeyh.name/code/moreutils/) reads its entire input before opening the output. That single difference makes the in-place pipeline safe: ```bash $ printf 'banane\npomme\ncerise\n' > f.txt $ sort f.txt | sponge f.txt $ cat f.txt banane cerise pomme ``` Same shape as the broken version, correct result. Because it soaks up all of stdin first, `sponge` also means the write happens in one go at the end, so a command that dies mid-stream leaves the original file untouched instead of half-rewritten. It appends too: ```bash $ echo datte | sponge -a f.txt $ cat f.txt banane cerise pomme datte ``` The pattern generalises to any filter. That is where it earns its place: ```bash grep -v '^#' config.ini | sponge config.ini jq '.version = "2.0"' package.json | sponge package.json ``` That `jq` line is the one I use most, because `jq` has no in-place mode at all: ```bash $ echo '{}' | jq -i . jq: Unknown option -i ``` `sponge` is the shortest correct substitute. ## Getting it `sponge` is not part of coreutils. It ships in [moreutils](https://joeyh.name/code/moreutils/), Joey Hess's collection of the tools that never made it into the standard set: ```bash sudo apt install moreutils # Debian, Ubuntu brew install moreutils # macOS ``` Nothing has it by default, macOS included, so a script that relies on it needs to say so. ## When not to use it The buffering that makes `sponge` safe is also its limit: the entire input lives in memory before anything is written. On a multi-gigabyte file that is a problem, and the temporary-file dance is the better answer. For the narrow case of editing a file with a stream editor, the tool already has it built in and you need neither: ```bash sed -i 's/foo/bar/' file.txt ``` Note that `sed -i` wants an argument for the backup suffix on macOS (`sed -i '' ...`) and does not on Linux. Which is a portability trap of its own, and another story. --- # xargs: turning a text stream into arguments _Because not every command reads stdin_ Pipes carry text into a program's standard input. That works beautifully for `grep`, `sort` or `wc`, and not at all for `rm`, `mkdir`, `kill` or `git`, which expect their targets as arguments. `xargs` is the adapter between them: it reads a stream and turns it into a command line. ## The gap it fills ```bash $ echo /tmp/old.log | rm rm: missing operand Try 'rm --help' for more information. ``` `rm` never looked at the pipe. It wants arguments, found none, and said so. `xargs` fixes the mismatch: ```bash $ echo /tmp/old.log | xargs rm ``` It reads the words on standard input and appends them to the command you gave it. That's the whole idea, and everything else is about controlling how it batches them. ## The space that breaks everything Here is the classic failure, reproduced on Ubuntu with two files, one of which has a space in its name: ```bash $ ls autre.txt 'mon fichier.txt' $ ls | xargs rm rm: cannot remove 'mon': No such file or directory rm: cannot remove 'fichier.txt': No such file or directory ``` `xargs` splits on whitespace by default, so `mon fichier.txt` became two arguments. Worse than an error: if a file named `mon` had existed, it would have been deleted instead. The fix is to change the separator to a byte that cannot appear in a filename, and the only such byte is the null one: ```bash $ find . -name '*.txt' -print0 | xargs -0 rm ``` `find -print0` terminates each path with `\0`, and `xargs -0` splits on it. Whenever the input is filenames, this pair is the correct spelling. Not "safer": correct. The whitespace version is the one that happens to work when nothing is unusual. ## Batching, and placing the argument By default `xargs` packs as many arguments as fit into one command line. `-n` caps it: ```bash $ printf 'a\nb\nc\nd\n' | xargs -n2 echo run: run: a b run: c d ``` Two invocations instead of one. When the command needs the argument somewhere other than at the end, `-I` gives it a placeholder: ```bash $ printf 'un\ndeux\n' | xargs -I{} echo "file [{}] processed" file [un] processed file [deux] processed ``` `-I` implies one argument per run. GNU tells you so if you combine it with `-n`: ```text xargs: warning: options --max-args and --replace/-I/-i are mutually exclusive, ignoring previous --max-args value ``` macOS accepts the same combination without a word, its own kind of unhelpful. ## Free parallelism The flag I reach for most is `-P`, which runs several commands at once. Eight one-second sleeps, four at a time: ```bash $ time (seq 1 8 | xargs -P4 -I{} sleep 1) real 0m2.020s $ time (seq 1 8 | xargs -P1 -I{} sleep 1) real 0m8.062s ``` Four times the throughput for four characters of typing, on both macOS and Linux. For anything CPU-bound, `-P$(nproc)` turns a serial loop into a parallel one without writing a single line of job control. Keep in mind that the outputs interleave. If each job prints more than one line, you get them shuffled together, so redirect per job or accept the mess. ## The portability trap This one deserves care, because it fails dangerously. Give `xargs` an empty input: ```bash $ : | xargs echo "CALLED ANYWAY" ``` On macOS, nothing happens. On Ubuntu and Debian, it prints `CALLED ANYWAY`: GNU `xargs` runs the command once even with no arguments. So... a script that reads `find ... | xargs rm -rf` and finds nothing does nothing on your Mac, and runs `rm -rf` with no target on the server. The portable answer is `-r`, short for `--no-run-if-empty`: ```bash find . -name '*.tmp' -print0 | xargs -0 -r rm ``` Two flags for two separate problems, and they stack. `-0` is the one from earlier, protecting the filenames. `-r` protects the empty case. GNU changes its behaviour to match, and macOS accepts the flag without needing it, since not running is already what BSD does. The same line then behaves identically on both. --- # tee: writing to a file you are not allowed to write to _The pipe that forks_ Everyone meets this one eventually. You need to drop a line into a file under `/etc`, you remember to prefix with `sudo`, and the shell tells you no anyway: ```bash $ sudo echo "net.ipv4.ip_forward=1" > /etc/demo.conf bash: /etc/demo.conf: Permission denied ``` You used `sudo`. You still got `Permission denied`. The reason is worth understanding, because it explains a whole family of shell surprises. ## The redirection happens before sudo `>` is not part of the command. It's the shell's own syntax, and the shell handles it first: it opens the target file, then forks and executes what's left. So the sequence is: 1. your shell, running as you, tries to open `/etc/demo.conf` for writing, 2. it fails, because you are not root, 3. `sudo` is never reached. `sudo` would have elevated `echo`, which never needed the privilege. The thing that needed it was your shell. That's why `sudo command > file` fails while `sudo command` alone works. ## tee, the T-junction [`tee`](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/tee.html) is named after the plumbing fitting. It reads standard input, writes it to standard output *and* to the files you name. Put it on the far side of the pipe and it becomes the process doing the writing, so `sudo` applies to the right thing: ```bash $ echo "net.ipv4.ip_forward=1" | sudo tee /etc/demo.conf net.ipv4.ip_forward=1 $ cat /etc/demo.conf net.ipv4.ip_forward=1 ``` Note that it echoed the line back: `tee` passes its input through, so it chains. Most of the time you don't want that on your terminal, so send stdout to the bin: ```bash echo "second line" | sudo tee -a /etc/demo.conf > /dev/null ``` `-a` appends instead of truncating, the same distinction as `>>` versus `>`. Forgetting it is how people overwrite a config file they meant to extend. ## More than one output The `sudo` trick is the famous use, but `tee` accepts several files at once, handy for keeping a log of something you are also watching: ```bash # wc -l is only here to prove the line came out the other side $ echo bonjour | tee /tmp/a /tmp/b | wc -l 1 $ cat /tmp/a /tmp/b bonjour bonjour ``` One input, two files on disk, and the stream still reaching `wc` at the end. This is the shape I use most: run a build, watch it scroll, and keep the full output for later. ```bash make 2>&1 | tee build.log ``` `2>&1` matters there. `tee` only ever sees standard output, so without it your errors go straight to the terminal and never reach the log, exactly when you need them. ## Where it stops Two limits worth keeping in mind. `tee` writes as it reads, so a crash halfway through leaves a half-written file. If you are transforming a file in place, that is the wrong tool, and it produces the classic empty-file disaster that deserves its own post. The other is that `tee -a` on a shared file is only atomic for small writes. Two processes appending long lines at once can interleave them. For real concurrent logging, let `syslog` or your init system handle it. Small caveat on exit codes, since it bites in CI: in a pipeline, the shell reports the status of the *last* command, so `make | tee build.log` returns `tee`'s success even when `make` failed. Add `set -o pipefail` in `bash`, or read `${PIPESTATUS[0]}`, otherwise a red build looks green. --- # units: how many furlongs per fortnight is that? _Three thousand units, one command_ Converting miles to kilometres usually means opening a browser tab. Your machine has had a tool for it since long before browsers, and it knows about three and a half thousand units, including several that exist only as jokes. ```bash $ units -t "10 miles" "km" 16.09344 ``` `-t` is terse mode: one line, just the number, ready to drop into a script. Without it you get an interactive prompt. ## It understands compound units The interesting part is not converting one unit to another, it's that `units` does dimensional algebra. You give it an expression, it works out whether the two sides are compatible: ```bash $ units -t "1 acre" "m2" 4046.8564 $ units -t "2 GB / 3 minutes" "Mbit/s" 88.888889 ``` That second one is a real question I have needed answered, and the arithmetic is the part I would have got wrong. And since the post needs to earn its title: ```bash $ units -t "furlongs/fortnight" "m/s" 0.00016630952 ``` A furlong per fortnight is about a sixth of a millimetre per second. The unit database is [8275 lines](https://www.gnu.org/software/units/) defining 3480 units, and it contains things like `smoot`, defined as `5 ft + 7 in` with the comment "Created as part of an MIT fraternity prank": ```bash $ units -t "1 smoot" "m" 1.7018 ``` ## The trap that will get you Write the most natural thing in the world and it fails: ```bash $ units -t "60 mph" "km/h" conformability error 26.8224 m / s ``` `mph` is fine. The problem is `h`, and the reason is beautiful: ```bash $ grep -E '^h[[:space:]]' /usr/share/units/definitions.units h 6.62607015e-34 J s # Planck constant (exact) ``` `h` is the Planck constant. You asked for kilometres per Planck constant, and `units` correctly told you that is not a speed. Spell it out and it works: ```bash $ units -t "60 mph" "km/hour" 96.56064 ``` This is what a tool looks like when physics matters more to it than your habits. ## Temperature is a function, not a factor Converting temperatures is different from converting lengths, because the scales have different zeros. Multiplying by a factor gives the wrong answer, so `units` exposes them as functions: ```bash $ units -t "tempF(72)" "tempC" 22.222222 ``` Note the parentheses. `degF` also exists and means something else: the *size* of a degree, for temperature differences instead of temperatures. ## What you get on each system Here the two implementations part ways. Linux has GNU Units: ```bash sudo apt install units ``` macOS ships its own, identifying itself as `Darwin units`, and the temperature functions are missing: ```bash $ units -t "tempF(72)" "tempC" unknown unit 'tempF(72)' ``` Everything else in this post gave identical results on both, including the Planck trap and the smoot. So conversions are portable, temperatures are not. On a Mac, `brew install gnu-units` gets you the full version as `gunits`. The honest caveat is that `units` answers the question you asked, and dimensional analysis is unforgiving about what that question was. A conformability error almost never means the tool is wrong. --- # dc: the calculator that is older than C _Reverse Polish, from the PDP-11_ Most people who need arithmetic in a shell reach for `bc`. Almost nobody reaches for `dc`. A shame, because `dc` is what `bc` was originally built on top of, and it is very probably one of the oldest program still shipping with your system. ## Older than the language UNIX is written in `dc` was written by Lorinda Cherry and Robert Morris at Bell Labs. It is the [oldest surviving UNIX language program](https://en.wikipedia.org/wiki/Dc_(computer_program)), written in **B**, and when Bell Labs got its PDP-11 it was the first language to run on the new machine, before there was even an assembler for it. It predates C. The operating system it ships with was rewritten around it. `bc` came later, also from Lorinda Cherry, as a friendlier front end: it took a C-like syntax, compiled it to `dc` notation and piped the result through `dc`. That plumbing is gone today. GNU `bc` is a standalone program, and running it under `strace` shows a single `execve`, its own. ## Everything is a stack `dc` uses reverse Polish notation. You push values, then apply an operator to what is on the stack, and `p` prints the top: ```bash $ echo "2 3 + p" | dc 5 ``` No parentheses, ever, because the order is unambiguous. Nesting is just ordering: ```bash # 5 + (3 * 2) $ echo "5 3 2 * + p" | dc 11 ``` That takes ten minutes to get used to and then becomes hard to give up. ## Arbitrary precision, with no ceremony This is the reason to keep it around. `dc` has no integer size limit: ```bash $ echo "2 200 ^ p" | dc 1606938044258990275541962092341162602522202993782792835301376 ``` Two to the two hundredth, exact, in a program from 1970. Ask your shell for the same thing and its 64-bit integers give up without saying so: ```bash $ echo $((2**200)) 0 ``` Not an error, not a warning. Zero. Decimals need you to say how many you want, with `k`: ```bash # 10 decimal places, then square root of 2 $ echo "10 k 2 v p" | dc 1.4142135623 ``` `k` sets the precision, `v` is the square root. The default precision is 0, so without `k` you get integer division and a lot of confusion. ## Base conversion in two characters `o` sets the output base, `i` the input base: ```bash $ echo "16 o 255 p" | dc FF $ echo "2 o 10 p" | dc 1010 ``` Hexadecimal and binary without `printf`, and it works for any base, not the three that `printf` knows. ## Where it lives `dc` is on macOS out of the box. Debian and Ubuntu ship neither `dc` nor `bc` by default, so both need installing: ```bash sudo apt install dc bc ``` The implementations differ, `dc 7.0.3` on my Mac against GNU's `1.4.1`, and every example in this post gave byte-identical output on both. The honest limit is readability. `10 k 2 v p` is fast to type and impossible to review six months later. That is exactly why `bc` was written. Use `bc` when a human will read the expression again, and `dc` when you want a big number right now, or when you want to feel briefly like it is 1970. --- # expect: when yes is not the answer to every question _When one 'y' won't do_ Last year I wrote about [the `yes` command](/posts/2024-02-yes-command/) and ended on a cliffhanger: when a script waits for different answers, `yes` won't help, and you may reach for `expect` instead. I left it there. Time to keep my word. ## Where yes gives up Here is a small installer that asks three questions. The first two take whatever you type, but the third one checks it, and wants the word `yes` in full: ```bash #!/bin/bash read -p "Project name: " name read -p "Use TypeScript? (y/n) " ts read -p "Confirm creation of '${name}'? (yes/no) " ok [[ "${ok}" == "yes" ]] || { echo "Aborted."; exit 1; } echo "Created ${name} (typescript=${ts})" ``` Save it as `setup.sh` and feed it with `yes`: ```bash $ chmod +x setup.sh $ yes | ./setup.sh Aborted. $ echo $? 1 ``` That's the whole problem in three lines. `yes` with no argument repeats `y` forever, so the project got named `y`, TypeScript got `y`, and the confirmation got `y` where the script wanted the word `yes`. One answer for every question means a wrong answer to most of them. ## Say what you expect [`expect`](https://core.tcl-lang.org/expect/index) was written by Don Libes at NIST, and presented at the Summer 1990 USENIX conference in a paper with the best title in the field: "expect: Curing Those Uncontrollable Fits of Interactivity". It drives an interactive program through a pseudo-terminal, so it believes a human is typing. You describe the conversation: wait for this prompt, send that answer. ```tcl #!/usr/bin/expect -f set timeout 10 spawn ./setup.sh expect "Project name: " { send "labs\r" } expect "Use TypeScript? " { send "n\r" } expect "Confirm creation of 'labs'? " { send "yes\r" } expect eof exit [lindex [wait] 3] ``` The first line is the shebang, which tells the kernel what to run the file with. Everything after it is Tcl: Expect is not a language of its own but an extension of one, so `set` and `lindex` are Tcl commands, while `spawn`, `send`, `expect` and `wait` are what Expect adds on top. `spawn` starts the program, each `expect` blocks until its pattern shows up, and `send` types the answer (`\r` is the Return key, not `\n`). The last two lines matter more than they look: `expect eof` waits for the program to finish instead of killing it mid-run, and `wait` gives you back its exit code so a CI job can act on it. Those square brackets are Tcl's command substitution, the equivalent of backticks in a shell. `wait` returns a list of four integers, the last of which is the status of the spawned process, so `[lindex [wait] 3]` takes that one and hands it to `exit`. Save it as `setup.exp` and run it: ```bash $ chmod +x setup.exp $ ./setup.exp spawn ./setup.sh Project name: labs Use TypeScript? (y/n) n Confirm creation of 'labs'? (yes/no) yes Created labs (typescript=n) $ echo $? 0 ``` Three questions, three different answers, and the real exit code. That's what `yes` structurally cannot do. ## Let autoexpect write it for you Writing those patterns by hand gets tedious. `autoexpect` records a real session and hands you the script: ```bash autoexpect ./setup.sh ``` You answer the questions once, and it writes `script.exp`, ready to replay. It's honest about itself, too. The generated header warns that it "does not guarantee a working script" and explains that it has to guess about timing and about which parts of the output are stable. Treat it as a first draft: it captures the conversation, you delete the noise and loosen the patterns that were too specific. ## Two traps worth knowing The first one cost me a few minutes while testing this post. On macOS I wrote the shebang as `#!/usr/bin/env expect -f` and it ran fine. The same file on Ubuntu: ```bash $ ./setup.exp env: 'expect -f': No such file or directory env: use -[v]S to pass options in shebang lines ``` Linux passes everything after the interpreter as a single argument, so `env` looks for a program literally named `expect -f`. Use `#!/usr/bin/expect -f` and it works on both. That `env` behaviour deserves its own post one day. The second trap is availability. macOS ships `expect` 5.45 out of the box, but only `expect`: no `autoexpect`, no `unbuffer`. On Ubuntu the `expect` package brings it all: ```bash sudo apt install expect # Debian, Ubuntu brew install expect # macOS, if you want autoexpect ``` And the honest limit: `expect` matches on output, so it's only as reliable as the prompts it waits for. Reword a prompt and the script hangs until `timeout` fires. When a program offers a `--yes`, a `--non-interactive` or a config file, that's still the better answer. `expect` is what you use when nobody left you one. --- # Back to write _But not to basics_ It's been about 6 months since my last post on this blog. Why? I was focused. Mostly because of work stuff, but also due to personal changes, managing my fun/office balance, exploring AI and how it changes my relationship to technology, the way I produce things. The main effect was that I lacked time to share what I was discovering on a regular basis, and that's bad. I devoted a significant part of my life to tech stuff in order to learn things and share them widely. I did so as a journalist for 20 years, and I still see this as a deep commitment. I took the opportunity of a short break to think about how I could reorganize things and be able to spend more time sharing my discoveries again. No, this blog won't be written by an AI or any other kind of agent. But maybe I'll explain how to create such things, or with better ideas. Wait n' see ;) Take Care. --- # How to create a CLI in V _That's what "batteries included" means_ As some of you may know, I'm a fan of [V](https://vlang.io/), a programming language inspired by [Go](https://go.dev/) but trying to do better on many fronts, with great tooling and native libraries. You can learn more in [its documentation](https://docs.vlang.io/introduction.html), using [its playground](https://play.vlang.io/) or watching [this quickie session](https://www.youtube.com/watch?v=YEiWEiamXrk) from Devoxx France 2024. I've already covered some aspects of V in a previous article detailing [how to create a tiny web server](/posts/2024-02-how-own-web-server-vlang/), which led me [to publish tVeb](https://github.com/davlgd/tVeb). More recently, I decided to explore the `cli` module of V, whose aim is to provide a simple way to create applications with commands, flags, help, man, etc. So I decided to create a simple CLI using another great included module: `compress`. As you may have guessed, it compresses data. In my case, files. To follow this tutorial [you'll need V](https://docs.vlang.io/installing-v-from-source.html) and a file editor, nothing more. ## Compressing files with V Let's create a new folder and a `main.v` file. For this first step, it creates a text file and compresses it with `gzip`. Here we'll assume the folder is empty, so we don't have to check if the file already exists for example: ```v import os import compress.gzip fn main() { // Define the file names filename := 'test.txt' compressed_filename := '${filename}.gz' // Create a simple text file // The `!` is a way to ignore errors os.write_file(filename, 'Hello, world!')! // Read the file, in bytes to pass to the compressor content := os.read_bytes(filename)! compressed_content := gzip.compress(content)! // Create the compressed file, or open it if it already exists // Here there is an error handling, change `create` by `open` to test it mut file_compressed := os.create(compressed_filename) or { eprintln('Impossible to access $compressed_filename') exit(1) } // Write the compressed content to the file file_compressed.write(compressed_content)! file_compressed.close() println("✅ File compressed to '$compressed_filename'!") } ``` Run it with `v run main.v`. You should see a `test.txt.gz` file in the same folder. You can check it has been compressed correctly with : ```bash $ gzip -d test.txt.gz -c Hello, world! ``` ## Zstd and its parameters The `compress` module provides a wrapper around [Zstandard](https://facebook.github.io/zstd/) (learn more [with Hubert](https://www.youtube.com/watch?v=BVM5vsPYbfg)). It allows to natively define a compression level and how many CPU threads to use. To do so, just edit some lines of the code above: ```v ... import compress.zstd ... compressed_filename := '${filename}.zst' ... compressed_content := zstd.compress( content, compression_level: 8, nb_threads: 4)! ... ``` You should see a `test.txt.zst` file in the same folder. If `zstd` is installed on your system, you can check it has been compressed correctly with: ```bash $ zstd -d test.txt.zst -c Hello, world! ``` ## Let's start to CLI What if we want to define compression level and number of threads as parameters and not hardcode them? It's where the `cli` module helps. It allows to define an application, its name, version, description, add commands and flags to it. Here we'll just define a main command to compress a file and add flags to define compression level and number of threads: ```v import os import runtime import cli { Command, Flag } fn main() { mut app_cli := Command{ name: 'Compressor' description: 'A tiny CLI to compress files with Zstandard' version: '0.1.0' execute: compress } app_cli.add_flag( Flag{ flag: cli.FlagType.int name: 'level' abbrev: 'l' description: 'Compression level' default_value: ['8'] required: false } ) app_cli.add_flag( Flag{ flag: cli.FlagType.int name: 'threads' abbrev: 't' description: 'Number of threads' default_value: [runtime.nr_cpus().str()] required: false } ) app_cli.setup() app_cli.parse(os.args) } ``` As you can see, we define short aliases for flags (`abbrev`), set default values, what's required or not, etc. In the case of the `threads` flag, we set the default value to the maximum supported by the CPU with `runtime.nr_cpus()`. Some flags are automatically added, like `help`, `man` or `version`. Then, we define the `compress` function, called when the command is executed. As `V` allows us, we'll do it in a dedicated file, named `compress.v`: ```v import os import cli { Command, Flag } import compress.zstd // The calling command is passed as a parameter // The return is void, with no error handling (`!`) fn compress(cmd cli.Command) ! { // Define the file names filename := 'test.txt' compressed_filename := '${filename}.zst' // Create a simple text file // The `!` is a way to ignore errors os.write_file(filename, 'Hello, world!')! level := cmd.flags.get_int('level')! threads := cmd.flags.get_int('threads')! println('Algorithm: Zstandard, level: $level, threads: $threads') // Read the file, in bytes to pass to the compressor content := os.read_bytes(filename)! compressed_content := zstd.compress( content, compression_level: level, nb_threads: threads )! // Create the compressed file, it will be open if it already exists // Here there is an error handling, change create by open to test it mut file_compressed := os.create(compressed_filename) or { eprintln('Impossible to access $compressed_filename') exit(1) } // Write the compressed content to the file file_compressed.write(compressed_content)! file_compressed.close() println("✅ File compressed to '$compressed_filename'!") } ``` As for `main.v` it will be considered as included in the `main` module used by default. You can also add `module main` at the beginning of both files if you prefer to be explicit. To run the project across files, use: ```bash v run . ``` Now it's time to compile and check flags are working: ```bash $ v -prod . -o compressor $ ./compressor version Compressor version 0.1.0 $ ./compressor -help Usage: Compressor [flags] [commands] A tiny CLI to compress files with Zstandard Flags: -l -level Compression level -t -threads Number of threads -help Prints help information. -version Prints version information. -man Prints the auto-generated manpage. Commands: help Prints help information. version Prints version information. man Prints the auto-generated manpage. $ ./compressor -l 22 -t 2 Algorithm: Zstandard, level: 22, threads: 2 ✅ File compressed to 'test.txt.zst'! ``` Of course, you can go further: add file management, error handling, more commands, options, etc. Just take a look at the `cli` module [documentation](https://modules.vlang.io/cli.html). You can also find a more complete version of this tool in [this repository](https://github.com/davlgd/vCompressor). --- # Chroot to any Linux (to test it) _Bring your own kernel_ In a previous article, I talked about [what's a (minimal) Linux](/posts/2024-05-whats-a-minimal-linux/) and explained how to launch such a system with only a compiled kernel, an initramfs and BusyBox to get some tools. You should now understand that a Linux based system needs a filesystem to be working and useful. Rules are defined in the [Filesystem Hierarchy Standard (FHS)](https://refspecs.linuxfoundation.org/fhs.shtml). That's what initramfs and BusyBox provided in a very basic way. But a Linux distribution brings more: lots of tools, libraries, config files, etc. Except from its prebuilt kernel, it's what makes it different from another. Thus, you can test any distribution on your local machine, without installing it, without any virtualization stack. You just need to get its file system and use it with your own kernel. There is a tool for that: [chroot](https://linux.die.net/man/1/chroot). The underlying system call was introduced in the 7th Edition of Unix (1979); the standalone command followed shortly after in BSD. ## I am chroot You don't trust it's so simple? Let's try it with Alpine Linux. It's a very lightweight distribution, based on musl libc and BusyBox, distributed [in many ways](https://www.alpinelinux.org/downloads/), including a [tarball]() containing its file system. Download it and extract it: ```bash wget https://dl-cdn.alpinelinux.org/alpine/v3.19/releases/x86_64/alpine-minirootfs-3.19.1-x86_64.tar.gz mkdir alpine tar xPf alpine-minirootfs-3.19.1-x86_64.tar.gz -C alpine ``` Then, you just need to `chroot` into it, launching `sh` shell as Bash is not included in Alpine by default: ```bash sudo chroot alpine /bin/sh ``` To see you're in a different environment, check `/etc/os-release`: ```bash cat /etc/os-release ``` You should see something like: ```bash NAME="Alpine Linux" ID=alpine VERSION_ID=3.19.1 PRETTY_NAME="Alpine Linux v3.19" HOME_URL="https://alpinelinux.org/" BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues" ``` What happens here is that `chroot` changes the root (`/`) directory of the current process to the `alpine/` folder. Thus, you see it as if you booted on an Alpine Linux system, after the `init` and login. You're still on your host system, with your kernel, your processes, etc. But no local network, Internet access, devices access, or whatever. To enable them, `exit` and mount some directories from your host system to the `chroot` environment (with `--bind` when we need to reflect the original). To get Internet, you'll also need a proper DNS `resolv.conf` file: ```bash sudo mount --bind /dev alpine/dev sudo mount -t devpts /dev/pts alpine/dev/pts sudo mount -t proc /proc alpine/proc sudo mount -t sysfs /sys alpine/sys cp /etc/resolv.conf alpine/etc/resolv.conf ``` Then, you can `chroot` again and install some packages, like `neofetch`: ```bash sudo chroot alpine /bin/sh apk update apk add neofetch neofetch ``` It should show some information about the system like the distribution ([ASCII](https://en.wikipedia.org/wiki/ASCII)) logo, kernel version, CPU, memory, screen resolution, uptime, etc. After exiting the `chroot` environment, you should `umount` the directories in the reverse order of the `mount` command: ```bash exit sudo umount alpine/{sys,proc,dev/pts,dev} ``` ## Chroot to any Linux How to do that with any Linux distribution? For many of them, you can't just download an archive and `chroot` into it. You need to get the filesystem from somewhere else. One easy way is to use the Docker registry, extract content from an image and `chroot` into it. There is a tool for that: `docker export`. Once Docker [is installed](https://docs.docker.com/engine/install/#supported-platforms) (or Podman with an alias), let's try with Arch Linux: ```bash mkdir arch docker create --name arch archlinux docker export arch | tar x -C arch ``` Then, you can `chroot` into it: ```bash sudo mount --bind /dev arch/dev sudo mount -t devpts /dev/pts arch/dev/pts sudo mount -t proc /proc arch/proc sudo mount -t sysfs /sys arch/sys cp /etc/resolv.conf arch/etc/resolv.conf sudo chroot arch ``` Then, you can update system and install some packages, like `neofetch`: ```bash pacman -Syu pacman -S neofetch neofetch ``` You can do the same with any other distribution, like Debian, Fedora, NixOS, Ubuntu, etc. You can also use any Linux based container image. But never forget: you're still on your host system, with your kernel. It's just a different environment, with its own file system, tools, libraries, etc. If you use tools like `ps -a` or `top`, you'll see host processes. ## A script to play with this easily To make it easier, I wrote a script to `chroot` into any Linux distribution, using Docker images. It's available on [GitHub](https://github.com/davlgd/chroot-from-image). For example, `chroot` into a system with `nginx` installed and launch it (port 80 by default). The script downloads the Docker image, extract the filesystem, mount directories, and `chroot`. When you `exit`, it will automatically `umount` the directories and remove the Docker container and the extracted content. ```bash git clone https://github.com/davlgd/chroot-from-image cd chroot-from-image # Use the script with the following syntax: ./chroot_from_image # After launch of nginx, you'll exit the chroot environment, refuse to clean it ./chroot_from_image nginx nginx ``` Then, you can check that `nginx` is running, accessing it from the host system: ```bash curl localhost # It works from host system, too! ``` To stop `nginx`, kill its processes and use the `clean_image` script to unmount, remove the Docker container and the extracted content: ```bash kill $(pidof nginx) ./clean_image nginx ``` --- # What's a (minimal) Linux? _KISS Linux, the manual way_ If you read this post, this blog, you certainly know (and use?) Linux. And even if you don't, it's probably part of your life. Because Linux won! These days, it's everywhere: in the Cloud, its servers, but also phones, tablets, TVs, cars, fridges, watches, cameras, routers, IoT devices, supercomputers, space stations, Mars rovers, nuclear submarines, airplanes, drones, robots, game consoles, smart speakers, smart homes, smart cities, smart grids, smart factories, smart farms, smart hospitals, smart cars, smart everything. Sometimes, in desktop computers too... ## Let's talk about Linux But when you discuss it with people, even confirmed users, you realize there are still a lot of questions and misconceptions about Linux. What is it, its kernel, why are there people yelling at you when you don't write GNU/Linux, what makes a distribution different from another, etc. So, let's try to clarify things a bit with a blog post series and some practical stuff. It won't be that technical, but I hope it will help some to better understand Linux and its ecosystem, and why it's so precious. Sorry BSD team, I won't elaborate more on it. Maybe later 😬 ## Do you GNU? As [stated by Wikipedia](https://en.wikipedia.org/wiki/Linux), Linux is not just A thing, it is "_a family of open-source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991, by Linus Torvalds_". I won't go into the details of [the history of Unix and Linux](https://www.youtube.com/watch?v=vjMZssWMweA), Wikipedia is far better than me for such things. But you got it: we use this name to talk about the kernel and operating systems (OS) based on it. In most situations, these OSes, or distributions, include tools from [the GNU project](https://www.gnu.org/gnu/gnu.en.html). There are a lot, [almost 400!](https://www.gnu.org/manual/blurbs.html). All with the same [Free Software philosophy](https://www.gnu.org/philosophy/philosophy.en.html). It's why you should then talk about GNU/Linux. ## (Compile) the kernel Kernel is the core of the system, the one talking to the hardware, managing resources, etc. It's (almost) the first thing that starts during boot. [Linux is open source](https://www.kernel.org/), so you can read it, modify it, compile it, and use it. By the way, let's demystify something: no, it's not that hard to compile and use your own kernel. You don't believe me? Let's do it! First, download a kernel [tarball]() (on a Linux based system), and extract it: ```bash wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.8.8.tar.xz tar xf linux-6.8.8.tar.xz cd linux-6.8.8/ ``` Configure with default settings and compile it (using all CPU cores): ```bash make defconfig make -j$(nproc) ``` Wait some minutes and... it's done! Installing it could be as easy as a `make install`. But many distributions prefer to package it or provide their own tools. Why? Because the hard part is the configure step. You never use a kernel with its default settings: you fine tune it, add some features, remove others, etc. And it's not that easy to do it right. But try it! ```bash # If you want check and/or modify kernel configuration # Result is stored in .config file make clean make menuconfig make -j$(nproc) ``` ## You almost have a Linux (system) Once compiled, Linux kernel is available in the `arch/x86/boot/bzImage` file, but also in the `vmlinux` file. The first one is to boot the system, the second one is [for debugging](https://en.wikipedia.org/wiki/Vmlinux). Both are ready to use. You wanna try? Let's download [qemu](https://www.qemu.org/docs/master/index.html), one of the fabulous tools [made by Fabrice Bellard](https://en.wikipedia.org/wiki/QEMU), and run your kernel in an emulated [x86_64](https://en.wikipedia.org/wiki/X86-64) machine: ```bash qemu-system-x86_64 -kernel arch/x86/boot/bzImage ``` You'll see it booting. But soon after that, it fails. Why? Because you need more than just the kernel to have a working system. You need a lot of things: a file system, a network stack, a shell, a package manager, tools, themes, wallpapers, etc. That's what distributions are for. They provide you all these things, pre-configured, ready to install from an ISO image. [They are a lot](https://upload.wikimedia.org/wikipedia/commons/1/1b/Linux_Distribution_Timeline.svg), for different purposes: servers, desktops, mobile, embedded systems. With different philosophies, preferences, strategies for packages management, etc. You want to discover them? There are 300+ listed on [Distrowatch](https://distrowatch.com/), find yours! ## Init it! To get a working system from a compiled kernel, we need an [initial ram disk](https://en.wikipedia.org/wiki/Initial_ramdisk) (known as `initrd` or more recently `initramfs`). It's a small file system containing an `init` command, loaded into memory at boot time. It contains tools to mount the real file system, and then start the OS. There are many ways to create it, from `mkinitramfs` to `dracut`. But let's keep it simple, using the `cpio` archive software and a "Hello, world!" C program. First, create a `initramfs` folder and a simple `init.c` file: ```bash mkdir -p initramfs cat > init.c < int main() { printf("Hello, world!\n"); return 0; } EOF ``` Compile it with `-static` to get a standalone binary in `initramfs` folder: ```bash gcc -static init.c -o initramfs/init ``` Then create the `initramfs` archive with `cpio` and [gzip it](https://www.youtube.com/watch?v=JARVYdwNSrI): ```bash cd initramfs find . | cpio -H newc -o | gzip > ../initramfs.cpio.gz cd .. ``` Now, boot the kernel with this `initramfs` file. Here we use `qemu` with more complete options to enable [KVM](https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine), native CPU instructions, a serial console to see the output of the `init` program. We also disable the graphical output, and ask to stop the system after the `init` program ends/panics: ```bash qemu-system-x86_64 -kernel arch/x86/boot/bzImage -initrd initramfs.cpio.gz \ --enable-kvm -cpu host -nographic -no-reboot \ -append "console=ttyS0 panic=1" ``` The kernel starts, the `initramfs` compressed file is mounted, the `init` script found and started. Then, the "Hello, world!" message is printed before the system stops (because there is nothing more to do). It's a good start, isn't it? But we want more than that... ![Hello, world! from qemu](/images/2024-05-linux-qemu.webp) ## Embed BusyBox We want a shell, some apps, etc. For this example we won't embed the full GNU toolset, but a subset of them: [BusyBox](https://busybox.net/). It's a single binary containing many common Unix commands, like `ls`, `cat`, `cp`, `mv`, `rm`, `grep`, etc. Download it, configure it with static linking, compile it, and install it in the `initramfs` directory: ```bash wget https://busybox.net/downloads/busybox-1.36.1.tar.bz2 tar xf busybox-1.36.1.tar.bz2 cd busybox-1.36.1/ # We configure and set CONFIG_STATIC=y to get a standalone binary make defconfig sed -i 's/# CONFIG_STATIC is not set/CONFIG_STATIC=y/' .config # Compile and install it in the initramfs directory make -j$(nproc) make install CONFIG_PREFIX=../initramfs ``` Add some folders to mount a working file system, and an `init` executable script to start `sh` shell from BusyBox: ```bash cd ../initramfs/ mkdir -p {dev,proc,sys} cat > init < ../initramfs.cpio.gz cd .. ``` Boot the kernel with this new `initramfs` file: ```bash qemu-system-x86_64 -kernel arch/x86/boot/bzImage -initrd initramfs.cpio.gz \ --enable-kvm -cpu host -nographic -no-reboot \ -append "console=ttyS0 panic=1" ``` You should see the system booting and get a shell prompt: ```bash [ 0.990881] Freeing unused kernel image (initmem) memory: 2680K [ 0.991694] Write protecting the kernel read-only data: 26624k [ 0.992787] Freeing unused kernel image (rodata/data gap) memory: 1568K [ 1.041711] x86/mm: Checked W+X mappings: passed, no W+X pages found. [ 1.042542] x86/mm: Checking user space page tables [ 1.090060] x86/mm: Checked W+X mappings: passed, no W+X pages found. [ 1.090875] Run /init as init process [ 1.093103] mount (50) used greatest stack depth: 13864 bytes left Welcome to my minimal Linux system! Linux (none) 6.8.8 #1 SMP PREEMPT_DYNAMIC Wed May 1 14:42:21 CEST 2024 x86_64 GNU/Linux ~ # ``` You can now use some commands: ```bash # Get information about CPU and memory # It's UNIX, everything is a file cat /proc/cpuinfo cat /proc/meminfo # List devices and binaries ls /dev/ ls /bin/ ls /sbin/ # Get system uptime uptime ``` It's your first minimal Linux system, it uses less than 15 MB of storage, less than 10 MB of RAM. Have fun with it! 🎉 _PS: You'll find the full script from this guide [here](https://gist.github.com/davlgd/a34d07c767ea4ca923964b31c6d83096)._ --- # Why I do not use WordPress for this blog _Better keep it simple_ Yesterday, there was a blog post published at this URL, entitled "Why I now use WordPress for this blog". Of course, it was an April fool's joke. I am not using WordPress for this blog, nor do I plan to. I'm using [a static site generator](/posts/2023-12-how-this-blog-was-built/), and I'm happy with it. Of course, I edit my Markdown files with an IDE, but I'm fine with that, and I'm free to write my posts with `nano` if I like. But the most important thing to me is that this website fits into a few MB, it's fast and clear to read. I can host it wherever I want, and I can move my Markdown content to another technology if I choose. Too many personal blogs involve big frameworks and many tools, for some requests per day/week/month. Not mine... KISS! --- # A road story to my first Exherbo Linux packages _My love letter to distributed open source and home made things_ As a kid, after years using an [Amstrad CPC 464](https://www.cpcwiki.eu/index.php/CPC_old_generation) (with a green/green screen), drawing rosettes in BASIC and playing video games through the cassette deck, I discovered the PC ecosystem through friends and family in the 90s. Shortly after, I got my own [Intel 486 DX2](https://en.wikichip.org/wiki/intel/80486/486dx2-66) (66 MHz with Turbo) and started to learn MS-DOS 5.x, reading the official [user guide](https://archive.org/details/microsoft-ms-dos-5) in my spare time. Then MS-DOS 6.x, Windows 3.x, Pentium, and so on. You know [what's next](https://www.davlgd.fr/39.html). ## Hello PC, and opening mind Although I also had a [Nintendo Entertainment System](https://en.wikipedia.org/wiki/Nintendo_Entertainment_System) (NES) for video games, I was definitely more interested in computers. Not only did I play with them, but I learned how to use them, to make (digital) things with them. The pleasure to talk to [Dr. Sbaitso](https://en.wikipedia.org/wiki/Dr._Sbaitso), create my first applications or explore `AUTOEXEC.BAT` editing to get more paginated memory for [Commander Keen](https://www.abandonware-france.org/ltf_abandon/ltf_jeu.php?id=273). As you may have noticed, I was mainly a Microsoft guy. My earliest memory of a Linux distribution I actually used, except from CD-ROMs we got in magazines, is Gaël Duval's [Mandrake](https://en.wikipedia.org/wiki/Mandriva_Linux). Like any geek of my generation, I also loved and played with [beOS](https://en.wikipedia.org/wiki/BeOS). It taught me a lot about the power of a well-designed operating system, and how good ideas do not always win. The teenager me wasn't aware of the key role of open source, how it changes our approach to software, communities, development, security, distribution, access to knowledge. Nobody explained it to me, I certainly lacked curiosity on the matter, and it wasn't as widely discussed as of today. ## davlgd ❤️ open source I discovered it over the years, during my time as an editor, covering the evolving trends in the IT market. I developed a real kink for communities and [distributed systems](https://next.ink/4851/de-linternet-distribue-a-victoire-plateformes/). Not the NFT Bro way, I'm more a [Merkle tree guy](https://next.ink/4998/de-git-a-bitcoin-en-passant-par-ipfs-derriere-foret-decentralisation-arbres-merkle/). Through Internet, I was convinced we could achieve a broad sharing of knowledge (and still am, in spite of the dark informational times we live in). This reminds me of the Libre Software Meeting (RMLL) 2018 with my friend [pyg](https://x.com/pyg), where I discovered there were hands-on workshops for teaching ARM assembly to teens. And how well it could go if done in a playful way. I wished I'd been raised at a time computers like the Raspberry Pi were affordable for everyone. But I was still happy I started out with BASIC long before discovering languages that gorged themselves with dependencies to create tools that, although simple, took up tens of MB once compiled. So please, tell your kids about open hardware/software and distributed, KISS principles as soon as they're old enough to be interested in computers. ## From simple Linux user... Over the past few decades, I've been using GNU/Linux distributions more and more. Sometimes as a day-to-day system, but mostly on bare-metal servers, in VMs/containers, on side desktop computers. I used Debian and Ubuntu, having fun with [Compiz](https://www.youtube.com/watch?v=7HmuMwfASD0). Then I discovered openSUSE, Fedora, Arch and their derivatives (thanks [Distrowatch](https://distrowatch.com/)). Although I prefer to stay hardware/software agnostic, and use multiple types of systems/tools, I'm now a macOS-first user. It brings me the best of both CLI/GUI worlds, and energy-efficient Apple Silicon SoCs. It helps me to learn working on ARM-based architecture, which is interesting for my job, yet sometimes a source of complexity. On the GNU/Linux side, my favorite distro these days is [Manjaro/GNOME](https://manjaro.org/download/): it's based on Arch, so up-to-date through rolling releases, and its default configuration and tools are great for my (moving) needs. During this open source journey, there was one thing missing: a source-based distribution. I've always been curious about them, but never had the time and enough motivation to try one. As I'm now part of the Clever Cloud team, I chose to test [Exherbo Linux](https://www.exherbolinux.org/), which we use all over our platform. ## ...to (distributed) packager I could have just used it in a container ([I sometimes do](/posts/2024-02-docker-alias/)), but I wanted to go further. When I started to read and learn about Exherbo, I was seduced by the philosophy behind it (explained by Bryan Østergaard [at FOSDEM 2009](https://www.youtube.com/watch?v=4KhJyEvD97s)). 15 years later, for sure it's not the most famous Linux distribution, nor the best documented. This explains that, and it could be far better about conviviality for newcomers. But it's definitely a good lightweight way to learn about Linux basics, intentionally maintained by a small core team. Thus, there is an important focus on distributed development, and user freedom, which are key values for me. So I started learning to install it, on a PC/VM, thanks to my teammates and the community. I read the [official documentation](https://www.exherbolinux.org/docs/install-guide.html) and some guides ([Alexherbo's](https://alexherbo2.github.io/wiki/exherbo/install-guide/), [S0ddy's](https://gist.github.com/s0dyy/905be36b2c39fb8c14906e15c05c68a3)) to write [my own script](https://github.com/davlgd/exherbo-setup). But my main goal was to learn using Exherbo's package manager, [Paludis](https://paludis.exherbolinux.org/index.html) and its client [Cave](https://paludis.exherbolinux.org/clients/cave.html) (some pronounce it "cawé"). Inspired by Portage, it's compatible with Gentoo, but opinionated on [some key differences](https://paludis.exherbolinux.org/faq/different.html). What seduced me about this is how easy it is for anyone to create their own local or remote exheres repository, through git. Packages are also simple to create (kinda). They're `exheres-0` files: shell scripts with special functions to use and some conventions to follow. They often benefit from `exlib`, a set of libraries to help you write shorter/simpler `exheres-0` files. - [Exheres for smarties](https://www.exherbolinux.org/docs/eapi/exheres-for-smarties.html) (a long, but complete guide) So I started to make and use [my own packages](https://github.com/davlgd/exheres/tree/main). ## Your own package repository The Exherbo official website [states](https://www.exherbolinux.org/docs/features.html) that: > A small team is one that can adapt quickly to changes and keep focus on the philosophy of Exherbo. The downside to this is that a team of ~20 cannot maintain 2000+ packages. Exherbo solves this issue by offering robust distributed repository management, opting for many small repositories that integrate seamlessly with everyday management. It's open source, give back to the community, contribute! The starting point is a [git local repository](https://next.ink/5730/apprenez-a-utiliser-git-bases-pour-suivre-evolution-dun-document/). At least it should contain: - `metadata/` - `about.conf`: information about you and the repository - `categories.conf`: list of packages' categories used in the repository - `layout.conf`: parameters for the repository - `profiles/` - `repo_name`: a file containing only the name of the repository If you're not sure of the content of these files, just clone an existing exheres repository and adapt it. Mine is available [here](https://github.com/davlgd/exheres/tree/main). In an Exherbo Linux system, you add a repository by creating a file in `/etc/paludis/repositories/` named `repo_name.conf` with the following content: ```bash format = e location = /var/db/paludis/repositories/repo_name sync = git+file:///path/to/your/repo sync_options = --branch=main ``` You can adapt the last line depending on your default branch name. If it's `master`, you can remove it. Once this is done, you can sync all your repositories, or only the one you just added: ```bash cave sync cave sync repo_name ``` Then, each time you make a new commit in your repository and `cave sync`, your new/updated packages will be available. If you want to access it from anywhere, you can push it to a remote git server (like Gitea, GitHub, GitLab, etc.) and edit your configuration file: ```bash sync = git+https://your.git.server/repo.git local: git+file:///path/to/your/repo ``` Once done, you can sync from different sources: ```bash cave sync repo_name cave sync -s local repo_name cave sync --source local repo_name ``` ## A simple package As they're designed for a source-based distribution, Exherbo's packages are a kind of script containing metadata and multiple steps to get the source code, compile it, test the result, install it, and clean up. Here I won't cover in detail how to create a complex package, I'll do that in a future post. On the contrary, I'll show you how simple it can be, with a Rust based example: [Static Web Server](https://static-web-server.net/). As it is available as [a crate](https://crates.io/crates/static-web-server), we can use `cargo` to build it and install it. And there is an `exlib` for that. `cargo.exlib` [is available](https://gitlab.exherbo.org/exherbo/arbor/-/blob/master/exlibs/cargo.exlib) in the `arbor` repository, the main one for Exherbo Linux, included by design. So you don't have anything to do to use it, you can just `require` it in your `exheres-0` file. But first, let's create it. The crate name is `static-web-server`, so the package name. As it's a web server, it will be in the [`www-servers`](https://summer.exherbolinux.org/packages/www-servers/index.html) category. We'll use the latest available version (2.28.0 at this time). So, we need to: - Add `www-servers` to the `metadata/categories.conf` file - Create a `packages/www-servers/static-web-server/` directory - Create a `static-web-server-2.28.0.exheres-0` file in it Now, what about the file content? It's quite simple. First, Copyright and License (you'll find official recommendations from Exherbo's team [here](https://www.exherbolinux.org/docs/eapi/exheres-for-smarties.html#copyright_lines)): ```bash # Copyright 2024 your_name # Distributed under the terms of the GNU General Public License v2 ``` Then, we include the `cargo` exlib and the minimum Rust version required, it will build/configure the application. We force the use of `github` exlib to get source code, as the `tests/` folder is not included in the crate archive, and we'll need it to run tests after build process: ```bash require github [ force_git_clone=true tag=v${PV} ] require cargo [ rust_minimum_version=1.74.0 ] ``` We configure the package (`${PN}` is a variable with the package name): ```bash SUMMARY="A cross-platform, high-performance and asynchronous web server for static files-serving" HOMEPAGE="https://${PN}.net/" UPSTREAM_CHANGELOG="https://github.com/${PN}/${PN}/blob/master/CHANGELOG.md [[ lang = en ]]" LICENCES="|| ( Apache-2.0 MIT )" SLOT="0" PLATFORMS="~amd64" DEPENDENCIES="" ``` And... that's it! At installation time, Cave/Paludis will use the file name to get the crate name and version, the `cargo` exlib will do the rest. Save, commit (and push) this file to your repository. Then you can install it: ```bash cave sync cave resolve -x static-web-server ``` To make this package available more broadly, [you can push it](https://www.exherbolinux.org/docs/contributing.html) to an official Exherbo's repository, there is one [dedicated to Rust tools](https://gitlab.exherbo.org/exherbo/rust). To go further and better learn how to create packages, look at those already available in the [official repositories](https://gitlab.exherbo.org/exherbo/), or in [the public packages list](https://summer.exherbolinux.org/). --- # Zig and WASM: your best friend here is the compiler _WASM is coming... everywhere!_ Some days ago, MJ Grzymek published an interesting piece on [his blog](https://blog.mjgrzymek.com/blog/zigwasm) about how Zig can be compiled in WASM and used for efficient web development. It reminded me I've wanted to write about Zig and WASM for months. Not because I'm in love with this language, I find it messy (and I'm not a low level guy). But its compiler is dope! Notably, it allows you to compile C/C++ code to WASM/WASI. And that could be a game changer! ## Zig compiler can compile C/C++ too Once [installed](https://ziglang.org/download/), `zig` help command will show you this message: ```bash info: Usage: zig [command] [options] Commands: build Build project from build.zig init-exe Initialize a `zig build` application in the cwd init-lib Initialize a `zig build` library in the cwd ast-check Look for simple compile errors in any set of files build-exe Create executable from source or object files build-lib Create library from source or object files build-obj Create object from source or object files fmt Reformat Zig source into canonical form run Create executable and run immediately test Create and run a test build translate-c Convert C code to Zig code ar Use Zig as a drop-in archiver cc Use Zig as a drop-in C compiler c++ Use Zig as a drop-in C++ compiler ... ``` As you can see here, `zig` can be used as a drop-in C/C++ compiler. For example, you can create a `guess.c` file: ```c #include #include #include int main() { int number, guess, attempts = 0; srand(time(NULL)); number = rand() % 421 + 1; printf("Guess a number between 1 and 421\n"); do { scanf("%d", &guess); attempts++; if (guess > number) { printf("Lower number please!\n"); } else if (guess < number) { printf("Higher number please!\n"); } else { printf("You guessed it in %d attempts\n", attempts); } } while (guess != number); return 0; } ``` Then compile it with `zig`: ```bash zig cc guess.c -o guess ./guess ``` And it works! You can do the same with a C++ code in an `analyze.cpp` file: ```c++ #include #include #include int main() { std::string line; int lineCount = 0, charCount = 0, charNoSpacesCount = 0, wordCount = 0; while (std::getline(std::cin, line)) { lineCount++; charCount += line.length(); for (char c : line) { if (c != ' ') { charNoSpacesCount++; } } std::stringstream ss(line); std::string word; while (ss >> word) { wordCount++; } } // Display results, formatted as JSON std::cout << "{" << std::endl; std::cout << " \"Line count\": " << lineCount << "," << std::endl; std::cout << " \"Character count (including spaces)\": " << charCount << "," << std::endl; std::cout << " \"Character count (excluding spaces)\": " << charNoSpacesCount << "," << std::endl; std::cout << " \"Word count\": " << wordCount << std::endl; std::cout << "}" << std::endl; return 0; } ``` Compile it with `zig` and run it: ```bash zig c++ analyze.cpp -o analyze ./analyze < analyze.cpp | jq ``` It will show you the number of lines, characters, words, JSON formatted. ## Compile (some of) your C/C++ code to WASM/WASI But the `zig` compiler also has a `-target` option, which can be used to compile previous C/C++ code to [WASM/WASI](https://github.com/WebAssembly/WASI) with no changes. Thus, it can run on any platform with a WASM runtime, such as [Wasmtime](https://wasmtime.dev/): ```bash zig cc -target wasm32-wasi guess.c -o guess.wasm zig c++ -target wasm32-wasi analyze.cpp -o analyze.wasm wasmtime guess.wasm wasmtime analyze.wasm < analyze.cpp | jq ``` Note there are some limitations, as I wasn't able to manipulate files. It's why I relied on `stdin` in the previous example. So, the following C++ code compiles and runs, but not with the `wasm32-wasi` target: ```c++ #include #include int main(int argc, char* argv[]) { std::ifstream file(argv[1]); if(!file.is_open()) { std::cerr << "Error: could not open file\n"; return 1; } std::string line; while(std::getline(file, line)) { std::cout << line << '\n'; } file.close(); return 0; } ``` The following C code compiles in WASM, but doesn't run, I get a `could not open file` error: ```c #include #include int main(int argc, char* argv[]) { FILE *file = fopen(argv[1], "r"); if(file == NULL) { fprintf(stderr, "Error: could not open file\n"); return 1; } char *line = NULL; size_t len = 0; ssize_t read; while((read = getline(&line, &len, file)) != -1) { printf("%s", line); } free(line); fclose(file); return 0; } ``` As WASM/WASI (`preview2` was [recently announced](https://bytecodealliance.org/articles/WASI-0.2)) and `zig` compiler evolves, be sure it will be possible to do more and more things. ## What about V? I couldn't end without trying to compile [V](/tags/v) code to WASM through `zig`. For the record, V compiler supports multiple backends (`c`, `go`, `js`, `js_browser`, `js_node`, `js_freestanding`) and `wasm`. Thus, the following `hello.v` file: ```v fn main() { println('Hello, WASM!') } ``` Can be compiled to WASM: ```bash v -backend wasm hello.v -o hello.wasm wasmtime hello.wasm ``` But this doesn't work with modules. For example, this `analyze.v` file: ```v import os import json { encode } struct Stats { mut: line_count int char_count int // Including spaces char_no_spaces_count int // Excluding spaces word_count int } fn main() { mut stats := Stats{} for line in os.get_lines() { stats.line_count++ stats.char_count += line.len stats.char_no_spaces_count += line.replace(' ', '').len words := line.split(' ').filter(it != '') stats.word_count += words.len } // Serialize `stats` to JSON json_str := encode(stats) println(json_str) } ``` Compiles well in V, but not with a `wasm` backend. I tried with the `c` backend and then with `zig cc`... it leads to errors. ## Go: the good balance for WASM/WASI? What's important here, is to see how WASM/WASI is gaining traction in multiple languages. Rust supports it as a target for a long time, there are movements from languages such as [OCaml](https://discuss.ocaml.org/t/announcing-the-ocaml-wasm-organisation/12676), [Python](https://github.com/python/cpython/blob/main/Tools/wasm/README.md), [Ruby](https://github.com/ruby/ruby.wasm), etc. Since last summer, [and the 1.21 release](https://go.dev/blog/go1.21), Go compiler natively supports WASM/WASI backend. And it works pretty well, it's my personal choice for multiples WASM projects, easy to bootstrap. For example, this `analyze.go` file: ```go package main import ( "bufio" "encoding/json" "fmt" "os" "strings" ) type Statistics struct { LineCount int `json:"Line count"` CharacterCount int `json:"Character count (including spaces)"` CharacterCountNoSpace int `json:"Character count (excluding spaces)"` WordCount int `json:"Word count"` } func main() { scanner := bufio.NewScanner(os.Stdin) var stats Statistics for scanner.Scan() { line := scanner.Text() stats.LineCount++ stats.CharacterCount += len(line) stats.CharacterCountNoSpace += len(strings.ReplaceAll(line, " ", "")) stats.WordCount += len(strings.Fields(line)) } if err := scanner.Err(); err != nil { fmt.Fprintf(os.Stderr, "reading standard input: %v", err) } jsonData, err := json.MarshalIndent(stats, "", " ") if err != nil { fmt.Fprintf(os.Stderr, "error marshalling stats to JSON: %v", err) return } fmt.Println(string(jsonData)) } ``` Compiles and runs well in WASM: ```bash GOOS=wasip1 GOARCH=wasm go build -o analyze.wasm analyze.go wasmtime analyze.wasm < analyze.go | jq ``` Note that the built WASM file is 2.5MB, compared to 3.3/3.4MB with `zig` from C/C++. The binary was 48K in native C++, 177K in native V with JSON module. --- # Having fun with (Franken)PHP _No dangerous code has been resurrected for this article_ [PHP](https://www.php.net/) is an old language, born in 1995. It's so old that when, as a teenager, I made my first "dynamic" curriculum vitae with a CRUD interface, it was based on PHP. I was proud of it, I was a web developer (kinda)! ## The former star returns Like lots of old languages, it progressively became a (heavy) mess, leaving the hype to newcomers. But unlike many others, it has been able to reform. Thus, its ecosystem is enjoying a breath of fresh air in recent years, with new tools and cool projects using them. And not just WordPress anymore. The [Laravel](https://laravel.com/) framework, came in 2011 to challenge [Symfony](https://symfony.com/) (2005). The [Composer](https://github.com/composer/composer) package manager, first released in 2012, quickly became the standard. PHP 7.x, born in 2015, was a big step forward. Around the same time, Facebook announced Hack, a PHP dialect, and [the HHVM virtual machine](https://github.com/facebook/hhvm). Then, everybody asked: why is such a big tech still using PHP and trying to enhance it? The fame was definitely back. The PHP 8.0 version, released in 2020, came with JIT and lots of new features. This branch is now getting better at every release, with [a pretty stable schedule](https://www.php.net/supported-versions.php). ## PHP for your CLI tools and scripts What never ceases to amaze me about PHP, it's how people forget it's a scripting language, and not just a companion of Apache. By the way, `mod_php` is only one of the many Server API (SAPI) out there. You can use PHP with FastCGI, FPM, CLI, etc. It's a powerful tool, not limited to web pages. For example, you can create a `urlCheck.php` file with this content: ```php \n"; exit(1); } $url = $argv[1]; $curlRequest = curl_init($url); curl_setopt($curlRequest, CURLOPT_NOBODY, true); curl_setopt($curlRequest, CURLOPT_HEADER, false); curl_setopt($curlRequest, CURLOPT_FOLLOWLOCATION, true); curl_setopt($curlRequest, CURLOPT_TIMEOUT, 10); curl_exec($curlRequest); $status = curl_getinfo($curlRequest, CURLINFO_HTTP_CODE); echo "The website at the URL '{$url}' is " . ( $status == 200 ? "available" : "not available\nHTTP response status code: {$status}" ). "\n"; curl_close($curlRequest); ``` And test it with [httpstat.us](https://httpstat.us): ```bash php urlCheck.php https://httpstat.us/200 > The website at the URL 'https://httpstat.us' is available php urlCheck.php https://httpstat.us/404 > The website at the URL 'https://httpstat.us/404' is not available > HTTP response status code: 404 php urlCheck.php https://httpstat.us/307 > The website at the URL 'https://httpstat.us/307' is not available php urlCheck.php https://httpstat.us/500 > The website at the URL 'https://httpstat.us/500' is not available > HTTP response status code: 500 ``` ## PHAR packages and dependencies Then you can "compile" it in the PHP Archive (PHAR) format with [Box](https://box-project.github.io/box/). You'll only need to create a `box.json` [configuration file](https://github.com/box-project/box/blob/main/doc/configuration.md#configuration) with this content: ```json { "main": "urlCheck.php", "output": "bin/urlCheck.phar" } ``` To build and execute it: ```bash box compile bin/urlCheck.phar https://httpstat.us ``` The result is a PHP script with a [shebang](), that you can execute (if PHP is installed on your system). Note that its size (1107B) is nearly twice the initial script's (599B). To generate a SHA-512 sum and check it: ```bash box verify bin/urlCheck.phar box check:signature bin/urlCheck.phar PHAR_FILE_SHA512_SUM ``` To distribute your PHAR packages, you can rely on [PHIVE](https://phar.io/#Install) (PHAR Installation and Verification Environment). If you're looking for PHP tools and dependencies, look at [Packagist](https://packagist.org/), the main Composer package repository. ## PHP native web server As many languages, PHP has its own web server. It's not as powerful as Apache or Nginx, but it's enough for development and small projects. You can start it with the `php -S` command. You'll need to specify the address and the port, but you can also set the root folder and a router script: ```bash php -S localhost:8080 -t public/ router.php ``` If you want to go further with the PHP CLI, just look at the `php --help` output. Here are some of my favorite options: ```bash php --ini # Show ini file paths and loaded files php -d memory_limit=128M # Set ini entry, e.g. memory limit php --info # Show PHP information php -s file.php # Output syntax highlighted source php -m # Show compiled modules php -a # Run an interactive shell php -r 'echo "Hello, world!\n";' # Run PHP code php -l # Lint a file ``` ## Portable PHP If you need to use PHP on a system where it's not installed, [Static PHP CLI](https://github.com/crazywhalecc/static-php-cli) can help you. It allows to build a PHP binary with no dependencies, including all the modules you need, supporting multiple platforms (`Linux`, `macOS`, `FreeBSD`, `Windows`) and Server API (`cli`, `fpm`, `embed` and `micro`). You can [download it](https://github.com/crazywhalecc/static-php-cli/releases) or get the latest version for your system from the GitHub repository with this script (`composer` and `git` needed): ```bash git clone https://github.com/crazywhalecc/static-php-cli.git cd static-php-cli && chmod +x bin/setup-runtime bin/setup-runtime bin/composer install -n chmod +x bin/spc bin/spc --version ``` Check and fix dependencies: ```bash bin/spc doctor --auto-fix ``` Build your PHP binary with the SAPI target and extensions you need: ```bash bin/spc download --for-extensions="bcmath,openssl,tokenizer,ftp,curl" bin/spc build "bcmath,openssl,tokenizer,ftp,curl" --build-cli --build-micro bin/spc build "bcmath,openssl,tokenizer,ftp,curl" --build-all --enable-zts ``` You can also compress the binary with `upx` adding `--with-upx-pack` (Linux and Windows only). The final PHP binary will be in the `build/` folder. ## Embed your code with PHP(micro) If you've read the Static PHP CLI documentation, you've seen the `--build-micro` option. It's a way to "_concatenate the produced binary with any php code then execute it_", based on a fork of the phpmicro project. For example with our previous `urlCheck.php` script: ```bash bin/spc micro:combine urlCheck.php -OurlCheck ./urlCheck https://httpstat.us ``` You can also use the built `micro.sfx` file in the `buildroot/bin/` folder: ```bash cat micro.sfx urlCheck.php > urlCheck && chmod +x urlCheck ./urlCheck https://httpstat.us ``` And now you get an independent binary, built from PHP code. The only problem here from my point of view ? [Its size](https://framagit.org/davlgd/the-sha-calculators-project): ~10MB on my macOS arm64 system. ## Can I deploy Nextcloud with that? All of this is fun for tiny scripts, but what about complete, real life projects, as [Nextcloud](https://nextcloud.com/)? Let's try with a portable PHP CLI and the web installer. First, build PHP with [needed and some optional](https://docs.nextcloud.com/server/latest/admin_manual/installation/php_configuration.html) extensions: ```bash bin/spc download --for-extensions="bcmath,ctype,curl,dom,exif,fileinfo,filter,\ ftp,gd,gmp,iconv,imagick,imap,intl,ldap,mbstring,memcached,openssl,pdo,\ pdo_mysql,pdo_pgsql,pdo_sqlite,posix,session,simplexml,sodium,xml,\ xmlreader,xmlwriter,zip" bin/spc build "bcmath,ctype,curl,dom,exif,fileinfo,filter,ftp,gd,gmp,iconv,\ imagick,imap,intl,ldap,mbstring,memcached,openssl,pdo,pdo_mysql,pdo_pgsql,\ pdo_sqlite,posix,session,simplexml,sodium,xml,xmlreader,xmlwriter,zip" \ --build-cli ``` Download the Nextcloud setup script, start the PHP server: ```bash mkdir nextcloud wget -q https://download.nextcloud.com/server/installer/setup-nextcloud.php -O nextcloud/setup-nextcloud.php buildroot/bin/php -S localhost:8080 -t nextcloud/ ``` You can access [http://localhost:8080/setup-nextcloud.php](http://localhost:8080/setup-nextcloud.php) with your browser to install Nextcloud as usual. Build an archive with the `buildroot/bin/` folder and the setup PHP script: you have a ready-to-deploy Nextcloud. ## Adding Caddy: welcome FrankenPHP! It's great for local tests, but nobody wants to use such an embedded web server in production. So let's try [Caddy](https://caddyserver.com/), a modern, open-source, easy to use, modular web server, built in Go. You can configure it with a simple [`Caddyfile`](https://caddyserver.com/docs/caddyfile), use it as a reverse proxy or as a static file server. It's PHP-compatible [through FastCGI/PHP-FPM](https://caddyserver.com/docs/caddyfile/directives/php_fastcgi), but such configuration can be difficult to make and is not very portable. It's where [FrankenPHP](https://frankenphp.dev/) can help us. ![FrankenPHP](/images/2024-03-franken-php.webp) Developed by [Kévin Dunglas](https://github.com/dunglas) (from [Les Tilleuls](https://les-tilleuls.coop/)), it is a supercharged version of Caddy including PHP as a SAPI through a [Go package](https://pkg.go.dev/github.com/dunglas/frankenphp). This server is available as a standalone binary or a Docker image. I saw it gaining traction in the recent months, so I tried it on some projects to look at the good ways to use it, its growing features and benefits. But [can we also use it with Nextcloud?](https://x.com/nomorsad/status/1764343696592339200). After downloading the binary as `frankenPHP`, make it executable, create a `www/` folder and get the Nextcloud setup script: ```bash chmod +x frankenPHP mkdir www wget -q https://download.nextcloud.com/server/installer/setup-nextcloud.php -O www/setup-nextcloud.php ``` Then start frankenPHP with the `www/` folder as web root: ```bash ./frankenPHP php-server -r www/ ``` And... it works! You can access [http://localhost:8080/setup-nextcloud.php](http://localhost:8080/setup-nextcloud.php) with no modules to install, no configuration to make, no PHP-FPM to start. It's possible to go further with a `Caddyfile` or using FrankenPHP's features to create a more complex setup, or [a binary](https://github.com/davlgd/frankenphp-binary-demo). Try it yourself 😉. ## To the Cloud and beyond! You're interested in FrankenPHP and want to deploy it easily on services like Clever Cloud ? Some days ago, [I was asked](https://x.com/davlgd/status/1763628921965158558) how it could be achieved. Of course, you can easily. I've published example repositories for both [standalone](https://github.com/davlgd/frankenphp-standalone-demo) or containerized versions, with [Laravel](https://github.com/davlgd/frankenphp-laravel-demo) or [Symfony](https://github.com/davlgd/frankenphp-symfony-demo) frameworks. Next step could be to package FrankenPHP for [Exherbo Linux](https://www.exherbolinux.org/), include it with dedicated options for all our customers. As we're revamping our PHP experience, it's an interesting idea. Let's discuss it. --- # Anchor () element: do you know its download and ping attributes? _I love to still have such n00b moments_ One thing I like about technology, is how I can discover new things every day, even about the most basic tools. For example, last week I read a tweet about the `` element and its `download` attribute. Although I've been creating web pages since the 90s, I didn't know about it. So, I made some tests. ## Download a file or open it: let's decide When you create a link to a file, maybe you want the user to open it directly in the browser or to get a download link. By default, the browser will take its own decision, based on MIME type, file extension, etc. But you can send an extra signal, by using the `download` attribute. For example: ```html Open the image Download the image Download as image.webp ``` It gives the following result: - Open the image - Download the image - Download as image.webp Of course, there are some limitations. For example the file must have the same origin as the page, and support for this attribute can vary from one browser to another. But it's a good way to give a hint about what you want. ## Ping a server when the user clicks on a link After discovering this, my first instinct was to check the [MDN documentation](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a). If there was an interesting attribute I didn't know about, there might be others. And I was right: I also discovered `ping`. As the name suggests, when the user clicks on a link, the `ping` attribute sends a request to another URL in the background. It can be a way to track usage without adding parameters to the URL or using JavaScript: ```html Read my blog ``` The `ping` request contains details about context, browser and the system: ```http POST /?action=open&file=external_webp HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate, br Accept-Language: fr-FR,fr Cache-Control: max-age=0 Connection: keep-alive Content-Length: 5 Content-Type: text/ping Cookie: token=; PHPSESSID= Host: localhost:4242 Origin: http://localhost:8080 Ping-From: http://localhost:8080/ Ping-To: https://labs.davlgd.com/davlgd-lab.webp Sec-Fetch-Dest: empty Sec-Fetch-Mode: no-cors Sec-Fetch-Site: same-site Sec-GPC: 1 User-Agent: sec-ch-ua: sec-ch-ua-mobile: ?0 sec-ch-ua-platform: "macOS" ``` Thus, it's often seen as a privacy issue and not widely used. But it's good to know it exists and to look for ways to avoid it. If you want to learn more or experiment with the `ping` attribute, I've published a [GitHub repository](https://github.com/davlgd/anchor-download-ping-demo) containing a demo website, a web server to host it and another to log the `ping` requests in a file. Have fun with it! 😉 --- # How I developed my own static hosting web server, in V _Not the better, but self crafted_ Those who follow me on social networks or work with me know how much I love [V](https://vlang.io/). I first heard of this language a few years ago, but really started to learn it last year, during [a small project about binary size](https://x.com/davlgd/status/1685757707213574146). ## Have you ever heard of V? I won't try to convince you how great it is, I don't need to. Just give it a try, you'll find out by yourself. [The GitHub repository](https://vlang.io/) contains [documentation](https://github.com/vlang/v/blob/master/doc/docs.md), [examples](https://github.com/vlang/v/tree/master/examples), how [to make scripts](https://github.com/vlang/v/blob/master/doc/docs.md#cross-platform-shell-scripts-in-v), resources about [vlib](https://github.com/vlang/v/tree/master/vlib). You'll find more about vpm packages [here](https://vpm.vlang.io/), the stdlib reference [there](https://modules.vlang.io/). An [online playground](https://play.vlang.io/) is available. It's easy to install and to run on any platform. Just `git clone` and compile (or download [pre-built binaries](https://github.com/vlang/v/releases)): ```bash git clone https://github.com/vlang/v cd v && make ./v symlink v run examples/hello_world.v ``` And to update: ```bash v up ``` ## vweb evolves, why not host my blog with it? One of the greatest powers of V is how it's "batteries included". It's not only a language, it's a complete ecosystem. So it includes a way to generate and serve web pages as a binary: vweb, [recently overhauled](https://twitter.com/v_language/status/1755135917956706487). It was thought [to dynamically render a website](https://song.cleverapps.io/), but also to serve static assets: css, images, documents, etc. But when I tried it I asked myself: "_Why not serve a complete static website?_". So I made some tests. And there was a problem: if you didn't ask for a file, nothing was served. For example, if you asked for https://labs.davlgd.com without ending it with `/index.html` you had a `404` response. It needed a small change, so I did it: ```v mut asked_path := url.path base_path := os.base(asked_path) if !base_path.contains('.') && !asked_path.ends_with('/') { asked_path += '/' } if asked_path.ends_with('/') { if app.static_files[asked_path + 'index.html'] != '' { asked_path += 'index.html' } else if app.static_files[asked_path + 'index.htm'] != '' { asked_path += 'index.htm' } } static_file := app.static_files[asked_path] or { return false } ``` What did this change? In the `serve_if_static` function of `vweb.v`, it creates the `asked_path` mutable variable, based on the `url.path`. If it ends with a `/`, it's a "folder", so I check if it contains an `index.html` or `index.htm` file. If yes, it's served. I also add a `/` at the end of the path when there's not and it doesn't contain a `.`, to handle more cases. ## It's OSS, contribute! I tried these changes locally and it worked. I was happy with the result, with how I achieved it, and started to use this modified version of V to build the static web server hosting this blog. But I didn't want to keep it for me, or wait for such a feature to be included officially in the language. So I added a test and made [a pull request](https://github.com/vlang/v/pull/20784). Here comes another powerful part in the V ecosystem: it's easy to contribute, reviewers are very reactive and open to newcomers. Quickly, it was reviewed and merged, I added some documentation and [an example](https://github.com/vlang/v/tree/master/examples/vweb/static_website). Now, anyone can use vweb to serve static websites. ## My way to Tiniest vWeb Server (tws) My final goal was to provide such a tool as a binary: a complete web server, as small as possible, with no dependencies, to serve static websites from a folder. Here started my work on [Tiniest vWeb Server](https://github.com/davlgd/tws). Its code is simple, mainly about managing the command line arguments, including and init vweb. It's as easy as: ```v module main import x.vweb pub struct Context { vweb.Context } pub struct App { vweb.StaticHandler } ``` The main part of the code fits in 4 lines of code, one for a log message: ```v mut app := &App{} app.handle_static(folder, true)! println("Server is started, serving '${folder}' folder") vweb.run[App, Context](mut app, port) ``` The complete code is available [here](https://github.com/davlgd/tws/blob/main/src/tws.v). It's really basic, but I'll try to make it better over time. The main point for me is that it only uses about 1 MB of storage once compiled, it's multiplatform, and able to serve my static website with no effort and a small footprint from anywhere. [I distribute tws as binaries](https://github.com/davlgd/tws/releases/) for Linux, macOS (ARM or amd64) and Windows. Give it a try, fork it and make it better. You'll see, it's fun! 😉 --- # Do you know the yes command? _Automate, the old-fashioned way_ Whether you are running GNU/Linux, macOS or even one of the BSD derivatives, you're using a UNIX-based system. Developed in the 70s, this family of multi-user, multitasking OS is now a major standard in the IT industry. ## Better know UNIX It comes with many tools. You probably use some of them on a daily basis, like `ls`, `cd`, `cp`, `mv`, `rm`, `mkdir`, etc. But there are many more, and some you may not be familiar with. A good example is the [`yes`](https://linux.die.net/man/1/yes) command. It outputs a string with a line break repeatedly until killed. It was conceived to address developers' need to automate application execution, since some applications didn't always offer a flag to prevent user interaction. It's therefore a kind of `repeat` command, you can use this way: ```bash yes # Repeat the "y" string yes yes # Repeat the "yes" string yes | ./script.sh # Repeat the "y" string and pipe it to a script ``` You can do the same with a longer string to write in a text file: ```bash yes "Lorem ipsum dolor sit amet, consectetur adipiscing elit. \ Aenean lacus est, laoreet et ornare eu, commodo sed nibh. \ Vestibulum ut eros tristique, consectetur nulla sed, ullamcorper diam. \ Integer tristique quis augue eget sagittis. Suspendisse velit urna, \ hendrerit eu mauris et, eleifend posuere augue. Curabitur eu suscipit lorem, \ ut iaculis diam. Vivamus aliquet arcu turpis, quis efficitur sem volutpat ac. \ Vestibulum iaculis, nisl ac molestie lobortis, turpis orci tincidunt ligula, \ eleifend volutpat nisl augue vitae erat." | head -n 10 > lorem.txt ``` ## Test it If you want an example script, here is a simple one that asks permission before executing a command provided as an argument: ```bash #!/bin/bash if [[ $# -eq 0 ]]; then echo "No command provided. Exiting." exit 1 else user_command=$1 fi echo "Do you want to execute: ${user_command}? (yes/no)" read proceed_answer echo echo "You answered: ${proceed_answer}" if [[ "${proceed_answer}" == "yes" ]]; then ${user_command} else echo "Permission not granted. Exiting." exit 1 fi ``` Save it as `ask.sh`, make it executable (`chmod +x ask.sh`) and run it with the `yes` command to answer "yes" or "no" to the question: ```bash yes yes | ./ask.sh "echo 'Hello, world!'" yes no | ./ask.sh "echo 'Hello, world!'" ``` Note you can use it on Windows through [CoreUtils](https://gnuwin32.sourceforge.net/packages/coreutils.htm) or its [Rust implementation](https://github.com/uutils/coreutils). ## A great tool, with limits to know Thus, it can be very useful to automate tasks, but there are some caveats. For example, you shouldn't use it when a script is waiting for different answers until it ends, because it will always answer the same thing. It's also not a good idea when you are waiting for a context-specific answer or dealing with a more complex case. In such situations, you may use `expect` or `autoexpect` instead... but these commands are another story. --- # How I use Linux distributions on my Mac, with a single command _Thanks to aliases and Docker_ As mentioned [in a previous article](/posts/2024-02-add-up-aliases/), I use aliases to ease my (CLI) life. In addition to day-to-day actions, they allow me to automate “boring” stuff, which requires me to remember multiple commands, launch them the same way over and over again. A good example of this is Docker. For me, Docker is a great tool for local development. If I need to test something in a specific distribution or context, it’s easy to download an image and use it. To cover my needs on GNU/Linux, I prefer [Podman](https://podman.io/docs/installation#installing-on-linux). I can use it [with the same command](https://archlinux.org/packages/extra/x86_64/podman-docker/) but no `sudo`. Under macOS, I use [Docker Desktop](https://docs.docker.com/desktop/install/mac-install/). Most of the time, once installed, I just want to access an image quickly, do my stuff and exit. For that, I need to pull it, create and start a container and run. But I found an easier way to do all that adding a function in my `~/.zshrc` file and use it as an alias (it works with `bash` or `fish` too): ```bash dssh() { local container_name=$1 local image_name=$2 local prefix="dock" local docker_command="docker" local launch_command=$([[ "${image_name}" == *alpine* ]] && \ echo "/bin/sh" || echo "/bin/bash") if ! command -v ${docker_command} &> /dev/null; then echo "Error: Docker is not installed" return fi container_exists=$(docker ps -a --format '{{.Names}}' | \ grep -q "^${container_name}$" && echo "yes" || echo "no") if [ "$#" -eq 1 ]; then if [ "${container_exists}" = "no" ]; then echo "Container ${container_name} does not exist" return fi elif [ "$#" -eq 2 ]; then if [ "${container_exists}" = "no" ]; then echo "Creating container ${container_name} with image ${image_name}..." ${docker_command} run -dit --hostname "${prefix}_${container_name}" \ --name "${container_name}" "${image_name}" ${launch_command} fi else echo "Usage: $0 [IMAGE]" return fi if [ "$(docker inspect -f '{{.State.Running}}' "${container_name}")" = "false" ]; then echo "Starting container ${container_name}..." ${docker_command} start "${container_name}" fi echo "Executing '${launch_command}' in container ${container_name}..." ${docker_command} exec -it "${container_name}" ${launch_command} } ``` What it does is check whether `docker` (or another command you can define) is available. If so, you can use it to access an existing container by name or create one from an image and access it. I always use `/bin/bash` as the start command unless I'm requesting an Alpine image where `/bin/sh` is the default Shell. This could be done better, with an optional variable for example, but it fits my needs as is for now. I usually don’t use this function directly as an alias, rather from other aliases. For example on a Mac, based on an Apple Silicon `aarch64` SoC: ```bash alias exherbo="dssh exherbo exherbo/exherbo-aarch64-unknown-linux-gnueabi-gcc-base" alias ubuntu="dssh ubuntu ubuntu:latest" ``` Thus, I can access an already configured Ubuntu whenever I want simply with the `ubuntu` command, or do the same with `exherbo`. And if I want to access another distribution: ```bash dssh alpine alpine:latest # To get latest Alpine dssh debian debian:latest # To get latest Debian stable dssh debslim debian:unstable-slim # To get next Debian in slim version ``` --- # Put an ‘up’ alias in your life (to start with) _And launch it as often as you want_ I’m an update junkie. I like my system and my applications to be up to date. In the new mobile « App Store » centric approach to the world, it’s quite simple. But that word is too bland and centralized to please me. I’m more of a terminal, scripts & packages manager guy. So, how do I update my desktop? ## Update everything with a short command On GNU/Linux and macOS I rely on aliases for this kind of stuff. My Shell is `zsh`, but it works too with `bash`, `fish` or whatever. And the most important alias to me is `up`. It’s the one I use to update all the things and launch actions I need to do on a regular basis to feel nice. To do the same, edit your `~/.zshrc` (or the file used by your Shell) and add: ```bash alias up='update && commands && you && want && to && use' ``` Once saved, run `exec zsh` or `source ~/.zshrc`. The `up` command is now available and will be when a new `zsh` Shell is started. For example, on Ubuntu I often add this `up` alias: ```bash alias up='sudo apt update && \ sudo apt full-upgrade -y && \ sudo apt autoremove && \ sudo snap refresh' ``` It updates the system, Snaps and clean unnecessary packages. ## Feel free, add complexity But you can go further and make it more complex by adding third-party packages managers, backup actions, etc. For example my `up` alias on a daily used macOS system is: ```bash up() { softwareupdate --install --all # macOS CLI updater brew update brew upgrade brew cleanup --prune=all -s bun upgrade npm update -g rustup update v up sync_git # A personal sync git repo function } ``` As you can see here, it’s on multiple lines and declared as a function to be more readable. You can add variables, other functions and more complex Shell stuff to such aliases. For example add a function call to `git pull` some repositories or update [your mirrors](/posts/2023-12-github-gitlab-framagit/) (yes, I do that). You can then run `up` on a regular basis through CRON or other mechanisms, and log outputs to check if anything went wrong. I prefer to launch it manually when I want to be sure everything is up to date (sometimes more than once a day)… but I have a compulsive disorder about that. ## Aliases are the spice of life Of course, you can improve your life by multiplying the aliases you use, making commands shorter and smarter. Creating `dotfiles` and sharing them. But it's another story, I’ll cover that in a future blog post. Nonetheless, here are some of my favorite aliases. And my personal advice: ask ChatGPT for new ideas, it’s good at this game. And it’s an interesting way to benchmark developer-focused LLMs 😏 ```bash # One letter is enough alias c='ncal -ws FR' alias f='find / -type f -name 2> /dev/null' alias h='history | grep' alias u='du -hsx * | sort -rh' # Replace `pbcopy` with the copy/paste tool of your choice alias pgen='gpg --gen-random --armor 2 32 | pbcopy' alias serve='python3 -m http.server' # Lots of git focused aliases alias gl='git log --oneline --all --graph --decorate' alias gac='git add . && git commit -m' alias gst='git status' alias gsw='git switch' alias gri='git rebase -i' alias dclean='docker ps -aq | xargs -r docker rm -f && docker images -q | xargs -r docker rmi -f' # My Clever Cloud Fast deploy command alias ccfd='git add . && git commit -m "Fast deploy" && \ clever deploy && clever open' checksite() { if curl --output /dev/null --silent --head --fail $1; then echo "$1 is online" else echo "$1 is offline" fi } mkcd() { mkdir -p $1 cd $1 } w() { if [[ "$2" == "--full" ]]; then curl "wttr.in/$1" else curl "wttr.in/$1?format=2" fi } ``` --- # Convert images to WebP from Finder in macOS, thanks to Automator and Quick actions _This blog's key to lightness_ On this blog, I often publish screen captures, pictures or AI generated images. Most of the time, I get files in JPEG or PNG format. Both are good, but not as optimized as I'd like for a thrifty static website. [WebP](https://en.wikipedia.org/wiki/WebP) is better for that, can be lossless and is widely supported. ## Keep it lean I could rely on Astro, which optimizes files during build. But this means I'd have to store large files with the source code and host them in my repositories. In such a situation, I prefer to solve "by design". I used to convert to WebP with [GIMP](https://www.gimp.org/downloads/) but it's a manual job and it can be a pain, especially with a lot of files to process. So I looked for an easier solution, on macOS (where I spend most of my Desktop time these days). The file explorer (Finder) has a built-in tool for such tasks in its `Quick actions` menu. Unfortunately, output formats supported are only HEIF, JPEG and PNG. But macOS is extensible, so you can add your own quick actions through another built-in tool: [Automator](https://support.apple.com/guide/automator/welcome/mac). ![Apple Automator Workflow to convert an image to WebP](/images/2024-01-apple-automator-workflow.webp) _Apple Automator Workflow to convert an image to WebP (in French)_ ## cwebp in a Shell script, 1-click away As its name suggests, it allows you to create lots of automations within the system. Here, we are looking for a way to select files in the Finder and convert them to the WebP format after clicking on a menu entry. The conversion part will be handled by [`cwebp`](https://developers.google.com/speed/webp/docs/cwebp?hl=en), which you can install with [HomeBrew](https://brew.sh): ```bash brew install webp ``` Check the targeted binary (it should be `/opt/homebrew/bin/cwebp`): ``` which cwebp ``` Then, we need to create a `Quick action` in `Automator` and add (with a double click) `Execute a Shell script`. Select `image files` and `Finder` in the process input, `/bin/zsh` as Shell, `Arguments` as data input and paste this script: ```bash for f in "$@"; do /opt/homebrew/bin/cwebp "$f" -o "${f%.*}.webp" done ``` Change the `cwebp` path if needed. Save the action (⌘+S), name it (`Convert to WebP` for example), it's now available! I tried this on 4 Dall-E generated images: I went from 12.9 MB to 909 kB, quite impressive. ## Automate all the things! Now I need to explore how I can use Automator (and Shortcuts) more on my Apple systems. This is definitely one of my good resolutions for 2024. --- # Zed: your next IDE to try _Not Otomo's_ Multi-purpose text editors and [IDE](https://en.wikipedia.org/wiki/Integrated_development_environment) market space is wide, from `vi` to Visual Studio. But some tools have always been massively used as they provide a good response to the current needs of developers and (Markdown) writers. ## Fresh days for IDE lovers 9 years ago, [Atom]() was the rising star, part of the GitHub tools family. Microsoft bought GitHub, driving efforts to make Visual Studio Code the next big thing. You know the rest of the ([déjà vu](https://killedby.tech/microsoft/atom/)) story. Since then, we've seen some efforts to challenge the situation. Particularly with the rise of LLMs in the AI ecosystem, supporting writers and developers all [around the world](https://www.youtube.com/watch?v=K0HSD_i2DvA). It's been a long time since I’ve witnessed such competition in the IDE space, with a steady stream of newcomers. As usual, there's a lot to sort through. All these contenders promise to reinvent the way we write/code, but only a few really succeed and convince beyond their marketing campaigns. And my new favorite here is [Zed](https://zed.dev/). ![Zed IDE](/images/2024-01-zed-ide.webp) Somehow it ticks all the boxes to be in the spotlight in 2024: [it’s Rust based](https://zed.dev/blog/beta) and promotes effectiveness, it includes AI, lot of (collaborative) [features](https://zed.dev/features), and there is [a beautiful story](https://zed.dev/about) about the team being part of those who built Atom. Sounds great, right? ## It's time to use Zed… I heard about it some months ago but didn’t take the time to look into it. This week, my teammate [Florian Sanders](https://twitter.com/flsan_) pointed out to me [it was now open source](https://zed.dev/blog/zed-is-now-open-source) (under a mixture of Apachev2, AGPLv3 and GPLv3 licences). The tool [evolved a lot](https://zed.dev/blog/why-the-big-rewrite) recently so I was curious. And now seduced. To be honest, I don’t care what language is behind a tool. It could have been developed in [APL](https://tryapl.org/) if the team thought it was fun to do. But I look at some metrics which are sometimes related to such a choice. Thus, Rust or Electron doesn’t carry the same mindset, goals, and it tells a lot about how the team evolved in its choices over time. And the result is great. Zed is fast, it has figures to prove it. It’s also easy as it’s new and light ATM. But it’s still a good job. What I like the most about it is how comfortable it is to use from scratch, without any customization. Of course, you can change a lot through settings (JSON way), globally, per project or per language. Zed is good at providing language-specific behavior without many extensions. We’ll see how this evolves over the coming months. AI and Git(Hub) integrations are light but well designed. I only regret some details: the mechanism to open/close panels, not relying on a cross to click/close or the absence of live preview for Markdown/HTML. And I hope to quickly be able to use open source local or remote LLM easily (I didn’t try so much, excuse me if there is already a 10 steps guide for that on the Internet). My only settings changes are (share yours): ```json { "theme": "Andromeda", "telemetry": { "diagnostics": false, "metrics": false }, "vim_mode": false, "ui_font_size": 16, "buffer_font_size": 16, "show_whitespaces": "all", "journal": { "hour_format": "hour24" } } ``` ## …or something else? Finally, many will regret it’s [only available on macOS](https://github.com/zed-industries/zed/releases) for now. But since Rust is cross-platform and the tool is now open source, I’m sure it will move fast. Anyway, this has convinced me to actively try out new IDE for my personal projects. Zed is my choice from now on. Maybe that will change. I hope to be convinced to try new cool things. Let’s discuss it! --- # EasyGit: a git server… based on iCloud Drive _How simple is that?_ You know how I love [git](/tags/git). In recent years, it has become the go-to [VCS](https://en.wikipedia.org/wiki/Version_control) for developers around the world. It’s fully distributed: anyone can act as a client or a server. But in most cases, we rely on external services such as GitHub, hosted GitLab, Gitea, etc. Humans remain gregarious creatures. But some developers looked for ways to integrate git with another kind of platform, more and more popular in recent years: Cloud storage services. It’s where I discovered [EasyGit](https://easygit.app/), a free to use solution that combines the power of git with the convenience of… iCloud Drive. - [EasyGit on the Mac App Store](https://apps.apple.com/fr/app/easygit/id1228242832?mt=12) ## git server made easy… Once installed, there’s nothing to do. Launch the application in a user session connected to an Apple account with iCloud Drive, that’s it! You can create a new local repository, EasyGit will host it and make it available through a `git://localhost/repo_name` URL. It works as any other repo: you can clone it, commit, push, create branches, add other remotes. EasyGit also offers to back it up as a copy folder with its `Save as` feature. ![EasyGit Interface](/images/2024-01-easygit-interface.webp) The only difference, except how easy it is to use, is that EasyGit repositories are synced through your iCloud Drive account. It’s where the git server data lives. Thus, install the tool on another Mac with the same Apple account and you’ll find the same repositories to clone and use. ## … in an Apple context Of course, you can also work with teammates and invite them as Contributors with a message or AirDrop. It will send them a link to iCloud Drive. Here is the only “issue”: EasyGit eliminates the need for server setup and allows seamless integration with existing workflows, but only in an Apple centric world. So it’s fine for personal use or if it’s how your team works. But if some use Linux or Windows, you’ll have to add other remotes. I also regret it can't be launched as a background service at startup. --- # Schedule article publication in an AstroPaper blog _Static is fantastic (sort of)_ As a tech editor, I used to write several articles a day. Most of the time, it was enough to prepare content to publish later. In some cases, I had to wait due to a NDA, requiring to hold off until a specific date/time. Thus, my common practice is to batch-write stories when inspiration strikes, and then spread them out over time. It's also how I work on this blog. But as it's a static generated website, things aren't that simple. ## Static is... static Why? Because it creates HTML files from Markdown content during deployment, applying a layout. If it's done on January 14th and you wrote an article for January 21st, it's built the same. And in AstroPaper theme, as in many others, the article will be displayed in the index and in many pages. I therefore had to edit some files. First, I added a number variable called `scheduledPostMargin` in `src/config.ts` and `src/types.ts`. It contains a delay in milliseconds between the publication date/time of a post and when it should be displayed. We'll see later why and where this can help. Next, I created a filter to apply after a JavaScript `map`: if a post is a draft or its publication date/time has not been reached, I consider it as a scheduled post and don't load it. There is one exception: if it's not a draft and the website is being previewed via the development server. The snippet is: ```javascript .filter(({ data }) => { const isPublishTimePassed = Date.now() > new Date(data.pubDatetime).getTime() - SITE.scheduledPostMargin; return !data.draft && (import.meta.env.DEV || isPublishTimePassed); }) ``` I added it to `src/utils/getSortedPosts.ts` and `src/utils/getUniqueTags.ts`. Thus, when Astro is looking for posts or tags, it doesn't load those that match scheduled content. I've also modified `src/pages/search.astro` to load posts from `getSortedPosts` and filter out those that are scheduled in the search. The full commit is [here](https://github.com/davlgd/labs/commit/99d0bd98c750f62d1dd6652be738f1d37eb920a0). I proposed it as a [PR](https://github.com/satnaing/astro-paper/pull/234) on AstroPaper (since merged). ## Build when a new post is ready Now, when the blog is built, scheduled content is created but not displayed anywhere. You must know the URL to access it. I didn't want to go any further because this allows me to check that everything is ok, share with some friends ahead of time and schedule posts on social networks. When it's time to publish, I just have to ask for a rebuild of my application on Clever Cloud, and two minutes later, it's done. It's why `scheduledPostMargin` is required: to build before release time. I also want to plan the restart of the application on a regular basis or at a specific date. Here I use [CRON](https://en.wikipedia.org/wiki/Cron). How to enable it will depend on your hosting platform. In Clever Cloud it's configured by [`clevercloud/cron.json`](https://developers.clever-cloud.com/doc/administrate/cron/). It can be defined to launch a rebuild every weekday at 7:42 or next January 21st at 13:37, for example: ```json ["42 7 * * 1-5 $ROOT/rebuild.sh", "37 13 21 1 * $ROOT/rebuild.sh"] ``` There are a few things to note here. First, the server time is UTC (France is UTC+1 or UTC+2). Second, `$ROOT` is not a variable as in a shell script. It's a value that's replaced by the path to the application root when the crontab is built. Third, it's recommended to use a `login shell` script (starting with `#!/bin/bash -l`) to get access to the application's environment variables. This is the one I use (don't forget to `chmod +x` it): ```bash #!/bin/bash -l clever link ${APP_ID} clever restart --quiet --without-cache ``` For this to work I had to set some environment variables: ```bash CLEVER_TOKEN: account token CLEVER_SECRET: account secret CC_OVERRIDE_BUILDCACHE: /dist:/clevercloud/cron.json:/rebuild.sh ``` Thus, `cron.json` and `rebuild.sh` will be present if the application is restarted from its cache. Thanks to `CLEVER_TOKEN` and `CLEVER_SECRET`, [`Clever Tools`](https://github.com/CleverCloud/clever-tools) can login and restart the application. If you need these values, just launch a `clever login`, you'll have to authenticate to obtain them in a browser. --- # asitop: monitor your Apple Silicon SoC, power consumption included _How low is it?_ When you want to monitor a UNIX system, `top` is a good command to know. It provides information about CPU, RAM, processes, storage and network usage, with some interesting details (in a messy way). [`htop`](https://htop.dev/) is a more modern and flexible alternative, as are [`btop`](https://github.com/aristocratos/btop), [`gtop`](https://github.com/aksakalli/gtop) or [`bottom`](https://github.com/ClementTsang/bottom) (previously `ytop`). But sometimes, your main concern isn't to use a cross-platform tool. You want "close to metal" data. One of the best-known examples of this is [`nvidia-smi`](https://developer.nvidia.com/nvidia-system-management-interface), familiar to Linux gamers and AI developers. ## All you need to know about your Apple SoC Recently I was looking for something similar for my Apple computers and discovered `asitop`, an open source Python tool, inspired by [`nvtop`](https://github.com/Syllo/nvtop): ![asitop on a Apple M1 Max SoC](/images/2024-01-apple-silicon-asitop.webp) It displays information about Apple Silicon SoCs in a graphical (CLI) way: how they're composed, Efficient/Performance CPU cores and GPU usage, their frequency. It installs via `pip` or [HomeBrew](https://brew.sh) and needs `sudo` rights: ```bash brew install asitop || pip install asitop sudo asitop ``` You also get information about RAM or Apple Neural Engine (ANE) usage. One of the interesting pieces of information provided is the real-time power usage of the CPU, GPU, ANE and of the whole package. If the chip is throttling (because it's too hot), you'll be notified. Finally, two things are missing: temperature and fan speed. Let's hope it's planned for a future release. --- # GitHub Desktop became my best friend to rewrite (git) history _Never do that... until you do it_ I love `git`. Versioning things is key in my world and `git` made this popular, distributed, thanks to [Merkle trees](https://next.ink/4998/de-git-a-bitcoin-en-passant-par-ipfs-derriere-foret-decentralisation-arbres-merkle/) (among others). But it's complicated too. You must practice a lot to really master such a tool. And when you're good enough to think you have it, you suddenly realize you're way off the mark. The good thing is, though, that you are constantly amazed by the new things & tricks you discover (`jq` does this to me, too). ![git versioning](/images/2024-01-git-versioning.webp) ## Master `git` your way But on a daily basis, you want to be efficient. Unfortunately, my brain is not when it has to deal with a whole bunch of arguments, flags and their potential combinations in order to get out of a complex situation. Thus, I tend to look for palliatives to balance things out. One is to use a graphical interface (GUI) for `git`. There are lots of them out there, but as I just said, I'm a [KISS](https://en.wikipedia.org/wiki/KISS_principle) made man. As I participate on several (personal and work) projects on GitHub, I use their [Desktop client](https://desktop.github.com/) which has the advantage of being [open source](https://github.com/desktop/desktop) and cross-platform (so do I). For some `git` actions [I prefer to use command line](2023-12-github-gitlab-framagit) or an [IDE](https://en.wikipedia.org/wiki/Integrated_development_environment). On the other hand, to open a repository, select a branch, commit and pull/push, GitHub Desktop is often my swiftest way to do things. And recently, I discovered it could help me with one of my favorite activities: rewriting `git` history. ![Git: squash your commits!](/images/2024-01-git-squash.webp) ## Squash your commits (but reorder them first) Let's be clear: revising `git` history is usually a bad move, especially for teamwork because it breaks the dynamics of distributed projects. But it can be helpful in some cases or when you're dealing with your own branch. In fact, I almost do this only for one thing: keep my `git` history clean. When I use `git`, I regularly commit to track changes and push on a remote to save my work. But when it's time to understand what has been done, or to prepare a merge request, such a behavior sucks. There is a rule for that: "_squash your commits!_". Basically, it's about merging several commits and describing the changes made on your files in a single step. It could be done with an interactive rebase (`git rebase -i `), but you sometimes need to rearrange commits first, amend/undo something, select only some lines for a commit, keep the others for a next one. It's where GitHub Desktop helps... a lot! You can do all this in a graphical way: ![GitHub Desktop Rewriting git history](/images/2024-01-github-desktop-capture.webp) If you avoid modifying the same files in multiple commits, you can reorganize and merge them at your convenience. If you already pushed before, you'll certainly need to `git push --force` on the remote (if you're allowed to). And do you have some `git` CLI/GUI tricks? Let me know! --- P.S.: As I said at the beginning of this blog post, there are lots of [`git` GUIs](https://alternativeto.net/software/github-desktop/) on the market. I'm not promoting GitHub Desktop as the ultimate solution here, just explaining how I use it and how it can save time to keep away from the CLI when you need to do tricky things. Find your favorite tools, let people know, explain why and how they help you. Share love ❤️ --- # How I've upgraded this blog to Astro(Paper) 4.0 _New year, new blog... and already a new version_ Some weeks ago, I decided [to launch this tech blog](/posts/2023-12-how-this-blog-was-built) based on the AstroPaper theme. After the release of Astro 4.0, it's been upgraded with VS Code snippets, `modDatetime` and `slug` support, packages updates, share buttons, back to top link, fixes, etc. You can learn more [here](https://astro-paper.pages.dev/posts/astro-paper-v4/). So, I decided to make the move. As it's a static blog with no 1-click process, I had to follow manual steps. First, backup some folders and files. I thank my idea to create [a commit](https://github.com/davlgd/labs/commit/6bd928bb5a83a0f442419ca49754d16e14847303) with items modified during configuration: - `src/assets` - `src/components/Header.astro` - `src/config.ts` - `src/content/blog` - `src/layouts/Layout.astro` - `src/pages/about.md` - `src/pages/index.astro` - `tailwind.config.cjs` After that, I deleted all the content except `.git` and launched the AstroPaper create process in a new folder: ```bash npm create astro@latest -- --template satnaing/astro-paper ``` I took all the files/folders created and moved them to the (almost) empty repository. Again, versioning helps and I was able to easily check each modified item in VS Code (but you can do it with any `git` GUI). Then, I reverted unwanted changes to recover the assets, config and content. I was ready for a final check with a local HTTP dev server: ```bash npm run dev ``` As everything was fine, I created [a new commit](https://github.com/davlgd/labs/commit/3a541ceb159f54dcb9e32a11b840f0222faf9080) and pushed it. The blog post you're reading is on AstroPaper v4! The next step is to review a few more settings, like the new social icons or custom share buttons. Maybe tomorrow. --- # How I mirror my GitHub repositories on GitLab (thanks to Framagit) _Publishing source code is good. Having a 3-2-1 backup system is better_ For [more than 10 years](https://github.com/davlgd?tab=overview&from=2011-10-01&to=2011-10-31), I use git and I publish my open-source projects on GitHub. The platform is great, its tooling too. But I don't know what tomorrow will bring, especially for a service from Microsoft. Thus, I prefer not to put all my eggs in one basket. So, some months ago I looked after an open-source alternative. I didn't want to rely only on my local repositories or on a self-hosted service. Of course, GitLab was one of the first to come to mind, but I needed a managed service. I already use [Heptapod](https://heptapod.net/) at work (GitLab with Mercurial support), but in a personal context I mostly rely on [Framasoft](https://degooglisons-internet.org/en/). So, I finally chose to create an account on [Framagit](https://framagit.org). But once it's done, how to backup my repositories? ![Degoogleify Framasoft](/images/2023-12-degoogleify-framasot.webp) ## Create a mirror repo with a GitLab push remote There is a GitLab [Mirror option](https://docs.gitlab.com/ee/user/project/repository/mirror/), but I want to rely on a lower layer. Thus, on my main computer, I have `GitHub/` and `GitLab/` folders. In the first, I simply clone my repositories. In the second, I do the same using the `--mirror` flag. As stated in the official `git` [documentation](https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---mirror): > Set up a mirror of the source repository. This implies `--bare`. Compared to `--bare`, `--mirror` not only maps local branches of the source to local branches of the target, it maps all refs (including remote-tracking branches, notes etc.) and sets up a refspec configuration such that all these refs are overwritten by a `git remote update` in the target repository. Then, I create an empty GitLab repository and use it as a push remote: ```bash git clone --mirror https://github.com/user/repo.git cd repo.git git remote set-url --push gitlab git@framagit.org:user/repo.git ``` ## Fetch/Push the content (via an alias) To sync your mirror from its local folder, you only need to: ```bash git fetch --prune && git push --mirror ``` You can use an alias declared in `.bashrc` or `.zshrc` to perform this sync action on multiple repositories from a single command: ```bash function sync_git() { GITLAB_DIR="/path/to/gitlab/directory" # Declare an array of repo subdirectories declare -a REPO_DIRS=("repo1" "repo2" "repo3") # Go in each of them and sync for repo_dir in "${REPO_DIRS[@]}"; do echo "Entering ${GITLAB_DIR}/${repo_dir}.git" cd "${GITLAB_DIR}/${repo_dir}.git" git fetch --prune && git push --mirror echo done printf "%s \e[32m✓\e[0m\n" "Script completed" } ``` After saving the file, reload your shell with `exec bash` or `exec zsh`. ## Go further You can launch this command manually or on a regular basis through [CRON](https://fr.wikipedia.org/wiki/Cron) for example (but you'll need a git authentication not asking for a passphrase). For my needs, I use this script in a more complete alias to update my system and tools (`brew`, `bun`, `npm`, `rustup`, `v`, etc.), with additional commands to clone some repositories I need to have locally, up to date. ## The most important Don't forget [to support Framasoft](https://framasoft.org/fr/#support). This is how their great actions are funded, along with [services](https://degooglisons-internet.org/en/) such as Framagit. --- # How this blog was built _Thanks to Astro, Clever Cloud and GitHub_ I've been publishing personal thoughts [on a blog](https://www.davlgd.fr/on-my-way-to-42.html) for several years, and I'd planned to post on a technical blog in English for quite some time. It's now done. I'm a static website guy. Thus, one of my main concerns was to find an easy-to-use generator with a theme I liked. After a few tests this summer, I found [AstroPaper](https://github.com/satnaing/astro-paper) and it was (dark) love at first sight. Astro is a great modern tool, easy to deploy. I cut trackers on this theme, and there is [an RSS feed](https://labs.davlgd.com/rss.xml)! Of course, I tried it on [Clever Cloud](https://clever-cloud.com) in many ways and I'm now close to what I consider a perfect pipeline. As my account is linked to GitHub, all I have to do is to edit my files and `git push` on the `main` branch. Less than 2 minutes later it's built and available online. If you want to know more, each step is explained [in the GitHub repository](https://github.com/davlgd/labs). And if you want to know more about how I host my other blog on an object storage service, take a look [here](https://github.com/davlgd/www.davlgd.fr).