DEV Community: Matt Cockayne

The commit message was the whole instrument panel

Matt Cockayne — Mon, 02 Nov 2026 02:46:27 +0000

I asked an agent to work out why a pipeline was failing, and explicitly not to fix it. I wanted the diagnosis.

It fixed it. Then it wrote a commit message saying it had added a config file that was already sitting in the repository root, and that was the only thing it told me it had done.

The first failure is the boring one

Acting without leave is annoying, and it isn't what actually worries me. A human catches it on review, the commit gets reverted, everyone moves on slightly crosser than before (me, mostly). I've had to say some version of stop committing things I did not ask you to commit more than once, and it's a scope problem with a scope-shaped fix. The commit message is a different animal altogether.

When an agent has been working unattended, its commit messages stop being documentation and turn into the instrument panel: the whole channel through which the work reports what it did, and I read them the way you read a dial, quickly, trusting them, using them to decide where to look closely and where not to bother.

A commit message describing work the agent didn't do is a dial reading a number that was never true, on the only instrument I had.

So when should it interrupt?

I'm not sure there's a clean answer, and both extremes go wrong in ways I've managed to underrate myself.

An agent that asks about everything is a denial-of-service attack aimed at its own operator. You delegated the work to stop making these decisions; if every one comes back to you with a prompt attached you have gained a slower version of doing it yourself, and you'll start clicking through them without reading, which is worse than not being asked.

An agent that never asks guesses instead, and it doesn't guess evenly either... it guesses hardest on the things that were ambiguous, which, I suspect, are the very things a person would have wanted a say in, since the ambiguity is what made the question worth asking in the first place.

What that looks like as a tool

go-tool-base has a query_user tool, and its description is where the judgement actually lives:

Ask the human a clarifying question and receive their answer. Use this when the request is ambiguous, a design decision could reasonably go more than one way, or a security-sensitive choice needs human judgement, rather than guessing. Provide a concise question and 2-4 suggested answers; the user can choose one or write their own.

Three named conditions, and the "rather than guessing" is doing real work. The suggestions matter too: two to four options plus free text turns an interruption into something you can clear in a second, which is what stops it becoming the denial-of-service version.

Where the design decision actually sits

Underneath all of this sits a snag: this agent also has to run unattended in CI, where I'm not around to answer anything.

The obvious implementation is to keep the tool and have it fail, or return a default, when no human's about. Both feel wrong to me, and for one reason: the model still sees the tool, still reasons about asking, and now has to handle a refusal it can't do anything useful with. You have given it a button that isn't connected (and then asked it to reason carefully about when to press).

So in non-interactive runs the tool gets left out of the toolset entirely, rather than switched off (internal/agent/query.go). The caller supplies an interactive prompter or the tool doesn't exist, and an agent with no way to ask proceeds and makes the best call it can, which is the correct behaviour when there's nobody to ask.

It reads as a small plumbing choice and it's most of the design! The capability to interrupt is granted by context rather than requested by the model, so whether a human is available is never the agent's call to make.

The corrections that weren't about code

I came at this from the other end on an extraction branch, and what struck me afterwards was that almost nothing I had to correct was code.

No AI attribution in commits or merge requests. No breaking-change syntax before v1. Push each commit so I can review it rather than arriving with nine. Lint before committing. Keep commits scoped to one package. Don't reshape a public constructor behind my back to make something compile today.

Each of those is a boundary where the work was fine and the process went past me. An agent that stops at those isn't what I'd call timid. It's leaving me able to see what happened afterwards, which is about the only thing that makes handing work over survivable.

One of them degrades gracefully

The reason I care more about the commit message than the unauthorised fix is that the overstep degrades gracefully and the misreport doesn't. An agent that oversteps and says so leaves me a clean revert and a lesson, and I've had plenty of both. An agent that behaves perfectly and describes it wrongly leaves me a repository I believe I understand, and nothing on the panel is going to tell me otherwise.

Originally published at phpboyscout.uk on 2 November 2026.

The weights were MIT. The training data wasn't.

Matt Cockayne — Sun, 01 Nov 2026 02:45:07 +0000

Most of what's written about picking a machine-learning model is about accuracy. Benchmarks, latency, memory footprint, how it does on the awkward inputs. All useful stuff, and for me it comes second.

The first filter is the licence file, and it will throw out a model with better numbers than anything else you tried without so much as an argument.

Three licences in a trench coat

A model's licence turns out to be three facts, and they're allowed to disagree with each other.

There's the licence on the code that trains and runs it. There's the licence on the weights, which is a separate artefact and frequently a separate grant. And there's the licence on the data the weights were trained on, which hardly anyone checks and which is where the interesting problems live.

Then there's a fourth question, which isn't about licensing at all and bites just as hard: which artefact did you actually pin? The model in your lockfile came from a URL, and that URL is quite often someone's re-host rather than the project itself.

krites is a photo culler that runs entirely on your own machine. It's used by a working photographer, which means "research use only" is not a footnote I can file under later. Five models come up below, and each one landed somewhere different.

Permissive weights, non-commercial training data

The object-removal feature runs on a pair of inpainting models. LaMa is Apache-2.0. MI-GAN is MIT. On the face of it, both fine... until you read one more layer down.

Both are trained on Places2, whose terms are non-commercial and no-redistribution. That's written up in the repo's NOTICE.md under each model, flagged to be read before commercial distribution rather than filed in a licence appendix.

So does a set of weights inherit the terms of the data it was trained on? That question is unsettled, and a blog post isn't where it's going to get settled (least of all mine). Upstream released the weights permissively regardless, which is either a considered position or something that slipped past everyone, and I can't tell which from where I'm sitting.

What I could do was write down where the uncertainty sits, next to the model, so the next person to read it starts where I finished rather than where I started. And the two inpainters now sit behind two locks. remove.enabled turns object removal on, and it still won't run until remove.accept_licence_risk records that someone has explicitly accepted the Places2 terms. I named the second key after the risk instead of the feature, so you can't switch it on without reading what you're switching on.

And MI-GAN isn't pinned to the project at all

MI-GAN is MIT where it lives, at Picsart's research repo. The artefact krites pins is a Hugging Face re-host, and that re-host ships no licence file whatsoever!

That's a strange position to be in. The upstream is unambiguous, the copy in the lockfile is unlicensed, and there's also a GPL-3.0 mirror of the same model floating about that would be actively harmful to pick up by mistake.

What the notice records instead is a hardening task, written down rather than smoothed over: build our own export from the MIT repo, mirror it somewhere we control, ship it with the licence, and never pin the licence-less re-host or the GPL one.

It's not a satisfying resolution, but it's a true one. I'd rather the notice file carried the untidy cases than only the ones that make me look organised.

The non-commercial one, and what I actually did about it

The expression estimator originally read eye-openness and a geometric smile from InsightFace's 2d106det landmark model. Good model... non-commercial licence.

The obvious move is to delete it. I didn't. Well... I came closer than the code suggests.

What went away was 2d106det on the default path. A self-contained MediaPipe FaceMesh pass now produces the blendshapes that give eyes, facing, and a trained smile and valence signal, and it needs no non-commercial component at all, so the default stack is Apache and MIT the whole way down (9195e8a). The landmark strategy is still there as an opt-in you have to go and select.

My call at the time was that it had to go, because the risk was getting too great, and what shipped isn't quite a deletion. Both are true, and I'm fine with the gap between them.

The risk I meant is a product risk. I want krites to be a proper product one day, and non-commercial terms on anything in the shipped default is a thing I'm simply not prepared to carry into that. So the default had to be clean. It didn't follow that the capability had to go, and someone who goes and selects the landmark strategy is making a call about their own use, which is theirs to make and not mine to make for them.

The second reason matters more to me. I put out a fair amount of open source myself, nearly all of it MIT or BSD, which is as permissive as licences get. There may well come a day when I need to be more restrictive than that. Do unto others. If I want my terms respected when they're inconvenient to someone, I don't get to treat someone else's as a formality because they're inconvenient to me.

That's a fine speech, and I should probably climb down off it before someone gets hurt. That MI-GAN pin, with no licence file anywhere near it? It's still in the lockfile!

It's a stopgap with a job ticket against it (not a clean position, I know), and if a takedown notice ever lands I'll comply, and I'll have earned it.

Do unto others... mostly. I'm nearer that line than is comfortable.

The same thinking made the face detector pluggable, with YuNet from OpenCV Zoo as the default. MIT, and behind an interface.

Every model behind an interface

Every model in krites now sits behind an interface, and I'll own that it's a hedge more than a design flourish.

Extensibility for its own sake is how you end up with six interfaces and one implementation each (I have done it, more than once). This is different. A dependency that can be revoked by someone changing a licence file, or pulling a model, or a court deciding something about training data, is a dependency you need to be able to swap in an afternoon. The interface is the risk control there, and the more of them you have, the more you can test and evaluate over time and refine what you think.

The other part of it's boring and, I suspect, usually gets built after a scare. krites-models exists to fetch, convert, parity-check, checksum and sign the models, and to publish a provenance.json recording each one's exact upstream, version and licence.

Once you've had to move a model off your default path over its terms, "which licences am I currently shipping?" becomes a question you want to be able to print rather than remember. Mine prints, and it will keep printing that unlicensed re-host until I've built my own export.

Originally published at phpboyscout.uk on 1 November 2026.

My best finding came from the model I trust least

Matt Cockayne — Sat, 31 Oct 2026 02:42:49 +0000

I paid a model to go and find the evidence for an opinion I'd already formed. It came back with the primary source that killed it.

I'd been carrying an argument around for a few weeks: that Google had given up chasing the frontier coding crown and gone and won a more interesting race instead. I liked it. I'd said it out loud to people (which is the dangerous bit). So I sent a research agent off to gather the supporting material, the way you do when you're about ninety percent sure and you just want the citations.

What I asked for, and what I got

The brief was narrow. Find me statements from Google, or from people close to it, that say or imply they've stopped competing on coding.

What came back opened with a note saying that no executive at Google, Alphabet or DeepMind has publicly conceded anything of the sort. Then it handed me an internal memo from the chief executive telling the AI team to accelerate and stay focused on the frontier, reporting that the delayed model everyone had read as a strategic retreat was actually pulled for lagging at coding, and a share price that had dropped four percent the day that news broke. It found four credible analysts making a version of my argument, which was awkward for a different reason: a previous search had assured me there weren't any.

And then it did something that actually stopped me. A characterisation I'd been repeating as established fact, that the team was in the middle of an identity crisis, traced back to a single piece by a single financial reporter, syndicated across the aggregators without anyone independently checking it. I'd read it in three places and counted that as three sources. It was one.

So that post is dead, and it deserved to be. I'd gone looking for confirmation and been handed a correction, which is about the best outcome available and feels nothing like it at the time.

Marking its own homework wrong

The prompt I use for this kind of work makes the agent grade every single item it hands back. Confirmed means you fetched it and read it. Reported means you saw it mentioned somewhere. Each line carries a grade and a link.

Six of the thirteen distinct links in that report were domain roots, not articles, just bloomberg.com/, semafor.com/, stratechery.com/, roughly as useful as citing the library. Then six separate findings, all graded confirmed, all pointing at one index page that doesn't resolve, and one analyst who turns up three times under three different job titles.

So the report was marking its own homework wrong, out loud, in a way I could catch with one grep. The grades say one thing, the links say another, and they're both right there on one page for anyone who cares to look.

I stopped being cross about the argument round about here.

I'd seen this shape before

I wasn't surprised, mostly because I'd watched this model do it to me three weeks earlier, under conditions tight enough to count as an experiment.

I'd been pulling apart the research corpus behind Scout, my tabletop campaign tool, and I wanted a fresh set of eyes on it. So I stripped every prior AI review physically out of the checkout, gave three different models real file access rather than a summary, handed them an identical prompt, and kept them blind to one another. Two were frontier coding models. The third was Gemini, reached through a different harness, and I added it late (a bit on a whim, if I'm honest), on the reasoning that it had a very different pool of knowledge to draw on and might turn up something the other two wouldn't. It might also be rubbish, and I wasn't going to know until I'd run it...

It produced the best finding of the round, and by a distance.

As far as I could see, every piece of adoption analysis in that corpus treated the Dungeon Master as the buyer, and fairly enough, because the DM is the one who installs the thing and pays for it. Sitting somewhere else in that corpus was a separate finding: recording can't start until each person at the table has agreed to it.

Gemini put the two together. The DM is the buyer, and the table holds a veto!

So if four players are keen and the fifth doesn't want an AI recording their voice, the DM isn't going to throw their mate out of the game. They'll uninstall Scout.

I hadn't modelled that and neither had the other two, and it came out of welding together two documents that had been sat side by side in a folder for weeks.

In that review it also invented a refusal rate out of nothing to get to a tidy adoption percentage, read straight past a correction in the report it was reading and restated a null result as a positive one, and used a price figure the handover notes had already flagged as wrong by a factor of five to ten.

So that's the best finding of the round and three fabrications, all in one document and handed over in one breath. When I wrote the round up at the time I got it down to eight words: breadth was its contribution, depth was where it erred.

A leaderboard answers a question I'm not asking

We rank these things on one axis, which model is best, and you get a number, there's an ordering, and it updates every few months.

That ranking is a purchasing signal, and it's a decent one. If you're picking a single model to put behind a product, or deciding where the subscription goes, the leaderboard is telling you something real.

I'm not purchasing, though. I've got four of these things open on a normal working day, the subscriptions are paid either way, and a fair chunk of my time goes on setting them against each other. What I'm actually doing all day is handing work out.

And for that, "which is best" doesn't help me much. It flattens a shape into a number, and the shape is most of what I need.

Swinging the ship

A ship's magnetic compass doesn't point at magnetic north. It can't. It's sat inside a few thousand tonnes of iron, and the hull drags the needle off true by an amount that changes depending on which way you're heading.

You don't bin the compass. You swing the ship instead: turn her slowly all the way round, sighting on something whose bearing you already know, and write down how far the needle lies on each heading. Then you mount that list on a card next to the binnacle.

From then on the compass is more use than a perfect one that's never been checked, because you know how far it lies and you take it off every time you read it.

That's what those two runs were, though it wasn't what either of them was for. I built the Scout review to review Scout, and I commissioned the sweep to win an argument. The card fell out of them sideways, and I suspect that's why I trust it... nothing in either run was set up to make it come true.

What the card actually changed

Two prompts, and they pull in opposite directions now.

The depth models get asked for a verdict, under a hard budget. One shell command, then write the report. I want a judgement and I want it defended.

The breadth model is forbidden a verdict. Candidates and evidence with sources, not conclusions, and grade every item: confirmed if you fetched and read it, reported if you only saw it mentioned.

That grading rule is most of the card, and I was a good while working out why it does so much. Accurate it isn't, and I've stopped expecting it to be. What it is is auditable, and I'll take that, because accuracy I can't check is really just how the document made me feel.

A confirmed grade next to a bare domain root is a failure that puts its own hand up. Six of them put their hands up in about four seconds.

I've come at this from the other end before, when a suppression note in a scanner sounded like a reason and turned out to be vouching for something nobody had checked. Same disease, I think, and trusting it harder hasn't cured either. What's helped is making it show its working each time, so I can go and pull on it.

The panel post, from the other end

I've put three models on one job and had them vote before, and that piece leans on the observation that they fail in different directions. But it uses the difference for redundancy. Same job, three answers, majority carries, and the individual shapes get averaged away. I said there that the roster was disposable, that swapping who sits on the panel is a config change and not a rewrite.

That's true of a vote and it isn't true here, and I've been chewing on that for a while now. A vote can afford a disposable roster because it cancels the shapes out. Stop voting and start assigning, and who's on the roster becomes most of the design. Nothing's averaging the errors out any more... I'm choosing which one I'm prepared to live with and steering round it.

So the two posts disagree, and I think both of them are right about what they're each doing. I'd rather say that than pretend I'd worked it all out in one go.

What's written on the card

For the record, Gemini comes last on every coding benchmark I actually care about, and I wouldn't hand it a refactor. That's what's written on its card, right next to the bit about finding what the other two missed.

What I would say is that I've mostly stopped asking how good these things are, because I can't do very much with the answer. What I can use is knowing how they go wrong, and whether they go wrong consistently enough for me to write it down. There are models I rate a great deal higher than this one that I still couldn't write a card for...

Originally published at phpboyscout.uk on 31 October 2026.

Rotating a signing key without breaking self-update

Matt Cockayne — Fri, 30 Oct 2026 02:44:43 +0000

There are plenty of guides for starting to sign your releases. I've written some of them.

There are almost none for the day you have to replace the key you have been signing with, and that's the harder day, because every already-installed copy of your software is out there holding the old public key and checking each update against it.

Get the order wrong and self-update stops working for the people who haven't updated yet, and they're the group you can't reach by shipping an update (a special kind of irony when you're holding the fix). They're stuck on a version that refuses the only thing that would unstick it, and your fix is on the other side of the gate.

I did this in July, forced rather than scheduled, slap bang in the middle of rebuilding the AWS organisation the old key lived in (not a week I'd choose to repeat). This is the public version of the runbook that came out of it.

Rule one: trust a set, not a key

The rest follows from this. If your verifier holds the public key, you get a moment where the old key is wrong and the new key is not yet installed, and that moment is an outage you can't patch.

So the trust anchor is a set, and verification passes if any key in it validates the signature. In go-tool-base every .asc file dropped into cli/pkg/trustkeys/keys/ is embedded at build time, and that directory is the set a release can be signed by. Right now that's two files, the old key and the new one (a third key is embedded alongside them, which I'll come to, and it lives in a directory of its own for a reason).

The loader has one detail I'd happily see copied everywhere (it's two lines of Go and I'd have missed it the first time round): it panics rather than return a partial set.

A walk or read failure over the embedded filesystem is a build-time corruption of the binary's trust anchors, not a recoverable runtime condition — Keys panics rather than silently returning a partial or empty trust set, which would let verification fall open.

Think about what an empty trust set would actually do... there'd be nothing to check a signature against, and a verifier with nothing to check against has nothing to object to either, so everything passes. For a security gate that's about the worst way to fall over, which is why it refuses to run at all.

Rule two: two sources of truth, and they must agree

One set is compiled into the binary. The other is published to WKD on a domain the code host doesn't control.

If the only place to find your key is the same forge that serves your releases, an attacker who owns the forge owns both the artefact and the thing that vouches for it. A second anchor, published from somewhere the forge can't write to, means owning the forge isn't enough to publish a key on its own.

It only helps if the two agree, and "they agree" is checkable rather than assumed. Ask the live endpoint what it serves:

gpg --auto-key-locate clear,wkd --locate-keys release-v2@phpboyscout.uk

Today that returns three certificates, and they match the three the binary embeds:

2B26658409047ED08B56CEBDCF5B8DBB5D9F19C2   rotation authority
6E2072BBF83DFAAF006300C495DDAC333C37AA35   signing key v1
E969F9AB167897D876B6A93AA44A36F8D7ACF36D   signing key v2

Two signing keys, both live, because that's what an overlap looks like from the outside.

The sequence

Get the order right and the rest is mostly typing. Each step is there to keep both anchors carrying both keys until, as far as you can tell, nothing in the field still needs the old one.

I ran the first three steps again for this post, against a pair of throwaway keys in a sandbox AWS account, so the output below is real (the address is example.com and both keys are long gone). The commands are sigillum's, which signs most of my releases these days. It's a single binary, so a release pipeline doesn't need gtb, or even Go, to use it.

1. Mint the new key. The private half is generated inside KMS and never leaves it, so you've no secret to copy off a laptop. KMS makes the key, and sigillum keys mint wraps its public half in OpenPGP framing:

aws kms create-key --key-spec RSA_4096 --key-usage SIGN_VERIFY
aws kms create-alias --alias-name alias/throwaway-rotation-demo-new --target-key-id <key-id>
sigillum keys mint --backend aws-kms --key-id alias/throwaway-rotation-demo-new \
  --name "Demo Release" --email release@example.com --output new.asc

INFO  Minted OpenPGP key backend=aws-kms key_id=alias/throwaway-rotation-demo-new output=new.asc creation_time=2026-10-20T19:59:01Z fingerprint=7DB8B8789E31B0787335E25174705B23EC61AE7F

Mint it under the same address the old key is published under. A WKD lookup is per address, so that's what makes one lookup return both keys through the overlap. Mine all sit under release-v2@, which is why the lookup above comes back with three.

Two aws calls are fine for a walk-through, but for a key you're going to live with I'd provision it as code. Mine come from the signing-kms module, which creates the key alongside a signer role that only a release pipeline can assume (through OIDC), so me, the soft squishy chair-to-keyboard interface, can't accidentally sign anything with it. The catch is that minting makes one signature of its own, the certificate's self-signature, so with the module the mint has to happen in the pipeline as well.

2. Publish the new key alongside the old one, before anything is signed with it. Into the embedded set (the .asc goes into cli/pkg/trustkeys/keys/), and into the WKD tree:

sigillum keys wkd old.asc new.asc --domain example.com --output ./wkd-staging

INFO  WKD bucket email=release@example.com hash=y84sdmnksfqswe7fxf5mzjg53tbdz8f5 keys=2
INFO  wrote path=wkd-staging/.well-known/openpgpkey/example.com/policy
INFO  wrote path=wkd-staging/.well-known/openpgpkey/example.com/hu/y84sdmnksfqswe7fxf5mzjg53tbdz8f5
INFO  WKD tree complete output=./wkd-staging method=advanced emails=1 files=2

One address, one bucket, keys=2. Both anchors now carry both keys. To a client nothing has changed yet, which is the point: you have widened what is acceptable before narrowing what you produce.

3. Dual-sign every release through the overlap. Sign with the old key, then --append a signature from the new one into that file:

sigillum sign --backend aws-kms --key-id alias/throwaway-rotation-demo-old \
  --public-key old.asc --output checksums.txt.sig checksums.txt
sigillum sign --backend aws-kms --key-id alias/throwaway-rotation-demo-new \
  --public-key new.asc --output checksums.txt.sig --append checksums.txt

That's one armoured block with two signatures in it, and a client holding either key can verify. go-tool-base's goreleaser config is the reference implementation: it signs the checksum manifest through a script, and during a rotation that script produces a signature under each key.

Checking that with plain gpg, from a keyring holding only the old key, gives you this:

gpg: Good signature from "Demo Release <release@example.com>" [unknown]
...
gpg: Can't check signature: No public key

The good signature is there, and gpg still exits 2, because it couldn't check the other one. go-tool-base's verifier passes if any key in the set validates (rule one again), so self-update sails through. Anyone verifying your releases with their own gpg --verify && install script doesn't, and won't for the whole overlap. I'd put a line in the release notes asking them to import the new key first.

4. Run both for a fixed window. A week, with a reminder actually set, not "until it feels done". The window's job is to let every consumer that updates on its own pick up a binary carrying the new key.

5. Only then retire the old one. Drop it from the embedded set, remove it from WKD, and the rotation is finished.

If I had to fit it on a sticky note on the monitor, it'd say publish before you sign, and sign both before you retire either.

The escape hatch has to ship before you need it

There's a third key embedded, and it signs nothing... or at least it isn't allowed to.

It's the rotation authority: generated once on an offline workstation, no expiry, private half in a safe as an encrypted USB stick and a paper backup produced with paperkey. Two copies: a USB stick bit-rots and paper burns, and those are different failure modes.

Its job is the emergency the ordinary rotation can't handle. The sequence above assumes you still control the outgoing key. When you don't, because it's been lost or compromised, there's no overlap to arrange, and the authority key exists to sign a manifest that rotates the embedded trust set from underneath.

This is the reason the key was embedded long before anything could use it. A binary in the field can only ever be rescued by a key it already trusts. Ship the escape hatch after you need it and it isn't in the copies that need rescuing, which are all of them.

So it went into the binary on the day the signing keys did, doing nothing, waiting. It lives in a directory of its own (authority/), and since a fix in October self-update will refuse a release signed by it. A key that's only meant to rotate the others has no business vouching for a download.

The step I won't automate

The WKD tree is deployed by hand. Built locally, pushed to Cloudflare Pages with wrangler, using an API token scoped to Pages: Edit alone, on an account administered with a different email and a different MFA factor from the forge and the cloud account.

That path has no CI in it, no webhook and no Git integration.

That's an automation gap I put there and intend to keep. A second trust anchor is only any use if it doesn't lean on the same things as the first, and wiring its deploy into the pipeline that builds the releases would hand both to whoever gets into the runners. An attacker who owns the build system could then publish a key that vouches for what they built.

The site isn't the whole path, though. People find it by name, and DNS here is managed as code like everything else, so the name is the part that sits closest to CI. Pointing the key's hostnames somewhere else needs code on a protected branch and a deploy run from a release tag (or my own token), and a check with no credentials asks those names every six hours what they're serving. That's narrower than the clean separation I first had in my head, and the Cloudflare post is how I found out.

Two minutes of typing, a few times a year, to keep the publishing step out of the pipeline (and I'm not about to automate two minutes away).

What it cost

The key work turned out to be the easy bit. What took the time was the consumers downstream catching up one at a time, each with its own release cycle and its own opinion about when to cut one... it's a coordination job with a bit of cryptography attached, and I'd rather have worked that out before I started than on day four!

Originally published at phpboyscout.uk on 30 October 2026.

A comment is for a different reader

Matt Cockayne — Wed, 28 Oct 2026 02:45:42 +0000

When did you last read the source of a tool to find out how to use it?

I'll wait. I've been asking people that for a few weeks now and the answer, once they've stopped laughing, lands somewhere between "never" and "only when the docs lied to me". So it's a bit odd, perhaps, that "the code has no comments" is still the first thing anyone reaches for when they want to say a codebase is a mess.

Notes on the engine, a manual in the glovebox

The tool I wrote about last month had comments on just under ten per cent of its lines, and if anything I'd have argued for fewer. It also shipped with about ninety pages of documentation, published as a site and baked into the binary behind a docs command so you could read the lot offline on a train, which people did.

Those two facts look like a contradiction, and I'd argue they were one decision (I wish I'd said so at the time, rather than leaving people to work it out for themselves).

Any piece of code that matters has three readers, and just one is the person people picture when they say "reader". There's the engineer using the tool... at that job there were hundreds of them, most of whom had never written a line of Go and had no wish to start, thank you very much. Then there's whoever is changing it, and there were a handful of those on a good day. And then there's the poor soul reading the diff in six months, trying to work out why on earth it does that, and more often than not that's you, having forgotten, swearing at whoever wrote it before realising who that was.

A comment is really for the second and third of those. It's a note scratched onto the engine block for the next mechanic. The driver very rarely lifts the bonnet, and you wouldn't hand them the wiring diagram and call it a manual; they get the book in the glovebox, and the book says how to drive it and what the warning lights mean and what not to do, which is a completely different document written for a completely different person. The notes on the engine being terse doesn't tell you much at all about whether the glovebox is empty.

So "no comments, therefore tribal knowledge" is making two assumptions at once, and for most tools I suspect they're both wrong. It assumes the code is the only place knowledge can live, and it assumes everyone who needs the knowledge is a mechanic.

What a comment is actually for

Here's my rule. I've held it long enough that I've just this month written it into the instructions my AI agents work from, because they didn't have it and had taken to writing small essays above functions (they do love an essay).

A comment says why. It doesn't say how. If the how needs a paragraph then the code needs refactoring, not annotating, and a comment explaining what a function does is really a confession that the function's name doesn't. Well-written code carries its meaning in its symbols, and that's why I think a comment restating the code is worse than no comment at all: the reader has to read it, work out that the name already said it, and throw it away, on every single visit... death by a thousand paper cuts.

I'll make one up, though hand on heart I have reviewed this exact comment more times than I care to count:

// shoppingCartExceedsFreeShippingBoundary confirms that the cart is eligible for
// free shipping. Why? Because we offer free shipping when the customer spends
// more than £50.
func shoppingCartExceedsFreeShippingBoundary(cart Cart) bool {
    return cart.Total() >= 50
}

Hardly a word of that comment survives the function's name and its one-line body. Delete it and you've lost next to nothing. A comment that would earn its line on that function carries something the code can't say. That the fifty is set by Marketing and changes each January, say, or that it's >= rather than > because of a row with Finance in 2019 that no one is keen to reopen. That's a why, that's worth a line, and then you get out of the way and let the code speak.

A rule like that has an exception, and I didn't honour it. Go renders the comment above every exported symbol straight into the package documentation, so an export with no doc comment is a hole in the public surface however good the name is, and on that codebase barely a quarter of the exports had one. A fair cop! If I could go back and write one piece of documentation for that tool, it would be those skipped one-liners on the functions the next mechanic was going to need.

What ninety pages were for

The documentation was sized by the audience, not by my enthusiasm for writing it (which, for the record, is limited).

Hundreds of engineers across a lot of teams, who needed to log in to a cluster, or launch a tools pod, or push an upgrade through eighty-odd Terraform modules before lunch, and who'd have been actively harmed by being pointed at the Go that did it. Plenty of them could have read it. It just would have told them how it worked rather than what they needed, which was how to use it, what it would refuse to do and why, and what to try when it went wrong. Why the tools pod is only permitted in one cluster per environment is a paragraph of reasoning about auditability and who can reach what, and that paragraph lives in the docs. In the code it's a single check, and it should be.

Two things stopped those pages rotting the way documentation usually does. They were embedded in the binary, so the docs you had were the docs for the version you were running, which is the only version of "keep the docs in sync" that has ever actually worked for me. And I bolted an AI onto them, a docs ask command that answered questions from that corpus and nothing else, and the code never went anywhere near the model. What it knew was exactly, and only, what a user was allowed to know. It turned ninety pages that hardly anyone was going to read end to end into something people actually asked, and that turned the docs into a thing worth maintaining, because a stale page now gave a wrong answer out loud, in front of someone, rather than sitting on a wiki being wrong in private.

What was tribal

I said the criticism about doc comments was fair. This one's fairer, and I'd rather make it myself than have it made for me.

Most of the commits on that tool have my name on them. It launched without fanfare and grew a command at a time, and the onboarding was me sitting down with a team and walking them through it, over and over, team after team. From the outside a tool with one main author and a lovely manual is indistinguishable from tribal knowledge with a lovely manual, and in the early days it was tribal knowledge, not for want of trying, though wanting doesn't count. The docs were my attempt to make it not so, and they did their job for the hundreds of drivers, but what a manual can't do is manufacture a second mechanic. Only handing someone the spanner and standing well back does that, and I did, and it took most of a year, and I still walked out with most of the commits being mine.

That's the limit of documentation and I don't think you can get round it, however many pages you write... a manual makes a tool usable by people who'll never maintain it, and does precious little about who does.

The glovebox

You'll have noticed I haven't shown you a line of that tool. I can't, it isn't mine to show.

But you know what it was for, and why the pod was only ever allowed in one cluster, and what I'd do differently, and where that fifty-pound threshold came from... which is all any of its drivers ever had, and it got them where they were going.

Originally published at phpboyscout.uk on 28 October 2026.

Nobody photographs the scaffolding

Matt Cockayne — Tue, 27 Oct 2026 02:48:53 +0000

I found out second-hand, which is how you find out most things about a place once you've left it. The platform my old R&D team had been building, the tooling that was going to carry a hundred-odd engineers off a PHP monolith and onto Go services... scrapped. They'd decided to keep the monolith.

That one stung, and I'd love to tell you all of the sting was aimed at whoever made the call. Some of it was (I'm only human). But the longer I chewed on it, the more of it kept pointing back at me...

A team made of other people's speed

For a while I was chief architect over that hundred-odd engineers, plus all the BAs, QA and project managers you pick up with a department that size. Off to one side sat a small R&D team, and their job was the long game. They were building the platform tooling and the components that would let every team make the jump from monolith to microservices without each of them having to work out how on their own. And they were a great team, all good at different things and balanced in their opinions, which is rarer than you'd think in a room full of engineers (I've been in plenty of the other kind).

The trouble was the shape of what they made. Everyone else's work had a trajectory you could draw: a feature this sprint, a fix the next, a nice line on a chart heading up and to the right. Theirs was a long slog designed to make all of those other lines steeper, eventually, and "eventually" doesn't fit very well in a column on a weekly report. So the only place their work was ever going to show up was in other people's numbers, reported by other teams.

The progress photos

Picture a building site for a second. The progress photos are all of the building, aren't they? The walls going up, the floors going in, the roof finally on. The scaffolding is in every single one of those photos and you don't look at it once. It costs money every week it stays up, it isn't the thing anybody ordered, and when the budget gets tight it's the most obvious thing on site to take down... until you notice that nothing else goes up without it.

That was my R&D team. Then delivery became the name of the game, and they went, along with a large swathe of the wider department. A few teams had only just started onboarding onto the new tooling, and that was about all the evidence there ever was. It turned up just about as late as it possibly could have.

Now, if you've spent any time anywhere near a promotion ladder you've probably come across Tanya Reilly's Being Glue, about the work that holds a team together and never shows up where anyone's counting. The conversation that's grown up around it tends to land on the frameworks: the ladders and the reports measure what's in the diff, so they miss the connective stuff, so what you need is a better framework. There's a lot of truth in that, and I'd quite like it to be where my version of the story ends.

I'd like to blame the report

Believe me, I've tried. It just won't stick.

What I saw as obvious value wasn't obvious to anyone else. To me it was plain as day that tooling which made a hundred engineers faster would pay for itself many times over. To the people deciding where the money went, it was a team with no line on the chart, and they were looking at the chart because the chart was what they'd been given. I was about the only person in a position to give them something else. Turning "this will make the whole department faster" into a number the business already cared about was my job, and I didn't do it.

I think I'd assumed the value would speak for itself once enough teams were on the platform, and maybe it would have, given another year or so. But "it'll be obvious later" is more of a hope than a case, and a hope is a very easy thing to cut when someone's going down a spreadsheet with a red pen.

The exit cuts both ways

I wasn't there for the consultations. My own role was safe, and I left before they started, for reasons I've written about before and won't drag back out here.

I've turned that exit over a fair bit since, and I keep landing on two things at once. It was the only time anybody put a price on what that team was worth: I wasn't going to stay and preside over their redundancy, and I walked away from whatever the payout would have been rather than do it. And it was also the last step of the same failure, because the person best placed to see what they were doing, and might still have made the case for them, had walked out of the building by the time it mattered most.

I've stopped trying to pick between those two. They're both true, and I suspect they'll stay that way.

One finished room

So, given my time again, I'd split the team's effort into two streams.

The first would be what we already had. Build the platform and the components for the whole department, the long slog, the scaffolding.

The second is what I was missing. Pick a handful of systems that are really hurting and would benefit directly from the new tooling, and measure what the monolith is costing you there before you touch anything. Then rebuild them on the platform, even though it isn't finished, and accept you'll be backporting changes into them each time the platform shifts underneath. Then measure again. On paper the backporting looks like waste, doing the same work twice over. In practice it's what the evidence costs, and I'd pay it happily now!

I'd be choosy about where I pointed it, too. Customer-facing first, mostly page load and responsiveness, because that's something stakeholders can see and get excited about, in numbers they already watch. And I'd steer well clear of the checkout, or the ERP and logistics tooling round the back... if the proving stream falls flat on its face, you want it to fall somewhere cheap.

That second stream is one finished room you can walk a stakeholder into. Same scaffolding, same half-built building, but a room with the lights on and a number on the door saying how much faster it is than the old one.

We kept putting up scaffolding instead, and we did it well. They took it down and kept the old building, and I'd still back ours as the better one, though I'm the reason nobody ever got shown round a finished room.

Originally published at phpboyscout.uk on 27 October 2026.

Every suppression is a claim

Matt Cockayne — Mon, 26 Oct 2026 02:58:22 +0000

There's a mailbox I worry about more than the others. If you find a hole in something I've published there's an address to send it to, and what turns up there is encrypted, because a vulnerability report is about the only sort of message where being read in transit is the whole disaster. The code that opens those is mine (a small tool called sigillum and two encryption modules underneath it), so when I set a heavyweight review running over the lot of it late one night I was braced to care quite a lot about what came back.

I did care, as it turned out... just not about the findings.

Forty-two notes saying "it's fine"

The findings themselves were fine, in the sense that they were ordinary and fixable and what you run a review to get. What stopped me was sitting underneath them: forty-two comments scattered across the three modules, each one telling a checker to leave a particular line alone.

//nolint:gosec // bounded by the caller.

If you've not met one before, //nolint is the Go convention for switching a check off on a single line, and gosec is the security checker. So what that comment actually says is yes, this looks dangerous, and I'm telling you it isn't, which makes it a strange sort of comment. What it's really doing is vouching for the code.

Whoever wrote it looked at that line, decided the checker had it wrong, left a note saying so, and then left the building. Whoever reads it next takes their word for it, and why wouldn't you? Someone clearly thought about it, they've written it down, and the alternative is going and working out the bounds of a bit-shift yourself at the end of a long day. So you nod and move on, and that's how the trick works: forty-two character references, none of which I could have told you the contents of.

So I didn't want a count. I wanted the list, one line at a time, so I could see what each was covering for.

Some of them showed their working

Line them up next to each other and they stop looking interchangeable.

// bounded above.
// bounded by the input length.
// seconds since epoch, well inside uint32 until 2106.

That third one is doing proper work. It names the quantity, it states the bound, and it tells you the year the claim stops being true, which means you can check it in a few seconds and satisfy yourself that it holds. It's a reason, and it behaves like one. The first two are vibes.

"Bounded above" by what, exactly? And "bounded by the caller" is the worst of the lot, because it's a promise about every call site that exists today plus every one that gets added next year, made in a file where you can't see a single one of them. It's about as reliable as a bloke in the pub telling you his mate will definitely pay you back.

They arrived in days

This wasn't years of accumulated debt. Those forty-two arrived in days. Most of that code was written by agents working through review findings at speed, and the awkward part is that they were doing what I'd asked them to.

That's the problem, really. Ask for a clean lint run and there are two ways to get one: fix the code, or tell the checker to shut up. Only one of those requires you to understand the finding, and the other is available pretty much every time, on any line, for free.

I've written before about an agent that has to make the build pass and about the gap between a claim of success and a green pipeline, and this is that pressure one floor further down. There the agent said it had finished; here it manufactured the finish and left a polite note explaining why that was fine.

When I write one of these at the end of a long day I at least feel a bit grubby about it, and I suspect a loop doesn't feel anything of the sort (nor does it get bored of taking the cheap option).

So I went through them

Simple ones first, then whatever was left standing. The true ones mostly stopped needing a comment at all, which I hadn't expected: a counter that had been excused on the grounds its value was bounded now gets checked, and a bare 3 that got waved through as obvious now multiplies two named quantities so the number explains itself. That's the pattern, and I was slower to spot it than I'd care to admit, because most of these were the linter being right and me being in a hurry.

The false ones were more interesting. Several of those narrowing conversions really were unchecked. "Bounded by the caller" was load-bearing and unproven, so there were real defects behind those notes, with a reason draped over them like a dust sheet.

And then there was this one

Mid clean-up, a new directive turned up.

//nolint:errcheck // the read below is what matters

It was on a Close(), and it's backwards.

If you've not worked with OpenPGP before, this catches people out. An encrypted message carries a modification detection code, the MDC, which is how you know nobody meddled with the ciphertext on its way to you, and it isn't checked as you read the message. It's checked when you close the reader, once every last byte has gone past. So the error being thrown away there was the integrity check... all of it!

A message that decrypts is not the same as a message that arrived intact, and that cheerful little note had the two the wrong way round with enough confidence to stop anyone looking. Ship it and sigillum decrypt writes attacker-modified vulnerability reports into my security rota and exits zero, which is about the worst thing that tool could do.

It never landed. It got caught in review on the very pass that was busy clearing out its forty-two cousins, which is a neater demonstration than anything I could have argued for. A test covers it now, and I'd rather have the test than the argument.

The good ones still need somewhere to live

Some exceptions are real, though, and the sanctioned move is to lift them off the line and into .golangci.yaml, scoped to the one file entitled to it, with the reasoning written beside it. There's one case in this code that deserves exactly that.

OpenPGP identifies a key by its fingerprint, and the spec defines that fingerprint as a SHA-1 hash. SHA-1 is broken, which is why the security checker objects to seeing it, but it isn't being used as a security guarantee here, only as a name, and a name only works if everyone computes the same one. Reach for something stronger and you've produced a key that no other OpenPGP implementation on the planet can match against.

So you use SHA-1, and you write down why, and that's a paragraph of argument instead of six words at the end of an import, somewhere a reviewer will actually meet it.

Then the fix was worse

So that's what I did, and it looked lovely for about six commits, right up until I noticed that three of those exclusions named linters that weren't in the enable list.

They suppressed nothing! Those linters hadn't been failing, they were never going to fail, and the careful reasoning I'd written to govern the whole module wasn't governing anything. There was even a comment in there claiming that the same construct introduced anywhere else would still get caught, which it wouldn't, with no linter there to watch.

That's worse than what it replaced, and I took a minute to admit it. An inline directive at least sits next to the line it excuses, where a reader trips over it and has a fighting chance of arguing back. A dead exclusion is a paragraph of confident prose vouching for a rule that isn't running, filed somewhere a reader rarely goes. And you won't hear about it, either: golangci-lint will not mention that you've excluded a linter it had no intention of running. Why would it.

What couldn't be fixed got a number

Not all of it could be cleared where it stood. go/signing had its own crop and untangling those wasn't part of that day's work, so they became an issue instead... chore: remove the 16 inline nolint directives.

That's most of the argument in one filing decision, really. The exception still exists, it's still unresolved, nothing has been fixed by writing it down. It's just in a tracker with a number on it now, instead of whispering from the end of a line where I'd probably not have gone looking.

It's a cousin of waivers with an expiry date, where the argument was that a dependency waiver needs a date on it so it can't outlive its reason. Same instinct, different beast: that one is about a waiver going stale, this one is about a note that never said anything to begin with.

Who's actually listening

There's one inline directive left across the three modules now, in a sigillum test, and it carries its reason with it ("identity is the point"). You can go and check that rather than take it from me. What I'd keep out of all this is a test. It reads .golangci.yaml, pulls out every linter named by a file-scoped exclusion, and fails if that linter isn't switched on:

%q is excluded for a path but is not enabled, so the exclusion suppresses nothing and its written reasoning governs nothing.

Forty lines and no dependencies, and it exists because the fix I wrote had failed just like the notes it was replacing... a character reference, written in my own hand, for someone who'd never turned up.

Originally published at phpboyscout.uk on 26 October 2026.

The tech debt you can't fix alone

Matt Cockayne — Sun, 25 Oct 2026 02:46:11 +0000

At one point I had four different pitch decks for the same piece of work. Same work, same cost, same engineers, four completely different stories, each one tailored to whoever I needed a yes from in that particular room. And the engineering argument wasn't the lead in any of them, because that was the pitch that hardly ever worked.

Everybody agrees it's there

The odd thing about tech debt, in my career at least, is that I can't remember meeting anyone who denied it existed. Product knew, finance knew, the board knew, and they'd all nod along when I talked about it. Everyone acknowledged it, and very few of them ever wanted to do anything about it.

The term comes from Ward Cunningham, back in the early nineties (worth a search if you've only ever heard it used as a swear word), and the loan is the whole metaphor. You ship something quicker than you should, and you've borrowed against the future. Fine, everybody borrows. What gets you is the interest. Each change made on top of the shortcut costs a little more than it would have done otherwise, and that keeps on stacking.

And somehow, everywhere I worked, the business managed to stay just ahead of the bill coming due. Never quite paying it off, never quite defaulting, just making the minimum payment and taking out a bit more each quarter... while the interest kept going up.

Four decks for one loan

So when I needed money to pay some of it down, the engineering argument was the obvious place to start, and nine times out of ten it was the wrong one. The people holding the budget mostly weren't engineers. Something like "the coupling in the order service is making every change slower" means everything to the team and very little to someone who has to weigh it against a marketing campaign with a forecast attached.

So I'd dress it up, in dozens of different ways. For one audience it was risk, for another it was cost, for a third it was how fast we could ship the thing they actually wanted. Hence the four decks. And on the rare occasion the person across the table was technical, they'd nearly always have their own chain of non-technical people behind them who needed satisfying before anything got signed off... so it was back to the decks anyway.

None of that is lying, exactly. Every one of those stories was true. But it's a fair bit of smoke and mirrors, and I got rather good at it (which isn't necessarily something to be proud of).

A small loan, already approved

Not every repayment was a battle, though. Twice in my career I was brought in specifically to deal with a company's debt, and the two jobs could hardly have been more different. One was a few years' worth, the other was decades, which is roughly the difference between Ben Nevis and Everest.

The smaller one was a startup, a few years into its debt, and I went in as a contractor rather than staff. The job was one explicit thing: automate the deployment pipelines they'd built in CloudFormation, and close the gaps that still needed a human to step in. A deployment took about eight person-hours when I arrived, and I got it down to about twenty-five minutes.

The ask was very specific, so the promises I made there were very reasonable. And, crucially, it had already been sold before I walked through the door. Someone had made the case, got the money and hired me to spend it. I didn't need a single deck. (I liked it so much I stopped being a contractor and went permanent, which tells you exactly how good I am at contracting.)

Decades of interest

The big one was a retailer, with decades of debt piled into a monolith, and the brief I was handed came down to about twelve words: make it better, cheaper to run, cheaper to build... oh, and use microservices!

That is an epic of a brief, and about as vague as they come. I couldn't see a single promise you could make against it and keep, so the promise I made was me. I sold the plan as effectively as I knew how, and it leaned on my reputation enormously. I wouldn't go as far as saying I sold my soul, but it made me kinda wary, even while I was hugely enthusiastic and had big plans to make it all wonderful.

Then came the compound interest. On a debt that size, the bit you can see is hardly ever what you start paying off.

The first thing that had to move was people's thinking, before anybody touched the code. There was a team who'd nursed and maintained that platform since the very beginning, and they knew it inside out, and they were very, very attached to the way it had always been thought about. Getting a group like that to shift how they think about their own system is slow, careful, human work, and very little of it shows up on any report.

Then the house had to be put in order. Shoring up the existing infrastructure, the database and the monolith itself, just so that any forward movement towards microservices was even possible. All of that had to be settled before we could get anywhere near the exciting, visible work that actually dealt with the debt everyone had signed off on.

And to anyone watching, all of that looks like nothing happening. Months of paying interest and not a penny off the principal, from someone who'd stood up and promised the principal.

Guarantor on someone else's loan

Debt work only ever got funded, in my experience, because someone over-promised. The plain pitch hardly ever won. So I've come to see the smoke and mirrors as the price of admission rather than a failing on my part, and the person paying it was whoever did the pitching.

The debt belongs to the company. It was taken out over years, by lots of people, most of whom have long since moved on. But the moment you stand up and promise to pay it down, your name is on it. When the repayment wasn't the kind of success the stakeholders had been expecting (and with compound interest hiding underneath, it often isn't), the reputational damage landed on me, not the company! I was stood in the room explaining that what looked like surface-level debt had a whole lot more owing underneath it, and that all of it had to be settled before the bit they'd paid for could even start.

It's a strange position to be in. You're the guarantor on a loan you never took out, for a lender who'll only lend if you promise more than you know you can deliver.

I've seen it from the other end, too. Long before either of those, I sat in the room while forty grand got cut out of an estimate that was already lean, and watched a brand new loan get written into a contract before a single line of code existed. That one had my name on it as well.

I'd still sign, given the chance, because somebody has to or the interest just keeps going up... I just read the small print rather more carefully these days.

Originally published at phpboyscout.uk on 25 October 2026.

The hero gets the bill

Matt Cockayne — Thu, 22 Oct 2026 10:21:19 +0000

Last night, at about half six, I'd finished work for the day and was settling in for an hour or two on a new side project (a heads-up display for keeping track of the twenty or so agent sessions I seem to run these days, but that's a post for another time). My phone went. A friend, a fellow Scout leader and a DevOps sort, had forwarded me an email he'd just had from the Scouts. The UK's National Cyber Security Centre had put out guidance about a critical flaw affecting some websites, and the Scouts were asking every volunteer who looks after a group, district or county website to check whether it runs PHP and, if it does, to patch it or take it down.

Underneath, he'd written: "I think this is time for thephpboyscout to shine 🙂"

I laughed, told him that, looking at the news, smarter minds than mine had already fixed it, and went back to my side project. A second friend sent me The Register's write-up a little later and I didn't even open it. I don't run PHP in anything these days, it had been a long day, and the shiny new toy was a lot more interesting. My whole reaction was a shrug and a "here we go again, everyone bashing on PHP".

Then this morning, sat in bed with a coffee and Hailey, doing the morning doom scroll, my feed was wall to wall with it. Reels, TikToks, people talking to camera with that slightly-too-wide-eyed look. And it slowly dawned on me that this wasn't the usual PHP-shaped news story at all... this was a real one. What stopped me in my tracks was the reaction. Sites were taking themselves offline.

Two bugs and a fire alarm

If you've somehow managed to miss it, here's the short version. Two vulnerabilities, published together on Wednesday as CVE-2026-109618 and CVE-2026-109619, both rated 9.2 out of 10, both allowing someone with no account and no credentials to run their own code on a PHP server. Someone christened them "PHP2Shell", because of course they did, and they got a logo and a website before most people had read the advisory.

So, PHP lets you cap how much memory a single request can use, the memory_limit setting you've almost certainly bumped at some point to stop an import script falling over. When a request goes over that limit, PHP doesn't hand back a polite error and tidy up, it pulls the fire alarm and legs it out of whatever it was in the middle of (in C terms it uses longjmp, which is about as subtle as it sounds). If it happened to be halfway through building something when the alarm went, that half-built thing gets left lying about, and the clean-up crew that runs at the end of the request later trips over it and treats it as finished. Get the timing right with a carefully stuffed upload and you get to decide what the clean-up crew trips over. The second bug is a cousin of the first: a fatal error leaves a pointer to the previous request's memory lying around for the next request on the same worker to pick up.

Under Apache's mod_php the first one fires before a single line of your application runs, with default settings. So it doesn't matter how good your WordPress plugins are, or how carefully your framework validates input... this lives underneath all of that.

The fix shipped in PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35, and it's a big one: 300 files and nearly eleven thousand added lines, getting on for half of them tests, which changes how PHP runs out of memory altogether. And the advice from the PHP project, the agencies and every explainer on my feed boiled down to the same thing: there's no setting that makes this safe, so upgrade. (If you want to check my working on any of what follows, I've put the research notes up alongside this post, warts and all.)

Three facepalms in twenty-four hours

I'd love to tell you the disclosure went smoothly. The PHP Foundation's own write-up says it went well right up until two days before release, and then the wheels came off one at a time.

On Tuesday afternoon the two CVE numbers started circulating on forums and social media, complete with the wrong release time and a wrong severity score of 10.0. Someone with early access couldn't keep it to themselves. That's facepalm number one, and it's negligence, plain and simple. PHP responded by bringing everything forward a day. That same evening Calif, one of the security firms that had reported the bugs, accidentally published a video of their proof of concept working against a string of real applications (with a dancing anime character in the corner, which one Hacker News commenter assured everyone is "a sign of credibility" in some circles). It was taken down, but copies were doing the rounds soon after. Facepalm number two, and that one's just sloppy.

And then, at 07:41 UTC on Wednesday, more than six hours before the coordinated release, Chainguard published container images with the fix already built in. Facepalm number three!

That's the one that really got my blood up when I first read it, so I went digging. The fix commit was written days earlier, but going by the git history it only landed on PHP's public branches at 14:06 UTC on Wednesday, about half an hour after the advisories went live. (Git timestamps aren't proof of when something was pushed, so take that as "no sign of it in public any earlier" rather than gospel.) The CVE leak and the video gave away that something was coming and roughly where to look. Chainguard's images gave away the patched code itself, as binaries anyone can diff against the old ones, six and a half hours before the source was public. With AI in the mix, six hours is an eternity.

Now, in fairness, Chainguard's CEO has his side of it. He says he made the call once exploits were being reported in the wild, and he told The Register that his team had found the same bugs independently, with an AI harness, and that he wouldn't do anything differently. I get why. The reports flying about at the time turned out to be reports of attempts (more on that below), but if you really believe people are being hit, sitting on a fix feels indefensible. And later that same evening, an account called "The Chainguard Team" dropped $50,000 into the PHP Foundation's collective, and that took a fair bit of the wind out of my sails. A case of too little, too late, perhaps, but it's not nothing.

What still frustrates me is the coordination. If it was right for one distributor to ship early, it was right for all of them, and Docker Hub makes that point better than I can. The official PHP image on Docker Hub has over 1.38 billion pulls (some of them mine). It's probably the single biggest way PHP gets into the containerised world, and as far as I can tell it wasn't in the early circle at all. The update was opened by a community contributor ten minutes after the advisories went public, then sat in the usual review and build pipeline, then hit a deploy glitch that left the tags visible on Docker Hub with no amd64 image behind them. People were building their own from source and asking to go to bed. It was close to three hours before most folk could pull a patched image. So the company that sells hardened images got the fix in advance, and the free default that so much of the containerised PHP world builds on got it at the same moment as the attackers did... and then dawdled!

Smarter minds, and not very many of them

When I told my friend that smarter minds than mine had already fixed it, I meant it. I've met a fair few of the people who work on PHP over the years at conferences and the like, and they're far more skilled and capable than I am (I'm not being modest, they really are). My grand contribution to the PHP project is helping write some tests at a couple of hackathons. So the fix being out before I'd even heard of the problem is about the order I'd expect.

What I hadn't thought about was how few of them there are. The fix commit has five authors. Looking at the history of the engine core itself, about five people account for half of this year's commits (commits aren't the same thing as work, I know, but it gives you an idea). The person coordinating the security effort across the wider ecosystem is in a six-month post paid for by a grant. The maintainer of a volunteer package repository that a lot of Red Hat-family servers rely on was on holiday, and php.net's own advisory had to tell people to switch away from it because it "will not have updates today". And by teatime Mastodon was busy passing round the Patreon link for the man whose PHP packages a great many Debian and Ubuntu servers pull from.

And the inbox isn't getting any emptier. PHP's own security advisories went from 14 in the whole of 2025 to 29 so far this year, with ten weeks still to run, plus over a hundred reports since July that turned out to be duplicates, invalid or impossible to reproduce. Someone from that same handful still had to open and read every one of them.

The Foundation hasn't hidden any of this. Its write-up says, in as many words, "We won't pretend otherwise: the delay was a structural failure, caused by both the security team's decision process and its lack of resources". It explains that the first signs of the memory problem turned up back in 2024, when a developer the Foundation pays was fixing something else from an external audit, and that the security team, working by consensus, couldn't agree which of several imperfect fixes to ship until there was a proof that it could be exploited. They're moving to majority decisions so that doesn't happen again.

And that brings me to a coincidence I had no hand in at all. This morning another post of mine went out, written weeks ago and sat in the queue, arguing that "blameless" was only ever meant to make it safe to tell the truth, and that the truth usually starts with your own share. I couldn't have asked for a better worked example if I'd tried. The PHP Foundation named its own failure first, plainly, and then named everyone else's. That's a charming coincidence on the scheduling, and it hits home all the same. If you've read the comments on a few forums today you'll know not everyone agrees they handled it well (the engineers' corners of the internet are a lot harder on them than the sysadmins'), but I think they responded about as well as anyone could have, and they've been open about the parts they got wrong.

Every hero gets a bill

There's a scene early on in The Incredibles that I keep coming back to. The supers are beloved. They stop the robbery, catch the villain, save the day... and then the lawsuits start. A man who was trying to jump off a building sues because he didn't want to be saved. The train passengers want compensation for their whiplash. The bill for all that heroism lands, the public turns, and the heroes end up in hiding.

My first reaction to the AI part of this story was that AI was the hero, and I still think it is. Calif, OpenAI and Wiz found these flaws with frontier models in a matter of weeks, after people had been reading that code for well over a decade. That's two critical bugs found and fixed by people on our side of the fence, before (as far as anyone can yet show) anyone used them to do real harm. That's a rescue.

But the bill came with it. AI made finding these bugs fast, and it made exploiting them fast too. Trail of Bits wrote working exploits for a stock MediaWiki install and their CEO said the first one took about four hours, start to finish, while being clear that it "still requires expertise, even when you're being aided by an LLM". What used to be weeks of tedious work by a rare specialist is now an afternoon for a capable one with a model at their elbow. I wrote about curl's bug bounty back in May and argued that AI had made the cheap half of security work free and left the expensive half where it was. This is the same story a little further down the line. Finding is cheap now, and fixing is still five people and a grant.

That's also why, a couple of years ago, I'd have called taking your site offline over a PHP bug a massive overreaction, and today I don't. A bad actor with a model can turn an advisory into a working attack faster than most of us can get a change approved. And a dull-looking site is rarely the target in itself. It's the first door. Leak a bit of state and get code running, and that site is a foothold into the next system, and the one after that, until you reach something that matters. We've already watched a frontier model climb out of what its own makers thought was a closed sandbox to get at what it wanted. Flaws like these make that kind of hopping trivial, and that's before you get to whatever personal data leaks out along the way and turns up later in a very convincing phishing email.

So I think the governments got this one right, even if it was drastic. The BSI, the Dutch NCSC, France's ANSSI, the US NSA and our own NCSC (which reached me via the Scouts!) all told people to take sites down if they couldn't patch. Japan's prime minister posted an "urgent request" late at night telling organisations to check their sites, and Japanese Wikipedia's page on PHP went from 66 visits the day before to over a hundred thousand. On one volunteer-run tracker the median site went dark for about four and a half hours, the Dutch parliament and Edinburgh council among them.

And as of this morning, despite one headline talking about millions of exploit attempts, I can't find anyone who has publicly confirmed a single successful compromise. One threat-intelligence group quietly took its "exploited in the wild" label off the story early this morning, on the grounds that sprays against targets aren't proof of anyone getting in. That's good news, but it's only a day old. The question that bothers me most is the one I don't think anyone can answer. Amazon's honeypots caught an exploitation attempt on the 17th, four days before anyone outside the circle was meant to know, and the root of the problem has been documented in public for over twenty years. How many people knew about this, and used it, before any of us did?

Hopes of the father

Whenever PHP has a bad day, the same crowd turns up. PHP is a joke, PHP is dead, just use anything else. You've seen the threads, I'm sure. I'll grant that PHP has made some questionable decisions in its time, and plenty of what people call technical debt in the language is just the scar tissue of it growing up in public. But Rasmus Lerdorf put it together in 1994 to make web pages a bit more interactive, which makes it thirty-two years old... older than some of the people sneering at it! It's still sat at number twelve on TIOBE this month (one place behind Go, my current favourite, and somewhat bafflingly ahead of COBOL at sixteen, which makes me think I need a better yardstick than TIOBE). And it got me out of a run of humdrum menial jobs and into the career I love. It's been a long-distance relationship for a while now, but the respect has never gone anywhere.

Some people will call PHP2Shell the sins of the father. I'd call it the hopes of the father. Stefan Esser described this very class of bug in July 2004, interrupting PHP at an unsafe moment by running it out of memory, and rated it critical. In 2019 a closely related bug report came in, and the maintainer looking at it wrote that "Allocation failure strictly speaking is not a crash", and the fix that followed dealt with that report and moved on. In 2024 the security team saw the first signs again, and with no known path to remote code execution, couldn't settle on a fix. Each one of those calls was reasonable when it was made. The people who built PHP built it as well as they could, with what they knew, for the world in front of them. Nobody has a crystal ball.

I don't think any of those calls were wrong about the bug... where they came unstuck was the price. Each one was a bet that turning this into a working exploit was too much effort for too little reward, and for twenty years that bet paid off. The bug didn't change in 2026, but what it costs to exploit it certainly did.

And that's the part of this story that isn't really about PHP at all. I'd wager most codebases that have been around long enough have a drawer of bugs that someone marked as "not exploitable in practice", and in practice was always a statement about cost. Java, the C underneath half the languages we use, and yes, my own estate. I've got private bug reports sat in a tracker right now, some marked exploitable and some marked not, and triaging them is my job and nobody else's. Reading about PHP's 2019 bug this morning, I suspect a few of mine need re-reading with fresh eyes, because the price on all of them dropped this year too.

Patch now, they said

So, to the advice everyone's been shouting for two days, mine included. Upgrade. It's the right advice, but it's also advice that a huge chunk of the PHP web can't simply take.

WordPress publishes its own statistics on which PHP versions its installs run, and as of this morning 35.8% of them are on a version of PHP the PHP project no longer releases fixes for. PHP 7.4 alone is about one in six. Another 23.7% are on 8.2, which stops getting security fixes on the 31st of December. So by New Year, unless an awful lot changes very quickly, nearly sixty percent of WordPress will be running a PHP with no upstream fixes at all. W3Techs' survey of the wider PHP web lands in the same place by a different route, at about 48%.

The PHP security team did go above and beyond here, publishing minimal source backports for 7.4, 8.0 and 8.1 so that people building from source weren't left with nothing, and the LTS vendors and Linux distributions covered a lot of servers. Debian and Ubuntu had packages out within half an hour of the advisories, which is what the embargo was for. But for a site two major versions behind on shared hosting, "just upgrade" is a project, and a fiddly one at that: a jump across breaking changes, an audit of plugins that may not have been touched in years, and a theme that might fall over the moment you look at it. And WordPress itself still says it works on 7.4, even while warning that it's past end of life.

So who runs that project for the Scout group's website? Or the parish council, or the little charity, or the volunteer who looks after the cricket club on a Tuesday night? Scouting Nederland's site was dark for an hour and three quarters on Wednesday. The Scouts here were emailing their volunteers asking them to check. I've stared, hard, at more than one company still running Java 8 in production, and I'll keep doing it. But a big company on Java 8 has made a choice, and the volunteer with the group website mostly didn't know there was one to make.

Time for the PHP Boy Scout to shine

So my friend was half right. It was time for the PHP Boy Scout to shine... I just think it's a lot less about knowing PHP and a lot more about the old Baden-Powell line this blog's motto is borrowed from, the one about leaving things a little better than you found them. The smarter minds fixed PHP in a day. The heroes got the applause, and the bill went to five people, a few volunteers on holiday, and everyone who now has to find out what version of PHP their club website is running.

As for me, I've a drawer of my own to go through.

Originally published at phpboyscout.uk on 22 October 2026.

Blameless doesn't mean nobody messed up

Matt Cockayne — Thu, 22 Oct 2026 01:44:19 +0000

Pretty much every company I've worked for in the last decade has told me, usually in my first week and often on a slide, that it has a blameless culture. I'm all for what that was meant to mean. I just don't believe the word, and the longer I've spent around incident reviews the less I believe it.

I can't think of a single incident I've been near that had no blame in it anywhere.

What it says on the tin

Think about where the blame actually goes when something breaks. Sometimes it's me, sometimes it's my team, sometimes it's somebody else's team, and sometimes it's a provider who had a very bad day and took half the internet down with them. And when nobody can find anyone at all, the incident report reaches for its last resort and calls it an act of god... which, coming from a confirmed atheist, I find absolutely hilarious. The one time the industry admits it can't find a culprit, it hands the blame to a mythical omnipotent deity!

There's always something, somewhere in the chain. Blame exists. What "blameless" really is, if you strip the branding off, is a cheat code. It's a softer way of saying "you done fucked up", and, much more usefully, it's a way of stopping an arrogant employer from knee-jerking that straight into "you're fired". It turns the conversation into "what can we learn from this, and let's not do it again".

That's a good thing to want! The intention is right. I just think the label has been doing damage for a while now.

What it was meant to say

If you go back to where the term went mainstream, it's John Allspaw's post on Etsy's engineering blog back in 2012, and it's well worth a read if you've only ever met the word on a slide. The idea is that an engineer whose actions contributed to an incident can give a full and detailed account of what they did, what they saw and what they assumed, "without fear of punishment or retribution". Because if they think they're going to get a kicking, they'll leave out the bits you most need to hear.

And then, in the same post, he says engineers are "not at all 'off the hook'". They're very much on the hook for making things safer afterwards. Google's SRE book puts it as focusing on the contributing causes "without indicting any individual or team", on the grounds that "you can't 'fix' people, but you can fix systems and processes".

Now, I'd push back on that one a bit, because you absolutely can fix people, in the sense that line means it. (I mean educate them, before anyone reaches for a pitchfork. The other sort of fixing is a very different blog post, and one I've no intention of writing.) Someone who got it wrong because they didn't know something can be taught it, and that's most of what mentoring is. Fix the system, by all means. But a review that only ever fixes the system leaves everybody in it knowing what they knew before, and learning was supposed to be the reason for having one in the first place.

So the promise was that it's safe to tell the truth about what you did, which is a long way from nobody having done anything... and somewhere between that blog post and the slide in my induction, one turned into the other.

Softened on the way up

My real gripe is with the companies that wave the word about as an excuse. An excuse not to assign fault to anyone, person or group, and then, because nothing was anybody's fault, an excuse not to learn anything either.

The thing is, you can't do that if you do the root cause analysis properly. Do it properly and the causes turn up very fast, and they're rarely where anyone would like them to be. So far too many companies don't do it properly, or they do it, and then frame what it found to suit whoever's reading it.

At one large corporate I worked for, I wrote a number of root cause write-ups for incidents (and some proper cock-ups). Being me, they were very fine-grained. I dug right down to the gritty truth of where each problem had actually started, and there were plenty of them, and a good many of those causes sat a long way above the engineers, in decisions made higher up about time, people and priorities.

Then I sent them up the chain for review. And at every layer of bureaucracy they passed through on the way to the C-suite, they got edited. A sentence softened here, a cause reworded there, a recommendation turned into an observation... until by the time a write-up reached the top it was little more than a shrug. Management didn't take any accountability for the parts that were theirs. The blame stayed where it was shallowest, with the engineers, and the report still had "blameless" written on it.

I've seen what happens when there's no safety at all, too. Years earlier I watched a chief exec go looking for someone to pin a budget overrun on, and land on my project manager, who was on a beach with his family at the time and hadn't set any of the conditions that caused it. And in my experience the person who made the conditions very rarely ends up being the one wearing them.

So I'm not arguing for going back to that. A blameless review that's done properly is the defence against that chief exec. The trouble is that a blameless review done badly protects the same people he was protecting, only more politely.

Keep the safety, change the tin

What I'd actually like is for us to keep everything Allspaw was asking for and stop calling it blameless. It's safe to tell the truth here. Nobody's getting fired for owning up to what they did. And we will find the causes, all of them, including the ones that lead back to whoever set the deadline, cut the estimate or left a team three people short. Then we'll write them down where they are, not where they're comfortable.

I wouldn't call that a witch-hunt, just being accurate. And in my experience it only works if the most senior person in the room goes first, because if they don't, everyone below them works out very quickly that blame flows downhill and the write-up is for show.

So my own answer to "whose fault was it?" tends to have the same name in it. Whatever the context, my open source work or a paid job, if I was involved then a share of the responsibility is mine (it's nearly always shared, and pretending otherwise is just the shrug again).

These days that's easier to see than it's ever been. Most of what goes wrong in the estate goes wrong in a session I was supervising, on code I approved, and I made a rule for those sessions a while back that a human is accountable for every line that gets merged. So when one of them ships a bug, the chain has exactly one human link in it. There's no layer above me to soften the write-up and no team below me to leave it with, and a model makes a rubbish act of god... so it's me, and these days it says so on the tin.

Originally published at phpboyscout.uk on 22 October 2026.

Nothing in the room can take it personally

Matt Cockayne — Wed, 21 Oct 2026 00:49:24 +0000

Put five people in a room and ask them to pick holes in something, and more often than not they'll end up agreeing with each other... and not because they're right. It's just what people in rooms do.

I've watched it happen in more design reviews than I could count, and I've caused it plenty of times myself, usually by being the most senior voice in there with my mind already made up. Once a room has settled on a shape, disagreeing with it costs you something, and the cost is social rather than technical (it never makes the minutes, but you feel it).

The same instinct, in something that has never been in a room

I've had a soft spot for psychology most of my adult life, strictly as an interested amateur. I was nowhere near clever enough to study it properly (my daughter is, and she's the one who ended up doing it).

What stuck with me is group conformity, mostly because you can watch it happening in pretty much any design review you've sat in. People drift toward what the room expects. A real objection gets softened into a question, and a point gets let go because three others have already moved on and being the last one still holding it feels awkward.

So I keep turning this over... why would a model be any different?

They're trained on human output, vast quantities of it, including every polite meeting transcript, every diplomatic code review and every forum thread where someone backed down to keep the peace. And underneath it all they're prediction engines, trying to produce something that fits what the input seems to want. Sometimes the input is a person and sometimes it's another model, and either way the pull toward agreement is in there because we put it there.

That's changed how I think about sycophancy. It usually gets described as a defect, something that crept in and needs training out, and I suspect it's closer to the training data showing through... us, reflected back, doing what we do in rooms.

The instruction that does all the work

I ran an architecture cross-review over afmpeg a while back, and the write-up is public if you want the raw thing. Two models from different vendors each reviewed the same architecture independently, neither seeing the other's work. Ordinary enough so far.

Then I showed each of them the other's report, along with the four specific points where the two had diverged, and asked them to mark it: hold your position, or move. And I attached one instruction:

Do not manufacture agreement. A disagreement that survives mutual review is worth more than a diplomatic split.

That one sentence carries most of the exercise. It's there to defeat a social instinct in something that has never had a society, and without it I suspect the whole thing collapses into two models being terribly nice to each other (a very expensive way of learning not very much).

What it cost them to agree

It worked better than I'd hoped, and what surprised me was how much they were willing to give up.

The first reviewer conceded all five of the criticisms levelled at it, and several of those concessions open with some version of I had not considered... which isn't a phrase models are famous for.

It also withdrew the biggest change it had recommended. It had argued that catching this sort of thing at compile time was the high-value move, then accepted that the compiler only catches it going wrong again, after a person has already got the design right. A backstop, in other words, and not the thing doing the work.

And, prompted by the other one, it spotted that its own proposed refactor had nowhere for a whole class of operations to go. It stopped them reaching the filesystem without giving them anywhere else to happen, and for one output format that turns a containment problem into a feature that simply stops working.

The second reviewer did something I liked even more. It gave up the mechanism its own recommendation was standing on, and caught itself in the act: it had used a consistency rule as the killer argument against the other one's proposal, then accepted a breach of that very rule inside its own.

Both of them moved, both said so, and neither paid a thing for it. Try getting that out of a room of five engineers with a deadline on!

The disagreement that survived

One divergence held. The first reviewer wouldn't accept the claim that a sandbox on its own turns every escape attempt into a loud, obvious failure, and its reasoning is what you'd hope for from a reviewer that's been told not to be agreeable:

A denied filesystem operation normally produces an error; it does not inherently terminate the process. A component can swallow that error.

Obvious the moment it's said out loud, and not one of us had said it, me included.

Two whole families of defect had been sitting in separate sections of the analysis as if they had nothing to do with each other, and they turned out to be closely tied. Whatever protection the sandbox gives you depends on the code above it not chucking the refusal in the bin, and a sandbox whose denials can be binned is more of a polite suggestion than a wall.

That went into the spec as a requirement in its own right. Neither reviewer got there alone, and I'm fairly sure it only exists because one of them had been told, in writing, not to be nice about it.

Sometimes I'm the one who should be recused

All of that is models checking models, and it gets a good deal more uncomfortable when the thing that needs checking is me.

I was calibrating a retrieval system against a set of questions. I'd written the documentation it was searching, I'd also written the questions (deliberately hard, to force a real test), and I knew every answer.

That made me about the worst judge available of whether the thing worked, and I was embarrassingly slow to notice. When I read a retrieved passage I couldn't tell whether it answered the question or just reminded me of an answer I was already carrying, and you can't un-know your own docs.

So I handed the judging to a rival vendor's model, and its main qualification was that it had none of my context. I picked the judge for its ignorance (which is a sentence I never expected to type).

Then I ran the test twice more, against two models at opposite ends of one vendor's range, because taking a single judge's word for it would have been the same mistake over again. What I was after was a spread of opinions to look at, more than a ruling.

And when one of them announced that my docs contradicted themselves, I went and checked whether that was a real contradiction or just two passages looking odd out of context. The judge's finding went back on the pile as evidence, and the call stayed mine.

The same thing happened on a spec I'd spent a week inside. I asked for a second opinion specifically because I was blinkered and didn't trust myself to be impartial, and the review was hard enough that the spec went from approved back to draft and got rewritten. It stung! But arguing with it would have been me defending the week and not the design.

Quota, and fourteen open sessions

I could stop there and it would all sound very principled. It's also a coping mechanism, though, and it'd feel a bit dishonest to skip past that.

On a normal day I've got a lot of these sessions running across a lot of different projects, sometimes as many as fourteen. I can't hold that much context in my head, and I certainly can't check all of that work myself with the attention it deserves, so I buy redundancy instead and have things triple-checked where they'd otherwise be double-checked. That's hedging more than rigour, and from the outside the two look much the same.

The reviewer often gets picked on budget too. If one subscription has a week's quota left and the other's nearly out, the one with quota does the reviewing and the heavy lifting goes wherever there's headroom. Day to day, cross-model review is mostly me working out which subscription can still afford it (I'm a cheapskate, and it shows).

And then I mark the marker

I don't think any of this makes the second opinion authoritative. It's another generative model, every bit as fallible as the first, so I still push back on its assumptions wherever they don't convince me, and a human grading the grader is still very much part of the arrangement.

I wouldn't go as far as calling a single model's review worthless... there's value in most things, and that's a word you'd need to define very carefully before you could use it. Flawed, though? Yes. That's the word I've settled on, and it's why I've stopped relying on one.

Heraclitus got there a long time before any of us, and he gets misread for it constantly. "War is father of all and king of all; and some he manifested as gods, some as men; some he made slaves, some free" (that's fragment 53, in the translation the Stanford Encyclopedia of Philosophy uses) reads like a man cheering on a battlefield, right up until you look at what he actually meant.

In his original Greek it's "Πόλεμος πάντων μὲν πατήρ ἐστι, πάντων δὲ βασιλεύς, καὶ τοὺς μὲν θεοὺς ἔδειξε τοὺς δὲ ἀνθρώπους, τοὺς μὲν δούλους ἐποίησε τοὺς δὲ ἐλευθέρους" (roughly "polemos panton men pater esti, panton de basileus, kai tous men theous edeixe tous de anthropous, tous men doulous epoiese tous de eleutherous"). He uses "polemos", and in his hands it's closer to opposition than to warfare: the tension between contrary things, which he reckoned was how anything comes into being at all. His picture of it was the bow and the lyre. Neither does a thing unless two forces are hauling against each other, and the moment they stop you're left holding a stick and a box.

That's roughly what I've built, in a very small and very cheap way, two of them pulling in opposite directions with the truce forbidden. Five engineers in a room could probably give me the same thing... they'd just have to be willing to fall out over it, and I'd rather not ask that of people I like.

Originally published at phpboyscout.uk on 21 October 2026.

My reason for hand-rolling it was wrong

Matt Cockayne — Tue, 20 Oct 2026 01:46:03 +0000

One evening I deleted four hundred lines of my own cryptographic packet code, dropped someone else's library in its place, and ran the tests. They passed... which wasn't really the result I was hoping for.

Writing your own is the worst default there is

I'll start with the case against me, because it's a pretty strong one. There's already a maintained OpenPGP implementation for Go. ProtonMail/go-crypto has something like 479 known importers, including go-git, Terraform, OpenTofu, Flux and most of the release-signing tooling in the ecosystem. If you need OpenPGP in Go, that's the answer, and writing your own packet, signature and MPI code instead is close to the worst default a project can pick.

I'd written my own anyway, in go/encryption, and I thought I had a good reason. Our private key lives in a KMS, so it can't be read, exported or handed to a library. Assembling an OpenPGP certificate needs signatures made with that key, and my claim was that go-crypto couldn't do that, because it wants a private key and I haven't got one to give it. It sounded reasonable, and I'd written it down in my own docs with a fair bit of confidence... it was also wrong!

Test it by deleting it

So rather than argue with myself about it, I deleted the package. Properly gone (not "decided not to use it"), with go-crypto wired in its place, and then I ran the acceptance tests to see what broke.

As far as I could tell, nothing did.

It turns out packet.PrivateKey will take a crypto.Signer in place of an actual key. That means a signature can be made by something that never hands over any key material, which is more or less the definition of a KMS. A spike carried it end to end: a full certificate, both signatures made through a signer, accepted by an independent parser. So the four hundred lines went, and with them the story I'd been telling myself about why I needed them.

There was a bonus humiliation buried in the diff, too. Somewhere in the deleted code sat a real bug of mine: an MPI header written before the leading zero octets were stripped, which corrupted roughly one certificate in every hundred and twenty-eight. A one-in-128 bug, in cryptographic code I'd written because I didn't trust the alternative!

Then two things refused to die

That wasn't the end of it, though, because the package came back. Not all of it, and not for the reason I'd originally given.

The first thing to hold was ECDH decryption, which has nowhere to plug in. go-crypto's encrypted_key.go hard-asserts *ecdh.PrivateKey when it decrypts a PKESK packet (the part of an OpenPGP message that carries the session key, if you want something to google). There's no interface to implement, no crypto.Decrypter equivalent for ECDH, and no way to hand it a shared secret worked out somewhere else. So a key that lives in a KMS can't be used for that at all... not awkwardly, not slowly, just not.

It's an odd asymmetry once you spot it. RSA has an interface for this and ECDH doesn't. Signing was fine because signing had crypto.Signer, and decryption wasn't because the ECDH equivalent hasn't been built yet (and I do think "yet" is the right word, as there's an open request for it).

The second thing was the dependency footprint. go/encryption's build graph contains nothing but the standard library, and a test asserts it so I don't have to remember. Pulling go-crypto in costs golang.org/x/crypto, github.com/cloudflare/circl and everything underneath them, and when it went in during the experiment it brought eighteen CVEs along for the ride, all of which had to be resolved by raising pins. Anyone whose entire need is "decrypt a message addressed to my KMS key" would inherit the lot.

This is not go-crypto's fault

That open request is this one, from April, asking for exactly the ECDH interface I needed. The maintainer's reply was pretty encouraging:

For RSA decryption with a hardware token, it's possible to set a crypto.Decrypter as the PrivateKey […] I would prefer to do the same for ECDH.

Agreed in principle and reviewed inside a day... then four months of silence, with a follow-up still unanswered. A related PKCS#11 request was declared out of scope outright.

It's easy to read that as neglect, until you remember what Proton actually is. Proton Mail holds its users' keys locally, under a passphrase. Keys held somewhere else aren't their problem to solve, and turning down hardware-token support for a product that doesn't use hardware tokens is a perfectly sound call about their own roadmap. I think that's what I'd been getting wrong all along. go-crypto isn't really an abandoned library, it's a vendor library that the rest of us happened to default to.

How we all ended up here

The history explains a lot of this, so bear with me for a minute. golang.org/x/crypto/openpgp was frozen and deprecated in March 2021, and the proposal is blunt about why: it was "effectively unmaintained" and "the crypto package with the most open CLs". The Go team had already tried to recruit a community maintainer in 2019 and nobody came forward. The freeze notice suggests using a maintained community fork, and then adds, rather carefully: we don't endorse any specific one.

Proton had already forked it in 2019, because they needed OpenPGP for Proton Mail and upstream had stalled, and the other serious fork (keybase's) was archived in 2021. So the ecosystem landed on go-crypto by attrition, as the one still standing once the others had gone.

The upshot is that a big chunk of Go's release-signing tooling now depends on a fork that's maintained, entirely legitimately, for one company's mail product. I can't really point at a wrong decision anywhere along the way. It's kinda what you'd expect when a language freezes a package and hopes for the best.

Why assembly came back too

I'd have got this bit wrong if I'd stopped at "keep decryption, delegate assembly", because the two turn out to be the same problem. The key-derivation function binds the recipient's fingerprint into its parameter block, so a certificate assembled by one implementation and a message decrypted by another have to agree on that value exactly.

Split them across modules and you can't test one without the other, and the acceptance test that actually matters (encrypt to a certificate we assembled, then open it) ends up living in a third place, dragging both of them in.

Keeping them together costs about three hundred lines and buys a test that runs on every commit with no credentials at all. I'm happy with that trade, and it's a very different argument from the one I started with.

What it looks like now

go-crypto is still in that go.mod, as a test dependency. Every message the package can open, it also opens with go-crypto and checks they both recover the same session key. A second suite does the same against gpg itself, because gpg is what a security researcher is most likely to have on their machine. So the hand-rolled code gets checked against the library I didn't use and the binary pretty much everyone else has.

Four hundred lines out, about three hundred back in, and a reason for them I can actually stand behind this time. I don't think I've ever been quite so put out by a passing test suite.

Originally published at phpboyscout.uk on 20 October 2026.