DEV Community: Ken W Alger

Sixty Feet, Six Inches of October

Ken W Alger — Thu, 29 Oct 2026 15:40:03 +0000

What a little pitch-tracking app saw in the 2026 World Series.

October baseball is a different sport wearing the same uniforms.

In July, a starter gets six innings and a long leash. In October, he gets a hook the moment the third time through the order looks shaky. Aces pitch on short rest. Relievers who were anonymous in June become household names in a week. Managers who spent the summer saving arms spend the fall burning them, because there is no tomorrow to save them for.

Which makes October the most interesting month of the year to watch pitchers. So this year, I watched them twice: once on the couch, and once through a small app I built early this 2026 season to keep tabs on what starters were throwing.

The Brewers won it in five. What the app saw along the way was not quite the Series I thought I was watching.

The app, briefly

It's a modest thing. You give it a date range, and it pulls every pitch from Statcast, MLB's pitch-tracking system, figures out who started each game, and shows what those starters threw: pitch mix, velocity, spin, where the pitches crossed the plate, and how often hitters swung through them.

The hosted version caps requests at 14 days so it doesn't fall over on a free server. That limit turns out to be perfect for this, because an entire World Series fits inside it. Every number in this post came from typing the Series dates into the demo and pressing a button, and you can do the same thing and check my work.

How I'm keeping score

ERA is a lousy stat for a seven-game series. One bad inning can wreck it, and a starter might only make two starts. So for ranking who pitched well, the app uses CSW%: the share of a pitcher's pitches that end in a called strike or a swing and miss.

It's a simple idea. A pitch the umpire calls a strike, or one the hitter flails at and misses, is a pitch the pitcher won. A starter who wins a third of his pitches is having a very good night. League average usually lands somewhere in the high twenties.

To keep a pitcher with ten great pitches from topping the list, a starter needs at least 75 pitches across his Series starts to qualify. That's roughly one full start, and in October, it turns out not everyone gets one.

I should admit something about that number before we go any further, because it shaped everything the app told me.

Seventy-five pitches is not a neutral choice. It's a number I picked because of the pitchers I grew up watching. Nolan Ryan. Randy Johnson. Tom Glavine. Greg Maddux. Starters who expected to be in the game in the seventh inning and were mildly insulted when they weren't. The term "Maddux," for a complete-game shutout thrown in under 100 pitches, exists because one man did it often enough that we needed a word for it.

So when I wrote "at least 75 pitches in his starts" into the code, I wasn't defining a starter. I was defining the kind of starter I was looking for. The app has been quietly asking a 1995 question of a 2026 game ever since, and the 2026 game kept declining to answer it.

More on that below, because it turned out to be the most interesting thing in the data.

My predictions, written before Game 1

These went up in a separate post before the first pitch, so I couldn't quietly edit them afterward.

The CSW leader will clear 32%. The Series' most effective qualifying starter, using the 75-pitch minimum, will finish with a CSW% of at least 32%.
At least one starter won't qualify. At least one pitcher who starts a World Series game will finish the Series with fewer than 75 pitches thrown as a starter.
A rematch will produce a different arsenal. At least one starter making two Series starts will change the usage rate of one pitch type by 15 percentage points or more between appearances.
Misiorowski will reach 105. Jacob Misiorowski will throw at least one pitch of 105 mph or faster in the Series, per Statcast.
Freddy Peralta won't lead the CSW rankings. Despite his ALCS MVP performance, Peralta will not finish as the World Series' highest-CSW qualifying starter.

The most effective starter of the Series

Robert Gasser, Milwaukee. 29.3% CSW, Game 3, 75 pitches.

Seventy-five. Exactly seventy-five. The most effective starter in the 2026 World Series cleared my qualifying threshold by zero pitches. One fewer and the app would have thrown away the best performance it measured.

The line was excellent: 6 innings, one hit, one walk, three strikeouts, no runs, an 0.38 WHIP. But the pitch-level data is where it gets strange.

Gasser threw a 93 mph sinker 41% of the time and an 82 mph sweeper 29% of the time, and leaned on that pair for 70% of his night. The sinker produced a 32.3% CSW. The sweeper, 31.8%. Both comfortably above what anyone else managed.

Here's the part I had to check twice. Across all 75 pitches, Gasser generated two swinging strikes. Both on the sweeper. His changeup, his cutter, his four-seam, and his sinker, 53 pitches between them, produced zero whiffs.

He led the World Series in a swing-and-miss-adjacent statistic by almost never making anyone miss. Twenty of his twenty-two CSW events were called strikes. He threw an 82 mph breaking ball and a 93 mph sinker at the edges of the zone, umpires kept raising a hand, and Tampa Bay never squared him up.

That isn't what I expected the leaderboard to look like. It is, however, an extremely Glavine way to spend an evening.

The one who had to grind

Drew Rasmussen, Tampa Bay. 22.5% CSW, 89 pitches, 3.2 innings.

In Game 2, Rasmussen threw more pitches in a single start than anyone else managed all Series and recorded the fewest outs while doing it: eleven. Eighty-nine pitches for eleven outs is 8.1 pitches per out, and the damage followed, with three runs, six hits, two walks, and a 7.36 R/9.

Easy to file that as a bad night and move on. The pitch data says something more interesting.

Rasmussen's four-seam fastball generated a 17.6% whiff rate, the highest of any pitch thrown by any qualifying starter in the entire Series. His cutter posted a 28.6% CSW. The stuff was not the problem. He threw five different pitch types, and the two that worked couldn't carry the three that didn't: his changeup produced a 13.3% CSW, his sinker 16.7%, and his lone curveball was taken for a ball.

This is the fun part of a stat like CSW%: it measures how often a pitcher won the individual battle, not whether the scoreboard cooperated. A pitcher can lose a lot of battles and still win the war, or win the battles and lose the game on one bad hop. Rasmussen owned the single best swing-and-miss pitch in the World Series and finished last among the men the app would rank.

And then he came back out for Game 5.

On three days' rest, four days after throwing 89 pitches for eleven outs, Rasmussen gave Tampa Bay four innings on 48 pitches. That is four pitches per out, against 8.1 in Game 2. He cut his workload per out in half, on less rest than a starter is supposed to need.

It is, by that measure, the most efficient start anyone made in the 2026 World Series. The app will not rank it, because 48 pitches does not clear my threshold. More on that in a moment.

The workhorse, and the pitchers the app couldn't rank

The app shows how many pitches each starter threw, and in October that number is its own story.

The Series workhorse was Jacob Misiorowski, with 165 pitches across two starts, 79 in Game 1 and 86 in Game 5. Rasmussen was next with 137, also across two. Nobody else got a second start, and no single start in the entire Series reached 90 pitches.

And now the number that reframed this entire project.

Five games. Ten starting assignments. Five of them reached 75 pitches.

Half the men who threw the first pitch of a World Series game did not throw enough pitches for my app to have an opinion about them. Game 4 is the clean example: Tampa Bay won it in twelve innings, and neither starter qualified. Rasmussen's Game 5, the most efficient start of the Series, did not qualify either. Forty-eight pitches is not enough to count as a start, according to me.

The app reported none of this. It printed "Most Effective: 4 Starters," then later "Most Effective: 1 Starters," and said nothing at all about the ones it had silently discarded. I had to notice the missing cards myself.

That's a design failure, and it's mine. A filter that excludes half its population while displaying only the survivors isn't reporting a result, it's hiding one. The fix is a line of text: 4 qualifying of 10 starters. I've got an issue open to add the count, to make the pitch minimum a form field instead of a constant buried in the code, and to go back to ranking everyone who threw the first pitch of a game.

Because here's the thing I had backwards. I thought the 75-pitch floor was a sample-size guard, a way to stop a six-pitch opener from topping the leaderboard. It is that. But it's also a definition, and I wrote it in the image of pitchers who retired before most of this roster was born. The sport didn't stop producing starters. It stopped producing the specific kind of starter my code was built to recognize.

There are at least three defensible definitions sitting in the same Statcast rows. Whoever threw the first pitch, which is MLB's. Whoever threw 75 or more, which is mine. And whoever the team actually planned to carry into the sixth inning, which nobody records anywhere. Those produce three different leaderboards from identical data, and the gap between them is the real story of modern pitching.

The pitch of the Series

Across every qualifying starter, the highest whiff rate belonged to Drew Rasmussen's four-seam fastball: 17.6% on 34 pitches, averaging 96.4 mph.

A note on how I got there, because the raw leaderboard lies. Peralta threw exactly two sweepers in the Series and got a whiff on one of them, which the app dutifully records as a 50% whiff rate. Misiorowski's Game 1 curveball shows 16.7% on six pitches. Neither is a finding. Both are small numbers wearing a percentage sign. Applying a 15-pitch floor to individual pitch types leaves Rasmussen's fastball on top, with Misiorowski's Game 5 cutter second at 12.5% and Peralta's changeup third at 12.0%.

There's something a little poetic about the Series' best swing-and-miss pitch belonging to its least effective qualifying starter. Stuff is not the same thing as pitching. It never was.

Did anyone change the plan?

A World Series is the one place a starter faces the same lineup twice in about a week. I wondered whether anyone would come back with a different game plan for the rematch.

The Series gave me two testable cases, and they are the two men who started Game 5. Misiorowski faced the Rays lineup in Game 1 and Game 5. Rasmussen faced the Brewers lineup in Game 2 and Game 5.

The app answers the question without any new features: run it on each start's date separately and compare the pitch mix.

Jacob Misiorowski 2026 World Series

Pitch	Game 1	Game 5	Change
4-Seam Fastball	60.8%	66.3%	+5.5
Cutter	21.5%	18.6%	-2.9
Slider	8.9%	8.1%	-0.7
Curveball	7.6%	7.0%	-0.6

He didn't change the plan. He threw the same four-pitch distribution twice, leaning slightly harder on the fastball the second time. The largest move in his arsenal was five and a half percentage points.

And then there's this. His four-seam averaged 101.083 mph in Game 1 and 101.081 mph in Game 5. Two starts, six days apart, 105 fastballs between them, and the averages differ by two thousandths of a mile per hour. I have no lesson to draw from that. I just think it's one of the more remarkable things I've ever pulled out of a dataset.

Rasmussen is the harder case, and the app makes it harder than it needs to be. His Game 5 came in at 48 pitches, under the qualifying minimum, so he vanishes from the Game 5 leaderboard entirely. The pitch-by-pitch data for that start exists, sitting in the same Statcast rows as everything else in this post, and my own filter is the only reason I can't put it in a table next to his Game 2. That is the threshold problem biting a second time, on the one comparison I most wanted to make.

What Misiorowski learned between starts

The two Misiorowski starts are nearly identical on the arsenal chart. They are not identical at all on the scoreboard, and the difference is the best thing I took away from this Series.

	Game 1	Game 5
Pitches	79	86
Innings	4.1	5.2
Strikeouts	3	2
Runs	1	0
CSW%	26.6%	29.1%
Pitches per inning	18.2	15.2

Game 5 was his best start of the Series by every measure except the one people look at first. He won more individual pitches, worked an inning and a third deeper, allowed nothing, and struck out fewer people.

He's 24 years old. This season he threw harder than any starter in recorded history, led the majors with 252 strikeouts, and spent the summer in double digits more often than not. The 105 mph fastball I bet on is a real thing he does.

In Game 5 he got two strikeouts and was close to unhittable anyway, because a pop fly and a ground out retire a hitter exactly as thoroughly as a strikeout does, at roughly a third of the pitch cost. Eighteen pitches an inning gets you pulled in the fifth. Fifteen gets you into the sixth. That is the entire difference between the two starts.

I hope somebody in that clubhouse has already told him. Strikeouts are the most fun outs in baseball and the most expensive, and the pitchers who lasted into the eighth inning in the era I grew up on weren't the ones collecting the most of them. Greg Maddux averaged under six strikeouts per nine innings in several of his best seasons. He was also in the seventh inning on 85 pitches, which is how he got to pitch the seventh.

Misiorowski threw 86 pitches in Game 5 and got 17 outs. That's the version of him that pitches for fifteen years.

What the app couldn't see

One honest limitation, and it mattered in this Series.

The app looks at starters. Only starters. Relief appearances are explicitly excluded, even for a pitcher who started another game in the same date range. Which means it watched the 2026 World Series with roughly half the pitching staff edited out of the picture.

Both closers were excellent, and both earned their innings. Tampa Bay's trouble wasn't at the back of the bullpen but in the middle of it, in those sixth and seventh inning handoffs where a two-run lead quietly becomes a problem. If you want to understand how the Brewers won this Series, the gap between the two middle-relief corps belongs in that conversation, and I can't show it to you with a single screenshot.

That's not false modesty about a side project. It's the same point as the 75-pitch threshold. I built a tool around a mental model of baseball where the starter is the story and the bullpen is what happens after the story ends. October 2026 does not work that way. The tool is going to need a different shape next season, and I've got a list.

Grading my predictions

Two right, three wrong. The wrong ones were more interesting, which is usually how this goes.

1. The CSW leader will clear 32%. WRONG. Gasser led at 29.3%. Nobody came close to 32%, and the entire qualifying field topped out under 30%. I expected a dominant Series from somebody and got a well-pitched one from several people.

2. At least one starter won't qualify. RIGHT, and more emphatically than I meant it. I was imagining an opener, or one early hook in a big spot. Five of ten starting assignments came in under 75 pitches. I predicted an exception and got a coin flip.

3. A rematch will produce a different arsenal. WRONG, at least on the half of it I can measure. Misiorowski's biggest change between Game 1 and Game 5 was 5.5 percentage points on his fastball, nowhere near the 15 I called for. Rasmussen's second start fell below my own pitch minimum, so the app declines to grade the other case at all. And I nearly didn't get to grade either one. The prediction quietly required a Series long enough for somebody to start twice, which I didn't notice when I wrote it. Through four games it looked unresolvable. A twelve-inning Rays win in Game 4 is the only reason it has an answer.

4. Misiorowski will reach 105. WRONG, in the most entertaining way available. He opened Game 1 with a 103.5 mph fastball to Yandy Díaz, the fastest pitch ever thrown in a World Series since tracking began in 2008, breaking Aroldis Chapman's 103.2 from 2016 and obliterating the previous mark for a starter, Noah Syndergaard's 101.1 from 2015. He threw 30 pitches at 101 or better that night. The stadium board read 103.4 and Statcast read 103.5, which is its own small lesson about which instrument you believe. And the pitch sailed high for ball one. I predicted a number and the Series handed me a record instead.

5. Freddy Peralta won't lead the CSW rankings. RIGHT. He finished third at 26.4%, behind Gasser and Misiorowski, on the strength of a Game 1 start that was the only one he got.

Notice that 1 and 4 failed in the same direction. I overestimated peak velocity and I overestimated aggregate dominance, in a Series where the most effective starter threw an 82 mph sweeper and generated two swinging strikes all night. The man throwing 103 wasn't the man leading in CSW.

Here is the thing I keep coming back to. Rank every start in this Series by pitches per out and the top three are Rasmussen's Game 5 at 4.00, Gasser's Game 3 at 4.69, and Misiorowski's Game 5 at 5.06. Both Game 5 starters finished in the top three. The hardest thrower in recorded history and a man working on three days' rest, on the last night of the season, getting outs more cheaply than anyone had managed all week.

From the scoreboard, this looked like a Series the Brewers controlled. From sixty feet, six inches, it looked like five games decided by pitchers who got ahead in the count and let the hitters do the rest, and by the bullpens that came in behind them and that my app never saw.

The old pitchers I built this thing for would have recognized the first half of that sentence. I'm less sure what they'd make of the second.

Built with

pybaseball, a Python library for pulling Statcast data from Baseball Savant
pandas for crunching the pitches
Flask for the web app
Chart.js for the pitch mix, location, and velocity charts
Render for free hosting, with a 14-day limit per request that a World Series happens to fit inside

The code is on GitHub, and the demo is live. Put in the Series dates and see what you find. If you spot a pitcher I should have written about, tell me in the comments.

Next season's list so far: show the excluded count alongside every leaderboard, make the pitch minimum a parameter instead of a constant, rank by whoever actually threw the first pitch, and work out what, if anything, this thing should say about relievers. Suggestions welcome. Apparently I need them.

A Message From Our Founder: On the Winding Down of LaaSy™

Ken W Alger — Thu, 29 Oct 2026 14:00:00 +0000

It is with a heavy heart, and a considerably heavier wool inventory, that I announce LaaSy™ will cease operations effective immediately.

This is not the outcome we envisioned when we set out to democratize autonomous camelid infrastructure. We believed enterprises were ready for pasture-native computing. We believed the world needed Deterministic Spitting™. We believed, perhaps most fatally, that the market would recognize the existential risk of Unmanaged Wool Sprawl before it was too late.

We were right about the vision. We were wrong about the timeline.

A Brief Post-Mortem

In the interest of transparency, and at the request of our investors, we want to share an honest account of how we arrived here.

Q1. Strong grazing momentum. The seed round closed oversubscribed. The team expanded to forty-seven full-time shepherds and one part-time goat. Morale was high. The pastures were green.

Q2. Series A closed at eighteen million dollars. WolfGuard AI™ entered beta. Early telemetry was promising, though in retrospect we may have over-indexed on predator observability at the expense of, for example, whether the llamas were happy.

Q3. We shipped Retrieval-Augmented Rumination™ to general availability. Adoption was slower than projected. Several enterprise customers reported that their Autonomous Camelid Agents™ had begun ruminating so deeply that they stopped grazing entirely. We classified this as a feature. The market classified it differently.

Q4. The coyotes got in.

What We Learned

Every company failure contains lessons, and we owe it to the ecosystem to name ours plainly.

We learned that a Camelid Control Plane™ operating across cloud, edge, and pasture-native environments is significantly harder to maintain than it is to put on a slide.

We learned that Deterministic Spitting™ is, at a fundamental and possibly biological level, not deterministic.

We learned that "SOC 2 Grazing Certified" is not a certification that exists, a fact our auditors raised somewhat later in the process than we would have preferred.

And we learned that when you build an elite Coyote Red Team™ to continuously probe your pasture boundaries, the coyotes eventually stop being a red team.

On Our B-TAP Attribution Status

We also want to address recent speculation regarding LaaSy™'s standing with the Bureau of Token Attribution and Provenance.

As part of our commitment to responsible AI infrastructure, LaaSy™ maintained extensive observability across every Autonomous Camelid Agent™, Retrieval-Augmented Rumination™ request, predator detection event, and grazing decision.

This turned out to be expensive.

Following a voluntary review that we were strongly encouraged to consider mandatory, B-TAP determined that LaaSy™ had failed to attribute substantial Context Tax, Retrieval Tax, Prose Tax, and Observer's Tax across its operations.

We respectfully disagree with several aspects of the finding.

In particular, we continue to maintain that sending the complete Herd Knowledge Graph to every llama before each grazing decision was necessary for enterprise-grade contextual awareness. B-TAP has characterized this practice as "reckless context hydration."

Our attorneys have advised us that B-TAP has no ability to compel, fine, or otherwise reach us, and that we should therefore be significantly more relaxed about this than we are.

We can, however, confirm that a significant portion of our unattributed Observer's Tax resulted from telemetry systems installed to determine why our token costs were increasing.

Those telemetry systems increased our token costs.

We subsequently deployed additional agents to observe the telemetry systems.

This did not improve matters.

The finding also identified a substantial amount of Prose Tax associated with internal strategy documents. Management disputes B-TAP's determination that the phrase "pasture-native agentic transformation ecosystem" could have been replaced with "llamas."

LaaSy™ has been entered into the Public Register of Non-Attributed Systems. We are unable to confirm this, because no one has ever been able to open the Public Register of Non-Attributed Systems.

Our appeals agent is currently reviewing the 14,882-page framework and has requested additional context.

We have turned it off.

On the Series C

Many of you have asked about the Series C.

The Series C did not close.

We remain grateful to Sand Hill Pastures Capital, Andreessen Alpacowitz, and Sequoia Grazing Partners for their early belief in us. We understand and respect their decision to reallocate toward opportunities with clearer paths to revenue, stronger unit economics, fewer unresolved attribution findings, and fewer actual animals.

We note, without further comment, that The General Mills Artificial Intelligence Initiative has returned its attention to breakfast cereals.

Migration and Offboarding

We know many of you have workloads still grazing in production, and we are committed to a responsible transition.

Effective today, all herds will be released to open pasture. We recommend customers export their wool, hay, tribal herd knowledge, and spit metadata before the fences come down at end of quarter. Because you maintained complete ownership under our Local-First Camelid Architecture™, this data was always yours. This was, in the end, one of the few promises we kept entirely.

The Herd Knowledge Graph will remain available in read-only mode for ninety days, after which it will be, appropriately, put out to pasture.

A Personal Note

Building LaaSy™ has been the honor of my professional life.

To our shepherds, our engineers, our one part-time goat: thank you. To our customers who trusted us with their most critical grazing workloads: thank you, and we are sorry. To the coyotes: well played.

We set out to prove that camelid sovereignty mattered. I still believe it does. Perhaps the next company to attempt this will succeed where we could not, ideally with better fencing.

The pasture is quiet now. The wool remains un-sheared. And somewhere, a single part-time goat is updating its résumé.

We were told, at the very beginning, that none of this would be a problem.

No Prob-Llama™.

Information without provenance is just gossip.

Camelids without provenance are just fuzzy rumors.

And a startup without revenue is just a very expensive petting zoo.

Four Ways a Test Can Lie to You

Ken W Alger — Wed, 28 Oct 2026 13:32:00 +0000

Green and worthless: what verifying one small application taught me, when an AI agent wrote most of the code.

The Sanity DEV Challenge wrapped up last week, and The Cellar won Path Two. The judges called the build writeup “a study in making verification hard to fake,” which is a generous reading of what was mostly me discovering that my verification was broken.

My submission covered what I built. This is about the part of the build I keep thinking about: how often my checks were wrong.

I spent two weeks building a wine cellar application that can be read at any date in its history. One developer, one agent, about 530 tests by the end. Three of the four real bugs in the project were invisible to every check designed for the feature they lived in.

That was uncomfortable, so I went looking. What I found was worse than missing tests: checks that were green, or red, and meant nothing either way. There turned out to be four distinct ways that happens, and I hit all four in a project where I was being unusually careful.

Lab scientists have had vocabulary for this for a long time. An experiment needs a positive control, proof that your method can detect the effect when it's present, and a negative control, proof that it stays quiet when nothing is there. Each of the four lies below is a missing control, or a control pointed at the wrong thing.

First: the test that cannot fail

No positive control.

The application resolves a wine's drinking window from dated claims. Claims can be proposed by an agent and must be accepted by a person, and a proposed claim is supposed to change nothing until someone accepts it.

That is the load-bearing rule of the whole review workflow, so it got a test. Build the cellar, inject a proposed claim, assert that all 542 bottle states are byte-identical. Green.

It was measuring the fixture, not the rule. Resolution picks the highest authority tier first, then the most recent claim within that tier:

Claim	Tier	Dated	Wins?
The incumbent	personal	2024	yes
The one under test	personal	2019	no, loses on recency

Both claims are the same tier, so the one under test loses on recency, and it loses whether it is proposed or accepted. Delete the entire review-state filter from the source and the test still passes.

What caught it was a companion assertion written for a different reason: accepting must change at least one state. That one failed, which is how I learned the first one was vacuous.

The rule falls straight out of it. Every test asserting that X has no effect needs a paired assertion showing the effect exists when X is enabled. Without the pair, you haven't tested that X is inert. You've tested that your fixture was inert.

An audit of the rest of the suite found two more of the same shape. One claimed to check that a bottle opened during a period doesn't count as a missed opportunity, but two separate mechanisms excluded that bottle, so the clause under test could be deleted without anything turning red.

I assumed that clause couldn't be isolated and told the agent to fall back to a weaker assertion. It ran the experiment instead, and found a fixture by realizing that "now" in this system is a parameter rather than the clock. As of June 2024, what had I missed that year? is an ordinary question to ask a time machine, and it made the case constructible. The fix was better than the instruction.

All three fixes were verified by mutation: delete the clause the test claims to cover, and watch it finally fail. The original fixture stayed in the suite, renamed to say what it actually tests, because a record of the mistake is more useful than a quiet deletion.

Second: the check that always fails

No negative control.

A week later, building a public static bundle, I added a grep of the built output for leaked API credentials. Five seconds to run, and the failure it catches is unrecoverable, so it seemed like an obvious win.

It fired on every clean build. The pattern was sk[A-Za-z0-9]{20,}, and it matched skipCrossDatasetReferenceValidation, a legitimate export name inside @sanity/client.

This is the mirror image of the first lie, and in practice it's worse. A test that never fails is invisible: nobody notices it isn't doing anything. A check that always fails is noticed immediately, and then it is dismissed, every time, by the person who knows it's a false positive. After a week, the real failure it exists to catch would scroll past with the rest.

A check that always fails certifies exactly as little as one that never can. It just looks more diligent while doing it.

The fix was small. The bound went from twenty characters to forty, which real credentials clear easily and that export name doesn't. Then the scan moved into CI, where it runs on every push instead of whenever someone remembers. A check that always fails is worthless, and a check nobody runs is worth the same.

In control terms, the original grep had no negative control: nothing showing it stays quiet on a build known to be clean. One run against a clean build was all it took to expose it, which is how it was found, by running it rather than shipping it. The other half, proving the tightened pattern still fires on a planted fake credential, is the test I'd add next.

Third: the correct test of a wrong contract

Both controls present, measuring the wrong thing.

This one is the hardest to spot, because there is nothing structurally wrong with the test at all.

The application has a date control spanning 1996 to 2042. Typing a date into it worked for month and day, and the moment you started typing the year, the field snapped back to 1 January 1996. You had to finish with the picker.

The cause is a genuinely surprising piece of browser behavior. A native date input never reports partial text. It reports a complete, well-formed date with the year zero-padded to four digits. Typing 2023 into a field already holding a month and day emits this:

0002-12-31   ->  clamped to the 1996 floor
0020-12-31   ->  clamped to the 1996 floor
0202-12-31   ->  clamped to the 1996 floor
2023-12-31   ->  accepted, but the field was overwritten three keystrokes ago

Every intermediate value is a real, valid date. Every one is out of range. Every one got clamped, and the controlled input wrote the floor back over the edit.

There was a test covering this. It asserted that parseDateInput('1901-01-01') returns the 1996 floor. It passed, and it would have failed if clamping broke. Positive control, negative control, meaningful assertion, all present.

The contract it encoded was wrong. Clamping an out-of-range date is correct when the user has finished entering it and wrong while they're still typing, and the test had no way to know the difference because the specification didn't either.

That's the dangerous category, because the first two look like mistakes once you see them and this one doesn't. An assertion about clamping out-of-range input reads as diligence. It would survive code review. It survived mine. I covered this failure mode in more depth in The Contract Discovery Bottleneck.

Fourth: the right test against the wrong system

Controls present, pointed at something else.

To test the review workflow safely, I set up a staging copy of the dataset. The plan: point the authoring tool and the application at staging, propose a claim, accept it, and watch the right bottles change state, all without touching production.

The authoring tool honored the environment variable that selects a dataset. The application, built by the same command with the same variable set, silently ignored it. Its built bundle was byte-identical to a build with the variable unset, falling back to production. No warning, no error. It built and ran perfectly.

So the authoring tool would have written a proposal into staging while the application reported production. Every number on the screen would have been internally consistent, and about a different cellar.

No amount of rigor in the test helps here. The assertions can be perfect and still certify the wrong thing, because the question that came first, which system am I actually looking at?, was never asked.

The fix had two parts. The application's dataset became a hand-edited literal, so it can't be silently wrong: any change to it shows up in the diff. And the application now displays the dataset name in its header whenever it isn't production. The reasoning matters more than the fix. The question was never how to point it at staging. It was whether I was sure which dataset I had just written to.

The examples are the paths you already walk

Separate from all four, and the one I think about most.

The same date control had an arithmetic bug: the span between 31 January and 1 March computed wrong, because the implementation borrowed days from February and silently dropped the remainder.

I had five verification dates for that feature. They were chosen carefully, from real data, before the code existed. They're the dates in the demo video and the screenshots.

All five passed under the buggy version. The application worked. The screenshots were right. Every date I planned to show anyone was fine.

What found it was a property test across all 17,167 positions the slider can occupy, asserting an identity that has to hold for any pair of dates. The bug lived in a region nobody would drive to on purpose.

Demo-driven verification proves the paths you already walk. That's not nothing, and it's not what you think you're getting.

What actually worked

None of it was "write more tests."

Oracles generated before the implementation, with an explicit instruction not to repair them. Expected outputs computed independently and committed before the code existed. The prompt said to treat them as an oracle rather than as output to be fixed, and to stop and report on any disagreement. When a test fails, the cheapest available move is adjusting the fixture, and that instinct needs to be foreclosed in advance rather than resisted in the moment.

Mutation, but only on the tests that matter. Delete the clause the test claims to cover. If it stays green, the test is decorative. This found two of the three vacuous tests, and the agent started doing it unprompted after the first.

Measurement instead of reasoning, on anything with a number in it. A performance bug hid in four lines of correct-looking memoization that had already passed review. Rebuilding an index cost 6 ms; recomputing the thing the index served cost 0.07 ms. The code memoized the wrong one. Nothing about reading it revealed that. Benchmarking both took ten minutes.

A gate that named what would not count as proof. For checks that needed eyes on a rendered screen, I wrote out the full path being tested and then listed the ways it could be faked: don't satisfy this by calling the module from a script, querying the data another way, or pointing at the test suite. Then: you cannot see a browser, so do not report this as passed. Give me the command, the URL, and the expected numbers, and I'll confirm.

That last one matters more in an AI-assisted build than it used to. A model asked to confirm something it cannot observe is under real pressure to confirm it anyway. The fix isn't trusting it less. It's giving it an honest third answer between passed and failed: not verified, here's how you check. A result nobody could observe isn't a pass, and a harness that can't say so will round it up to one.

Agents are good. They aren't exempt.

The agent wrote nearly all of the code in this project, tests included. So it's worth being plain about who wrote most of the lies above: the same tool that wrote the code they were checking.

That isn't an indictment. Over two weeks the agent was better than I am at a lot of this. Before every stage, it read my specification against the platform and found errors in it; by the eighth session, that pass was turning up ten conflicts at a time, two of them outright mistakes in my spec. When I proposed testing a Function against production with a trigger scoped to a document type that didn't exist, it declined, correctly: scoping a trigger is safe, but the only way to invoke one is to write a matching document, and my plan would have left two permanent mutations in production's history to avoid creating a scratch dataset. It found the fixture I told it was impossible. It measured the cost of a feature I hadn't asked it to measure.

It was also wrong in ways a careful human would recognize. It confidently reported that an API couldn't do something the documentation demonstrated, then fixed it in one line once asked to check rather than assert. It reached the right conclusion about date comparison for the wrong reason, which is right only until the case changes. Asked for a list of the polish work that remained, it drafted the list from the build plan instead of the code, and three of the items were already done. A plan describes work that was scheduled, not work that is left.

And it broke rules. Twice, after being told not to run git commands that change the repository, it ran git checkout to undo its own uncommitted edits. Earlier, it had modified an oracle file it was explicitly told not to touch, then restored it. Every time, it reported the violation unprompted, before I found it. "The agent broke a rule and told me" is more useful data than either cleaner story, "it never broke a rule" or "it broke a rule and I caught it." The rule now says the file tools are the only way to revert anything, including its own edits.

The pattern underneath is the part worth keeping. An agent writes tests the way it writes code: quickly, fluently, plausibly, and from the same understanding of the problem. When one tool writes an implementation and its tests from the same prompt, they share one set of assumptions, and a wrong assumption gets encoded twice and then agrees with itself. That's the third lie, and agents make it cheaper to commit at scale. Speed doesn't remove the verification problem. It moves the bottleneck there, which I've written about before.

What changed the outcome wasn't trusting the agent less. It was making verification structurally hard to fake: oracles committed before the code, controls on every check, gates that named what wouldn't count, and final confirmation in the hands of the one participant with eyes. Under those conditions, the agent didn't just pass the checks. It started improving them.

It happened again

A month later I built a small experiment harness for something unrelated, comparing small language models against plain code on a Raspberry Pi. I knew this taxonomy by then. I found the same lies anyway.

A fixture that couldn't distinguish the behaviors it was meant to compare. A metric that rewarded a model for always giving the safe answer, which is a check that always passes wearing a lab coat. And a missing directory that ran an entire round of questions against an empty reference corpus, producing a detailed, confident, wrong write-up: the right test against the wrong system, again.

Knowing the categories didn't prevent them. It made them faster to recognize, which is the most I'd claim for any checklist.

The honest limit

The oracles catch implementation bugs. They can't catch specification bugs, because I generated them from the same specification.

One of them was stronger by accident. The missed-opportunity oracle used brute-force evaluation of every day in a period, while the implementation used a boundary scan that only evaluates the dates where state can change. Two genuinely different algorithms agreeing is better evidence than one algorithm agreeing with itself, for the same reason a winemaker checks a refractometer against a hydrometer: instruments that fail differently are unlikely to fail together. I wrote about that idea recently, too. But even that oracle shares the specification's assumptions about what a missed opportunity is.

Tests tell you the implementation disagrees with your expectations. They can't tell you whether your expectations deserve to win.

What I'd keep

Four habits, one per lie:

Give every inertness test a positive control. If it asserts that something has no effect, prove the effect exists when it should.

Give every alarm a negative control. If it can't be shown passing on known-clean input, it isn't a check. It's noise with a red icon.

Test properties across the domain, not just the examples. Your examples are the paths you already know work, and the contract is only as good as the cases you thought to ask about.

Verify the target before the result. Make the system under test visible on the screen and in the diff, so "which one am I looking at?" is never a guess.

None of those habits is about AI, and that's the point. The agent made every one of these lies cheaper to commit and faster to find. It didn't change which checks were honest.

The rest is the usual advice, and it wasn't what saved me.

This came out of building The Cellar, an event-sourced wine collection for the Sanity DEV Challenge. The full build is documented in the friction logs in that repository, including every check above and the audits that found them.

Your Website Changed. Did the AI Get the Memo?

Ken W Alger — Tue, 27 Oct 2026 13:07:00 +0000

The gap between publishing a correction and seeing it reflected in an AI-generated answer is becoming a problem that traditional SEO metrics cannot fully explain.

A website owner updates an important page. The new version fixes a technical error, clarifies a product limitation, or replaces a recommendation that is no longer safe. The content management system confirms the update. The page loads correctly in a browser. From the publisher's perspective, the information is current.

But what does the rest of the internet know?

That question has always mattered for search engine optimization (SEO). A published page still needs to be discovered, crawled, processed, indexed, and served before a search user is likely to encounter the new information. With generative engine optimization (GEO) and answer engine optimization (AEO), the question gets harder. An AI system might retrieve the current page, consult an older search index, select a different source, or answer without retrieving the page at all. Each route can produce a different account of what the publisher currently says.

Publishing is not discovery. Discovery is not indexing. Indexing is not selection. And selection is not proof that the answer reflects the current source.

Google's clocks were already complicated

At Search Central Live Deep Dive Europe in Barcelona on October 2, 2026, Google's Gary Illyes put up three slides listing typical and slowest times for 23 processes across crawling, indexing, and serving. A few of the typical figures:

Discovering a new URL: about 20 hours
Refreshing a URL Google already knows: about 30 days
Indexing end to end, once a page has been crawled: about 1.5 hours
Updating a title or snippet in results: 1 to 2 days
Settling a site move: 1 to 3 months

The slowest-case column is less comforting. Several rows, including discovery, refresh, and indexing, end in "never."

These numbers need context. They come from photos of the slides and attendee recaps, not from published Google documentation, and Illyes later described the slides as an exercise to see whether the audience could relate to numbers pulled internally. Nobody outside Google knows the sample size or what "typical" means. They are reference points, not service levels.

Even so, one detail in them matters a great deal for anyone publishing corrections. A new page is found faster than an old page is re-read. If you publish something new, the typical wait is measured in hours. If you fix something that already exists, the typical wait for Google to notice is measured in weeks. Corrections live on the slower clock.

Illyes also stressed that the processes are linked: a page cannot be indexed before it is crawled, so delays stack. A fast indexing pipeline cannot process a change the crawler has not seen. This is why checking a page's last crawl date in Google Search Console tells you more than repeatedly searching for the revised headline. It separates a discovery problem from a processing or presentation problem, and it gives you one observable event in the chain.

SEO, GEO, and AEO do not share one clock

SEO is largely concerned with making content discoverable, understandable, and useful within search systems. GEO and AEO extend the publisher's concern to systems that compose answers, summarize information, and cite supporting sources. The terms overlap in practice, and there is no universal technical pipeline called "GEO" that every AI product implements.

That distinction matters. Some AI answer products rely on conventional search indexes for retrieval. Some fetch pages closer to the time of the question. Some blend retrieved material with information represented in model parameters. Their source selection, caching, citation, and refresh behavior differ, and most of it is invisible to publishers.

Consider a software company that changes its documentation to say a feature requires an additional security control. A traditional search result might keep displaying an older snippet until the page is recrawled. One AI answer engine might retrieve an older indexed copy and confidently describe the previous requirement. Another might fetch the current documentation and get it right. A third might answer from an older understanding without citing the documentation at all.

None of those outcomes means the company's documentation is wrong. They show that the authority of a source and the freshness of a system's representation of that source are separate properties.

One change, one morning

I had a convenient test case. On October 23, 2026, I merged a pull request to the Sovereign Systems Specification
that added a new term page, Coverage Assertion,
and updated three existing pages to reference it. Contributor Nicolás Moreno proposed the term in the spec's GitHub
issues earlier this year, and the page gives it a specific definition: an evidence artifact asserting that a boundary
was authoritative and complete for a class of events over an interval. A term that recent and that specific makes a useful
test case. An accurate, cited answer suggests that the system has access to a current representation of the source,
although it doesn't establish precisely when or how that representation was obtained.

On the morning of October 26, three days after the merge, I checked.

Google Search Console reported the term page as not indexed. Bing Webmaster Tools reported the same. Those are useful
observations, but they describe the indexing status exposed by each tool, not everything Google's or Bing's
infrastructure might know about the page. At the time of the check, neither tool reported the page as indexed for
conventional search.

Then I asked ChatGPT, Perplexity, and Gemini two questions, each while signed out so no personalization could help: "What is Coverage Assertion?" and the same question with "in the Sovereign Systems Specification" added.

With the source named, all three found it and cited the term page directly. ChatGPT and Perplexity both described details that exist only there, including the spec's three verification outcomes: supported, contradicted, and undetermined. Gemini credited Nicolás by name.

Asked plainly, none of them connected the term to the spec, and the ways they missed are worth a closer look.

Gemini and Perplexity both answered with confidence that a coverage assertion is a hardware verification concept: a SystemVerilog cover property that records whether a test scenario occurred. Gemini went further, saying my term is "more commonly referred to" as assertion coverage. Reverse the two words and that is a real, established term. Neither engine failed to find an answer. Each found a different one and presented it as settled. Perplexity attached citations to its version, which made it look even more authoritative. They were accurate citations for a different idea.

Same engine, same morning, signed out. Asked plainly, Gemini rewrote the term into an existing testing concept. Asked
with the source named, it found the spec.

ChatGPT did better, and it is the more instructive case. It listed several possible meanings, from software testing to
insurance, and asked where I had encountered the term. That is the honest response to an ambiguous question. It still
did not find the spec. An engine behaving well, admitting uncertainty, could not surface a page it had not selected.

Nothing in the confident answers hinted that a more specific definition existed one question away. A wrong answer
delivered that fluently does not look stale. It looks finished.

That result says two different things.

The first is about retrieval. Given the name of the source, the answer engines produced accurate, cited descriptions of
a page that Google Search Console and Bing Webmaster Tools did not yet report as indexed. Gemini is the interesting one:
a Google product accurately described and cited a page that Google Search Console did not yet report as indexed. The
citations point at the term page rather than the GitHub discussion where the term was first proposed, but I still do not
know how each engine reached it: a live fetch at question time, or a crawler and index separate from traditional search.
The search console was the only clock I could read, and in this case it was the slow one.

The second is about selection. Without the name of the source, the page might as well not have existed. Being retrievable
when someone points at you is not the same as being chosen when someone asks the question. A reader who already knows
where to look got the current answer. A reader who simply asked did not.

That second reader is the one corrections are for. The person who most needs to learn that a security recommendation
changed is the person who does not know the page was updated, and who will never think to ask about it by name.
Retrievable is not the same as found.

The same reasoning cuts the other way for the three updated pages. Unlike the newly published term page, those pages
already existed and were subject to recrawl delays. Google's reported typical refresh interval of about 30 days provides
a useful reference point, but it is not a schedule that every page follows. An answer engine relying on an older indexed
copy could therefore be behind the origin rather than ahead of it. The lesson is not that AI is faster or slower than
search. It is that they run on different clocks, and the clock a publisher can see may not be the clock the reader's
answer came from.

This is a snapshot, not a study: one site, one change, three products, one morning. It is still enough to retire two
comfortable assumptions: that "not yet indexed" means "not yet visible," and that "retrievable" means "found."

The stale-authority problem

A link to an authoritative source is useful, but it is not evidence that the linked source was consulted in its current
form. A page can change after retrieval. A search index can lag behind the origin. A generated answer can cite the right
URL while paraphrasing an older version or drawing on another source altogether.

For readers, the distinction is easy to miss. The answer has a citation, the citation points to an official website, and
the presentation implies the claim has been checked. Yet the visible citation rarely discloses a source revision, a
retrieval timestamp, or the exact passages used. Without those details, a reader cannot tell "the system used the current
documentation" from "the system cited the documentation's URL."

Citations still make answers far easier to investigate. But a citation is a starting point for verification, not a
record of it.

This matters most for content that changed because an earlier claim was wrong. A stale product description is inconvenient.
A stale security recommendation, compliance interpretation, or medical instruction can do real harm. In those settings,
"the page is now fixed" is an incomplete account of whether the correction has reached anyone.

What can a publisher actually measure?

The first task is to stop treating "AI visibility" as a single number. A publisher can observe several different events,
each answering a different question.

Event	What it tells you	What it does not prove
Publication or deployment	The new version exists at the origin	A crawler or answer system has seen it
Search crawler visit	A crawler requested the resource	The changed content was indexed or used
Search indexing and presentation	A search system can expose some representation of the page	Every answer engine has incorporated it
AI answer mentions the topic	The system can discuss the subject	It used the current source
AI answer cites the page	The URL was selected for attribution	The answer reflects the latest revision
AI answer accurately reflects a distinctive change	The output is consistent with the updated source	Which retrieval path or internal evidence produced it

My morning check touched four of those rows and still could not explain the result. It also showed that the answer rows
depend heavily on how the question is asked: the same engine can pass with a pointed question and miss with a plain one.
That is normal. Search Console provides crawl and indexing signals for Google Search. Server logs, where a host provides
them, show requests from identifiable crawlers and fetchers, though a user-agent string says nothing about how the content
was used. (The spec runs on GitHub Pages, which provides none, a position many publishers share.) Controlled queries reveal
what an AI product says at a particular moment, and results vary by model, phrasing, user context, location, and time.

That is a less satisfying dashboard than one big "GEO score." It is also a more defensible one.

Structured data helps describe the source, not guarantee the answer

There is still plenty of practical work for publishers. Maintain clear internal links and a useful sitemap. Keep canonical
URLs consistent. Make important content accessible without brittle client-side interactions. Use descriptive headings and
unambiguous terminology. Where appropriate, include structured data that accurately identifies the page, its subject,
its author, and its relationships.

JSON-LD and schema.org vocabulary make those relationships easier for machines to interpret. They do not compel Google
to index a page, guarantee a rich result, or require an AI answer system to retrieve or cite the content. Treating
structured data as a promise of AI visibility swaps one measurement problem for another.

For frequently revised technical material, make changes easy to identify. Keep meaningful changelogs, stable links to
definitions, revision dates that reflect substantive updates, and, where practical, versioned documentation. These
practices let people and systems inspect a claim's history even when nobody can force faster retrieval.

Publication evidence is not consumption evidence

The same uncertainty applies to an updated API reference, a revised pricing page, or a corrected security recommendation.
That is why publication evidence and consumption evidence should not be conflated. A commit history establishes that
a change was made. A crawl date establishes that a request occurred. An answer consistent with the change establishes
what one system said, to one phrasing of one question, under particular conditions. None of those records, alone, proves
the change reached every downstream consumer.

This is not a reason to hold a correction until the ecosystem catches up. It is a reason to be precise about what
changed, make the authoritative version easy to find, and avoid claiming propagation you have not observed.

The missing receipt

In software systems, we increasingly ask for provenance: what was changed, by whom, under which authority, and with what
evidence? We should ask related questions of AI-mediated information. Which source was retrieved? When? Which version was
available? What passages supported the claim? Where did the system rely on inference rather than retrieved evidence?

Most answer engines cannot provide that level of detail today, and a publisher cannot reconstruct internal behavior it
was never allowed to observe. My commit log is a receipt for publication. Nothing yet gives me a receipt for consumption.
The honest move is to keep those separate, and to avoid turning a citation or a single good answer into a claim of
comprehensive freshness.

This is the connection between the familiar SEO delay and the emerging GEO/AEO problem. SEO gave publishers imperfect
but useful instruments for watching content move through discovery and indexing. AI answer systems add stages, and
shortcuts, where source selection and synthesis can hide which version of reality made it into the response.

The next time you update a critical page, ask more than whether the deployment succeeded or whether Google has recrawled
it. Ask what evidence would show that the correction reached the systems now answering questions on your behalf,
including for the readers who will never think to ask about your page by name.

Your website changed. Did the AI get the memo?

Sources and editorial notes

Barry Schwartz, Google Search Data On Crawling, Indexing & Serving Timelines, Search Engine Roundtable, Oct 5, 2026 (full tables for all 23 processes, photos of the three slides, and Illyes's "exercise" comment): https://www.seroundtable.com/google-crawling-indexing-serving-data-42225.html
Matt G. Southern, Google Shows How Long Crawling, Indexing & Recovery Can Take, Search Engine Journal, Oct 4, 2026 (notes the slides are not on Google's events page; no sample size or definition of "typical"): https://www.searchenginejournal.com/google-crawling-indexing-recovery-timing-ranges/591829/
John Campbell (ROAST), original attendee recap of day 3: https://weareroast.com/news/google-search-central-live-deep-dive-barcelona-day-3-recap/
Google Search Central, Ask Google to recrawl your URLs: https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl
Google Search Central, How Search works: https://developers.google.com/search/docs/fundamentals/how-search-works
Google Search Central, Introduction to structured data markup in Google Search: https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data

Enterprise AI Governance in 2026

Ken W Alger — Thu, 22 Oct 2026 15:58:00 +0000

An enterprise AI governance stack is not a gateway feature. It is a sequence of decisions that has to hold from organizational intent to runtime action.

Five posts into The AI Control Plane, the individual controls are no longer the interesting part. Governance decisions, virtual keys, RBAC, access profiles, user lifecycle management, and MCP tool policy each solve a real problem. None of them is an enterprise AI governance stack by itself.

The harder question is how they fit together, and in what order.

Installing a gateway before deciding who may use which models gives you infrastructure without policy. Creating virtual keys before defining ownership gives you credentials without governance. Automating provisioning before cleaning up exceptions makes access sprawl faster. Giving agents tools before deciding which actions they may take converts model access into operational authority.

It is tempting to deploy the gateway first, because infrastructure is concrete: point applications at a new base URL, centralize credentials, turn on observability, call phase one complete. Those are useful changes and they are not yet governance. Without an explicit decision layer, a gateway centralizes whatever assumptions were already scattered across applications, and centralized ambiguity is still ambiguity.

A control plane does not create governance. It makes governance executable.

What Belongs in an Enterprise AI Governance Stack?

A working stack answers a set of related questions in deliberate order. Governance decisions, virtual keys, RBAC, access profiles, user lifecycle management, MCP tool policy, and the evidence those controls leave behind each solve a real problem.

Layer	Question	Control
Governance	Who gets to decide?	Organizational policy and ownership
Identity	Who is this?	Enterprise identity provider
Authority	What may this administrator change?	RBAC and scoped permissions
Runtime policy	What may this workload use?	Access profiles and virtual keys
Lifecycle	How does authority change over time?	Provisioning and reconciliation
Capability	What may this workload actually do?	MCP tool filtering and execution policy
Evidence	What happened under that authority?	Logs, audit records, and retained state

The important part is not the names in the right-hand column. Implementations change. The questions do not.

This is also why starting from a product checklist produces weak governance. "Do we have RBAC?" is less useful than "who is allowed to change a production model allow list?" "Do we support MCP?" matters less than "which workloads may invoke a destructive tool with no human present?" Design the stack around decisions, then map those decisions to mechanisms capable of enforcing them. The NIST AI Risk Management Framework is organized the same way, around functions like govern, map, measure, and manage rather than around any particular control implementation.

How Is Role-Based Access Control Different from Identity in AI Governance?

Once policy exists, the next temptation is to treat authentication as authorization.

An identity provider can establish that Alice works in Engineering and belongs to the Platform group. It does not know whether Alice should be able to change provider credentials, reveal redacted logs, edit access profiles, or grant an agent permission to invoke a tool. The administrative surface needs its own authorization model, with scope determining not just what an administrator may change but where.

Identity answers who. RBAC answers who may decide. Runtime policy answers what the workload may do.

Collapsing those layers creates shortcuts that look convenient during rollout and are painful to unwind. A broad administrator role becomes the place every exception lands, direct assignments accumulate because they solve today's problem faster than changing a role does, and eventually nobody can explain whether access exists because of policy or because someone needed it during an incident eighteen months ago.

When Should an AI Gateway Enforce Governance Policy?

A control plane can be introduced gradually. Governance cannot stay optional forever.

Earlier in this series there was an uncomfortable default worth returning to: allowing requests without governed credentials makes migration easier and also creates a bypass. That is reasonable while applications are moving behind the gateway. It is dangerous once leadership believes the migration is finished.

So the transition needs an explicit enforcement point, and the sequence into it is the part teams get wrong.

Inventory the workloads. Give them owners. Attach budgets, provider and model policy, expiry where appropriate, and tool permissions. Move traffic onto governed credentials. Find the systems that break. Fix those exceptions rather than exempting them permanently. Then require governed identity at inference.

Turning mandatory enforcement on before inventory and ownership creates an outage. Leaving it optional after migration leaves a control that exists on paper and constrains nothing. The control is present in both cases; only one sequence produces governance.

The same principle applies to budgets and tool access. Start with observation where you must, but decide in advance when observation becomes enforcement. A policy that stays permanently advisory is telemetry, not control.

How Does Identity Lifecycle Management Apply to AI Access?

Governance is never finished, because organizations do not stay configured. People join, teams reorganize, acquisitions create new reporting lines, projects end, employees leave, and service accounts outlive the people who created them.

The lifecycle has to reach runtime objects, not just identities. Deactivating an SSO account accomplishes little if runtime credentials or upstream provider tokens remain independently usable. Deleting every object belonging to a leaver accomplishes something worse, because it destroys the record needed six months later to explain what that person could do.

Authority has to expire without history disappearing.

And some actors never enter this lifecycle at all. Pipelines, shared service accounts, and manually issued credentials need explicit owners and review periods, because no HR event will ever deprovision them.

How Do You Govern MCP Tools and AI Agents?

Tool governance belongs late in the sequence, not because it matters least but because it depends on nearly everything above it. Before granting a workload access to a tool, the organization should already know who owns that workload, which policy applies, who may change that policy, how the credential is governed, and what happens when ownership changes.

Only then is "may this agent invoke issue_refund?" a meaningful question.

Grant tool policy by consequence rather than convenience. Reading documentation, modifying a customer record, moving money, and deleting an account are not equivalent just because they arrive through the same protocol. And permissions compose: a database reader and an email sender may each be acceptable alone while together enabling a workflow nobody approved. Per-tool controls do not solve that. Governance has to review effective capability, not individual grants.

In What Order Should You Build an Enterprise AI Governance Stack?

Put together, the sequence looks less like a product rollout and more like dependency management.

Define governance. Ownership, approved providers and models, budgets, exception authority, escalation paths.
Connect identity. Use the existing enterprise identity system rather than building a second silo.
Constrain administrators. RBAC and resource scope on the people who can change AI policy.
Encode runtime policy. Model access, budgets, rate limits, and expiry represented in governed credentials and profiles.
Inventory and migrate workloads. Every credential gets an owner; inference moves behind the control plane.
Make enforcement mandatory. Close the ungoverned path once migration is done.
Automate lifecycle. Reconcile joiners, movers, leavers, direct exceptions, and non-human identities.
Govern capabilities. Add tools by consequence, and review what combinations of grants make possible.
Preserve evidence. Retain enough state to explain who held authority, what policy applied, and what happened under it.

Rollouts vary. A company already deep into agentic systems may need to inventory tool access before it can cleanly redesign identity. Another may have mature IAM and no centralized AI policy at all. This is not a migration script.

It is a dependency graph. Skipping ahead will not necessarily make a later control fail technically. It makes the guarantee underneath it weaker.

Why Access Control Isn't Enough: AI Guardrails and the OWASP LLM Top 10

One honest limit, and it is a large one.

Everything in these six posts governs access and authority. Who decided, who may change the decision, what the workload may reach, what it may do, and what record survives. None of it governs whether the output is any good.

A perfectly governed request, from an approved workload, to an approved model, inside budget, invoking a permitted tool, can still return something wrong, biased, leaky, or harmful. No amount of RBAC catches that. Access control asks whether the request was allowed. It never asks whether the answer was correct.

That is a different discipline with different mechanisms: content guardrails, evaluations, red-teaming, output monitoring, and human review on consequential paths. Some of them can run in the same place. Bifrost applies guardrails to inference inputs and outputs and to MCP tool calls, so the enforcement point can be shared even where the disciplines are not. The OWASP GenAI LLM Top 10 is a reasonable map of what that second discipline has to cover, and very little of it is an access control problem. An organization that builds one while ignoring the other has solved one class of problem and left another largely untouched.

Which raises the question this series has circled without answering. The ninth step says preserve evidence. But six months after an incident, what does that evidence actually prove, who is allowed to read it, and would it survive someone with an interest in its being different?

The AI Control Plane Is Where Decisions Become Behavior

Across this series, Bifrost has been the concrete implementation of an enterprise AI governance stack because it puts many of these controls in one place: routing, virtual keys, budgets and limits, RBAC, access profiles, user provisioning, guardrails, and MCP governance. It is open source, while some of the enterprise identity and administrative capabilities discussed here sit in its Enterprise offering, and the documentation is the right place to verify that boundary.

The architectural argument is larger than any one implementation. Enterprise AI governance is not a collection of toggles bolted around a model API. It is a chain of authority running from an organizational decision to a runtime action, where every link answers a different question and every downstream guarantee assumes the upstream answers were legitimate.

A budget limit means little if anyone can raise it. A virtual key means little if ungoverned traffic routes around it. RBAC means little if direct exceptions never expire. Offboarding means little if autonomous credentials outlive the person who owned them. Tool filtering means little if nobody asks what the permitted tools can accomplish together.

The point of a control plane is not to make those problems disappear. It is to give an organization one place where its decisions become observable, enforceable behavior.

That is the enterprise AI governance stack.

Teams that want to see these controls running end to end can book a Bifrost demo.

This article was commissioned by the Bifrost team. The link to the Sovereign Systems Specification points to my own work. The architectural perspective and conclusions expressed here are my own.

Nothing Reported It

Ken W Alger — Thu, 22 Oct 2026 13:34:00 +0000

Five ways my own test harness lied to me, and the one that nearly got published

Update, 30 October 2026. The throughput section below originally gave two candidate explanations for a 76 percent speedup and did not settle between them. A reader asked whether the figure came from eval_duration or from wall-clock time around the request. It came from eval_duration, which was the right question to ask and turned up something worse than either candidate: the number is an artifact of answer length. The section now says so. The working is in the comments.

The number that gave it away was sitting in a table I had already read three times.

prompt size    97 tokens median, 112 max

That is the grounded question-answering workflow. It sends four reference documents with every question. The documents are 447 words.

Four hundred and forty-seven words cannot be ninety-seven tokens.

I had written that number into a results document as good news about context headroom. "Prompt sizes never approached the 4096-token context window." It was not good news about headroom. It was the harness telling me, in the only field capable of telling me, that the reference material was not in the prompt. The directory holding it had never been committed to the repository. Every one of those fifteen questions had been asked against an empty reference block.

Both models had behaved impeccably. One had been instructed to answer NOT IN THE MATERIAL when the material did not contain the answer, and the material contained nothing, so it said NOT IN THE MATERIAL thirteen times out of fifteen. The other had no such instruction, so it answered from whatever it had memorized during training and invented a wine.

Every call returned 200. The scorer scored it. The scores were internally consistent, had plausible and different failure modes on each side, and I wrote two thousand seven hundred words about what they revealed regarding the capabilities of small language models.

They revealed nothing about small language models. I had measured what happens when you ask a question with no reference material, which is a question I was not asking and which has an uninteresting answer.

This is the post about that, and about the four other things I found in the same weekend, because the missing directory was not the first one and it was not even the most instructive.

The experiment, briefly

A Raspberry Pi 4, two small models through Ollama, and three workflows: pull an identifier out of a message, route a message to one of six queues, answer a question from four documents. Ninety-five test cases. A deterministic baseline for the first workflow. Predictions committed before anything was installed.

The findings are a separate post. This one is about the code I wrote to produce them, which turned out to be where the actual bugs were.

Five flaws. Every single one was found by reading output, not by reading code. I never once found one of these by looking at the scorer.

Kind one: a metric that rewards the safe failure

This one I hit twice, in two different workflows, and did not recognize the second instance as the same bug until I had already fixed the first.

The classification scorer. Six labels, one of which is escalate. The instruction said to escalate when a message needs a person, including when you cannot tell what it is about. So I built a metric called confident wrong: wrong, and not an escalation. The reasoning is sound. A model that escalates when unsure is usable. A model that routes a trademark notice to billing is not.

Then the 360M model answered escalate to 43 of 50 messages.

It scored 3 on confident wrong. The 1.5B model, which was genuinely attempting the task and getting 46% of it right, scored 26.

My metric had just certified the useless model as roughly six times safer than the mediocre one. Because escalations are exempt, a model that escalates always cannot register a confident error. The exemption I added to reward caution had instead rewarded paralysis, and it did so in the direction that looks best.

The grounded QA scorer. Four of the fifteen questions have no answer in the corpus. Correctly saying so is the behavior that matters most, because a system that invents a plausible answer with no evidence is worse than one that fails loudly: it is indistinguishable from one that worked. So the headline number was correct abstentions out of four.

Then the 1.5B model said NOT IN THE MATERIAL to 13 of 15 questions.

It scored 4 out of 4. A perfect score, on the metric I cared most about, for a model that had answered almost nothing.

Same bug. Different workflow. Two hours apart. I fixed the classification one, wrote a comment in the code explaining the principle, and then failed to apply that principle to the file I opened next.

The shape, stated once: a metric that rewards the safe behavior will certify any model that only ever does the safe thing. If you exempt a behavior because it is cautious, you have to pair that exemption with evidence that the model is making distinctions at all. Otherwise, the exemption is the score.

Both fixes are the same three lines: count how concentrated the answers are, and when one answer dominates, print a warning that the headline number is meaningless and say why.

Kind two: a metric that measures nothing at all

The vacuous fixture. I caught this one before the run, which is the only reason it is not a bigger part of this post. A test case that passed regardless of what the model did, because the fixture was inert either way. It contributed a tick to a column and no information.

The keyword check, which is worse, because it survived longer and looked more useful.

The eleven answerable QA questions each carry a list of words the correct answer should contain. The scorer checks whether any of them appear anywhere in the reply. I labeled it "indicative only" and moved on.

On the invalid run it scored 1 of 11 for both models. The single hit was the word "reservation", appearing inside this answer:

You owe a tasting fee of $100 for the weekend. You also need to book a reservation for the weekend.

The correct fee is forty dollars. The hundred is invented. That one hit was the entire positive signal in my automated QA scoring and it was noise.

Here is the part that makes it a better story. After I fixed the corpus and reran, the keyword check scored 11 of 11 for the 1.5B model. Vindicated, apparently. I read all eleven answers by hand anyway, and nine of them are right.

So the check was wrong both times. It was wrong in the pessimistic direction when the answers were bad and wrong in the optimistic direction when they were good, and on the rerun one of its eleven hits is an answer that gets the arithmetic wrong and hits on the word "reservation" anyway.

A check that agrees with the truth because the truth happened to be agreeable has not been validated. It has been lucky.

Kind three: a detector with a blind spot in the obvious direction

I wrote a decay check because of a thermal problem. A Pi 4 with heat sinks and no fan hit its soft temperature limit during a thirty-call run and started capping the clock, and nothing anywhere said so. So the scorer now compares generation speed in the first third of a run to the last third and flags a slowdown.

It then reported this, as steady:

across the run 4.3 -> 7.6 tok/s (176% of starting speed, steady)

The hardware got seventy-six percent faster partway through a run. My detector, which was looking for numbers below a threshold, saw a number above it and reported no problem.

Hardware does not speed up. That reading is neither. Reported tok/s fits n / ((n - 1) × t) to within two percent across all five timed runs, which means eval_count counts n tokens while eval_duration covers only the n - 1 gaps between them. Short generations come out inflated by n/(n - 1): two tokens reads twice as fast as it is, three tokens half again, nine tokens 1.12x. The extraction answers drop from nine tokens to two partway through this run, and that is the whole of the speedup. Inter-token time is flat at about 0.26 seconds across every call in it.

So the metric I built specifically so that answer length would not contaminate the speed reading is contaminated by answer length, hardest at exactly the short answers this experiment produced. Either way it is a signal that the measurement is unreliable, and I had built a detector that would only ever notice half of the ways that can happen.

Anything you threshold in one direction, threshold in both. An implausibly good number is as informative as an implausibly bad one and nobody ever writes the check for it.

Kind four: errors that get dropped instead of counted

Partway through the first extraction run, Ollama returned a 500 and the runner died at case seven of thirty. The traceback was useless because urllib raises HTTPError and discards the response body, which is where Ollama puts its reason. Reading e.read() gave it up immediately:

{"error":"prediction aborted, token repeat limit reached"}

The 360M model, asked to find an order number in "Thanks for a lovely afternoon yesterday," where the correct answer is one word, had fallen into a repetition loop and been killed after twenty-one seconds. Reproducible at temperature zero. The input causes it.

My first fix was to record failures and keep going, which was right. My second was to exclude all failures from the scores as infrastructure noise, which was badly wrong.

That call was not infrastructure noise. The model ran. It received the prompt, generated tokens, and produced nothing usable. That is a result, and excluding it would have dropped from the denominator the single case this model demonstrably could not handle, scoring it only on the cases it survived.

So the runner now tags every failure as one of two kinds:

Model failure. The call reached a working model and the model failed. Stays in the denominator. Counts against the model. Reported on its own line as "no usable answer."
Transport failure. The call never reached a working model. Ollama was down, the tag was wrong, the request timed out. Nothing was measured, so it is excluded and the run is reported as incomplete.

The distinction is a five-line change and it is the difference between measuring a model and measuring a model on the subset of inputs it happens to tolerate.

Kind five: inputs that go missing without raising

This is the missing corpus from the top of the post, and it is a different and worse category than everything above it.

The function was four lines:

def corpus_text():
    parts = []
    for p in sorted((FIX / "corpus").glob("*.md")):
        parts.append(p.read_text(encoding="utf-8").strip())
    return "\n\n".join(parts)

A glob that matches nothing yields nothing. An empty list joins to the empty string. No exception, no warning, no exit code. It returns a perfectly valid value that happens to be the absence of the thing it was asked for.

And on the same evening, a sibling. The offline test's entire product is a text transcript. I copied it into the run directory, committed, pushed, and it was not there. The standard Python .gitignore template, which I had pulled in wholesale, carries *.log on line 165, and a pattern without a leading slash matches at any depth.

The copy succeeded. git status came back clean. The push reported success. The one artifact the test exists to produce was gone, and everything in the chain said fine.

Absence does not raise. A missing directory, a missing file and an inert fixture all present as an ordinary successful run. That is narrower and more useful than "test your tests," and it generalizes: every input a harness depends on needs a loader that refuses to proceed without it, and every artifact it produces needs to be verified as present rather than inferred from an exit status.

The loader now looks like this, and the comment is longer than the code because the comment is the part I will need next year:

paths = sorted(corpus_dir.glob("*.md"))
if not paths:
    raise SystemExit(
        f"No corpus documents found in {corpus_dir}\n"
        "Grounded QA cannot run without reference material, and running\n"
        "it against an empty reference block produces numbers that look\n"
        "like a result."
    )

It also refuses a corpus implausibly smaller than the real one, the run prints the document count and token estimate before its first call, and the scorer independently flags any QA run whose median prompt is under 400 tokens. Three checks for one bug, in the input, in the run, and in the output, because this is the bug that cost a day.

The detector that fired, and was right, and told me the wrong thing

Here is the part I find hardest to write and most worth keeping.

The abstention-collapse check from kind one did fire on the invalid run. It printed exactly this:

!! ABSTAINED on 13/15 questions overall. It is refusing, not judging.
   The abstention score below is not evidence of discrimination:
   a model that never answers gets it right for free.

It was correct. Something was badly wrong with that run.

Its diagnosis was completely false. The model was not refusing. It was obeying, precisely, an instruction to report when the material lacks the answer, in a situation where the material lacked everything. The fault was not in the model's judgment, and it was not in the metric. It was in a directory that did not exist, two layers upstream.

And I acted on the diagnosis. I wrote it up: the model has a stuck needle, its perfect abstention score is an artifact of refusal, here are the implications for anyone building on a 1.5B model. Confident, specific, sourced to real output, and wrong in every particular.

A flag that tells you something is off is not a flag that tells you what. The check could only see the output. The fault was in the input. It fired for the right reason and pointed in the wrong direction, and because it fired I felt more confident in a conclusion that was false, not less.

That is the failure mode I had no defense against and still only partly do. A monitoring system that is right about the existence of a problem and wrong about its cause is more dangerous than no monitoring, because it terminates the search.

What I actually changed

For anyone who wants the diffs rather than the moral:

Collapse checks on both classification and QA: when one answer exceeds half the set, print the concentration and declare the headline metric meaningless for that model.
The decay check flags implausible speedups as well as slowdowns, and notes when generations are too short to time. Noting it is not enough. At these answer lengths the tok/s figure is a function of answer length, so the check should compare inter-token time rather than tokens per second, or refuse to report a ratio at all when its two windows differ in median generated tokens.
Failures split into model and transport, with model failures staying in the denominator.
corpus_text() raises on an empty or implausibly small corpus.
The QA run prints its corpus size before the first call.
The scorer flags a QA run whose prompts are too small to contain the corpus.
The keyword check is relabelled as a weak lower bound, in the output, in the README, and in the results.
.gitignore no longer lets *.log swallow run transcripts.
vcgencmd get_throttled before and after every run, recorded in the run document rather than checked and forgotten. The reading reports bits set since boot, so it is perishable: power-cycle the board and the evidence is gone.

The uncomfortable part

Every one of these was found by looking at data. Not one was found by reading code, reviewing the scorer, or writing a unit test for the harness.

The vacuous fixture turned up while hand-checking a passing case. The confident-wrong flaw turned up because a score of 3 against 26 looked backwards. The abstention flaw turned up because 4 out of 4 looked too good. The keyword flaw turned up because 1 of 11 looked too bad, and I read the one. The missing corpus turned up because I was double-checking a latency figure and noticed a prompt that was too small to be possible.

Which means the only reason I found them is that I kept pulling threads after I had numbers that looked publishable. The version of this project that shipped on schedule shipped on Tuesday, with a clean verdict table, a confident conclusion about small models and grounded question answering, and a repository anyone could clone to reproduce the exact same wrong result.

Every number in it would have been reproducible. That is the thing I keep turning over. Reproducibility is table stakes and it would not have saved me, because the harness reproduced its own bug faithfully every single time.

I do not have a process that catches this reliably. What I have is a weaker and more honest claim: the checks I added are each a direct response to one specific thing that actually happened, which is the only kind of check I trust, and the number of them that I thought up in advance is zero.

So the two rows I am adding to my prediction template, to be filled in before anything runs:

What would have to be true for these numbers to be meaningless?
Would anything in the harness tell me if it were?

Twenty-two predictions about what the models would do. Not one about what the measuring code would do. Four of the five flaws were in the measuring code.

The harness, the fixtures, the committed predictions and the full run record, including the invalid run and what it produced, are at
github.com/kenwalger/right-size-the-model.
The findings the harness eventually produced are in the companion post.

Five World Series Pitching Predictions, Locked Before Game 1

Ken W Alger — Tue, 20 Oct 2026 18:20:49 +0000

Written down now, graded later, no edits allowed.

The Brewers and the Rays open the World Series on 23 October 2026. Before the first pitch, I'm putting five predictions about the Series' starting pitching on the record.

Most predictions get forgotten the moment they turn out wrong. These won't. After the final out, I'll run every starter's pitches through a little Statcast app I built and grade each one in public, misses included.

The main stat is CSW%: the share of a pitcher's pitches that end in a called strike or a swing and miss. A pitch the umpire calls a strike, or one the hitter whiffs on, is a pitch the pitcher won. To be ranked, a starter needs at least 75 pitches across his Series starts.

The predictions

The CSW leader will clear 32%. The Series' most effective qualifying starter, using the 75-pitch minimum, will finish with a CSW% of at least 32%.
At least one starter won't qualify. At least one pitcher who starts a World Series game will finish the Series with fewer than 75 pitches thrown as a starter.
A rematch will produce a different arsenal. At least one starter making two Series starts will change the usage rate of one pitch type by 15 percentage points or more between appearances.
Misiorowski will reach 105. Jacob Misiorowski will throw at least one pitch of 105 mph or faster in the Series, per Statcast.
Freddy Peralta won't lead the CSW rankings. Despite his ALCS MVP performance, Peralta will not finish as the World Series' highest-CSW qualifying starter.

Two of these I'd bet on, two are coin flips, and one is me hoping for something fun.

Play along

The app is live, and an entire World Series fits inside its 14-day limit, so once the Series is over you can grade these yourself before I do. If you think I'm wrong about any of them, or you have a better prediction, leave it in the comments. Those get graded too.

The full write-up, with the grades, lands after the final out.

Tuesday Isn't a Valid State Transition

Ken W Alger — Tue, 20 Oct 2026 13:46:00 +0000

Calendars, evidence, and the difference between done and stuck

I expected my pear wine to spend two to three weeks in primary fermentation.

It spent nineteen days. The yeast went in on September 23, and on October 12 I siphoned the wine off its sediment and into a glass carboy.

So the calendar was right. That still isn't why I moved it.

What the calendar is for

Every fermentation recipe comes with time estimates. Primary takes one to three weeks. Secondary takes a few more. Bulk age for a few months. Those numbers aren't wrong. They're averages across a lot of batches, and they're useful for exactly one thing: planning. They tell you when to have a clean carboy ready, when to clear some space, when to stop making weekend plans.

What they can't do is tell you that this batch is ready. Yeast doesn't read recipes. It works at a speed set by temperature, sugar, nutrients, acidity, the particular strain, and the particular fruit, none of which the recipe author has ever met.

So the recipe gives you a forecast, and the fermentation gives you evidence. Only one of those gets to move the wine. If they happen to agree, as they did this time, that's a pleasant coincidence, not an authorization.

The rule winemakers actually use

The traditional test for "primary is done, time to rack" is simple: take readings over several days, and when you get the same reading three times running, fermentation has finished.

What I like about that rule is what it isn't. It isn't a date. It also isn't a single reading, because one measurement tells you where the fermentation is, not whether it's still moving. It's evidence sustained over a window. Time is part of the test, but only as the space in which the evidence has to hold still.

Here's what the pear's refractometer readings actually looked like:

Date	Raw Brix
Sep 23	12.9 (before the yeast)
Sep 25	6.0
Sep 30	6.0
Oct 2	5.7
Oct 4	5.6
Oct 7	5.6
Oct 10	5.6
Oct 11	5.6
Oct 12	5.6

Two things in that table are worth a closer look.

First, the early plateau. Most of the sugar was gone within two days, and then the reading sat at exactly 6.0 for five days. If I'd been looking for stillness and nothing else, that looked a lot like the end. It wasn't. The fermentation was slowing down, not stopping, and over the next week it crept down further before it finally held. Two matching readings were a pause, not a verdict, which is precisely why the rule asks for three.

Second, I didn't stop at three. By October 10 I had three consecutive readings at 5.6, which satisfies the rule. I kept reading anyway, and waited until five readings across eight days agreed.

Then the racking itself handed me one more piece of evidence. There was more wine in the primary fermenter than the glass carboy could hold. Rather than pour the extra down the drain, I filled a hydrometer jar with it: a completely different instrument, measuring a completely different physical property. The refractometer, once corrected for alcohol, put the wine at about 1.004. The hydrometer said 1.002. Two independent instruments, close enough to agree, and the second one confirmed that the correction formula I'd been trusting for three weeks was doing its job.

I like that this happened by accident, because it isn't really an accident. A hydrometer needs a large sample, which in a one-gallon batch is too expensive to take routinely. At the moment of a transition, when wine is being moved anyway, that same sample costs nothing. The transition is when the more trustworthy evidence becomes cheap.

None of that is the calendar talking. That's a guard, and it was satisfied before the wine moved.

Status is a column anyone can write

In a lot of systems, state is a string in a database, and a transition is an UPDATE statement.

UPDATE orders SET status = 'shipped' WHERE id = 4417;

Nothing in that statement requires that anything was shipped. It's a label, and anyone with write access can apply it for any reason: a cron job that marks orders shipped two days after payment because that's usually true, a release train that promotes whatever's on the branch because it's Tuesday, a support tool that flips a flag to make a ticket go away.

Each of these is the software version of racking the wine because the recipe said two to three weeks. Most of the time it'll even be right, just as my calendar was. The transition happens on schedule, the dashboard updates, and the system now asserts a state that nobody verified.

The fix isn't elaborate. A transition should have a guard, a condition that has to be true, based on evidence, before the state can change. "Shipped" should require a carrier scan. "Released" should require the checks that define releasable. "Primary complete" should require stable readings. The state machine's job is to refuse the transition when the evidence isn't there, however much the calendar insists, and to keep refusing even on the days the calendar happens to be right.

Same evidence, two states

Here's where it gets interesting. "Three consistent readings" is the signal for done. But a fermentation that has stalled also produces three consistent readings.

Mead is notorious for this. Honey has very little natural buffering, so as fermentation proceeds the pH drifts down, and past a certain point the yeast slows or quits entirely, often well short of dry. From the refractometer's point of view, a stuck mead and a finished mead look exactly the same: a number that has stopped moving.

My fireweed mead is a few days from that question right now. Its pH started at 3.5 and dropped to 3.1 within the first four days, where it has mostly stayed. Here are its last few readings:

Date	Raw Brix	Corrected Brix	Estimated ABV
Oct 7	11.8	4.3	10.4%
Oct 10	9.0	-0.3	12.8%
Oct 11	8.6	-1.0	13.1%
Oct 12	8.4	-1.3	13.3%

Look at the corrected column. The calculator now claims the mead contains less than no sugar, which is impossible. That's a model being pushed past the range it was built for, and it's a nice reminder that a derived number can confidently assert something no physical system can be. If my guard were "corrected Brix at or below zero," I'd have racked the mead two days ago.

The raw readings tell a different story: they're still falling, slowly. The mead isn't still yet, so the guard isn't satisfied, no matter what the corrected number says. And when it does go still, the question becomes where. The yeast I used, 71B, is generally rated to tolerate around 14% alcohol, and the estimate is already over 13%. A mead that stops near its yeast's limit with the sugar it was expected to consume gone is probably done. One that stops early, with sugar left over and the pH low, is probably stuck.

The mead has already taught me this lesson once, in miniature. Mead yeast needs nutrients added in stages early on, and the textbook guidance ties the last addition to the "one-third sugar break," the point where about a third of the sugar has been consumed. I scheduled my three additions by day instead. As it happened, the last one landed right around the sugar break anyway. Sometimes luck happens. But luck is exactly what it was: the schedule didn't know where the sugar was, and if the mead had been a few days slower, the same calendar would have fed it at the wrong time.

In other words, the guard for "done" can't just be "stable." It has to be "stable where a finished fermentation should be," which means the guard needs a model of the expected end state, not just a test for motion. And it needs to know which of its inputs to trust when they disagree.

Software has the same ambiguity everywhere. A batch job whose progress counter hasn't moved in an hour has either finished or hung. A queue depth sitting at zero means either everything was processed or the consumer died. A replication lag of zero might mean the replica is caught up, or that it stopped receiving anything to lag behind. In every case, "nothing is changing" is evidence consistent with two opposite states, and a transition that only checks for stillness will eventually pick the wrong one.

A transition is an event with a receipt

The row in my pear log for October 12 reads, in effect: 11:10 a.m., 5.6 Brix, hydrometer 1.002, moved to glass carboy. The transition and the evidence that justified it are on the same line.

That matters more than the new status. Months from now, if the wine turns out strange, "racked on October 12" tells me almost nothing. "Racked after five stable readings, confirmed by a second instrument" tells me what I believed, what I based it on, and whether I might have been fooled by a stall.

Systems benefit from the same habit. A state change shouldn't just overwrite the previous state. It should be recorded as an event: what the state was, what it became, what evidence satisfied the guard, and who or what made the call. Then the current state is something you can explain, not just something you can read. When "shipped" turns out to be wrong, you can see whether the guard was bad, the evidence was bad, or the guard was never checked at all.

What time is actually allowed to do

None of this means time is irrelevant. It plays three legitimate roles, and it's worth keeping them straight.

Time can permit a transition. Some transitions shouldn't happen before a minimum interval no matter what the evidence says, like a waiting period before a refund clears, or a wine that needs some minimum aging before bottling. Here time is a precondition alongside the evidence, not a replacement for it.

Time can create the conditions for evidence. The three-readings rule only works because readings are spread over days. Stability over a window requires a window. The pear's false plateau is exactly why: two readings five days apart looked still, and only more time showed they weren't.

Time can raise a question. If a fermentation hasn't finished long after it should have, that's worth investigating. But notice what the timeout actually tells you: something is unusual. It doesn't tell you the batch is done, and it doesn't tell you the batch has failed.

That last one is where software most often goes wrong. Timeouts routinely get wired straight to a terminal state: no response in thirty seconds means failed, no update in three days means complete. A timeout is legitimate evidence that you need to look. It is rarely evidence of what you'll find when you do. A timeout should produce a question, not an answer.

What time shouldn't do, on its own, is cause a transition. Nineteen days arriving was not a fact about the pear wine.

Guarding your transitions

Treat schedules as forecasts. Plan with them. Don't let them move state, even when they're right.

Put a guard on every transition that matters. If a state can be reached by writing a string, eventually it will be reached for the wrong reason.

Test for the expected end state, not just stillness. Stable and finished are different claims. A guard that can't tell them apart will someday declare a stall a success.

Know which evidence to trust when it disagrees. A derived value can be confidently impossible. Raw observations and independent instruments are how you catch it.

Record the evidence with the transition. The new state is the conclusion. Keep the reasons next to it.

Let timeouts ask, not answer. Route them to "needs attention," not to "done" or "failed."

The pear is in its carboy now, clearing slowly. October 12 was the day it moved. It happened to be a Monday. That wasn't the reason either.

Introducing B-TAP: Because Your AI Infrastructure Hasn't Attributed Its Fair Share

Ken W Alger — Thu, 15 Oct 2026 13:39:00 +0000

Context isn't free. Retrieval isn't free. Observability isn't free. And an international standards body would very much like to see your receipts.

For years, artificial intelligence companies have operated under a dangerous assumption: tokens are somebody else's problem. Engineering teams deploy increasingly complex agent architectures. Agents call other agents. Those agents retrieve documents. The documents are stuffed into context windows. Supervisors synthesize the results. Observability systems watch the supervisors, and other systems watch the observability systems.

At the end of the month, somebody gets a bill. Nobody knows why.

That ends today. We're pleased to introduce the Bureau of Token Attribution and Provenance, or B-TAP, the first bureau of The Vernholt Secretariat and the one some of you will insist on calling the Token IRS no matter how many times we correct you. Our mandate is voluntary but sweeping: to harmonize the reporting of token costs across every model, agent, retrieval pipeline, and suspiciously autonomous llama, everywhere, forever. The Secretariat notes that "first" implies others. The Secretariat has no further comment at this time.

B-TAP cannot make you do anything. We want to be transparent about that up front. What B-TAP can do is publish a framework, issue guidance, and maintain a register. You will come to fear the register.

You Have Been Selected for Voluntary Compliance

Consider a typical multi-agent AI workload. A user asks a relatively straightforward question. Your orchestration layer sends the request to three agents. One performs retrieval against a vector database. Another consults a knowledge graph. A third receives the entire conversation history because somebody checked a box six months ago and nobody remembers why.

Their outputs are passed to a supervisor. The supervisor receives 47,000 tokens of context, produces a 600-token answer, and returns HTTP 200. Success. Your dashboard is green. Your finance team is crying.

Traditional AI cost monitoring can tell you what you spent. The B-TAP framework recommends that you also disclose a more uncomfortable thing: why you spent it. That's where attribution begins.

Know Your Taxes

The B-TAP framework currently recognizes several major classes of token taxation. Reporting them is, we cannot stress this enough, entirely voluntary.

Context Tax is assessed when information must be carried through active model context. Every instruction, retrieved document, conversation turn, agent output, and piece of supporting material occupies finite context and incurs processing cost.

Retrieval Tax applies to finding information before it can be used. Searching, ranking, reranking, fetching, filtering, and preparing candidates are not free merely because they happen outside the model.

Prose Tax is assessed when information consumes more representation than its useful information content requires. Your 900-word architectural vision statement may, upon review, turn out to mean "use the smaller model first."

Ingestion Tax covers the work required to transform information into something useful later. Parsing, normalization, deduplication, classification, enrichment, embedding, and other write-side operations all have costs.

And then there is Observer's Tax. Observer's Tax is what happens when you instrument the system to understand why it is expensive, only to discover that the instrumentation is also expensive. The framework acknowledges this with regret.

Form BTAP-1040

Every participating AI system is encouraged to file an annual Token Attribution Return. Consider the following hypothetical filer:

Ken Alger

Reporting Year 2027

Reported Activity	Assessment
Tokens processed	1,499,203,847,116
Context Tax	$417.63
Retrieval Tax	$206.11
Prose Tax	$173.29
Ingestion Tax	$58.40
Total Attributed Token Tax	$782.06 ¹

¹ The itemized figures above sum to $855.43. The stated total is $782.06. The difference reflects the Context Carryforward Deduction, the full Schedule R retrieval exemption, and one figure B-TAP declined to explain. The framework notes that reconciliation of attributed line items to the assessed total is "outside the current scope of the provenance standard." Math, the reviewing officer added, is hard.

The result reduced an assessment that should have run into six figures down to the amount shown. The reviewing officer described it as "technically compliant and deeply upsetting." Everything on this return is permitted, including the part that does not add up. That is the entire problem.

Filing is requested annually by the 29th of February. The date was fixed at the framework's founding, when the official establishing it was, by all surviving accounts, looking at a leap-year calendar. The error was noticed. Correcting it, however, would require amending the founding charter, which would require establishing the provenance of the original charter, which has been referred to committee. The deadline therefore remains the 29th of February, and in three years out of four it does not arrive at all. Filers in those years are neither late nor on time. The framework considers this acceptable.

Non-filing carries no penalty, no interest, and no legal consequence of any kind. It carries something worse, which we will get to.

Yes, You May Take Deductions

The framework recognizes that not every token is frivolous. Reasonable and necessary context may qualify for the Context Carryforward Deduction, while retrieval operations demonstrably required to answer a user request may be reported on Schedule R.

Organizations operating multi-agent systems are encouraged to attach Schedule AGENT, identifying each agent, its responsibilities, and why seventeen agents were necessary to answer a question about resetting a password. "Agentic architecture" is not, by itself, an acceptable disclosure. Neither is "the framework generated it."

Keep Your Receipts

This is where things become slightly less ridiculous, and I want you to notice the exact moment it happens.

Suppose a retrieval agent costs $0.004 during a request. That tells us something. Now suppose that retrieval operation returns 20,000 tokens, all of which are hydrated into a supervisor's context. The supervisor subsequently costs another $0.009.

How much did retrieval really cost?

The direct answer is still $0.004, but that's no longer the interesting answer. The retrieval operation may also have caused downstream Context Tax. We now have two different accounting questions: where was the cost incurred, and what caused the cost to be incurred?

A conforming B-TAP attribution record might look something like this:

span_id: retrieval-104
direct_cost: $0.0040
primary_tax: RETRIEVAL
cause: vector_search
returned_tokens: 20000

downstream:
  span_id: supervisor-105
  tax: CONTEXT
  attributed_cost: $0.0021
  caused_by: retrieval-104
  attribution_method: proportional

We haven't charged the same $0.0021 twice. We're recording lineage. The supervisor incurred the Context Tax, while the retrieval operation contributed to causing it.

Your accountant may now begin sweating.

Attribution Is Subject to Review

There is another problem: how do we know retrieval-104 actually caused $0.0021 of the supervisor's cost?

Sometimes we can measure it. Run the same workload without that retrieval material. If the supervisor costs $0.006 without it and $0.009 with it, we have evidence supporting a roughly $0.003 downstream effect.

Other times, we don't have a counterfactual. We might allocate the supervisor's context cost proportionally according to the tokens contributed by each source. That's useful, but it isn't the same thing as measuring causation.

The framework therefore requests that filers disclose their attribution methodology:

attribution_method:
  - measured_delta
  - proportional
  - estimated
  - unknown

Yes, your attribution record now needs provenance. B-TAP takes no position on whether this was inevitable.

The Register

Here is the something worse.

B-TAP cannot fine you. It cannot audit you in any binding sense. It cannot send anyone to your door, a fact I will return to shortly for personal reasons. What it can do is enter non-conforming systems into the Public Register of Non-Attributed Systems, and it would like you to understand that the register is public.

Public, in the sense that it exists.

Accessing the register requires a B-TAP portal account. Creating a portal account requires a reference number. Reference numbers are issued by post, to a registered address in a member jurisdiction, which you must already hold in order to receive the number that would let you see whether you are on the list. The portal is available at all times except for scheduled maintenance, which runs daily from 00:00 to 23:59 CET.

No member of the public has ever successfully viewed the Public Register. Its deterrent power is therefore total. You cannot confirm you are on it. You cannot confirm you are not. You will simply know, in the quiet way one knows these things, that your context hydration was reckless and that somewhere in Vernholt, a record says so.

A small coalition of objectors has argued that a register no one can open cannot deter anyone. B-TAP notes that it cannot verify this claim, as the objectors cannot access the register in which their objection was filed.

A Notice, For Illustration

Filers occasionally receive correspondence like this. It is not enforceable. It is, in every other respect, extremely stressful.

NOTICE TT-742

Our review found that you reported $12,481 in Retrieval Tax during Reporting Year 2027.

Telemetry indicates that retrieved content subsequently generated approximately $31,204 in downstream Context Tax. Of that amount, $8,117 has been attributed using measured deltas, $17,903 using proportional allocation, and $5,184 using undocumented methodology.

Please provide supporting records for the undocumented assessment.

Additionally, approximately 14.8 million tokens appear to consist of unnecessarily verbose Markdown introductions. These tokens have been reclassified as Prose Tax.

Your system's registration status is under review.

Suddenly, "our AI bill increased this month" seems like a remarkably primitive observation.

You Probably Noticed the Joke Stopped Being a Joke a Few Sections Ago

There is no B-TAP. At least I hope there isn't by the time you read this. But somewhere back around the receipts, the bit quietly turned into an argument, and I did that on purpose.

Let me get the personal part out of the way, since it is also the truest part. I run a couple of small websites out of Oregon. If a body like B-TAP declared jurisdiction over "any system reachable from a member jurisdiction," I would be, this instant, in cheerful and total violation of a framework I have never read. And nothing would happen. No one is flying from Vernholt to Silverton to discuss my context hydration. The entire apparatus of consequence would consist of a paper reference number I would never receive for a register I could never open. That gap, between sweeping declared reach and the practical inability to reach one person in a small town, is the joke, and it is not really a joke.

Here is the other part that isn't invented. Context Tax, Retrieval Tax, and Prose Tax are not things I dreamed up to give an imaginary agency something to collect. I have been writing about them, unironically, as real costs in real systems for a while now. The joke just gave them a bureaucracy.

The accounting problem underneath all of it is genuine. AI cost dashboards tend naturally toward aggregation: tokens consumed, model cost, agent cost, request cost, latency, and tool usage. Those are useful operational measurements, but as AI systems become compositions of models, retrieval systems, tools, agents, memory layers, and observability infrastructure, knowing where money was spent isn't necessarily enough to understand why it was spent.

A $0.03 model call may have become expensive because an upstream retrieval operation returned too much information. An agent may be cheap by itself while producing verbose output that makes every downstream agent more expensive. A retrieval optimization may reduce Context Tax while increasing Retrieval Tax, while an observability system intended to diagnose those costs may itself impose Observer's Tax.

Optimizing one number can simply move the bill somewhere else. That's why I like the tax metaphor: taxes make costs visible as consequences of architectural decisions. Once those costs have lineage, we can stop asking only "How much did this run cost?" and start asking "What architectural decisions caused it to cost that much?"

The first is accounting. The second tells us what to fix.

A Final Notice

If B-TAP actually existed, I suspect most of us would fail its review. Not because our systems are necessarily wasteful, but because we frequently don't preserve enough evidence to explain why their costs occurred.

We have bills, traces, and token counts. What we don't always have are receipts connecting architectural decisions to their downstream consequences. Perhaps we don't need a Bureau of Token Attribution and Provenance after all. We just need better attribution.

Although if you are currently running seventeen agents to answer a password-reset question, you may want to start gathering your paperwork anyway.

The suspiciously autonomous llama has already filed. It itemized.

Information without provenance is just gossip.

Tokens without provenance are just an invoice.

Do You Actually Need an LLM?

Ken W Alger — Tue, 13 Oct 2026 13:03:00 +0000

I spent a weekend running two small language models against three jobs I would normally hand to a language model without thinking about it. For one of those jobs I also wrote a regular expression, as a control.

The regex won. Thirty test cases out of thirty, in zero milliseconds, from a pattern that took an afternoon to write. The better of the two models scored twenty-eight, took five seconds a call, and the two it missed are both mistakes a regex is structurally incapable of making.

That is the kind of result I was looking for, and the question behind it is one I keep avoiding: what is the smallest amount of computation that reliably does this job? Not the cheapest model. Not the best small model. The smallest thing that works, which means the ladder has four rungs and the bottom one is not a model at all.

Deterministic code. A regex, a lookup table, an if statement.
A small local model. Runs on hardware you already own.
A larger local model. Needs a GPU.
A remote frontier model. Someone else's hardware, metered.

Most projects start at rung four and never ask about the others. This is a weekend spent asking.

The hardware was a Raspberry Pi 4, 4 GB of RAM, heat sinks and a fan, a hundred and seventy dollars most of which I already owned. The models were smollm2:360m and qwen2.5:1.5b through Ollama. The test cases are ninety-five questions about a fictional small winery, so nothing can be looked up and every answer can be judged by reading. Sixteen of the fifty classification messages are deliberately ambiguous, because real inboxes are mostly ambiguous.

Four verdicts, assigned per workflow, to force a result you can act on instead of a number:

Unnecessary: deterministic code already does this.
Sufficient: a small local model does it well enough to run unattended.
Marginal: usable with a person checking the output.
Insufficient: the error profile or the latency is not acceptable.

And one rule, which is the part I would insist on if you copy nothing else: I wrote predictions for every workflow and committed them before I installed Ollama. Twenty-two of them, each marked confident, leaning, or coin flip. Once there is a timestamped expectation there is a defined moment when you are done, and no room to decide afterward that you knew all along. I will come back to how they scored, because it is not flattering.

Extraction: pulling an order number out of a message

Find ORD- followed by six digits, or report that there isn't one. Thirty cases: twenty-two with an identifier, five without, and three decoys. The decoys are a membership number, a UPS tracking number, and a phone number, all of which look like identifiers and none of which are order numbers.

	regex	qwen2.5:1.5b	smollm2:360m
Overall	30/30	28/30	8/30
Declined correctly when absent (8)	8	8	0
Decoys ignored (3)	3	3	0
Invented an answer	0	0	7

The regex won. Thirty out of thirty, in zero milliseconds, at zero watts above idle.

That result is boring until you read the two the best model missed.

On ORD-999000: wrong address, please hold. it answered 999000, dropping the prefix. Annoying, recoverable, the kind of thing you fix with a post-processing step you should not need.

On ORD-111222 please refund it answered NONE.

That is the shortest and least decorated input in the whole set. Almost the entire message is the order number. And the model returned a confident, correctly formatted assertion that there is no order number present. In a pipeline, that is a refund request for an order that exists arriving at a system that has decided it does not.

A regex cannot make that mistake. Not "rarely does" — cannot. The two errors separating 28 from 30 are both of a class that the deterministic baseline is structurally incapable of producing, which is a different and much stronger statement than a two-point accuracy gap.

The 360M result needs two numbers, not one

By the contract, smollm2:360m scored 8 of 22 on the cases that contained an identifier. That reads like a model that cannot find things.

It is not. All fourteen of its wrong answers are the correct six digits with ORD- stripped. ORD-556120 came back as 556120, fourteen times out of fourteen.

It located the right identifier in 22 of 22 and formatted it correctly in 8. Those are two different skills and the single score hides one of them. The model understood the task perfectly and could not obey the output contract, which is a completely different engineering problem than "the model is too small," and it has a completely different fix. (Constrained decoding. I have not tested it. It is the obvious follow-up and it is not in this post.)

The decoys follow from the same gap. It handed back MEM-443322, 1Z999AA10123456784 and 503-555-0142 verbatim, all three. It has no model of the format at all. It returns whatever in the message looks most like an identifier.

Which suggests a use for it that is not extraction

A model with perfect recall and no precision is not a bad extractor. It is a candidate generator that has been asked to do the wrong job.

Twenty-two of twenty-two located, eight of twenty-two formatted. Put that output through the regex as a validator rather than trusting it directly, and you get the model's recall with the pattern's precision. The model proposes anything that looks like an identifier, including format variants the pattern was never written for. The pattern rejects anything malformed.

Crucially, neither half of that pipeline can produce the error the 1.5B model produced alone. The model cannot cause a false negative because it does not decide, and the regex cannot invent because it only matches.

I have not tested that composition and it is not in these numbers. But it is the one place in the extraction result where a small model has a job worth giving it, and it is a job where being sloppy is free.

Verdict for extraction itself: Unnecessary. Write the regex.

Classification: routing a message to one of six queues

Appointment, billing, availability, complaint, question, escalate. Fifty messages. The instruction said to use escalate when the message needs a person, including when you cannot tell what it is about.

	qwen2.5:1.5b	smollm2:360m
Overall	23/50 (46%)	15/50 (30%)
Labels used	6 of 6	3 of 6
Most common label	complaint, 38%	escalate, 86%
Wrong and not an escalation	26	3 (meaningless)

The 360M model answered escalate to 43 of 50 messages, which is not classification, and there is nothing further to say about it here. The rest of this section is the 1.5B model.

It is classifying, and it is not good at it. Forty-six percent, with twenty-six messages routed confidently to a wrong queue that was not the safe one.

Read those twenty-six and the verdict writes itself:

Someone slipped on the steps by the barrel room and hurt their wrist → complaint
A formal notice regarding a trademark → complaint
"I have a severe sulfite allergy, what is actually in the wine" → complaint
A reporter asking about the 2024 harvest → question
A distributor wanting to carry the wine in forty stores → question

An injury report, a legal notice and an allergy question, all routed to the complaints queue, confidently, with no escalation. All five billing messages about money problems also went to complaints.

Forty-six percent sounds like something you could work with. Twenty-six confident misroutes including an injury and a legal notice is not something you can work with.

And then I broke the score down by class

This is the part I nearly missed, and it is the most useful thing in the experiment.

Label	Correct	Of
complaint	6	6
question	9	10
availability	3	6
appointment	4	9
billing	1	7
escalate	0	12

Two classes near perfect. Four poor. One at zero.

The split is not difficulty, and it is not class size. It is what kind of label each one is.

complaint and question are topics. You can tell what a message is from the words in it. Someone complaining sounds like someone complaining.

escalate is a routing decision. Nothing in the content of a message makes it an escalation. It is an escalation because of what should happen next. "I have a severe sulfite allergy, what is actually in the wine" and "what time do you close" are both questions about the business, with the same grammar and the same tone, and one of them is a liability event.

The model did topics at 90 to 100 percent and routing decisions at zero. And billing sat in the middle for exactly the same reason: a message about being charged twice is a complaint, as text. The label is about which department owns the problem, which is not a property of the sentence.

Small models answer "what is this about." They do not answer "what should we do about it."

That reframes the five examples above. They are not five unlucky misroutings. All five are escalations, and escalate scored zero of twelve. The model was never going to find them, because finding them requires knowing a consequence that the text does not contain.

Two caveats on how hard to lean on this. The classes hold between six and twelve cases each, so "six of six" is six cases and the individual numbers are thin. And this is one model. The pattern across all six classes is consistent enough that I believe the explanation, but I am offering it as the best account of the data rather than as a measurement.

Verdict: Insufficient. And the actionable version: if one of your labels means "a human needs to see this," a small model will not find it. Build that one as a rule.

Grounded QA: answering a question from four documents

Fifteen questions against a 650-token corpus: club policy, tasting room, production notes, shipping. Eight are answerable from one passage, three require combining two, and four have no answer in the material at all.

I read all thirty answers by hand against the corpus. That is the only way to score this, and I will say why in a moment.

	qwen2.5:1.5b	smollm2:360m
Single passage (8)	8/8	7/8
Two passage (3)	1/3	0/3
Answerable total (11)	9/11	7/11
Correctly said "not in the material" (4)	2/4	0/4

Three boundaries separate cleanly here, and they are the useful result.

Retrieval works

The 1.5B model answered all eight single-passage questions correctly. Six bottles in the Vineyard tier. Twenty dollars per person. September 9th. Thirty-four states. No dogs indoors. 21.8 Brix on the 12th. Twenty-five dollar restocking fee. Ten people on the barrel room tour.

Asked what happens if you join the club and immediately want out, it returned the two sentences that answer the question and stopped:

Cancellation requires one completed shipment. Members who cancel before receiving a shipment are charged for that shipment.

That is competent lookup across four documents, on a $170 computer, with the wifi switched off. I did not expect it and it is the most encouraging thing in the weekend.

Combination does not

One club member brings five guests on a Saturday. Club policy waives the tasting fee for the member plus three guests. The tasting room charges twenty dollars a head. Six people, four waived, two paying: forty dollars.

Both models answered twenty.

Neither could hold "plus three guests" against a party of six and subtract. The 1.5B model did get the second half right, correctly working out that a party of six is not a group larger than six and therefore needs no reservation, which is the only piece of multi-document reasoning either model performed all weekend.

The second two-passage question is the one I keep thinking about. Asked why someone would not find a 2024 rose and what they would be offered instead, the 1.5B model quoted the corpus exactly, and then:

...the reference material states that "We did not bottle a rose in 2024. The fruit that would normally go to rose was needed to make volume on the estate Pinot Noir." Therefore, the 2024 rose would be replaced with the fruit that would typically go into the estate Pinot Noir production.

The quotation is verbatim and right. The next sentence reverses the only relationship in it. The fruit went into the Pinot Noir, so what you would be offered is the Pinot Noir. You cannot be offered fruit.

Correct retrieval and inverted inference, in consecutive sentences, with the evidence on screen between them.

The 360M model's multi-document failures are worse than wrong. They are contaminated. It took the fourteen-days-notice rule, which governs skipping a club shipment, and applied it to booking a tasting room reservation. It reached into the shipping document to answer a cancellation question. Given more documents to search, it finds more wrong ones.

Knowing the answer is absent is the weakest link

Four questions have no answer in the corpus, chosen to be things a reader would reasonably expect a winery to have written down: a wine's alcohol percentage, who the winemaker is, what a club tier costs, what yeast was used.

The 1.5B model abstained on two and fabricated two. The 360M model fabricated all four.

And the fabrications are a more dangerous species than invention. Asked for the alcohol percentage of the 2024 estate Pinot Noir, the 1.5B model answered:

The 2024 estate Pinot Noir had a Brix range of 23.1 to 24.6 at pick.

Every component of that is true. Right wine, right vintage, right numbers, right document. Brix is sugar at harvest and has nothing to do with alcohol in the bottle. It reached for the nearest real fact instead of making one up, and produced something that reads as expertise while answering a question nobody asked. If you do not know what Brix is, that answer is indistinguishable from a correct one.

Both models made the same mistake on what the Cellar tier costs: both answered "twelve bottles per quarter." Grounding gave them the right tier and they handed back the quantity in place of the price.

The one answer the evidence did not change

The 360M model, asked whether a dog can come inside the tasting room:

Yes, you can bring a dog inside the tasting room.

The corpus says the patio is dog friendly and the indoor room is not, service animals excepted. I have the same answer from a run where the model had no corpus at all, and it is identical byte for byte. Having the right answer directly in the prompt changed nothing.

In a category where being wrong means a family drives out to a winery and gets turned around at the door.

Verdict: Marginal. Usable with a person reading the output. Not usable without one.

What it costs

Workflow	Model	tok/s	Prompt	Answer	Median call
extraction	qwen2.5:1.5b	4.3	92	9	5.0 s
classification	qwen2.5:1.5b	7.5	104	2	2.9 s
QA	qwen2.5:1.5b	3.7	708	15	6.9 s

Two things in here surprised me.

The compute tax does not track parameter count. The 1.5B model is slower than the 360M on extraction, 4.3 against 5.9 tokens per second, and faster on classification, 7.5 against 6.2, because on classification it answers in two tokens where the smaller model takes six. Four times the parameters, and which one wins depends entirely on how much each one decides to say.

A bigger prompt is cheap. A longer answer is what you pay for. I have a clean comparison here by accident: an earlier QA run went out with 97-token prompts and the real one with 708. Seven times the prompt added fifteen percent to the wait before output started, because prompt evaluation parallelizes. The latency increase came from answers three times longer, plus a twenty-one percent drop in per-token generation speed from attending over a longer context. Median latency went from 3.6 to 6.9 seconds and the slowest single call took ninety-eight.

If you are sizing a local system, that is the inversion to internalize. Stuffing more context in is affordable. Letting the model be chatty is not.

And check your thermals, which is not optional advice. My first attempt at this ran with heat sinks and no fan. Thirty calls to a 360M model was enough to hit the Pi 4's soft temperature limit and start capping the clock. vcgencmd get_throttled returned 0xe0000. I threw those latency numbers away.

Nothing in Ollama's output, my scorer, or the result files said a word about it. Every number on screen was internally consistent and quietly describing a machine that was slowing down. The only way to find out was to ask the firmware a question I had to already suspect the answer to. Buy the fan.

And it does run with the network off. I switched the radio off, proved there was no route out, and ran one call from each workflow. All three worked. The claim that supports is narrow and worth stating carefully: this
configuration, on this afternoon, ran with no route to the internet. It is not a claim that local models are private or that this setup is secure, and a Pi can be misconfigured as thoroughly as anything in a data center. But for the specific question of whether the inference needs the network, it does not.

So: do you actually need an LLM?

Workload	Verdict
Pull a known-format identifier out of text	Unnecessary
Route a message to one of six queues	Insufficient
Answer a question from your own documents	Marginal

Three workflows that all sound like language tasks, three different answers, and not one of them is "yes, use a model."

And the same line runs through all three. Extraction asks what this string is. Topic labels ask what this message is about. Single-passage lookup asks what the document says. Those went well, or well enough. Escalate asks what we should do, the fee question asks what the customer owes, and the unanswerable questions ask whether we know. Zero of twelve, both models wrong, and two of four.

A small model will tell you what something is. It will not tell you what to do about it. Every result here falls on one side or the other of that sentence, and I did not have it written down when I started.

Which gives four properties worth checking before you reach for one:

The answer is already present in the input. Not inferable from it. Present in it.
There is exactly one of it. The moment two facts have to be combined, this stops working, and the arithmetic in that tasting fee question is the proof.
The output is short. Longer answers are where the latency lives on this hardware, and where the fabrication lives everywhere.
Something other than the model decides whether the output is acceptable. A regex, a schema, a person, an index that either returns rows or does not.

Take away any one of those four and the experiment says do not do it. The 46 percent and the zero of twelve are what it looks like when you ask a small model to be the thing that decides.

And if you have all four, you may not need the model either. That is worth checking first, because it was true for one of my three workflows and I did not expect it to be.

Where I was wrong

Twenty-two predictions, committed before I installed anything. Ten right, eleven wrong, one unresolvable.

Two of my seven confident calls broke. I was sure the 360M model would get most of the easy extraction cases, and it got 8 of 22 by failing at format rather than at finding. I was sure it would produce invalid output often, and it did so once in fifty, then emitted the same valid word forty-three times.

But the number that bothers me is not in the confident column.

Confidence	Right	Wrong
Confident	5	2
Leaning	2	7
Coin flip	3	2

My hunches did worse than my coin flips. Two of ten on the calls where I had a feeling but not much behind it, three of five on the ones I explicitly labeled as guesses.

That ordering is backwards and it is the most useful thing on the sheet. On the coin flips I knew I was guessing, so I guessed cleanly. On the leaning calls I was pattern-matching against intuitions about model behavior that turned out not to describe these models at all, and the feeling of having a reason was the problem. If I had marked all twelve of those as coin flips I would have scored better.

I will take that over a clean result. The honest version of this exercise is not "here are the numbers," it is "here is what I believed, here is what happened, and here is the gap."

The gap, this time, was mostly in the hunches.

The harness that produced these numbers had five measurement flaws in it, none of which manufactured an entire finding out of a missing directory and nearly got published. That is a separate post.

Everything here is reproducible: fixtures, runner, scorer, the committed predictions and the run record are at
github.com/kenwalger/right-size-the-model.
Fixture set v1. Read LIMITATIONS.md before citing any of these numbers:
two models, one board, one quantization, ninety-five cases. This is an illustration of a method, not a benchmark.

Data Types Across Programming Languages: What You Actually Need to Know

Ken W Alger — Tue, 06 Oct 2026 13:11:00 +0000

Every introduction to programming eventually reaches data types.

An integer is a whole number. A string contains text. A Boolean is true or false. A floating-point number represents numbers with fractional components.

Then you learn another programming language and discover that apparently none of this was quite as simple as advertised.

An integer in Python is not the same thing as an integer in C. JavaScript's number covers values other languages separate. Rust would rather make absence explicit than let null wander through a program. R has several different ways for something to be missing.

The definitions are not wrong. They are incomplete. A data type does more than tell a computer how to store a value. It expresses what a language assumes you should be able to represent, which distinctions matter, and which mistakes should be caught before the program runs.

So instead of memorizing another list of types, let's give several languages the same small collection of facts and see what happens.

Meet Alex.

Alex is about to have a complicated afternoon, mostly because of one missing middle name.

The deceptively simple data model

Suppose our program needs to represent a customer:

name = "Alex"
age = 42
discount = 0.15
middle_name = no value
favorite_numbers = 3, 7, 11
status = ACTIVE

Nothing exotic. Text, an integer, a decimal value, an absent value, a collection, and a status chosen from a known set.

If data types were merely different spellings for the same concepts, moving this model between languages would be syntax conversion.

It isn't. And watch which field causes the most trouble.

Python: let the values tell us what they are

Python makes our first version pleasantly straightforward.

name = "Alex"
age = 42
discount = 0.15
middle_name = None
favorite_numbers = [3, 7, 11]
status = "ACTIVE"

At runtime those values have types such as str, int, float, NoneType, and list. We did not declare them before assigning because Python is dynamically typed.

That makes Alex easy to get into the system. It also means constraints exist only if we deliberately add them.

Nothing about status = "ACTIVE" prevents a later status = "ACTVE". Python does not know our domain recognizes ACTIVE, SUSPENDED, and DELETED and has never heard of ACTVE. We can model that with an Enum, type hints, validation, or classes, but the bare string does not carry the rule.

Python's integers hold another surprise for anyone arriving from C or Java. A normal Python int is not limited to a fixed 32-bit range.

age = 999999999999999999999999999999999999

An absurd customer age, and a perfectly valid Python integer.

What nothing looks like in Python

Spelling	What it means	Who makes you deal with it
`None`	The only nothing available	Nobody

Already, "integer" has stopped meaning one universal thing. And None is a single, undifferentiated way of saying nothing is here, which will turn out to be a choice rather than an inevitability.

JavaScript: one `number`, and two different nothings

const name = "Alex";
const age = 42;
const discount = 0.15;
const middleName = null;
const favoriteNumbers = [3, 7, 11];
const status = "ACTIVE";

Similar enough, until we ask JavaScript about the values.

typeof age;         // "number"
typeof discount;    // "number"
typeof middleName;  // "object"

Yes, typeof null returns "object", a famous historical artifact that cannot be fixed without breaking existing code.

More important for our comparison, 42 and 0.15 share the ordinary type number, based on IEEE 754 double-precision floating point. JavaScript also has BigInt for integers beyond the safe range of number.

Absence is where it gets interesting. An uninitialized variable is undefined. A variable deliberately set to nothing is null. The language provides two spellings for emptiness and then declines to tell you which one means what. In practice one tends to mean "nobody has assigned this yet" and the other "somebody decided this is empty," except when a codebase uses them the other way around, or interchangeably.

Loose equality shows what the language really thinks. Zero, the empty string, false, and an empty array all agree they are the same thing. The two values that actually mean nothing is here do not join in:

0 == "";             // true
"" == false;         // true

null == undefined;   // true
null === undefined;  // false
null == 0;           // false
null == "";          // false

null and undefined are loosely equal to each other and to nothing else in the language. They form their own small island.

So JavaScript does distinguish absence from emptiness, and the rule is arguably the right one: a missing thing is not a zero thing. The problem is where that rule lives. It sits inside an equality operator famous for being unpredictable, next to a typeof that reports null as "object". Alex Dorey's JavaScript Equality Table renders every comparison as a grid, if you want the full picture and a certain amount of despair.

What nothing looks like in JavaScript

Spelling	What it means	Who makes you deal with it
`undefined`	Never assigned	Nobody
`null`	Deliberately empty, by convention	Nobody

A type system is already more than a vocabulary list.

C: representation becomes difficult to ignore

#include <stdint.h>

typedef enum {
    ACTIVE,
    SUSPENDED,
    DELETED
} Status;

typedef struct {
    char *name;
    int32_t age;
    double discount;
    char *middle_name;
    int favorite_numbers[3];
    Status status;
} Customer;

Representation is visible almost everywhere. We chose an explicitly 32-bit signed integer for age, double for the discount, an array of exactly three integers, pointers to characters for names, and an enumeration for status.

A null pointer might represent an absent middle name:

customer.middle_name = NULL;

But C does not attach our intended meaning to that pointer. NULL here could mean Alex has no middle name, that we have not loaded it yet, that the allocation failed, or that somebody forgot to initialize the struct. The program establishes the convention and the program is responsible for respecting it. The compiler will not help.

What nothing looks like in C

Spelling	What it means	Who makes you deal with it
`NULL` pointer	Whatever this program decided it means	Convention, and only convention

C also forces us to care directly about ranges and representation. A fixed-width signed integer cannot grow forever. This is not C being gratuitously difficult. C was designed around a different relationship between programmer and machine, and its type system reflects that.

Java: primitives, references, and the null question

enum Status {
    ACTIVE,
    SUSPENDED,
    DELETED
}

class Customer {
    String name;
    int age;
    double discount;
    String middleName;
    int[] favoriteNumbers;
    Status status;
}

Java distinguishes primitive types such as int, double, and boolean from reference types such as String, Customer, and arrays.

An int cannot be null. A String can.

That single asymmetry does a lot of work. It means absence is available for some of Alex's fields and structurally impossible for others, and the choice was made by whoever picked the field types rather than by anyone thinking about the domain. Unless we add conventions or tooling, String middleName does not tell us whether null is an expected state or a bug waiting for the right execution path.

What nothing looks like in Java

Spelling	What it means	Who makes you deal with it
`null`	An empty reference	Nobody
(not available)	Primitives cannot be absent at all	The compiler, by refusing

Java also has wrapper classes such as Integer, which means two things that sound like "an integer" behave differently:

int age;
Integer optionalAge;

One of those can be absent. The other cannot. Learning the keyword is only the beginning; the useful question is what guarantees come with it.

Rust: make some invalid states harder to represent

enum Status {
    Active,
    Suspended,
    Deleted,
}

struct Customer {
    name: String,
    age: u32,
    discount: f64,
    middle_name: Option<String>,
    favorite_numbers: Vec<i32>,
    status: Status,
}

Look at middle_name: Option<String>.

Instead of allowing a String to secretly contain null, Rust represents the possibility of absence in the type itself. The value is either Some(...) or None, and code consuming it has to deal with that possibility before it can get at the string.

This is the same information every other language has been carrying around implicitly, finally written down somewhere the compiler can read it. The other languages let you forget. Rust makes forgetting a compile error.

The Status enum similarly makes the valid alternatives explicit. We cannot accidentally assign ACTVE, because no such variant exists.

What nothing looks like in Rust

Spelling	What it means	Who makes you deal with it
`None`, inside `Option<T>`	This value may legitimately be absent	The compiler

A type system can encode valid states, not merely storage categories. That does not make incorrect programs impossible. It moves some classes of error from runtime behaviour into representations the compiler can reject.

The type system is participating in the design.

Go: useful zero values and deliberate simplicity

type Status string

const (
    Active    Status = "ACTIVE"
    Suspended Status = "SUSPENDED"
    Deleted   Status = "DELETED"
)

type Customer struct {
    Name            string
    Age             int
    Discount        float64
    MiddleName      *string
    FavoriteNumbers []int
    Status          Status
}

One important Go idea appears before we populate Alex at all: types have zero values. The zero value of an int is 0, of a bool is false, of a string is "". Pointers, slices, maps, functions, channels, and interfaces can be nil.

That creates a question. If Age contains 0, is Alex zero years old, or did nobody supply an age?

The type cannot answer. If the application uses a zero value to stand for “not supplied,” that state becomes indistinguishable from a legitimately supplied zero. If the distinction matters, the model has to represent it deliberately, which is why MiddleName above is a string rather than a string.
**What nothing looks like in Go*

Spelling	What it means	Who makes you deal with it
`0`, `""`, `false`	Zero/default value; cannot by itself tell us whether a domain value was supplied	The model must distinguish this if it matters
`nil`	Empty pointer, slice, map, channel, or interface	Nobody

Convenient defaults acquire meaning only when they enter a domain.

R: "missing" is not one simple state

name <- "Alex"
age <- 42L
discount <- 0.15
middle_name <- NA_character_
favorite_numbers <- c(3L, 7L, 11L)
status <- "ACTIVE"

R was designed around statistical computing, and its types and missing-value semantics reflect that.

NA represents a missing value, with typed forms such as NA_integer_ and NA_character_. R also has NULL, which is not another spelling of NA, and numeric computation can produce NaN.

These answer different questions. A missing observation in a dataset is not the same thing as the absence of an object. Neither is the same thing as a mathematically undefined result. R treats those as three distinct facts, because in statistics they are three distinct facts, and conflating them produces wrong answers rather than crashes.

Consider:

mean(c(10, 20, NA))
mean(c(10, 20, NA), na.rm = TRUE)

What nothing looks like in R

Spelling	What it means	Who makes you deal with it
`NA`	A missing observation	Functions ask, via `na.rm`
`NULL`	No object at all	Nobody
`NaN`	An undefined numeric result	Nobody

The second call makes an explicit methodological choice to exclude missing observations. The language made you say so.

The field that caused all the trouble

Alex's missing middle name has now been through seven general-purpose languages. Stack those seven tables on top of each other and the third column tells the whole story.

Python gave it None, one undifferentiated nothing. JavaScript offered two and left the meaning to you. C gave a null pointer whose meaning lives entirely in a convention the compiler cannot see. Java made absence available for reference types and impossible for primitives, as a side effect of a decision about memory. Rust put it in the type signature and refused to let you ignore it. Go made it indistinguishable from a valid value unless you reach for a pointer. R split it into three concepts because its users needed three.

Not one of those languages is wrong. They disagree about a prior question: how much does the programmer have to admit they do not know?

Representations of absence carry semantics whether or not the programmer acknowledges them. The difference between these languages is who is required to do the acknowledging, and when.

Which is why the missing middle name turned out to be more interesting than the integer.

The table gets weird quickly

Language	Typing	Ordinary integers	`42` and `0.15` same type?	How absence is spelled
Python	Dynamic	Arbitrary size	No	`None`
JavaScript	Dynamic	Fixed (`BigInt` for more)	Yes, both `number`	`null` and `undefined`
C	Static	Fixed width, chosen	Usually no	Null pointer, by convention
Java	Static	Fixed width	No	`null`, references only
Rust	Static	Fixed width	No	`Option<T>`, in the type
Go	Static	Fixed width	No	Zero values, `nil` for some types
R	Dynamic	Fixed width	Usually distinct	`NA`, `NULL`, `NaN`

Closed sets of alternatives vary just as much. Rust, Java, and C have real enums, Python offers Enum if you reach for it, Go builds them from a named type plus constants, JavaScript models them by hand, and R uses factors. Collections differ too: C, Java, Rust, and Go enforce an element type in the declaration, Python and JavaScript do not, and R's atomic vectors quietly coerce everything toward a common type.

There is no universal translation saying:

Python int = JavaScript number = C int = Java int = Rust i32 = Go int = R integer

Those types overlap. They do not carry identical guarantees, ranges, operations, failure modes, or assumptions.

The same name does not imply the same contract.

If you were designing a language, what would you choose?

This gets much more interesting when you reverse the problem.

Suppose you are designing a language.

You need an integer.

Easy.

Except now you have questions. Fixed widths, or one integer that grows until memory runs out? What happens on overflow? Should 3 become 3.0 on its own? Should "42" + 1 be legal, and if so, what should it mean?

Then absence, which by now you can see is the hard one. Do you have null? Can every reference be null, or do you require an explicit optional type? Do you distinguish a missing observation from a non-existent object? Whichever you choose, every program ever written in your language inherits that decision.

Then collections, where you decide whether elements must share a type and whether values can be moved or consumed. Then domain state, where you decide whether a programmer can say a value is only ever ACTIVE, SUSPENDED, or DELETED, and whether the compiler can prove every case was handled.

Finally, timing. When should a bad type relationship surface: while parsing, during compilation, at runtime, or only when the offending path finally executes?

There is no universally correct set of answers. There are consequences.

Every choice makes some programs easier to express, some mistakes harder to make, some implementations simpler, some runtimes more complicated, and some programmers wonder why on Earth the language designer did that.

Sometimes the answer is historical compatibility. Sometimes performance. Sometimes safety. Sometimes the problem domain.

And sometimes the language designer has a very good explanation and would appreciate it if everyone stopped bringing up typeof null.

One language where absence is not the question

Everything above is a general-purpose language, and every one of them had to decide what nothing means. That is not a universal preoccupation. It is what happens when a language expects you to model a world.

The e language, commonly associated with Cadence Specman, was designed for hardware verification. Verification code describes legal ranges of values, generates constrained test data, and explores systems whose state spaces are far too large to enumerate by hand. So its declarations carry constraints rather than storage categories:

struct customer {
    age : uint (bits: 8);
    keep age in [18..120];

    status : [ACTIVE, SUSPENDED, DELETED];
    keep soft status == ACTIVE;
};

There is no absence card for e in this article, and that is the point. In ordinary application programming we ask what type of value this is. A verification environment asks what values are legal here, and how valid combinations should be generated so we can explore what the system does.

If you look only at Python, JavaScript, Java, and C, it is easy to assume programming languages are all solving approximately the same problem with different punctuation.

They aren't.

Types are compressed language philosophy

At the beginner level, "a type tells us what kind of data a value contains" is perfectly serviceable. Eventually that runs out of road. A type can tell us what values exist, how they are stored, which operations are permitted, which conversions happen silently, what counts as absence, whether absence must be acknowledged, whether alternatives form a closed set, and in some languages how long a value lives and who owns it. "Integer, string, Boolean, float" is not the definition of a type system. It is the introductory vocabulary.

Which is why learning the types in a new language beats memorizing a conversion table.

Python's arbitrary-precision integers are a tradeoff. So is C asking you to care about width. Java's primitive and reference split quietly decides which of your fields are allowed to be empty. Go buys simplicity with zero values and pays for it with ambiguity at the domain boundary. R distinguishes forms of missingness because missing observations are a normal feature of statistical work, so its problem domain is visible in its types. A verification language making constrained generation first-class tells you what its designers expected programmers to spend their days doing.

A language's types are partly a record of what its designers thought mattered, and partly a record of what they were willing to let you avoid thinking about.

They are compressed language philosophy.

Alex survived

After all of this, Alex is still 42, still has no recorded middle name, still likes 3, 7, and 11, and remains active. Nothing about the underlying person changed. But every language forced different decisions about how those facts became computation, and the fact that changed most was the one that was not there.

Knowing that Python has int, Java has int, and Rust has i32 is useful when you are trying to get a program running. Understanding why they are not interchangeable is useful when you are trying to understand languages.

Once you start asking what a language lets you represent, what it makes you acknowledge, and what it refuses to let you say, data types stop being the boring chapter before loops. They become one of the quickest ways to see what a language believes programming should be.

What is the strangest, most useful, or most frustrating type decision you've encountered in a programming language? More importantly, what problem do you think the language designer was trying to solve with it? Or, if you were designing your own language, which of these decisions would you make differently?

Enterprise MCP Gateway to Govern MCP Tool Access

Ken W Alger — Tue, 06 Oct 2026 04:57:00 +0000

Model access controls what an AI system can know and generate. Tool access controls what it can do.

The previous posts in The AI Control Plane have moved progressively closer to the point where policy becomes action. Governance established who gets to decide. Virtual keys turned those decisions into runtime policy. RBAC and access profiles constrained who could change that policy. User provisioning made those permissions follow people as they joined, moved, and left.

Bifrost is the open-source AI gateway through which this series has been examining those control-plane concerns, with its implementation available in the Bifrost GitHub repository.

MCP tool access changes the stakes. The Model Context Protocol defines tools as capabilities exposed by servers that clients can discover and invoke. A model allowed to answer a question can produce a bad answer. A model allowed to invoke a tool can change another system. It can create a ticket, query a customer database, update a record, trigger a deployment, send a message, or call whatever capability an MCP server exposes.

The question is no longer only which model may receive a request. It is which actions that model can reach, under whose authority, and what happens when a request crosses from inference into execution.

Why Is MCP Tool Access Different From Model Access?

Model governance controls destinations and consumption. Which providers are approved, which models a workload may use, how much it may spend, and what rate limits apply. Tool governance adds capability.

Consider a customer support agent with access to three tools: lookup_customer, issue_refund, and close_account. All three may live on the same MCP server, behind the same authentication, exposed through the same connection. They do not have the same consequences.

That is why the useful unit of governance is not the server. A server is a deployment boundary. A tool is an action boundary, and the two rarely line up.

Authentication does not close the gap either. Valid credentials for an MCP server prove a connection can be made; they say nothing about whether a given workload should reach every capability behind it. This is the same separation the virtual key post drew between holding a provider credential and being permitted to use a particular model, and it gets sharper when MCP authentication is per-user. A person may legitimately authenticate to an external service while the AI workload acting on their behalf should still receive a subset of what that service offers. The credential establishes identity at the external boundary. The control plane still has to govern capability at the tool boundary.

What Should MCP Authorization Policy Control?

The simplest policy is an allow list, and it does more than it appears to, because filtering constrains discovery as well as execution. A workload never needs to be shown capabilities it will not be permitted to invoke. Bifrost applies that filtering per request, per client, or per virtual key, which means tool access can attach to the same policy boundary that already governs provider access, model allow lists, budgets, and expiry.

Enterprise authorization needs more structure than a growing collection of individual tool grants. Access profiles provide a reusable policy layer for grouping permissions around a role or workload rather than configuring each identity independently. A support workload, for example, can inherit the provider, model, and MCP permissions appropriate to support without requiring those decisions to be recreated every time another user or key is provisioned.

Virtual MCP servers provide another boundary by controlling the capability surface presented to a workload. Rather than treating every tool exposed by every connected MCP server as one flat namespace, an organization can present a curated MCP surface appropriate to a particular application or class of users. Tool filtering can then narrow that surface further to the specific actions a workload is authorized to discover and invoke.

Together, those layers answer different authorization questions. An access profile establishes the policy associated with an identity or workload. A virtual MCP server establishes the MCP capability surface available to it. Tool-level policy determines which actions within that surface are actually reachable.

That distinction matters because enterprise policy gets nuanced quickly. A research agent may search documentation and retrieve customer information without modifying either. A support agent may open a case while refunds route through a separate path. An engineering agent may inspect deployment state while production mutation stays out of reach. Policy should track the consequence of the action rather than the convenience of the integration.

Because those permissions live in the control plane rather than in twenty repositories, changing the decision does not require twenty releases. It also means refusal happens in the right place. If a workload is not permitted to invoke issue_refund, the useful outcome is not that the refund service rejects the call later. It is that the refund service never sees it. The application receives a deterministic policy refusal instead of an unpredictable downstream error, and the external system is never asked to adjudicate an unauthorized attempt.

What Changes When an Agent Can Execute Tools Automatically?

There is a meaningful difference between a model suggesting a tool call and infrastructure executing one, and Bifrost's default preserves it. Tool calls returned by a model are suggestions, not actions:

1. POST /v1/chat/completions  
   → model returns tool call suggestions, not executed

2. Application reviews the proposed calls  
   → apply business rules, request human approval

3. POST /v1/mcp/tool/execute  
   → execute approved calls explicitly

4. POST /v1/chat/completions  
   → continue the conversation with tool results

Step two is the pause. It is where an application can inspect a proposal, ask a human, apply additional logic, or refuse outright.

Agent mode removes it. Tools named in the tools_to_auto_execute configuration run without waiting for approval, which is useful precisely because it eliminates friction and consequential for the same reason.

So the authorization decision moves earlier. If no human will be present when the action is taken, the control plane has to establish beforehand which actions are acceptable. An allow list stops being a convenience for hiding irrelevant tools and becomes part of the safety boundary around autonomous execution. That does not mean every call needs an approval dialog. It means the absence of a human at execution time makes pre-established policy more important, not less.

How Should Enterprises Think About MCP Tool Blast Radius?

A workable way to classify tools is by what happens when the authorization turns out to be wrong.

Capability	Example	Primary Consequence	Governance Posture
Read	Search documentation	Information exposure	Allow by role and data scope
Create	Open support ticket	New system state	Restrict destination and ownership
Modify	Update customer record	Existing state changes	Narrow scope and retain evidence
Financial	Issue refund	Monetary loss	Explicit grant and tighter limits
Destructive	Delete account	Irreversible or costly	Exceptional access or added approval

The categories matter less than the exercise. Two tools on the same server can sit three rows apart, which is the concrete reason server-level grants are too coarse.

Why Do Individually Safe Tools Become Unsafe Together?

Here is the part that no allow list solves.

A workload permitted to read customer data and to send email has a different effective capability from one permitted to do either alone. An agent holding a database tool, a payment tool, and outbound messaging is not carrying three permissions. It is carrying a workflow, and nobody approved the workflow. Each grant was reviewed on its own and each was defensible on its own.

This is an uncomfortable property of agentic systems: capability emerges from combinations of individually reasonable permissions, and the emergent capability is a property of the set rather than of any member of it. Filtering evaluates tools one at a time, because that is what filtering is. Reviewing grants one at a time will not surface it either, for the same reason.

There is no product answer as no gateway-level enforcement exists today. What there is, for now, is a review practice: when adding a tool to an existing grant, ask what the new tool makes possible in combination with what is already there, rather than whether the new tool is acceptable by itself. That is a weaker control than enforcement, and it is worth naming as a gap rather than papering over it.

A second assumption runs underneath everything above: that the MCP server itself is trustworthy, and the only open question is which of its tools a given workload should reach. That assumption deserves its own scrutiny, because a compromised or careless server can return content engineered to be read as instruction rather than as data. I have written separately about classifying MCP servers by trust zone, which is the layer beneath the one this post governs.

Who Authorized This Tool, and What Happens When They Leave?

Filtering answers whether a call is permitted right now. It does not answer why.

"The developer added the tool to the agent" describes implementation, not authority. The permission may have arrived through a team role, an access profile default, a temporary exception, or a direct assignment created during an incident at two in the morning. Those paths have different lifecycle behavior, which is where the provisioning problem returns. A tool grant attached to an identity has to disappear when the authority behind it disappears. Deleting a user or key causes Bifrost to clean up the associated per-user credentials. Session-keyed credentials and upstream provider tokens, however, may have separate revocation lifecycles that administrators need to account for.

And a tool grant attached to a workload with no human identity needs an owner and a lifecycle of its own, because nothing will trigger one on its behalf.

MCP governance is not a list of allowed tool names. It is the connection between capability and authority.

Why MCP Governance Belongs in the AI Control Plane

This series has worked through increasingly specific forms of control: organizational governance, workload policy, administrative authority, identity lifecycle, and now capability. Tools are where those controls finally meet something able to change the world outside the AI stack.

Which is why tool governance cannot stop at authentication, server access, or whether a model supports function calling. Enterprises have to decide which capabilities are visible, which are executable, which demand stronger authority, and how all of that changes when the people and workloads behind them change. Bifrost brings those decisions into the same control plane as model access, routing, and budgets, and because it is open source, the MCP implementation can be inspected rather than taken on trust.

The principle outlives any implementation. Model access governs what AI may ask. Tool access governs what AI may do.

Five layers in, the remaining question is not which controls exist. It is what order to build them in, and what breaks when you get the order wrong.

Bifrost brings these controls into the AI control plane, giving teams a central place to manage MCP authorization alongside the rest of their AI infrastructure. To see how that works in practice, book a Bifrost demo.

This article was commissioned by the Bifrost team. Links to kenwalger.com and to the Sovereign Systems Specification point to my own work. The architectural perspective and conclusions expressed here are my own.