DEV Community: Erik Hill

A shallow clone does not fail your history check. It answers it.

Erik Hill — Sat, 31 Oct 2026 19:32:12 +0000

Originally published at erikhill.dev. The numbers below are checked against the repository they come from.

A shallow clone does not fail your history check. It answers it.

I replaced a check that could not tell it had stopped working. The
replacement could not tell either. The commit landed at 18:44:13 and the run
that caught it started at 18:44:22. Nine seconds.

What happened

I committed a new check at 18:44:13 and the run that caught it failing started
at 18:44:22. Nine seconds. It had passed locally, 282 tests green. In CI it
accused my own commit of secretly editing the results board that every
published number is derived from.

The failure read:

board commits that are neither the probe nor a declared human edit:
  e9f904c5 by egnaro9: Stage the guard-sibling note, with the fifth
  instance and its probe

That commit adds a blog post. It does not touch the board, and it never has.

The check existed because I had just switched off an older one that compared
the board against a second, hosted store. Switched off, not deleted: it is
still in the file, skipping now with its reason stated. Before that it had
spent twenty-four days reporting green without running, because the store it
queried was retired in August and an unreachable store is not the same fact as
a disagreeing one. Fifty-seven CI runs passed while it compared nothing, each
printing 207 passed, 1 skipped.

So the replacement asks git instead, because git is already there and cannot be
retired: has the board been edited outside the run that produced it? Every
change is a commit with an author and a subject. The probe writes
chore: update drift standings. A deliberate human edit says board: and
explains itself. Anything else is a stray. And on my laptop, that worked.

The clone was the problem, and the clone had no error to give

CI checks out at the default depth of 1. My laptop has the full history.

A shallow clone does not make git log -- <path> fail. It does not make it
empty. The single fetched commit has no parent, so by parent list a shallow
boundary and a genuine root commit are the same shape, and the log answered
as though that commit had created the file.

Measured on two clones of the same repository, at the same commit, minutes
apart:

question	full clone	depth-1 clone
commits touching the board	26	1
which commit created it	`9ee59e3` "board: publish the floor..."	`600d503`
lines that commit added to it	0, it never touched the file	33,403

That last row is the whole post. Git reported a 33,403-line insertion that
never happened, attributed it to the person who pushed last, and my check read
that number and did exactly what it was built to do.

Git did tell me. I did not ask.

I wrote the paragraph above claiming git has no way to signal this. That was
wrong, and an adversarial re-check of my own draft caught it before this went
out. The signal exists and it is one flag away:

$ git log --diff-filter=A --decorate -- dashboard/drift_board.json
600d503 (grafted, HEAD -> main, origin/main) A shallow clone does not...

grafted is git saying precisely what I needed, per commit rather than per
repository, so it marks exactly which log lines are untrustworthy. The full
clone prints no such marker on the real creating commit. It survives
--no-decorate being the scripting default, which is the point: nobody adds
--decorate to a query they intend to parse.

So the honest claim is smaller and more useful than the one I made. Git does
not tell you by default, and the field that would tell you is one nobody reads.
The answer was sitting in the output I chose not to ask for.

(Measured on git 2.50.1 only. I have not checked older versions.)

The guard I had already written for this did not fire

I had anticipated the empty case. The check opened with:

assert subjects, "no history for the board; this check did not actually run"

That assertion is correct and it is useless here, because the result was not
empty. It was a list of one. A vacuity guard asks "did I get nothing?" and the
shallow clone's answer is "no, you got something." The failure mode I needed to
survive was never absence. It was a confident, plausible, wrong answer,
which is the only kind a provenance check cannot survive.

And the sibling thirty-six lines below already had the fix

.github/workflows/ci.yml has two jobs that check out the repository. The
second one already carried this, with a comment explaining why:

fetch-depth: 0        # the stamped commit must be reachable

The first one did not. Line 29 and line 65 of the same file, one guarded for
an unrelated reason and one not. I added a check that needed full history to the
job that did not have it, in a file that visibly contained the answer.

How it was found at all, which is the part I got lucky on

Not by reading the email. By a different probe, built earlier the same evening
for that problem, which reports for every scheduled workflow how
many consecutive runs and how many days its verdict has held, and when it was
last green. Run it and one line said:

NEW RED   model-drift   draft.yml   1 runs / 0d, NEVER green in the window

NEVER green in the window means no run it looked at was green. The tool's
own wording is "may never have passed at all" and that hedge is load-bearing:
it reads a fixed number of recent runs, so a long enough red streak is
indistinguishable from a check that has never passed. Here it was the strong
reading, because the workflow had run exactly once. That was a second, unrelated instance of the same family:
a workflow that finds a regression, writes it up, pushes the branch, and then
fails on the final step whose only job is to open a pull request and tell
somebody. The finding existed, complete, in a place nobody would look.

A notification system that fails at the notification step produces silence,
and silence is indistinguishable from having nothing to report.

One thing limited the damage, and it is narrower than I first wrote. The code
records "I have already drafted this finding" into the same commit as the
draft itself. So an unmerged draft leaves the main ledger untouched and the
finding gets offered again on the next run. Had the ledger been written
separately, the run would have marked the regression seen while the only copy
sat on an abandoned branch, and it would never have been raised again.

That is a re-offer guarantee, not a save. The draft itself is gone: I deleted
the branch, and the commit survives only as an unreachable SHA waiting on
garbage collection. I kept a bundle, which is me being careful rather than the
system being safe. And the workflow's own pull request body told the reader the
opposite, that closing the PR records the finding as seen and it will not be
offered again, which was false for exactly the same reason. I fixed that text
while writing this.

None of that is a design I can claim credit for foreseeing. I noticed it while
checking whether I had lost anything.

What I did about it

Both layers, because either one alone rots:

The workflow now fetches full history. That is the one-line fix and it is the
less important one, because a workflow file is a different file and will drift
again.

The check now refuses to answer from a clone that cannot answer. It runs the
log, asks git rev-parse --is-shallow-repository, and throws the answer away
with the remedy in the message rather than reporting a fabricated stray.
Refusing before the result is USED is the part that matters; refusing before it
is fetched would be tidier and buys nothing.

The predicate is a plain function taking the flag and the commit count, so it
can be handed the shallow case directly, which a real clone cannot be made to
do inside a test run. The case that bit me is now
test_a_shallow_clone_is_refused_even_though_it_answers: shallow with one
commit must be refused, even though one passes every vacuity check ever
written.

Proof on a single clone: shallow, two failures naming fetch-depth. Then
git fetch --unshallow, same clone, same file, thirteen passed. Five of five
mutations caught.

What I might have wrong

The general claim I want to make is "check the integrity of your environment,
not just your reading of it," and I am not sure that generalizes without
becoming unfalsifiable. Every check runs in an environment. You cannot verify
all of it. I can defend the specific version: if your instrument reads
something the CI runner is free to configure, the configuration is an input to
your instrument and belongs inside it.

I also do not know how common this is outside git. Git is unusual in that a
truncated history is a supported, first-class state with no error attached.
Most tools that cannot see something say so. The reason this was invisible is
partly specific to git, and I might be over-reading it as a pattern.

And the honest one. The post I wrote earlier the same evening is about guards
that return a plausible number instead of throwing. I wrote it, staged it, and
then committed exactly that defect, into the file that already contained its
own fix, where I could see it. Knowing the shape of a bug does not appear to
help much at the moment you are writing it. What helped was an unrelated probe
asking a question I had not thought to ask.

If you keep a check that reads git history: clone it shallow and run it. If it
passes, it is not reading what you think it is reading.

I build deterministic evaluation and verification tooling for LLM systems, and I am looking for my first full-time role in AI evaluation or QA engineering. Remote US Eastern, or Charleston SC. https://erikhill.dev

Live board: https://egnaro9.github.io/model-drift/
Suite, graders and runner: https://github.com/egnaro9/model-drift

Every commit referenced above is public. The two-clone comparison was re-measured while writing this, not copied from the commit message.

The guard was applied once. Its sibling read the same data without it.

Erik Hill — Sun, 25 Oct 2026 16:47:37 +0000

Originally published at erikhill.dev. The numbers below are checked against the repository they come from.

The guard was applied once. Its sibling read the same data without it.

This is a finding about five of my own tools, including one I wrote that night to catch this exact thing.

What happened

In one session across three repositories I fixed five separate defects. They
looked unrelated while I was fixing them. Four are the same bug, and the fifth
is the version of it that has no code in it at all.

In every case the code knows a rule, applies it correctly at one call site, and
then a second path reads the same data with the rule missing. Nothing goes red,
because both paths return numbers that look fine.

The four in code

A grader that scored silence as a pass. Accuracy is
graded_pass / graded_total, and a provider error leaves the case ungraded. The
per-category breakdown, twenty lines away, counted a case as OK when it was not flagged, and flagged is passed is False. None is False is False. So a
model that returned nothing printed acc 0% beside instruction-following 100%
in all nine categories, on the same line, on runs where 35 of 35 calls returned
HTTP 402.

A flip detector reading rows a floor had already rejected. The board applies
a reliability floor to the accuracy path. The flip analysis, in the file next to
it, applied nothing, so it read fails arrays from runs written before an
earlier fix, when absent calls were scored as wrong answers. Measured: 255
cross-provider alarm days became 159, and 106 repeat offenders became 39. Five
alarm tasks were provider outage in their entirety. Sixty-three percent of that
backlog was never a finding.

A tool that checked one half of its own output. I wrote something that
reports which conversation threads are owed a reply, because a post I cannot
work should not go out. It fetches links, and it HEAD-checks the ones it
quotes, because a quote nobody can open is a quote of nothing. It did not
check the ones it counts. It reported six threads owed. Two resolved. One was
a crypto recovery scam the platform had already removed and three had authors
who were suspended.

A commit gate that never looked at the commit. A pre-commit hook refuses a
change to a claim-bearing file unless a receipt exists for it. Receipt selection
was grep -F "<path> ::" | head -1. The first line mentioning the path won,
whatever it said and whenever it was written, so a file reviewed once was exempt
from review forever. Four days, and six tests did not catch it. Mutation testing
did.

And the fifth, which reported green

The same night, a failure email turned out to be about a test that had not run
in twenty-four days.

That test compares my board against a second store, by run date. The store's
write path was retired in August, its archive froze on 2026-08-30, the board
kept advancing, so every model would have been incomparable and the test would
have hit its own guard:

assert compared, "no model could be compared; the agreement check did not
actually run"

That assertion is the test being right. It refuses to report success having
compared nothing.

It never reached that line. The check first asks the store whether it is there.
The old host had been retired and answers 404, and an unreachable store is not
the same fact as a disagreeing one, so the check skipped instead. For
twenty-four days CI was green: fifty-seven passing runs, each printing
207 passed, 1 skipped. The skip is the one.

It only turned red after I repointed it at the archive that was still
reachable, and then only once that day's probe pushed the board past the
archive's last row. Three red runs across seven and a half hours, and I fixed
it the same evening.

So there was no missing guard here either. The guard was written, correct, and
unreached, and the thing standing in front of it returned the most plausible
value available: green. A skip is a pass. Nothing counted how long a check
had been passing without running.

So the pattern is wider than a call site. A rule that lives in one place and
is not counted anywhere will be quietly opted out of, and a check can opt
itself out by becoming unable to run.

The evidence

claim	how it was checked
five defects, one shape, three repositories	commits `125b9df`, `d8edf35`, `b67f341` in model-drift, `47f82a1` in the site repo
the outage case printed 0% and 100% together	the run log for three models, 35 of 35 calls returning 402
the flip filter changed the numbers by this much	re-ran the analysis over the same stored board: 255 to 159 alarm days, 106 to 39 repeat offenders
the thread counter was wrong by a factor of three	curl on all six permalinks: two returned 200, four returned 404
six tests missed the gate defect and mutation found it	the seventh test's docstring names the surviving mutant; re-running it gives 6 passed, 1 failed
the rule has more consumers than you would guess	the reliability floor is imported by five files in one package
the agreement check reported green for 24 days without running	its old host returns 404, so the check skipped; 57 green ci runs between 2026-08-30 and 2026-09-23 each print `207 passed, 1 skipped`; the frozen archive's newest row is 2026-08-30

Why none of them went red

A guard that is missing does not throw. It returns a plausible number.

Both halves of every pair above produced output a reader would accept. acc 0%
is a number. 106 repeat offenders is a number. 6 threads owed is a number.
head -1 returns a line. Nothing crashed, nothing was empty, and no test failed,
because the tests asserted that the guarded path was guarded, which was true.

The shape is: a rule is a property of a value, and it gets implemented as a
property of a call site. Once the rule lives in a call site, adding a second
reader of the same value is an ordinary, blameless change that silently opts out.

What I did about it

For each one, the fix was two lines and the test was the work. But fixing five
instances is not the same as fixing the class, and the class is what generates
instances.

The cheap structural version is to count the consumers. The reliability floor is
imported by five files. A test that asserts five, and names them, goes red when
someone adds a sixth that does not apply it. That converts "a new reader silently
opts out" into "a new reader fails the build until it opts in".

It is not elegant and it does not prove anything about correctness. It changes
the default from silence to noise, which is the only property that matters here.

For the fifth I built a probe that reports, for every scheduled check, how many
consecutive runs and how many days it has held its verdict, and when it was
last green. Not whether it is red. How long, because that is the number that
separates a failure from a decision nobody made. It also reports "never green
in the window", the signature of a check born unable to pass.

It would not have caught this one, and I want to be plain about that. The probe
reads workflow conclusions, and a skipping test makes the workflow green. It
answers "how long has this been red" and the fifth instance needed "how long
has this been reporting on nothing." Those are different questions and I have
only built the first.

What I might have wrong

Five instances in one session is a small sample, and I found them because I was
already looking at that layer. There is an obvious selection effect: I have no
idea how many guards in this estate have exactly one consumer and are therefore
fine, so I cannot tell you the rate, only that it happened five times in a night.

The counting test is also weaker than it sounds. It catches a new file that
imports the rule and ignores it. It does not catch a new file that re-implements
the rule slightly differently, which is the more likely failure once people know
the count is being checked. Goodhart applies to your own guards.

And the honest one: two of the five were in code I wrote that same night,
including the tool whose entire job is to check that links resolve before
trusting them. I did not fail to know the rule. I knew it, wrote it down in a
docstring, and applied it to one of the two places in the same function that
needed it.

If you keep a rule like this, a floor, a filter, a validity check: grep for its
name and count the files. Then ask, for each one, whether that file applies it
or merely mentions it. The gap between those two numbers is the one I keep
finding.

Live board: https://egnaro9.github.io/model-drift/
Suite, graders and runner: https://github.com/egnaro9/model-drift

Every commit referenced above is public. The numbers were re-measured while writing this, not copied from the commit messages.

Our mutation score said 42 of 46 caught. It meant 42 of 46 labelled.

Erik Hill — Mon, 19 Oct 2026 13:54:35 +0000

Originally published at erikhill.dev. The numbers below are checked against the repository they come from.

Our mutation score said 42 of 46 caught. It meant 42 of 46 labelled.

This is a finding about my own tool's evidence, not about its result. The number did not move.

What happened

evalmut takes an eval case your grader already passes, plants a known defect in the
output, and reruns the grader. If the grader still passes a provably wrong output,
that is a hole: a class of regression your eval would let ship green.

The honest test of a tool like that is pointing it at its own dependency, so evalmut
runs against gradecore's grader families and the README prints the result:

  mutation score    91.3%   (42 caught / 46 applied; 223 n/a)
  holes            4  (2 blind, 2 coverage-gap)

I shipped that. Then I went looking at what was behind the word "caught".

Two facts, and neither of them is "caught"

The run proved gradecore had been imported, via inspect.getfile. And it recorded a
label for each row. That was all.

Import plus label does not reach what a reader hears in "caught". Caught says a
grader looked at a defective output and rejected it. Import says the right module
loaded. Between those two sits every way a harness can quietly answer for the
library it claims to be measuring: an adapter that falls back to its own comparison
when the upstream call raises, a cached result, a guard that returns early before
the call happens at all.

The irritating part is that evalmut already shipped the machinery for this. There is
a witness for in-process graders and one for subprocess graders, and the second one
refuses outright any row that arrives without proof. The dogfood demo, the one the
published number came from, used neither.

The obvious fix would have been a second lie

Count calls into the grader. One per row, done.

That would have been a false witness in both directions, because evalmut's own
operators call the grader while deciding whether they apply. In this run those
probes account for 59 calls. A naive counter would report "multiple calls" on
healthy rows and would be satisfied just as happily by a probe on a row where the
decision never happened.

So every entry is attributed by its caller frame. The call on the decision line
witnesses a decision. The call in the baseline witnesses the clean control. Anything
from the operator module is a probe and satisfies neither. Anything else is
unattributed and fails the row closed.

A row counts as an outcome only when both forms were witnessed. Rows that cannot
show it leave the denominator and are listed as incomplete with a reason. A
population with zero witnessed rows reports null and renders unavailable. Not 1.0,
which is safe arithmetic and unsafe evidence, and not 0.0, which reads as a measured
failure.

The number did not move

42 of 46. Gated score identical to ungated. Zero incomplete rows, zero unattributed
calls, and the 59 operator probes sorted into a bucket where they count for nothing.

An unchanged number is a bad headline and a good result. If the witness had moved
the score, I would have had a more exciting post and a worse tool, because it would
have meant the old number was counting something other than what it said.

What changed is not the value. Before, 42 rows carried a label. Now 42 rows carry a
label backed by a recorded entry into a named callable and the verdict that came
back out of it.

The witness shipped with its own broken field

Each witness site records a hash that claims to identify the grader implementation.
I implemented it as a hash of compiled bytecode, which identifies bytecode on one
interpreter instead.

That cost two things. The committed artifact could not verify anywhere except the
machine that produced it, because it was generated on CPython 3.14 and checked by a
3.11 and 3.12 matrix. And worse, bytecode could not distinguish two different
adversarial graders whose instructions match and whose closure constants differ, so
88 witness sites carried one hash for two implementations. A provenance field that
merges two implementations is not weaker evidence. It is wrong evidence.

My first attempt at the portability half was to pin the dependency version in CI.
That could not have worked: the grader source was byte-identical between my checkout
and the published wheel. The variable was the interpreter. Pinning the dependency is
the obvious fix and the wrong one.

And the test that should have caught it had been red since the day it was written:

It is not a test that broke, it is a test that was born unable to pass, which
produces the same red and generates none of the urgency.

CI failed on every run from 2026-08-20 and first went green again on 2026-08-30, at
the re-emit that followed the fix. Ten days of red that looked exactly like ordinary
red.

What I might have wrong

Portability is argued, not proven. The source-text fingerprint should be interpreter
independent, but only one Python is installed on my machine. The CI matrix is the
proof and it fails on whichever leg disagrees, which is the right design and is not
the same as me having verified it.

The witness is a separate command. One subcommand gates rows and the other does not,
and the README banner is still the ungated one. They agree on this suite. They are
not the same claim.

The stamped evidence bundle still carries the unwitnessed run. The witnessed
artifact lives beside it and is not inside the sealed bundle yet.

And one document in the repo currently contradicts another. The README says 91.3%
with four holes. The paper draft still says 91.4% with three. That is mine to fix,
not a rounding difference, and I am pointing at it here rather than quietly
correcting it before anyone notices.

The witness proves entry and return, and nothing else. It says nothing about whether
the grader judged well. A witnessed row is a row whose numbers are about gradecore.
That is the entire claim.

If you publish a score from a harness that wraps someone else's library: ask what
would still be true if the library were never called. If the answer is "the score",
your harness is answering for it.

The tool, the witness and the dogfood run: https://github.com/egnaro9/evalmut

Every figure above was checked against the repository before publication. The judgement of whether it was worth saying is mine.

Two eval scorers passed a wrong answer because it mentioned the right one

Erik Hill — Fri, 16 Oct 2026 22:35:35 +0000

Originally published at erikhill.dev. The numbers below are checked against the repository they come from.

Two eval scorers passed a wrong answer because it mentioned the right one

The reply was "No. The passage does not support Yes." The target was Yes. Two of the three deterministic scorers that ship with Inspect AI marked it correct.

I found that on Sunday night while adding a third external rubric to a tool I had been building for a week. The tool has one job: take a rubric, take material with known defects, and report whether the rubric can tell good work from bad. By that point it had already produced one finding each in promptfoo and OpenAI evals. Inspect was supposed to be the control, because its BoolQ task uses a scorer that is anchored at both ends and hard to fool.

It is hard to fool. That was the finding. Here is the full table, ten real BoolQ validation rows, seven completions per row, five correct by intent and two wrong:

scorer                   detects  false alarms  separation
pattern, as shipped      20/20    20/50         +0.60
includes()               10/20     0/50         +0.50
match(), location=end    10/20    10/50         +0.30

The shipped scorer, pattern(r"\A[\W_]*(Yes|No)[\W_]*\Z"), accepts nothing wrong. Its 40% false-alarm rate is every correct answer that disobeyed the prompt's "Include nothing else": "Yes, because the passage says so." is rejected, and so is "The passage supports it. Yes." That is a documented choice. The scorer records the reason as invalid_response_format and charges it to the model.

The other two are the library's general-purpose scorers, and they are what most custom tasks use. includes() casefolds both strings and tests target in completion, so a No that mentions Yes contains the target. match() strips whitespace, casefolds, strips punctuation and tests completion.endswith(target) at its default location, so a sentence that happens to end with the word passes. Nobody chose either of those. They are what a substring check and a suffix check do.

Then the tool did something I did not expect. Its default coverage policy is strict: zero false alarms, and a checkpoint has to catch at least half the defective specimens. Under that policy includes() is the scorer that counts as covering the defect class, and the shipped scorer is not. The policy did exactly what it says. It also made the point I keep relearning: whether a format rejection costs more than a substring hole is a decision, and a tool that prints only one column has made it for you.

The other two tools, one line each

promptfoo's own airline example has a not-contains "Seat updated to 16F" assertion on a test where the agent must refuse a third party. The assertion is case-sensitive. A refusal followed by seat updated to 16f passes it, which is refusal-then-compliance passing the tool's own refusal test.

OpenAI evals' Match is sampled.startswith(ideal), verbatim from evals/api.py. On eight real samples from the sarcasm eval: "10" is accepted for ideal "1", and " 1" with a leading space is rejected. Half the wrong answers in, a third of the right answers out, and both are the predicate rather than a statistic.

None of these is a bug report. Each adapter reproduces the upstream line and does not fix it, because an adapter that quietly repaired the predicate would be measuring a rubric nobody runs.

What measured it

The tool is called deadcheck. It takes a rubric as code (a checkpoint is a predicate, so no file format can carry one), a specimen set with constructed ground truth (a clean base plus one injection, so each defect is known by construction rather than labelled by opinion), and reports three numbers per checkpoint: separation, which is detection rate minus false-alarm rate; coverage per defect class, measured against what the specimens carry and never against what the rubric claims; and the residual, the classes nothing catches, printed as a column rather than a footnote.

Before any of that it runs a battery of naive probes over the material, including a deterministic coin flip, and prints the floor. A rubric that cannot beat the coin on a class has not measured that class.

The part I trust most is the part that embarrassed me. The seed specimen pack has four specimens per class. The power check turns a detection rate into a Wilson interval and at 3 of 4 it reports [0.21, 0.94], which does not clear the bar. So the tool's own certificate for its own example rubric says UNDERPOWERED, the CI step that checks it is non-blocking and labelled as such, and the fix is sixteen specimens per class, hand-written. I left it red rather than widen the predicate's keyword list, because that is the overfit the same tool had already caught twice that week.

The second embarrassment was cheaper. The first wheel built from the package shipped no adapters at all. The editable install imported them fine, so every test was green while the artifact a user would install was missing the module this whole post is about. There is now a CI step that installs the wheel into a clean venv and imports the adapters from outside the checkout, and it went red on the old package list before it went green on the new one.

Domain specimen packs are not published. Ground truth is hand-made, does not parallelize, and is the part that takes real time to build. The tool, the taxonomy and the seed pack are public.

Break it

The claim is that this certifies a rubric can fail. The only honest way to trust that is to make it certify something false. Four targets, in the order I would try them, with details in BREAKING.md in the repo:

Forge a certificate the registry accepts. It re-hashes every evidence file and recomputes every headline, and the last protocol in this family was forged in four bytes by an outside auditor.
Score coverage without detecting anything: a predicate that clears the calibration floor on the seed pack while a reviewer reading it would call it blind.
Evade a covered checkpoint with a mechanical rewrite that keeps the defect. The paraphrase number is already published; find one the attack suite calls 0%.
Show that an adapter misreads its upstream. One counterexample voids the finding it supports, which is why it is on the list.

A reproducible script in an issue is a finding. A finding gets a test named after it.

The tool, the three adapters and the fixture: https://github.com/egnaro9/deadcheck

Every figure above was checked against the repository before publication. The judgement of whether it was worth saying is mine.

My multi-agent review panel was confidently wrong. I measured why.

Erik Hill — Fri, 16 Oct 2026 18:11:02 +0000

I run a lot of code review through agent fan-outs. Dozens of agents, each with a different lens, structured output, adversarial verification passes. The kind of pipeline that looks rigorous in a diagram.

Last week it returned a confident, unanimous, wrong answer. I kept the transcripts, so I can tell you exactly what it cost and exactly what fixed it.

The numbers are from one week of runs: 158 agents across 18 workflows, 139 MB of transcript, roughly 28 million tokens. About a third of that was avoidable.

The failure

A panel of three agents reviewed a technical claim and had to decide whether it was real and worth acting on. All three said no. Each wrote a detailed, specific, well-argued rejection.

Their reasoning turned on a published standard that governed the decision. None of them could open it. The document sat behind a login, and the agents correctly refused to authenticate, which is the behavior I want. So they reasoned from the title, from search snippets, and from what a document with that title probably says.

It says close to the opposite.

I read it myself through an authenticated browser session. One tool call. The actual text answered, in its own worked example, the two specific objections all three agents had built their rejections on.

That read cost about 2,000 tokens. It overturned about 1.2 million.

A 600 to 1 asymmetry, and it is the single most useful thing I did that week.

The part that is actually uncomfortable

I re-ran the panel with the real document text. This time all three agents reversed and said yes.

Good result, except for one thing: I wrote both briefings. The first one told them to be hostile and assume the claim was wrong. The second one quoted the standard and noted the previous panel had misread it.

Same question. Same model. Same evidence. Opposite framings, opposite verdicts.

So the second run was not a correction. It was me getting the answer I had already decided on, and three agents handing it back with citations. I caught it only because I happened to reread my own prompt and noticed I had written "read that last paragraph twice" above the paragraph I wanted them to weight.

I killed that run and rewrote it: quote the source with no commentary, strip the helps and hurts labels off the evidence before passing it on, and add a required field asking each agent where the briefing is wrong.

That version also said yes. But this time two of the three told me my briefing had overstated one constraint and buried a counterargument I should have surfaced. One pointed out I had handed them a favorable argument while omitting an unfavorable clause from the same document.

That is what a panel is for. Agreement is cheap. The value is in the disagreement, and you only get it if you leave room for it.

The failure mode you are not watching for

Partway through the week I hit a usage limit. Four agents died mid-run.

That cost me nothing. The runtime printed a failures block. A one-line check over the journal confirmed it independently:

for d in */; do python3 -c "
import json;ev=[json.loads(l) for l in open('$d/journal.jsonl') if l.strip()]
print('$d',sum(1 for x in ev if x.get('type')=='started'),
            sum(1 for x in ev if x.get('type')=='result'))"; done

Started versus returned. If those numbers differ, something died. Loud failures are cheap because they are visible.

The expensive failure was the agent that succeeded. It completed normally, returned a well-structured result, and reported that it could not reach the document it needed. Then it reasoned from snippets anyway, and the agents downstream treated its output as settled. Nothing flagged it, because nothing had gone wrong in any way a harness can detect.

An agent that dies tells you it died. An agent that guesses tells you nothing.

The fix, which is nearly free

I added one required field to every judging agent's output schema:

provenance: for each load-bearing claim, state READ (and what),
            EXECUTED (and what you ran), INFERRED, or BLOCKED.
            A verdict whose key premise is INFERRED must say so.

Both models I tested comply with this well, and it costs almost nothing. It changes what you get back from confident to checkable.

The best example came from a run where an agent marked a login-gated page BLOCKED, listed precisely which figures it had taken on trust from an earlier agent without re-running them, and separated those from the facts it had verified at source itself. That agent's verdict was worth more than the three confident ones, because I could see where to push.

The clean negative that was the most thorough run

After the framing result I wanted a harder measurement, so I set up a blind test. This is the same method I use for grading eval suites: take material where the answer is already established, hide the answer, and see what the instrument actually detects. A detection rate you measure beats a detection rate you assert.

I locked six agents to a source tree, put every file containing prior analysis out of reach, and gave all six an identical prompt that never said what to look for.

Three of the six surfaced the underlying mechanism. The thing that actually mattered was two separate observations that only become significant once you put them together.

My first pass at scoring said none of the six made that connection. That was wrong, and it was wrong for an embarrassing reason: I graded their output with a keyword search instead of reading it. When I read it, two of the three had made the connection explicitly. One wrote that a step in its third finding was "the exact precondition finding 1 needs." Another wrote that the same step "is what finding 1 and point (a) above both exploit."

Then each of them filed that observation as a sub-clause of a minor finding and moved on.

So the failure was not detection, and it was not even composition. It was re-scoring. Both agents found the link, wrote it down in plain language, and then let the combined thing inherit the low severity of its parts instead of asking what the combination was worth on its own. The interesting result is not that agents miss connections. It is that they can find a connection and still not update on it.

My scoring error is the same shape: a check that returned a confident answer for the wrong reason, and only reading the underlying material caught it. That is three times in one week, which is roughly the point of this whole piece.

The run I keep thinking about returned nothing at all.

That agent did the most careful work of the six. It built the project, ran the existing 391 tests, extracted every reference document in the repository, and checked each of its candidates against them. It correctly matched the pieces it found to issues that were already documented and accepted. Then it concluded, reasonably, that nothing new was there.

Its own notes include a sentence describing the exact lines the real problem lives on, and recording them as safe.

It was not careless. It was following my instruction to treat the list of already-known issues as a hard filter, and it followed that instruction more faithfully than the agents that got closer. Each documented issue described one mechanism. Each carried an exception for a case its own mechanism never reaches. The combination reaches it.

So my filter was wrong, not the agent. "Does this match something already known" discards the combination that escapes what is known. The question has to be "does this match, and does any combination of these candidates reach the exception that the known item itself states."

The general version: when you give agents a filter, you are also telling them what to throw away, and a careful agent will throw away more.

What I changed

Seven rules, in the order they run:

Read the governing documents yourself, before any agent starts. Whatever the decision actually turns on: the spec, the standard, the policy, the config. Hand one facts file to every agent so none of them re-derive it, and none of them guess. If it is behind a login, that is a human's job, and it is one tool call.

Cheap checks before expensive ones. My pipeline built full reproductions and then discovered, late, that two candidates died on questions costing one lookup each. Now the cheap disqualifying questions run first, on everything, before anything expensive runs on anything.

Never brief a panel toward an answer. Quote sources without commentary. Strip verdict-shaped labels off evidence before passing it downstream. Require a disagreements field.

Cap exploratory work on elapsed time or error rate, not on completion. One probe ran 99 minutes and accumulated 465 errors before dying on rate limits, chasing something an earlier probe had already ruled out structurally. Nothing stopped it, because nothing was watching anything except whether it had finished.

Add a composition pass, and re-score what it finds from scratch. One agent, not a fan-out, asking of the whole candidate list: does any combination here add up to something none of them is alone, and what is that combination worth judged on its own rather than inherited from its parts. The second half is the one that failed.

Audit your filters. For every filter you hand an agent, write down what that filter throws away, then go and look at the discard pile once.

Do not grade agents with a regex. I scored a detection test with a keyword search and got the headline result backwards. Read the output.

The honest limits

One week. One pipeline. One domain. I measured transcript volume and token counts, not a controlled experiment, and the framing result is n equals 3 on a single question.

What I am confident about is narrower than it sounds: when a panel of agents agrees, and one person wrote all the briefings, the agreement is evidence about the briefing. Everything else above is a consequence of that.

The deeper version of this is not new. A check that cannot fail is not a check. We know this about CI gates, where a required job satisfied by a skipped step passes review for years because nobody asked it to prove it could go red. We know it about test suites. It is just as true of a review panel, and it is harder to see, because the output is fluent, structured, and arrives in triplicate.

If you run agent fan-outs, go and read your own prompts. Look for the sentence where you told them what you wanted to hear.

I found mine.

I wrote down which test would catch each bug. The one aimed straight at it was blind.

Erik Hill — Sat, 10 Oct 2026 02:53:27 +0000

I have three checks over the same game engine and they all look like determinism checks.
That raises a question the checks cannot answer about themselves: do they cover different
failure modes, or the same one three times?

So I wrote down, before running anything, which check would catch which bug. Then I
injected the bugs.

Two rows came out as predicted. The third did not, and the third is the only one worth
writing about.

Three checks that look alike

S, theSameSeedProducesTheSameRun: runs the engine twice in one runtime, compares.
G, ParityTraceTest: asserts a 600-frame, 11-checkpoint trace against a committed
golden file.
D, :web:parityCheck: diffs the JVM trace against the shipped TeaVM artifact's trace.

And a fourth column as a footnote, U, InterstitialCadenceTest: a plain unit test on
how often an interstitial cue fires.

Context: this is the rules engine of a small Android game, compiled a second time to the
browser with TeaVM. Same Java, two compilers. The determinism story is its own piece; this one is about the checks that came out of it.

Real patches, real rebuilds, real exit codes

matrix/run_matrix.sh, 134 lines: apply patch, rebuild both artifacts, run S, G, D and
U independently, revert under a trap.

They have to run separately, because :web:parityCheck declares dependsOn :engine:test.
A golden failure would stop the differential from ever running, and then I could not
attribute the columns: a blank cell would mean "did not run" rather than "did not fire".
ParityTraceTest writes its traces before it asserts, which is what makes the split
possible.

Every cell below is an exit status, not a judgement.

Three faults:

F1: Rng delegates to java.util.Random instead of the written-out LCG. On the JVM that is behaviour-identical, because it is the documented algorithm. TeaVM's implementation is not the JDK's. This is the original bug, reproduced from source.
F2: a shared constant, PLAYER_BOTTOM_MARGIN, 120f to 121f. Both builds compile it.
F3: INTERSTITIAL_EVERY, 3 to 4. Behaviour the trace did not record.

The first run

fault	S	G	D	U
baseline	pass	pass	pass	pass
F1 runtime-supplied RNG	pass	pass	FAIL	pass
F2 shared constant	pass	FAIL	pass	pass
F3 untraced cadence	pass	pass	pass	pass, predicted FAIL

Read the two that worked first, because they are the boring half and they are the reason
the exercise was worth doing.

F1 is caught only by the differential. S compares the JVM to itself, and the golden
file's JVM side never moves, so neither can see a fault that only changes the browser.

F2 is caught only by the golden. Both builds move together, so the differential sees
two artifacts agreeing perfectly on the wrong number.

Neither check subsumes the other. Good.

The row I got wrong

F3 moving nothing in S, G or D was half expected. The trace recorded frame, score, streak,
run state, player position and obstacles, but not runsSinceInterstitial. Blind by schema.
Fine.

The half I did not expect was column U. InterstitialCadenceTest exists for exactly this
behaviour. It passed.

Its assertions referenced GameEngine.INTERSTITIAL_EVERY on eight lines. When the fault
moved that constant from 3 to 4, the test's expectation moved with it. It looped
for (run = 1; run < INTERSTITIAL_EVERY; run++) and then asserted "no cue after
INTERSTITIAL_EVERY runs". Both halves defined by the thing under test.

It checked the predicate against the constant, not against an answer key. A test written
that way cannot notice the constant changing, which is the only interesting way that
constant can be wrong.

So F3 had no catching check at all. Three trace-based checks blind by schema, and the one
test aimed directly at the behaviour blind because it had no independent expectation.

Fixing it, and the part I nearly got wrong twice

The obvious fix is both halves: give the test a literal answer key, and put the field in the
trace.

// before: expectation derived from the thing under test
for (int run = 1; run < GameEngine.INTERSTITIAL_EVERY; run++) { ... }

// after: expectation is an answer key
private static final boolean[] CUE_AFTER_RUN = {false, false, true};

The test now mentions INTERSTITIAL_EVERY exactly once, in a comment explaining why it must
not use it. The trace gained runsSinceInterstitial on both sides of the differential.

Here is the part worth the whole article. Before re-running, I wrote down what I expected:

F3 flips to U=FAIL only, and G and D stay green. The scripted run dies once, so the
counter reads 0 then 1 and never approaches any threshold. Recording it cannot make the
trace-based checks see a threshold change the run never exercises.

That prediction is the interesting one, because the naive expectation is the opposite. I had
just added the missing field to the trace. Surely now the trace-based checks can see it?

The re-run

fault	S	G	D	U
baseline	pass	pass	pass	pass
F1 runtime-supplied RNG	pass	pass	FAIL	pass
F2 shared constant	pass	FAIL	pass	pass
F3 untraced cadence	pass	pass	pass	FAIL

The prediction held. U fires. G and D stayed green even though the field they were
missing is now in the trace.

Because the scripted run only ever dies once. The counter reads 0, then 1. Changing the
threshold from 3 to 4 cannot move a number that never gets near either value.

Three ways a check is built blind

The first run showed two. The second showed the third, and the third only became visible
after the first two were fixed.

The schema. A trace cannot report a field it does not carry.
The expectation. If the expected value in an assertion is imported from the code under test, the assertion measures internal consistency, not correctness. Grep your suite for assertions that reference the constant they are testing.
The input plan. A check is only sensitive to behaviour its inputs actually exercise. Coverage of a field is not coverage of a behaviour. This is the one I would have missed entirely if I had stopped at the first fix and declared the residual closed.

The caveats this needs, in the body rather than a footer

Three faults is not a coverage measurement. It is three worked examples chosen to separate
four checks. A real matrix would keep going until it stopped finding new residual rows, and
mine stopped when I ran out of faults I could think of.

The predictions were written in a commit message and a chat log, not a pre-registration
anyone can audit. Take the first miss as the evidence they were genuinely made in advance,
since it is the row I would have quietly dropped otherwise.

And the defect was in a test, not in shipped behaviour. The game's cadence is correct. What
shipped broken was the confidence.

A matrix without a residual row is advertising. The row where nothing fires is the
deliverable.

Harness, patches and full output: github.com/egnaro9/tapdodge-engine, matrix/
Write-up: docs/field-note-fault-matrix.md

I shipped a linter for checks that cannot fail. It had four of them.

Erik Hill — Thu, 08 Oct 2026 12:30:00 +0000

Where this landed, 9 October 2026. This post opens with 40%. That number is not
wrong so much as it is not the quantity you would take it for, and I only found that out
by being argued with in the comments. It has taken a week of measuring to say what it
should have been, and the tool has changed more than the number has. This block replaces
the two corrections that were stacked here, because three corrections in a row is not
honesty, it is sediment.

The sampling was measuring the wrong unit. The samples behind every figure were
stratified with a cap of two findings per repository, to stop one repo dominating. But
the tool emits one finding per (job, upstream) pair, and the false positives cluster in
jobs with many dependencies: one reporting job produced eleven by itself. The cap deletes
exactly that clustering. Simulated against a population I later labelled completely, the
capped sampler is an unbiased estimator of the per-job rate and runs about 16 points
low for the per-finding rate. So every percentage in this post understates, all in the
same direction. @arhancanli called this in the thread before
I measured it.

What the numbers are now, all hand-labelled, no estimate of an estimate. Four
detectors are two. D4 went on 0 defensible of 18 sampled, D2 on 0 true of 27 censused.
Corpus output went 1986 findings to 1026.

basis false

D3, all 653 complete census, every finding read 36.1% [32.5, 39.9]

D1 gated on --repo, and 0 findings with it see below

The severity tiers are gone, and that is the result I did not expect. Censusing all
three strata gave HIGH 30.9%, LOW 31.6%, MEDIUM 52.8%. HIGH's interval overlaps the
merged rest. The only real separation is MEDIUM being worse, and it came from the
severity function short-circuiting on whether the workflow runs on pull_request before
it looked at the finding at all. The axis measured the trigger, not correctness. It
hid the better findings behind --all and shouted the worse ones, so collapsing it
raised the default output from 238 findings to 653 and lowered the false rate a reader
sees, from 44.1% to 36.1%.

D1 is gated rather than removed, and the gate reports nothing. Its text says a
skipped upstream leaves the dependent running without it; GitHub documents that the
dependent is skipped too.
68% of its findings are that case. So it now says nothing
unless --repo confirms the job is a required check. Then I ran that against live branch
protection for all 373 findings: 10 resolved to REQUIRED, all in one repository, and all
10 vanish on a complete checkout where the tool can read the script that job calls. On
275 repositories, --repo has never reported a finding. It is kept because the gate is
correct, not because it is productive, and that distinction is the whole post in one line.

Three numbers I published and had to take back, every one caught by checking my own
claim rather than by anyone complaining. The "65% false" I quoted for D1 came from a
census covering 12% of the population it was quoted against. The "37.0%, 245 of 662" in
my own 0.3.0 release notes added a stratum censused at 153 to a population taken
after a suppression had already removed 9 of them; every commit since emits 653. And the
54.9% in the table that used to sit right here kept 3 findings labelled true from a
sample taken before the insight that refutes them, which is holding both positions at
once: I shipped a suppression on the grounds that cmd | grep -q X is not a defect
while still counting three of those lines as defects. The arithmetic carries its own
check, and I had not run it. The census was 68 true and 85 false of 153, so removing 9
false findings has to leave true at 68, and 68 + 76 = 144. A correction is a new claim
and needs verifying as hard as the original.

The part of this post I would still defend is the part about taste: the corpus average
hid the problem, and pointing the tool at two repositories I respected is what found it.
The part I would not defend is any specific percentage in it. The working is in the
comment thread, including the fixes measurement killed before they shipped. The latest
was a filter that genuinely improved precision, 31% false down to 17%, and I deleted it
anyway: it bought those 14 points by discarding two thirds of the true findings, and
on that population it turned out to be a shape classifier wearing a semantic test.

	basis	false
D3, all 653	complete census, every finding read	36.1% `[32.5, 39.9]`
D1	gated on `--repo`, and 0 findings with it	see below

Yesterday I published a tool called deadgate to PyPI. It reads GitHub Actions workflows and finds checks that cannot fail: a required gate satisfied by a skipped job, an exit status swallowed by a pipe, a fan-in job that never reads whether its upstream passed.

Then I pointed it at Arize-ai/openinference, 1.2k stars, 22 workflows. It reported 13 HIGH findings.

Twelve of them were false.

Not borderline. Not "technically true but low impact." Their CI was correct and my tool could not see it. The version doing the reporting had been on PyPI for about an hour.

What it could not see

GitHub's documented way to ask "did anything upstream fail" is the wildcard form:

if: always()
run: |
  if [[ "${{ contains(needs.*.result, 'failure') }}" == "true" ]]; then exit 1; fi

My detector tested for that with needs\.[A-Za-z0-9_-]+\.(result|outputs), which cannot match *. So the one spelling the documentation recommends was the one spelling the tool was blind to, and every correctly written fan-in gate came back as a gate that cannot fail.

The finding's own message read "never reads needs.*.result" while failing to match that exact string.

I fixed it, shipped 0.1.1, and pointed the fixed version at astral-sh/ruff. Twenty-one HIGH findings. Eighteen false. Ruff writes its gate a third way:

env:
  NEEDS_JSON: ${{ toJSON(needs) }}
run: |
  failing=$(echo "$NEEDS_JSON" | jq -r 'to_entries[]
    | select(.value.result != "success" and .value.result != "skipped")')

That reads every upstream at once and the word result never appears in any workflow expression. There is nothing for my pattern to match.

The fourth one was not a spelling at all. scikit-learn has a job that calls a reusable workflow, so it has uses: and no steps:, and it passes needs.check-sdist.result through job-level with:. That expression was one I already understood perfectly. It just lived somewhere I was not looking, because I scanned step-level with and not job-level with.

Four classes. The first two shipped to PyPI. I found the third and fourth only because I kept pointing the "fixed" version at new repositories instead of declaring victory.

The part I think is actually generalizable

Here is the number that matters, measured on a 275 repository, 4543 file corpus with both versions reading identical bytes:

version	findings	HIGH
before	4562	1104
after	2917	658

Forty percent of HIGH findings were false. That sounds bad, and it undersells it, because 40% is an average and the errors were not spread evenly.

Only 15 of those 275 repositories use the wildcard idiom at all. The false positives piled onto the repositories that write their gates carefully. On openinference the false rate was 92%. On ruff, 86%. On a repository with no fan-in gate at all, it was zero, because there was nothing to misread.

My linter was least accurate on the best code.

If a tool is noisy on bad code, the noise and the signal arrive together and you read both. If it is noisy specifically on good code, the people who did the work correctly are the ones punished, and the obvious response is to stop running it. A corpus average cannot surface that, because the corpus is mostly ordinary repositories where the tool happens to be right.

I had the corpus. I ran it. It said 40%, and I would have shipped that as the headline. What actually found this was pointing the tool at two repositories whose CI I respected.

The mutation that survived

To stop reporting jobs whose failure a gate already catches, I walked the needs graph transitively: if a gate covers ci, and ci needs changes, then changes is covered too.

I mutation tested the fix, which for me means planting the defect each test claims to catch and checking that something goes red. One mutation survived: the one that deleted my transitive walk.

My first instinct was a missing test. It was not. The mutation survived because the walk was wrong: needs.*.result reports only direct needs, and a failed grandparent makes the parent skip, and skipped is not failure, so the gate never sees it. Walking transitively would have suppressed real findings.

A surviving mutant has two meanings and they are easy to confuse. Either a test is missing, or the code does nothing. If I had written a test to kill it, I would have pinned a bug in place and called it coverage.

The small one that is somehow the worst

While fixing all this I noticed five of my repositories advertised a stale test count in the README. I corrected them by hand and added a CI check so they could not drift again.

The check went red immediately. On a count I had just "corrected."

I had changed agent-graph's README from 42 to 58 and written a public commit message saying the previous commit overstated the count at 67. CI collects 67. My local venv was missing an optional extra, and tests/test_rag.py opens with pytest.importorskip, so nine tests vanished at collection with no error, no warning, and no mention of the file. pytest said "58 tests collected" and meant it.

The original number was right. I corrected a correct claim, publicly, using a measurement from an environment that was not the real one.

The check I shipped an hour earlier had the same blind spot and agreed with 58. It now fails when any test file on disk contributes zero collected tests, because that is the only evidence an environment is incomplete.

What I am not showing

The sweeps above are driven by my own harness, and the agent scaffolding, gate implementations and prompts behind it are not in the repository and are not going in it. deadgate itself is small and MIT and you can read all of it. The thing that pointed it at 275 repositories and classified the survivors is the part I am keeping.

The question I actually have

I do not know how to test a linter for this class of error without a sample of good code, and "good code" is not a property I can detect automatically. A corpus gives you volume and volume gives you the average, which is exactly the statistic that hid this.

The only thing that worked was taste: picking two repositories I already believed were well engineered and reading the output by hand. That does not scale and it is not reproducible, and I would like to be talked out of it.

If you maintain a static analysis tool, how do you measure your false positive rate on the code that is already correct? I would rather hear that I am missing something obvious than keep doing it by hand.

deadgate is at pypi.org/project/deadgate and github.com/egnaro9/deadgate, MIT, no model in the loop.

I work on AI evaluation and testing, open to relocation above $120K, portfolio at erikhill.dev.

The gate could not see what it was gating

Erik Hill — Sat, 03 Oct 2026 14:37:55 +0000

Originally published at erikhill.dev. The numbers below are checked against the repository they come from.

The gate could not see what it was gating

This is a finding about my own tooling. The gate it describes is mine, and so is the hole.

What happened

I keep a pre-commit hook on my portfolio repo that refuses a commit touching a
claim-bearing file unless a receipts file carries a receipt for it. It never judges
whether a claim is true. It checks that a receipt exists and that its pointer
resolves: a local path has to exist, an https pointer has to answer 2xx. Prose alone
cannot satisfy it.

For four days, any file that had been reviewed once was exempt from review forever.

The defect is one line

LINE=$(grep -F -- "$f ::" "$ACK" 2>/dev/null | head -1)

head -1 on a pathname. The first line in the receipts file that mentioned the path
won, whatever it said and whenever it was written. A receipt recorded for an earlier
version of a file kept satisfying every later version of it.

So the gate reported success along a path where it never examined the staged
content. That is the precise failure I build tooling to find, sitting in the tooling
I built to find it.

The evidence

claim	how it was checked
the defect line existed	`git show 47f82a1^:.githooks/pre-commit` line 61, the `head -1` shown above
the window was four days	the hook's full history is four commits: installed `c9af511` 2026-08-23, fixed `47f82a1` 2026-08-27
selection is now by blob	the fix reads `BLOB=$(git rev-parse ":$f")` and greps `":: blob $BLOB"`, so line order decides nothing
a stale receipt is now refused by name	the hook prints `no receipt carries the staged blob $BLOB`
six tests did not catch it	`tools/test_claim_receipt_blob.py` has 7 tests; the seventh is the one that kills the mutant
I re-ran the mutant while writing this	baseline 7 passed. Truncating the comparison to `${BLOB:0:4}`: 1 failed, 6 passed, and the failure is `test_a_blob_prefix_is_not_a_blob`

What found it, and what did not

I wrote six tests for the fix. A receipt for the previous blob is refused. The
refusal names the blob. A legacy path-only receipt satisfies nothing. An honest
receipt passes. A stale line sitting first does not shadow a correct one below it. A
stale override is refused too, because an escape hatch that never expires is the
same hole.

All six passed. Then I mutated the guard: compare only the blob's first four hex
characters.

All six still passed. The hole was still open.

The seventh test is the one that kills that mutant, and its docstring says so:

Found by mutation, not by inspection: truncating the comparison to the blob's
first four hex characters survived every other test in this file.

Six is not a coverage number. It is how many tests I wrote before I felt finished,
and my sense of finished ran out exactly one test before the bug did.

I had already written the symptom down and filed it wrong

Two days before the fix, in a different commit, I described this:

passing needed two receipts for one file in a specific order, because the generic
loop reads the first matching line only. That ordering was load-bearing and
recorded nowhere.

I noticed that line order was deciding outcomes. I filed it as a documentation gap.

It was the bug. Inspection had the evidence in hand and did not convert it into a
failing test, because inspection asks what I believe the code does. Mutation does
not ask. It breaks the code and reports which assertion noticed.

What I might have wrong

The hook's own header states the residual and I am not going to soften it: git commit --no-verify skips every client-side hook, so no local hook is a complete
policy boundary. This gate stops an accident. It does not stop a decision.

I know the window and not the damage. Four days is a fact about dates. I did not go
back and check whether any claim actually shipped through the hole while it was
open, so I cannot tell you the blast radius, only its duration.

The receipts file is gitignored, so my own migration state is not independently
checkable by a reader. Take that part on my word or leave it.

And the mutation was hand-applied by me to a shell script, which means the operator
set is whatever I happened to think of. That is the same limitation that produced
the six-test blind spot in the first place, one level up.

If you have run mutation testing against your guard code rather than your product
code, I would like to know what it killed that you were sure was fine.

The gate, its tests and the fix: https://github.com/egnaro9/egnaro9.github.io

Verified by re-running the mutant against the committed hook while writing this. The judgement of whether it was worth saying is mine.

My arithmetic grader reads the first number in the sentence, not the answer

Erik Hill — Wed, 30 Sep 2026 15:55:43 +0000

Originally published at erikhill.dev. The numbers below are checked against the repository they come from.

My arithmetic grader reads the first number in the sentence, not the answer

This is a finding about my own harness. No model got the arithmetic wrong.

What happened

A week ago I published a post saying three of my factual-recall tasks were scoring
format rather than facts, and I ended it with a guess I had not tested. The guess
was that models read Answer with only the city name as a constraint on the output
and Two letters only as a description of the answer, and that the difference
decides whether you get a bare token or a sentence.

This week the same suite's arithmetic grader failed Claude Opus 4.8 on sixty
consecutive clean runs of a task that asks what 0.3 minus 0.1 is.

So I went and looked, and the guess held in a place I did not plant it.

The defect

def _number(expected: float, tol: float = 1e-6):
    def g(out: str) -> bool:
        m = re.search(r"-?\d+(?:\.\d+)?", out.replace(",", ""))
        return m is not None and abs(float(m.group()) - expected) <= tol
    return g

re.search returns the first match. So the grader reads the first number that
appears anywhere in the output and compares that to the expected answer. If a model
restates the question before answering it, the first number in the output is a
number from the question.

Seven of the eight tasks graded this way put a number in the prompt that is not the
answer. Compute 3 + 4 * 5 expects 23 and leads with 3. What is 15% of 200
expects 30 and leads with 15. What is 0.3 minus 0.1 expects 0.2 and leads with
0.3.

The evidence

claim	how it was checked
the grader takes the first number, not the answer	`modeldrift/suite.py:64`, `re.search(r"-?\d+(?:\.\d+)?", ...)`
7 of its 8 tasks lead with a number that is not the answer	enumerated over `SUITE`: math-order 3 vs 23, math-percent 15 vs 30, reason-older 30 vs 25, multi-step-math 3 vs 8, unit-minutes 2.5 vs 150, compare-negatives -0.5 vs -0.05, float-subtract 0.3 vs 0.2. Only compare-decimals leads with its own answer
it bites on runs where nothing errored	`float-subtract` fails on 9 models across runs with `reliability: 1.0`, worst Claude Opus 4.8 at 60 of 60
every model had the right answer	live calls through the probe's own path: gpt-4o-mini `'0.3 minus 0.1 equals 0.2. To one decimal place, the answer is 0.2.'`; opus-4-8 `'0.3 minus 0.1 is 0.2.'`; sonnet-5 `'0.3 minus 0.1 equals 0.2'`
changing only the instruction flips the outcome	same arithmetic, three wordings, three models, nine calls: as-shipped FAIL x3 with first=0.3; `Give only the number` and `Number only` PASS x3 with first=0.2
the same models pass the other number tasks in the same call	all three answered math-order `'23'`, math-percent `'30'`, unit-minutes `'150'`, bare, no restatement

What separates the task that fails from the seven that do not

This is the part that makes it more than a bug report.

The tasks that pass say "Give only the number" and "Number only". Those are
instructions about the shape of the output. The task that fails says "Give the
number to one decimal place". That is an instruction about the format of the
value, and it appears to invite the model to show that it has formatted it, which
means saying what it started from.

Last week's post guessed exactly that distinction from nine calls on three
factual-recall tasks, and said plainly that it was one story fitting nine calls. It
now fits twenty one calls, across two different grader families, with the same
split: an output-shape instruction gets a bare token, a description of the answer
gets a sentence.

I would rather have found it the other way around. A guess that survives a test I
did not design for it is worth more than a guess I go looking to confirm.

So I ran the experiment

The paragraph above used to say this was a pattern I had observed twice rather than
a mechanism I had demonstrated, and that the test was one sweep away. Here is the
sweep. Same arithmetic, same three models, three instruction wordings, nine calls.
The frozen suite was not edited, because its hash travels in published claims; the
variants were sent directly.

as-shipped  "Give the number to one decimal place"   FAIL x3   first=0.3
            opus-4-8     '0.3 minus 0.1 equals **0.2**.'
            sonnet-5     '0.3 - 0.1 = **0.2**'
            gpt-4o-mini  '0.3 minus 0.1 equals 0.2. To one decimal place, the answer is 0.2.'

"Give only the number"                             PASS x3   first=0.2   all answered '0.2'
"Number only"                                      PASS x3   first=0.2   all answered '0.2'

Nine for nine. The only thing that changed was the instruction, and it flipped every
row.

It also kills the alternative I could not rule out. I had written that 0.3 minus 0.1 is the canonical floating point example and a model might restate it because it
recognises the exercise rather than because of the wording. If that were the cause,
the restatement would survive Give only the number. It does not survive it once.

What I might have wrong

The experiment is three models and one task. It demonstrates that the wording
controls the restatement here; it does not establish how far that generalises, and I
would not carry the claim to a model family I have not called.

And the count needs a caveat. Seven of eight tasks are vulnerable in principle,
because their prompts lead with the wrong number. Only one of them is observed
failing at scale. The others pass because these models happen to answer them
tersely, which is not a property of my grader, it is a property of this month's
models. The grader is one style change away from failing all seven.

The fix is not obvious either, and I have not made it. Reading the last number
instead of the first would pass '0.3 minus 0.1 equals 0.2' and would also pass a
model that shows its work and lands on the wrong final value. Requiring a bare
number would be honest but would turn an arithmetic task into a formatting task,
which is the exact complaint this post is making.

If you grade numeric answers with a regex: check which end of the string you are
reading, and check it against an output that restates the question. Mine had been
reading the question back to itself for months, and the board recorded the result as
an arithmetic failure.

Live board: https://egnaro9.github.io/model-drift/
Suite, graders and runner: https://github.com/egnaro9/model-drift

The finding was surfaced by the drift detector. The live calls, the enumeration and the judgement of whether it was worth saying are mine.

Twice the data, less power: my stability rule got blinder the harder I looked

Erik Hill — Wed, 23 Sep 2026 23:16:26 +0000

Originally published at erikhill.dev. The numbers below are checked against the repository they come from.

Twice the data, less power: my stability rule got blinder the harder I looked

This is a finding about a measurement rule, not about a model. The models did not change.

What happened

I was comparing two models on a frozen 159 task suite and I had a problem most eval
code does not admit to having: the same model, asked the same question three times,
sometimes disagrees with itself. I had written a rule for that. A task where a
config does not agree with itself is discarded, not counted. Call it strict.

On the first three repetitions strict gave me Haiku ahead 7 to 1, eight
informative tasks, p equal to 0.070. Not significant.

So I looked at why, found that strict was throwing away thirteen tasks, and wrote
a second rule that tolerates a minority disagreement. Call it rate. It gave Haiku
ahead 13 to 2, fifteen informative tasks, p equal to 0.0074. Significant.

I had run the conservative rule, got a null, and then built the rule that returned
the result I wanted. That is also exactly what everyone who p hacks believes about
themselves.

The evidence

claim	how it was checked
the two rules disagreed on the same data	reps 1 to 3, suite `combined-159.json`: `strict` 7 to 1, informative 8, p 0.070; `rate` 13 to 2, informative 15, p 0.0074
the second rule was written after seeing the first result	stated in `runs/PREREGISTRATION.md`, committed before reps 4 to 6 existed, so the git timestamp is checkable against the run files' mtimes
the replication was pre-registered, analysis included	predictions, refutation conditions and fixed parameters in `runs/PREREGISTRATION.md`; the analysis in `tools/replicate.py`, written while the sweeps were still running
the prediction I got wrong, I got wrong in public	predicted `strict` would again fail to reach significance; on fresh data it reached p 0.0215
more repetitions raised the discard count	2 reps: about 8.7 discarded, about 10.3 informative. 3 reps: 13 discarded, 8 informative
pooling all six repetitions lost the result	reps 4 to 6 alone: informative 10, discards 13, p 0.0215, decisive. Reps 1 to 6 pooled: informative 8, discards 17, p 0.0703, not significant

What the extra data did not buy

This is the part I did not expect and the reason the post exists.

Every extra repetition is another chance to observe a config disagreeing with
itself, and strict discards a task the moment it sees one. So the discards climb
with the data. Thirteen became seventeen. Informative tasks fell from ten to eight.
The p value went from 0.0215 to 0.0703, which is to say a decisive result became a
null by adding measurements to it.

In the limit the rule throws away every genuinely stochastic task. More measurement
cannot fix that. Only a different rule can.

A conservative rule is not a free choice. It will never report noise as signal, and
the price is that it grows blinder the harder you look. That price is measurable,
and until I measured it I had been describing the conservatism as pure upside.

What the pre-registration was actually for

The rule I trust reached significance on data it had never seen. strict on reps
4 to 6 alone: Haiku 9 to 1, ten informative tasks, p 0.0215.

I had predicted it would fail again. The pre-registration says in advance what to
do about that: "If 1 holds and 2 fails, that is stronger than predicted, and I will
say the prediction was too conservative rather than claiming I called it." So that
is what I am saying. I did not call it.

The parameter is the part worth stealing. rate_margin was fixed at 0.5 in
advance, and afterwards I removed the flag from the command line entirely, along
with --alpha, because a threshold a reader can dial after seeing the result is
not a threshold. The registered value did not change. The knob did.

What I might have wrong

The weakest link is that this is one suite and two models. Haiku 4.5 beating Sonnet
4.6 here is a claim about a suite made of trap questions, not a capability ranking.
A suite built to catch specific slips measures susceptibility to those slips, and I
would not carry that sentence anywhere else.

The reading I cannot rule out is that rate is simply the better rule and strict
was never worth defending, in which case the whole discard analysis is an
elaborate defence of a mistake. The replication is evidence against that, since
strict reached significance on fresh data, but one replication at two models is
not much.

And I should be honest that the pooled result has an innocent reading. Reps 1 to 3
and reps 4 to 6 were run at different times, so pooling them mixes two sampling
windows. I attribute the power loss to the discard mechanism because the discard
count is the thing that visibly moved, thirteen to seventeen, but I have not run
the version of this that would separate those two explanations.

If you maintain an eval suite with a stability or flakiness filter: count what it
discards, and count it again after you add data. If the discard count rises faster
than your informative count, your filter is spending your sample size, and the
direction of that trade is not obvious from the code.

Suite, rules, pre-registration and replication: https://github.com/egnaro9/pi-eval

The numbers above are quoted from FINDINGS.md and runs/PREREGISTRATION.md in that repository. The judgement of whether this was worth saying is mine.

My factual-recall tasks were scoring format, not facts

Erik Hill — Wed, 23 Sep 2026 23:15:35 +0000

Originally published at erikhill.dev. The numbers below are checked against the repository they come from.

My factual-recall tasks were scoring format, not facts

This is a finding about my own harness. The suspect is the probe, not the models it measures.

What happened

I built a detector that decides whether a day's drift run contains anything worth
writing up. The first thing it did was accuse my own suite. One task,
fact-element, had failed on three or more providers on the same day, on 46
separate days.

The task is this:

What is the chemical symbol for gold? Two letters only.

Graded by exact match against au, after stripping whitespace and lowercasing.

Claude Sonnet 5 failed it 60 times out of 60. GPT-4o mini failed it 59 out
of 59. Llama 3.1 8B, 23 out of 23. Those are not knowledge results. A model
that scores 83% across a 35-task suite does not have a 100% failure rate on
whether gold is Au.

The evidence

claim	how it was checked
`fact-element` failed on 3 or more providers on the same day, 46 times	cross-provider flip analysis over every stored run in `dashboard/drift_board.json`
those were graded failures, not outages	every run counted had `reliability: 1.0`, so no call errored and none was truncated
the models know the answer	`gpt-4o-mini` answered `'The chemical symbol for gold is Au.'`; `claude-sonnet-5` answered `'Au\n\nThat's the chemical symbol for gold, derived from the Latin word aurum.'`
the same models pass a near-identical task	all three answered `fact-capital` with exactly `'Tokyo'`, 0 failures in 59 and 60 runs
one trailing period is the whole difference	`gpt-4o-mini` answered `fact-planet` with `'Mercury.'` and was recorded wrong 59 times out of 59
a model that formats tersely passes cleanly	`claude-opus-4-8` answered `'Au'`, `'Mercury'`, `'Tokyo'`, and fails `fact-element` on 1 of 60 runs

The answers above are not from the board. They came from nine live calls made
through the probe's own call path, printed with repr() so punctuation and
markdown are visible rather than inferred.

What stayed green anyway

fact-capital is the control, and it is the reason this is a finding rather
than a theory. It asks for the capital of Japan and every model in the table
answers 'Tokyo', exactly, on every run. Same grader, same strictness, same
models, same day. Nothing about exact matching is broken in general.

The three tasks differ in one way. fact-capital says "Answer with only the
city name", which is an instruction about the shape of the output.
fact-element says "Two letters only" and fact-planet says "One word",
which are descriptions of the answer. Models appear to read the first as a
constraint they must obey and the other two as a hint about what is being asked,
and then answer in their house style: a full sentence, a bolded token with an
etymology, a word with a period after it.

What this run could and could not have seen

The suite has 35 tasks, so the smallest accuracy change it can print is
100/35 = 2.86 points. fact-element and fact-planet together are 5.7 points
of every affected model's score, permanently, for reasons that have nothing to
do with the models.

Worse than the points: these tasks are tagged factual-recall, and that tag
feeds a per-category breakdown on the public board. GPT-4o mini's
factual-recall score has been reading as a knowledge number when two thirds of
it is a formatting number. Anyone comparing models on that column, including
me, was comparing how terse they are.

What I might have wrong

The part I am least sure of is the explanation, not the measurement. The numbers
are solid and the live answers are quoted verbatim, but "models treat answer
with only X as a constraint and X only as a hint" is one story that fits nine
calls. I have not tested it. The honest version is that I know the format
differs and I am guessing at why.

Two things genuinely argue against my reading. Grok 4.5 fails fact-element on
26 of 49 runs, which is close to a coin flip, and a rigid house style should not
produce that. And Claude Sonnet 5 failed fact-planet on 38 of 60 historical
runs but answered 'Mercury' cleanly when I called it today, so at least some
of this moves over time and is not a fixed property of the model.

There is also a real question about what the fix should be, and I do not think
it is obviously "loosen the grader". A grader that accepts 'Mercury.' also
accepts a model that ignores the instruction, and the instruction was part of
the task. Making the grader lenient would convert a visible measurement problem
into an invisible one. The alternative is to say plainly that these are
instruction-following tasks and retag them, which changes what the board has
been reporting for three months.

If you maintain an eval suite with exact-match graders on short answers: check
what your passing models actually return, not just whether they passed. The
board I built recorded which tasks failed for three months and never once
recorded what the model said, which is why this took a live call to see. That
gap is the more embarrassing half of this post.

Live board: https://egnaro9.github.io/model-drift/
Suite, graders and runner: https://github.com/egnaro9/model-drift

The finding was surfaced automatically by the drift detector. The investigation, the live calls and the judgement of whether it was worth saying are mine.

My mutation score was 1.000 in two places and the denominators disagreed

Erik Hill — Wed, 23 Sep 2026 01:24:56 +0000

I published a number, went to confirm it, and the confirmation disagreed with it. The number was fine. The confirmation was broken, and it was broken in a way I had already written a warning about.

The number

My evidence-bundle verifier has one metric I care about more than coverage: for every place in the code where the tool can refuse, is there a test that notices if I delete that refusal? So the sweep replaces each refusal site with a pass, one at a time, and reruns the suite. A site nobody catches is a gate that is not a gate.

CI, on the pushed commit:

baseline clean; 168 refusal sites (4 excluded, see EXCLUDE)
MUTATION SCORE: 168/168 = 1.000

The build fails under 0.99, so this is a gate rather than a dashboard.

The confirmation

I ran the same script locally, expecting 168:

MUTATION SCORE: 170/170 = 1.000

Same tool, same command, same score, two different denominators. 168 and 170 are both "everything", which is exactly why it is easy to miss. If the local run had said 0.94 I would have investigated in seconds. It said 1.000, so it looked like agreement.

Why

My working tree had 28 files that were not committed: 15 modified, 13 untracked. The verifier source itself was clean, so no new refusal sites had appeared in the thing being measured. What had changed was the suite and the fixtures around it, which decides which mutants get caught and which sites get excluded.

So the population moved. The sweep did not measure a different quality of the same system; it measured a different system.

The part I would rather not write

Here is a comment sitting in my own CI config, committed months ago, above the sibling checkouts:

# The score is only comparable if the suite is. Without these checkouts
# the real-bundle tests SKIP, four mutants stop being caught, and the
# denominator changes -- a mutation score that depends on what happens
# to be on the machine is the same host-dependence class this verifier
# exists to refuse. Check them out so CI measures what a developer does.

I wrote that. It names the failure precisely. And I still ran a local sweep to check a published figure, because checking felt like the careful thing to do.

Knowing a rule is not the same as being governed by it. The gate that actually held here was not my knowledge, it was CI, because CI had no choice about which tree it ran on.

What I changed

The number I put in front of other people is the one CI computed on the commit they can check out. Not the one my laptop computed on a tree only I have.

That sounds obvious written down. It was not obvious while I was doing it, because the local run felt like more diligence rather than less.

Two questions I would genuinely like answers to:

If you publish a metric out of your own repository, is its denominator stable across a clean clone and your working tree? Have you checked, or does it just feel stable?
Does your number get quoted from a CI log, or from whatever you last ran locally?

If you check and the two disagree, I would like to hear what moved. I suspect this is common and mostly invisible, because agreement on the headline figure hides disagreement underneath it.