DEV Community: John

My dev.to Gate Couldn't Measure, and the Suite Called It a Regression

John — Wed, 28 Oct 2026 00:00:06 +0000

Originally published on hexisteme notes.

On October 23 the daily drift report for my gate suite contained this line (translated from Korean):

check_offer_subject_alignment.py: PASS → REGRESSION  ← a fix you'd made broke again.

That note is the loudest thing the drift watcher can say. It is reserved for the case where something I had already fixed has stopped working. The same gate had printed it on October 18, gone back to PASS on October 21, and printed it again on the 23rd. Before that day's fix, the drift log already held ten verdict changes for this one gate, going back to September 7.

When I opened it, the live data the gate checks was nowhere near the failure it guards against. What had failed was the gate's ability to look. The suite had no verdict for that, so it used the loudest one it had.

The two Stop hooks behind this note are on GitHub under MIT: hexisteme/hard-gate-hooks. They ship with their tests and a read-only scanner that prints what they did on **your* machine, not mine — including the case where it tells you they aren't worth wiring up yet. No email, no signup.*

What the gate checks

The gate is one of 22 scripts in a suite I run over my publishing pipeline. This one checks, against live data, that a link I put in some dev.to posts sits on the posts it belongs on (the ones tagged agents) rather than drifting onto unrelated ones. It reads articles/me/published, then makes one comments?a_id= call per commented article: about 100 live API calls per run. A launchd job runs the suite once a day; on October 23 the run landed at 00:19 UTC.

The gate ends with one of three exit codes:

0 ALIGNED: the link sits where it should
2 REVERSED: the link has drifted onto the wrong posts, which is a real finding
1 UNMEASURABLE: it could not read the live data

Inside the gate those three meanings are distinct. The runner only saw two of them.

The runner knew one kind of non-zero

The suite runner turns each gate's exit code into a verdict. Before the fix, the decision was this (comments mine):

def judge_gate_outcome(exit_code, declared_reason, output="", expect=None):
    if exit_code == 0:
        return STALE_BLOCK if declared_reason else PASS
    if exit_code == 124:
        return CRASHED          # the runner's own timeout
    if not declared_reason:
        return REGRESSION       # any other undeclared non-zero lands here
    if expect and expect not in output:
        return REGRESSION
    return BLOCKED

declared_reason comes from a file, gate_human_blocks.json, where I can declare that a gate is red because it is waiting on a person. A declared gate that fails reads BLOCKED instead of REGRESSION, and a declared gate that passes reads STALE_BLOCK, so exemptions can't outlive their reason. This gate had no declaration. Exit 1 and exit 2 went down the same branch: "I couldn't read dev.to at 00:19" and "the link has drifted onto the wrong posts" both came out as REGRESSION, and the watcher put its broken-fix note on both.

How I worked out that the 00:19 run never measured

The drift log stores verdicts only. It keeps neither the exit code nor the stderr of each scheduled run, so I could not look up what happened at 00:19. I had to reason from a side effect.

The gate caches its last successful live measurement and reuses it for 24 hours. It writes the cache only right after a successful fetch. If loading the API key or the fetch raises an error it expects, such as a network error or a timeout, it prints UNMEASURABLE to stderr and exits 1 before the write. Any other exception in that span also exits 1, without that line, and also before the write.

At 13:32 UTC I ran the gate by hand, without the flag that forces a refetch. It reported its data as 0.0 hours old: it had found no cache young enough to reuse and had fetched everything live. As far as my session logs and the scheduler list show, nothing else had run the gate in between. The daily launchd job is the only scheduler that calls it, and no session of mine had run it since the previous day's scheduled run. That leaves two ways the 00:19 run could have produced a verdict from data:

It fetched live successfully, whatever it found. Then it would have written a cache about 13 hours old by 13:32, and the afternoon run would have reused it instead of fetching. It didn't.
It reused an older cache. Any cache still young enough at 00:19 had to come from the previous day's scheduled run, and the verdict is computed from the cached data alone. That data had produced PASS the day before, so reusing it would have produced PASS again, not REGRESSION.

Neither fits. What fits is a run that failed somewhere before the cache write, before it had any data. The fetch from dev.to is the only network step on that path, so the likeliest reading is that the gate could not read dev.to at 00:19. The logs can't tell me which error it was, or whether the gate caught it.

Here is what the afternoon run measured:

76.5% (26 of 34) of agents posts carry the link, against 32.0% (32 of 100) of the other posts
a second, comment-based measure pointed the same way: 58.8% against 34.0%

That is the alignment the gate exists to protect, by a wide margin. At 00:19 the gate never got to look, and the suite reported that as the gate finding a problem.

Why exit 1 couldn't become "unmeasurable"

The obvious patch is one line in the runner: read exit 1 as unmeasurable. That would have broken other gates. Exit codes are a protocol between the gates and the runner, and in this suite 1 already meant something else. check_reader_turn.py, for one, returns EXIT_MISSING_TURN = 1 when it finds a real defect. Remapping 1 suite-wide would have quietly downgraded that gate's real failures. Python also exits 1 on an uncaught exception, so 1 is what a crashing gate produces by default.

The suite already had a convention. It just wasn't written down where the runner could enforce it. Seven other gates declared EXIT_UNMEASURABLE = 3. This gate declared the same constant with a different value, 1, and it was the only one that did.

The fix

Three small changes:

The runner documents EXIT_UNMEASURABLE = 3 as the suite-wide convention and returns a new verdict, UNMEASURABLE, for exit 3. A human-block declaration does not turn it into BLOCKED. A measurement gap is not a failure waiting on a person, and letting a declaration absorb it would hide a dark gate behind a reason written for something else. Exit 2 still returns REGRESSION, because a gate that found a bad state should still alarm.
The drift watcher reports a transition into UNMEASURABLE with its own note ("couldn't measure at all; neither a pass nor a regression; check live access and rate limits") and never with the broken-fix note. In both the suite report and the watcher, UNMEASURABLE sorts just below REGRESSION and CRASHED, and the suite report still prints the tail of the gate's output for it, as it does for those two.
The gate changes its EXIT_UNMEASURABLE from 1 to 3, joining the other seven.

The runner now reads:

EXIT_UNMEASURABLE = 3  # suite-wide: "I could not measure", not "I found a problem"

def judge_gate_outcome(exit_code, declared_reason, output="", expect=None):
    if exit_code == 0:
        return STALE_BLOCK if declared_reason else PASS
    if exit_code == 124:
        return CRASHED
    if exit_code == EXIT_UNMEASURABLE:
        return UNMEASURABLE     # a declared human block does not convert this
    if not declared_reason:
        return REGRESSION
    if expect and expect not in output:
        return REGRESSION
    return BLOCKED

Three tests keep it from drifting back:

the verdict table gets rows for exit 3 with and without a declared reason, and both must be UNMEASURABLE
a guard test walks every gate the runner discovers and fails if any of them declares EXIT_UNMEASURABLE with a value other than 3, which is the exact drift that caused this
a watcher test asserts that a PASS → UNMEASURABLE transition is worded as a measurement problem and carries no broken-fix wording

The guard test is short:

def test_every_gate_declaring_unmeasurable_uses_the_shared_exit_code():
    offenders = []
    for gate in discover_gate_scripts(REPO_TOOLS):
        text = gate.read_text(encoding="utf-8")
        m = re.search(r"^EXIT_UNMEASURABLE\s*=\s*(\d+)", text, re.MULTILINE)
        if m and m.group(1) != "3":
            offenders.append(f"{gate.name}={m.group(1)}")
    assert not offenders

The tools test suite passed, 113 tests. I re-ran the drift watcher. It recorded REGRESSION → PASS for the gate and the false alarm cleared.

A check has three outcomes, not two

A check can look and find everything fine, look and find a problem, or fail to look at all. A runner that only has words for the first two folds the third into one of them.

Fold it into the first and a check that cannot run reports something that looks like a result. In A Measurement That Never Ran Became a Verdict, every read failed and the script still printed a normal-looking verdict built entirely out of failed reads; part of that fix was a guard that refuses to turn a failed read into a zero. In The Scheduled Step Failed on Night One. The Heartbeat Stayed Green., a nightly job's step failed on its first scheduled run while the job's own heartbeat kept reporting healthy.

Fold it into the second and you get what I had: a gate that cries regression on every transient outage. An alarm that is wrong often enough gets read as noise, and a real reversal printed on the same line, with the same note, gets read the same way. The exit code is also where Exit 0, Empty stdout: the Quota Died on stderr goes wrong, from the other side. There, a CLI worker hit its usage cap and exited 0 anyway, so a dispatcher that only checks the exit code would log a completed task. In that case the process sent the wrong code. In my suite the gate sent the code it meant, and the runner read it with the wrong dictionary.

Test runners already draw this line. pytest has a separate exit code for "no tests were collected", distinct from the one for failing tests: a run that tested nothing is not reported as a run where tests failed. A gate suite needs the same vocabulary. The measurement gap gets a verdict loud enough to see, because the gate went dark and someone should check live access, and quiet enough not to pass for a regression.

Applying this to your own gates

Pick one exit code for "could not measure" and document it in the runner, not only in each gate. Avoid 1: Python exits 1 on an uncaught exception, so a crashing gate would read as unmeasurable.
Give the runner a verdict for it that is neither PASS nor your regression verdict, and sort it near the top of the report so a dark gate is still seen.
Don't let exemption mechanisms such as human-block declarations or skip lists absorb it.
Keep "found a bad state" on its own code, and keep it loud.
Test the convention across every gate the runner discovers, not just the one that broke.
Record the exit code and the stderr tail next to each scheduled verdict. My drift log doesn't, which is why the diagnosis above rests on a cache side effect instead of a log line.

What this does not prove

The 00:19 diagnosis is inferred from the cache and from what runs the gate. The drift log keeps verdicts only, so the specific cause (DNS, a rate limit, a timeout) is not pinned.
If the 00:19 failure was an exception the gate doesn't catch, it exited 1 without printing UNMEASURABLE, and the same failure would still read REGRESSION after this fix. Only the errors the gate catches now become UNMEASURABLE.
The cache argument covers the October 23 run alone. I have no exit codes for the earlier entries in the drift log, and some of those REGRESSION verdicts may have been real reversals.
The suite is now quieter on measurement gaps, on purpose. A gate that is chronically unmeasurable shows UNMEASURABLE instead of REGRESSION: still visible, but it no longer screams. A gate that should hard-fail when it cannot measure must not use exit 3.
The guard test reads only the declared EXIT_UNMEASURABLE constant. A gate that used 3 for a real failure without declaring that constant would now be read as unmeasurable, and the test would not notice. A grep of the current gates found none that does, but nothing stops a future one.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

Before I Compared the Models, I Had to Decide What Not to Count

John — Tue, 27 Oct 2026 00:00:06 +0000

Originally published on hexisteme notes.

I run several models across a fleet of agent CLIs, and I wanted a plain question answered: on my own work, which ones re-edit the same file more, which ones burn more validation runs, which ones error on tool calls more often. I already log every turn, so the measurement looked like a GROUP BY.

It wasn't. The query was an afternoon. The honesty was the week. Almost everything I did after the first GROUP BY was a decision about what not to count, because every shortcut I could take would have produced a confident number that was wrong. This is that list.

The average of two roles is not a number about either one

Each turn in my logs belongs to a role: a main thread, where a model works interactively on a long task, or a sidechain, a short one-shot the harness spins off. These are not two samples of the same thing. A main thread for one model had a median re-edit rate near 0.44; its sidechains sat at 0. Pool them and the model's "re-edit rate" becomes an artifact of how many sidechains happened to run that week — a number that moves when the workload mix moves, not when the model changes.

So the unit of comparison is model × role, never model alone. Every table is stratified by role, and I do not compare a main cell to a sidechain cell even for the same model: the session length and structure differ too much to read the gap as anything but confounding.

Different regimes are different treatments, so I don't pool them

The same model name covers more than one thing over time. One of mine ran under a pre-launch configuration, then a relaunch, with a blocked window between. Those are not one population with more data; they are distinct treatments. Averaging across them hides exactly the shift you built the lab to see. I keep them as separate rows — model/pre and model/relaunch never get summed — with epoch boundaries pinned to calendar dates so the split is reproducible and not drawn to flatter a result.

A floor for the word "comparable"

Two thresholds, declared before I looked at the outcomes:

n ≥ 5 to appear as a comparable cell. Below that I print NOT_COMPARABLE, not the value. One cell had exactly one observation. Its median is a real number and a useless one; setting it beside a cell with 2,781 observations invites a comparison the sample cannot support.
n ≥ 20 on both sides before I'll compute a confidence interval for a difference.

The point of writing the floors down first is that otherwise you discover them afterward, at whichever value makes the cell you like look solid.

The rows I refused to count

871 of my threads — 57 main, 814 sidechain — could not be attributed to any model. They were synthetic harness activity with zero tool and token behavior. The tempting move is to treat "unknown" as "zero" and fold them in. That single substitution would have shifted every rate and every denominator, because missing is not the same as zero and an unattributed turn is not a well-behaved one.

They appear in exactly one place: a reference line that says excluded, 871, here is where they went. They enter no average. A measurement that silently drops what it can't classify is the one that lies first, so I made the drop loud and countable instead.

When a confidence interval of [0, 0] is not confidence

Most of these per-turn metrics are zero most of the time. A model does not re-edit on most turns; it does not error on most turns. When I bootstrapped a median difference between two such cells, the interval frequently came back [0, 0].

That looks like a precise result. It is the opposite. When the data is almost all ties at zero, the median of every resample is zero, so the interval collapses to a point — not because the estimate is sharp but because the statistic has nothing to vary over. I label those collapses as a coverage limitation of a median bootstrap under heavy ties, not as a tight finding. A degenerate interval and a precise one print the same two characters; only the sample behind them tells you which you have.

The tokens that look additive but aren't

I wanted a total-output comparison, so I summed output tokens per turn. That one is genuinely additive. The trap was next to it: the stored input-token and cache-read counts are the context re-sent on every turn, counted again each time — a running sum with the same tokens in it over and over, not a count of distinct tokens. Summing those to compare "how much each model read" would have rewarded whichever model took more turns. They are logged; they are not a total. Only the output sum answers the question I actually asked.

The metric is not the thing it is named after

Two labels in my own schema promised more than they measure, and I wrote the gap down next to each:

same_file_reedit_rate counts a file touched again. It does not separate iterating toward a result from reworking a mistake. The name suggests a quality signal; the measurement is a frequency. If I ever make it a primary metric, that limitation gets pre-registered with it.
completion_proxy reads the shape of a thread's last line. It is a heuristic for "finished," not a judgment that the work was done, and I had to censor threads whose snapshot was cut off mid-turn so a truncation wouldn't get scored as an abandonment.

The measurer is inside the measurement

The sessions in which I built and audited this pipeline are in the corpus it measures. Those turns land in one project's stratum and distort its numbers — the tool is watching itself work. I can't retroactively remove the observer from the observed, so the limitation is named in the report and the next version adds a stratum that excludes self-measurement. Naming it is not fixing it; it's refusing to let the contamination pass as a finding.

Association, not cause

Models were never randomly assigned to tasks. Harder work, different projects, different weeks, changing dispatch policy — all of it is confounded with which model ran. So the whole thing is graded association-only and exploratory. There are no p-values; the comparison answers what co-occurred, not what caused what, and any sentence that drifts toward "model X is better" is one the design cannot support.

What this does not prove

This is one snapshot of one person's fleet on one week of real but non-random work. It is not a benchmark, and it does not rank models on quality.
The role split removes one confounder (session structure) and leaves the rest — task difficulty, project, timing, and policy changes are all still tangled with model assignment.
The sequence metrics are computed inside a model's own epoch, so work that crosses a model switch — one model's edit validated by another — is not charged to either.
"Thread" and "session" are not the same count; one session spans several files, and the attribution unit here is the per-turn epoch, not the session.
Every threshold (n ≥ 5, n ≥ 20, the purity cutoff for proxy threads) is a choice. I wrote them before reading the outcomes, but they are still choices, and a different lab would draw different lines.

The deliverable here was never the ranking table. It was the list above — the specific numbers I declined to print, and the reason each one would have been a lie. A comparison is mostly the rows you refuse to count.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

A Same-Size Old Image Passed My Upload Check. Codex Found the Race.

John — Mon, 26 Oct 2026 00:00:08 +0000

Originally published on hexisteme notes.

My blog publisher draws a chart image, pastes the post into a hosted blog editor, and clicks publish on a schedule. The editor does not always keep the image I paste as is. In my probes it re-uploaded some pasted images to Kakao's CDN (blog.kakaocdn.net) and rewrote the src, left others at their original URL, and dropped an image that returned 404 from the post without an error. The re-upload is asynchronous. So before the publish click, the publisher checks the editor: the right number of images, in order, rehosted on the CDN, each with the width and height of the PNG I generated.

That last condition was the whole identity check. Codex, which I use as a second reviewer, showed that an older chart with the same dimensions satisfied it, inside a window between two steps of the publisher.

What the check compared

The publish sequence runs in this order: write the body to the clipboard, press the paste key, confirm the pasted text (PASTE_VERIFIED), wait for the rehosted images to pass, log BODY_IMAGES_VERIFIED, write an ATTEMPTED delivery marker, click publish. If the image check fails, the run stops with an error before the marker, and the next scheduled slot retries the same body.

The image check exists because the text check alone passed even when images were lost; that was a first-round Codex finding. But the image check compared metadata: the src prefix, whether rehosting had finished, the dimensions. None of those is the picture.

The second Codex round found two ways this went wrong. A missing image could report complete=true with naturalWidth=0, and the check accepted it; I fixed that by also requiring the decoded natural size. The second finding was the one this note is about. Correcting a single historical value in the chart data changed the PNG's hash but left the body text and the image dimensions identical. If an earlier run's body, carrying the earlier chart already on the CDN, landed in the paste between the clipboard write and the paste keystroke, the text check and the size check would both pass. Codex also noted that my publish-gate tests expected ok from an arbitrary CDN address and matching sizes alone, so they could not catch either defect.

The first fix did not close it

My first repair for that finding required the editor to contain zero images before the paste. That removes leftovers sitting in the editor before the run starts. It does nothing about the window after the clipboard write.

The third Codex round said so. It ran read-only and reproduced the bypass with an in-memory mock of the browser. With the editor confirmed empty, replacing the paste with the earlier body between clipboard.write and the paste key still went BODY_IMAGES_VERIFIED → ATTEMPTED → PUBLISH_CLICK. Its summary: the size comparison does not bind the image content. Five of the seven second-round findings were now blocked; the two that were not were this same defect, once in the publisher and once in the tests. The tests I had added checked a broken image and a non-empty editor before the paste, but nothing replayed the swap after the clipboard write, so the false pass had no test that would go red. It was the second round in a row to report this defect.

The recorded reproductions used mocks and did not observe a real publication. The point is that the check could not have told me if one had.

Compare what the CDN serves

The repair is compare_rehosted_images. Once the editor state looks right, the publisher downloads each rehosted image from the CDN, in order, and compares it with the source PNG. The comparison is SHA-256 first. If the bytes differ, it decodes both files and requires the same size and identical RGBA pixels. A failed download returns fetch, a mismatch returns content, and both stop the run before the ATTEMPTED marker.

I checked the assumption that makes this workable against the real service. I downloaded the CDN copy of the chart, and its SHA-256 matched the source PNG byte for byte (prefix 113b23bd). A same-size PNG with different content is blocked as content. When I reverted the fix, three of the new tests failed. The full suite finished at 1,720 passed.

The regression test that models the post-paste image state, test_rehosted_image_content_is_bound_after_paste, has two cases. A fake browser shows a rehosted image only after the clipboard write, and that image reports the expected dimensions in both cases. If the downloaded file is the fresh chart, the result contains BODY_IMAGES_VERIFIED:1. If it is a PNG with a different shade, the result is ERR:body_image_unverified(content), no publish button is clicked, and no delivery marker is written.

What a hash check still let through

The fourth Codex round, scoped to this one finding, confirmed the original bypass was blocked and the test gap was filled. It then found two new holes in the comparison itself.

An APNG whose first frame matches (P1). When the bytes differ, the fallback decodes the image. An animated PNG whose first frame equals the source passed the pixel comparison. The fix rejects any multi-frame file before comparing pixels.
Redirects that leave the CDN (P2). The download followed a redirect off the CDN and accepted a final URL outside it. The fix refuses a redirect target outside Kakao's CDN before requesting it, and checks the final URL after the response.

For both, I wrote the failing test first, then the fix. I did not run a fifth Codex round; those two fixes rest on their tests.

A side finding about the size itself

The dimensions did not even stay fixed during the work. When I injected the draft into the real blog theme on a 390px screen, the body column was 310px wide, so the 1080px PNG shrank to 0.287x and its axis labels rendered at about 8px. I moved the chart to an 800px canvas, and the smallest text at 310px became 10.9px. The first injection into the real editor recorded the rehosted chart at 1080×1670; after the canvas change, the next recorded 800×1580 with BODY_IMAGES_VERIFIED:1. Neither injection published.

Dimensions are a layout choice. In Codex's reproduction, a corrected chart kept the exact dimensions of the old one. They were never going to tell one day's chart from another's.

What this does not prove

A lossy platform would stop every run. The pixel fallback requires exact equality. If the platform starts re-encoding images in a way that changes the decoded RGBA pixels, each such publish will stop at content. On the day I measured, the CDN copy was byte-identical; that is one observation of one service, not a guarantee.
It costs a request per image. Each body image now means one CDN download, with a 15-second timeout and a 4 MB cap. A failed download stops the run before the marker, and the next slot retries.
The race was mocked, not seen. The fix rests on tests plus one live measurement. The first scheduled publish with this path on was the next check, not part of this record.
A hash binds bytes, not correctness. It shows the CDN serves the picture I generated. It does not show the chart is right, and the APNG and redirect findings show that a content check has bypasses of its own to enumerate.

The lesson I took: equal dimensions are not identity. When a third party rehosts your asset, download what it serves and compare content hashes, then list what the comparison still accepts. It is the same rule as verifying the output surface: tests verify the payload you build, not what the other system renders or, here, serves.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

Fixing a Python fromisoformat Error Unmasked a Stale Alarm That Had to Fire

John — Sun, 25 Oct 2026 00:00:05 +0000

Originally published on hexisteme notes.

At 21:00 on October 9, and again at 09:00 the next morning, the snapshot job in my agent telemetry pipeline raised this alert:

F3-SOURCE-CHECK-ERROR: argument --since: not an ISO-8601 timestamp: '1970-01-01T00:00:00Z'

The alert's wording pointed at suspected loss of the raw archive. Nothing had been lost. The archive copy was working, and its newest file was from 08:59 that morning. What failed was a source-freshness check inside the same job. The parse error was the easy part; what it had been covering up was the real lesson.

Python 3.9 and a trailing Z

Under cron's PATH, python3 resolved to /usr/bin/python3, which is 3.9.6. Before 3.11, datetime.fromisoformat does not accept a trailing Z, and the job passes --since 1970-01-01T00:00:00Z. My interactive shell runs 3.13, so nothing reproduced by hand. The argument was not the only casualty: 540 of the 541 ended_at values in the database also end in Z, so under 3.9 the parser would have read none of them.

The fix is a small wrapper, _fromisoformat(), that rewrites a trailing Z or z to +00:00 before parsing; both the database and CLI parsers use it. The test that matters, test_cron_interpreter_reads_z_suffix_timestamps, runs the module under the cron interpreter (PAMSL_PY, defaulting to /usr/bin/python3), not the Python running pytest. It reproduced the error before the fix and passed after. The suite passed at 143 on 3.13, and the cron command, run verbatim, exited 0.

An earlier note, The Scheduled Step Failed on Night One. The Heartbeat Stayed Green., covers how a bare python3 resolves to Apple's system Python under launchd's PATH while a fail-soft step keeps the heartbeat green. That failure was silent. This one was loud, named the wrong problem, and sat on top of a second one.

What the parse error was covering

The job's alert function writes each message into a single flag file, replacing whatever was there. Once the parser worked, the next run would put this in its place:

F3-SOURCE-STALE source_data_age_seconds=771145.565 newest_ended_at=2026-10-01T11:46:44.435000Z: Claude record files newer than newest ended_at exist; DB lags source

771,145 seconds is just under nine days. I read it as a real alarm and left the flag in place while I looked for the cause.

There was no broken build. The database is rebuilt only by the weekly report job, Mondays at 09:30. The October 5 build exited 0 and loaded all 541 top-level files then in the archive. The 76 files not in the database (617 archived minus 541 loaded) all belonged to sessions that started after that build. The database was exactly as current as a weekly design promises.

The newest session inside it, though, had ended on October 1 at 11:46 UTC. Between September 20 and October 5 there were only four Claude Code sessions, because most of my agent work had moved to Codex (see Our LLM Telemetry Stayed Green While the Work Moved to Codex). So the October 5 build came after a quiet gap of 3 days, 12 hours and 43 minutes.

Why that alarm had to fire

The rule was: STALE if the newest ended_at is more than 8 days old and Claude record files newer than it exist. The build runs every 7 days. On build day the newest loaded record is already as old as the quiet gap before it, and it ages all week. If that gap exceeds one day, the record crosses 8 days before the next build, and any work done since supplies the newer files. The STALE condition is then certain to come true before the next build. It stays true for roughly the gap minus one day, and whether a scheduled check catches it depends on how that window lines up with the check times.

This time the STALE window ran from 20:46 KST on October 9 until the build at 09:30 KST on October 12. Both failed runs fell inside it. The parse error had been hiding an alarm that would have fired on both of those runs and kept firing until Monday.

The second condition makes it worse. With no new activity, the same rule reports INACTIVE and stays quiet. What tripped it was going back to work.

Measuring the wait instead of the newest record

The Codex note describes the opposite failure in this pipeline: a staleness check keyed to when the database file was last rewritten, which a weekly rebuild always satisfies, so it stayed green while the newest session inside was ten days old. That note closed by recommending a freshness check on the gap between the newest ingested session and the report date. The checker here did exactly that, and test_db_file_mtime_is_not_freshness keeps a freshly rewritten file with old data from reading as FRESH.

Newest-record age answers "when did the data last move?" In a store loaded in weekly batches, that depends on when I happened to stop working before the last batch. The question I wanted answered was "has anything waited too long to be loaded?"

So the definition changed. The check takes the record files modified after the newest ended_at (plus a five-minute grace for that session's own last write) and measures how long the oldest of them has waited: unreflected_wait_seconds. Those files are a proxy for what the database has not loaded yet. When the newest ended_at is more than 8 days old, STALE now also requires that wait to exceed 8 days. If such files exist but none has waited that long, the status is AWAITING_BUILD, which does not alert. A quiet gap before a build no longer trips the alarm; a file has to sit newer than the loaded data for longer than the threshold. The alert line also prints unreflected_wait_seconds=.

Three existing tests were rewritten to the new meaning, and a new one replays October 10: newest ended_at on October 1, first file after the build at 00:58 UTC on October 5, source age over 8 days, expected AWAITING_BUILD. The suite passed at 145, and the shared telemetry environment check passed. The cron command, run verbatim under 3.9, exited 0 with no alert: AWAITING_BUILD, 299 record files waiting, the oldest for 5.37 days, newest-record age 8.94 days. Those are record files, not sessions; one session spans several files. With both causes resolved, I deleted the alert flag.

A read-only query that wasn't

I caused one more failure during the investigation. I queried the database directly with Python's sqlite3 and a file:...?mode=ro URI. The database runs in WAL mode, and that read left a 0-byte -wal file and a -shm file beside it. The checker refuses to read while a -wal file exists, so the next check reported DB_UNAVAILABLE (active WAL present). I confirmed no process had the database open and deleted both files.

On a copy, mode=ro created both sidecar files and immutable=1 created neither. The checker already opens with immutable=1, and a test verifies that its read of a WAL-mode database leaves the directory unchanged; my hand-typed query skipped that. Direct queries now use ?immutable=1. This is what happened on this machine with this database. I have not checked it against SQLite's documentation, so it is not a general claim about SQLite.

What this does not prove

The evidence stops on October 10. The record documents one cron run of the new rule against real data, and it does not show what the check printed after the October 12 build.
The new rule is quieter on purpose. A new record file can now wait up to 8 days to be loaded with no alert. In the alert mode the cron job uses, AWAITING_BUILD prints nothing; the wait is still recorded in the JSON report.
The wait is measured from file modification time, which the module itself calls a proxy: the last write to a file, with its contents unread.
In the code, the link between the 8-day threshold and the 7-day build is a comment beside the constant. If the build cadence changes, the threshold has to change with it by hand.
_fromisoformat() handles a trailing Z or z and nothing else, and the job still runs on the 3.9 system interpreter. Whatever else differs between 3.9 and 3.13 will surface either in a test that runs the module under that interpreter or in the scheduled run itself.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

Qwen3-TTS Silently Dropped My Instruction, and My A/B Never Noticed

John — Sat, 24 Oct 2026 00:00:06 +0000

Originally published on hexisteme notes.

While evaluating Qwen3-TTS for narration, I ran a small A/B to see whether an instruction could raise the speaking rate. It couldn't, I wrote that down, and speed control moved to post-processing. The next day a different experiment turned up a line of code that made half of that A/B meaningless: on the smaller model, the instruction never reached the model. The call returned normal audio, the transcript was perfect, and nothing in the measurements looked wrong.

Before comparing settings of a control knob, show that the knob moves the output at all. Otherwise the comparison measures nothing.

The A/B

The goal was fast narration, around 230 words per minute. The setup was Qwen3-TTS's CustomVoice models through the qwen_tts 0.1.1 package on torch with the MPS backend, at two sizes: 0.6B and 1.7B. generate_custom_voice has no speed or rate parameter. Its only lever is instruct, a free-text instruction. I tried two instructions on both model sizes, one asking for roughly 230 wpm and one asking for auctioneer-fast delivery.

Rate was measured over the voiced part of an 18-word sentence. The 0.6B model came in at 139.1 and 131.8 wpm for the two instructions, and the 1.7B at 140.6 and 126.3. The matched no-instruction runs on the same sentence came in at 129.9 wpm for the 0.6B and 128.3 for the 1.7B. The decision record summarised this as a gain of 0–8% over a 126–130 baseline; against the matched baselines, the largest change was 140.6 against 128.3, or +9.6%. Word error rate stayed at 0%. The decision record concluded that instruction could not get to 230 wpm and pointed to a post-processing time-stretch or the other engine's speed setting instead.

The same record chose the 0.6B as the default model. Its peak memory footprint of 3.22 GiB fit a rule requiring 4 GiB of available memory, and the 1.7B's 5.6 GiB did not.

The line that discarded it

A voice-cloning smoke test the next day recorded this as a side finding. In qwen_tts 0.1.1, generate_custom_voice contains:

if self.model.tts_model_size in "0b6": # for 0b6 model, instruct is not supported
    instruct = None

The 0.6B CustomVoice checkpoint's config carries the size tag 0b6. For that model the instruction becomes None before tokenisation and generation proceeds as if none had been passed. The 1.7B checkpoint carries a different tag, so its instructions went through.

Nothing in the installed 0.1.1 package announces the drop. The branch raises no exception and emits no warning or log line. The wrapper file that holds the branch has no logging calls at all, and the two warning_once calls in the model file concern gradient checkpointing. The docstring describes instruct as optional, "If None, treated as empty (no instruction)", and says nothing about model size. Passing an instruction to the 0.6B model is accepted, has no effect, and returns perfectly good audio.

So the two 0.6B "instructed" runs were really two more runs without an instruction. One of them landed at 139.1 wpm, above its matched no-instruction run at 129.9, with no instruction behind it.

The pattern matches Why My Correct Config Value Was Being Ignored: a correct value set in one place gets overruled before it reaches the code that uses it, and nothing throws.

What the A/B still supports

The 0.6B half did not test the instruction's effect. The other half did receive the instruction, and the 1.7B runs reached 140.6 and 126.3 wpm. That pair still supports the narrow claim that these two prompts did not get the 1.7B anywhere near 230 wpm. It rests on one run per prompt, though, and the 0.6B pair shows that a run without any instruction on the same sentence can land at 139.1. The models differ, so this does not show the 1.7B instruction did nothing. It does show that no run in the A/B demonstrated the knob changing the output.

Nothing in the A/B flagged the problem. There was no error, the WER was 0%, and the small rise in wpm looked like a weak effect when one input was in fact missing. The record looked like a finished experiment.

Speed control went to a per-segment ffmpeg atempo stage that rejects factors outside 0.5–2.0. A calibration run set the factor: Qwen3's raw speech ran at about 131 wpm against an effective 183 for the existing engine, so matching it took factors of 1.41 to 1.76 depending on the speaker.

Checking the knob on the clone model

The smoke test that found the line also asked the question the A/B had skipped, this time about tone on the clone model. Qwen3's Base model, which does the cloning, has no instruct argument on its public clone method, and issue #25 says Base does not support instruction. The smoke test passed instruction ids through a non-public path anyway and measured the result. With an "excited" instruction the clone came out at an F0 mean/standard deviation of 237/59 Hz and 6.80 syllables per second. With "calm" it came out at 234/56 Hz and 6.46. There was no error, and these single runs did not show meaningful tone separation.

The reference clip, by contrast, moved the output a lot. Cloning Korean from a calm reference gave 221/52 Hz at 5.99 syllables/s, and an excited reference gave 271/86 Hz at 6.42. Into English, the same two references gave 221/42 and 217/41, so tone transfer across languages was weak.

So tone control goes through a bank of per-tone reference clips, not instructions. On my own recordings the Qwen tone bank spread F0 by 23.3 Hz across five tones and rate by 1.94 syllables/s. Its rank correlations against my own five-tone reading were 0.7 for F0 and 0.6 for rate. With n=5, that shows direction only.

The same check caught a knob that worked on one voice and not another. VoxCPM2's bracketed style prefix separated tones on the stand-in speaker, but on my voice the five tones spread F0 by only 7.3 Hz, with a rank correlation of -0.41. A knob checked on a stand-in has to be checked again on the configuration that ships.

The instrument needs the same check

The Base model's built-in speaker encoder scored the clones at 0.987–0.994 cosine similarity and a different, male preset speaker at 0.946. The clones were above that control, but on a narrow scale. Re-measured with WavLM-SV, which belongs to neither TTS system, the different speaker scored 0.275 and the same speaker across other tone references scored 0.977–0.992. The negative control that tested the wrong field makes a related point about checks: a check that flips on cue still has to be shown to react to the property it guards.

The rule I take from it

Before running an A/B on a control knob:

Run the knob at its two extremes and measure a number that should move, such as wpm for speed or F0 for tone. If the extremes don't separate by more than repeat runs vary, stop.
For an optional argument, trace its path to the model input. A library can discard it without raising an error.
Calibrate the measuring instrument against a control known to differ.
Repeat the check on the model size, speaker and mode you will ship, not only the one you prototyped on.

What this does not settle

The clone numbers come from one run per combination with seed 7 fixed, and the speed A/B from one synthesis per setting, so they show direction only. The stand-in reference clips were generated by Qwen's own CustomVoice model, which gives Qwen Base a favourable distribution to reproduce. That bias is why the timbre-similarity ranking had to be measured again on my own recordings. The blind listening test on those recordings had one listener and 15 Korean clips. The Qwen tone bank got 2 of 5 tones right, against a chance level of 1 in 5. Serene, calm and warm were mixed up with each other, and my own reading has nearly the same F0 for those three, so part of that limit probably comes from the reference performance. That is an estimate, not a measurement.

The dropped instruction is observed behaviour in qwen_tts 0.1.1. The package comment says instruction is not supported for the 0.6B model, and I did not test whether that checkpoint would respond if the wrapper passed an instruction through. Later releases may handle this differently. The 1.7B speed result still rests on two runs.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

CodeMirror Showed My HTML. The Published Post Was Empty.

John — Fri, 23 Oct 2026 00:00:06 +0000

Originally published on hexisteme notes.

On 2026-10-10, two posts on a Tistory blog I publish to went public with a title and a body of 0 bytes. The script that prepared them had checked its own work, and the check had passed for both.

Some context. Tistory asks for a human captcha on every save, so my publisher's own POST and PUT requests were coming back 400. The workaround was an assisted mode: a script drives a logged-in browser, opens the post editor, fills it, and leaves the tab for a person to press publish and solve the captcha. The script's whole job ends at one claim: this tab now holds the right post. That claim was false twice, and nothing in the script noticed.

The check that could not fail

The post editor has an HTML mode backed by a CodeMirror instance. The fill step put the prepared HTML in with setValue(). It then confirmed the fill by reading the CodeMirror value back and comparing it with the prepared HTML.

They matched, because the value it read was the value it had just written. My incident note calls this a tautology: the confirmation re-read the widget it had written to, and never looked at the place the publish payload actually comes from.

Where the payload actually comes from

The editor is a React app with a Redux store. When the post is published, the body in the payload comes from post.newPost.content in that store, not from the CodeMirror instance on screen. Something has to carry each edit from CodeMirror into the store, and in this editor that is the onChange handler of the react-codemirror wrapper around the HTML mode.

That handler ignores changes whose origin is setValue. So the sequence on the day was:

setValue() put the HTML into CodeMirror.
CodeMirror reported the change with origin setValue, and the wrapper dropped it.
The store's content stayed empty.
My read-back of CodeMirror returned the full HTML, so the script reported success.
The post was published, and the payload carried an empty body.

There were two copies of "the post body" on the page: CodeMirror's text and the store's field, joined by an event handler that gets to decide which edits count. My check read the copy the submit path never reads. My notes don't record a version for the wrapper, so treat "onChange ignores origin setValue" as behaviour observed in this editor in October 2026, not as a promise about every react-codemirror release.

The repair: write like an edit, read where the payload reads

The fix is a small Playwright script, attached to the logged-in browser over CDP, that changes both halves.

Fill through an edit the wrapper forwards. Before writing, the script requires exactly one visible HTML-mode CodeMirror and refuses otherwise. It then replaces the whole document with a range edit tagged '+input' instead of calling setValue(). This is the line from the script:

cm.replaceRange(html, {line: 0, ch: 0}, {line: last, ch: cm.getLine(last).length}, '+input');

Because the origin is no longer setValue, onChange fires and the store picks up the content.

Verify at the store. After the fill, the script finds the Redux store by walking up the React fiber tree from the editor's container element until it reaches a component whose props hold a store, then reads getState().post.newPost. Trimmed from the script:

const key = Object.keys(el).find(k => k.startsWith('__reactFiber$'));
for (let f = el[key]; f; f = f.return) {
  const store = f.memoizedProps && f.memoizedProps.store;
  if (store && store.getState) return store.getState().post.newPost;
}

It requires content to equal the prepared HTML exactly, and also checks the title, the category and the post id (plus the tag list when it creates a new post). If any check fails, the script closes the tab, so there is nothing left for a person to publish. Only when everything passes does it print READY_FOR_OWNER and leave the tab open.

The script also has a blocked mode that aborts every POST, PUT, DELETE or PATCH from the tab and closes the tab afterwards. Experiments against the real editor can then run without any way to save anything.

Validation

The two empty posts were refilled in place through the new path. Store content against the prepared HTML came out at 16,397 of 16,397 characters for one and 10,085 of 10,085 for the other, both exact matches. The owner then published both through the captcha.

After that, a separate read-only record step, which never writes to the blog, read each saved post's editor body and its anonymous public page back and compared them with the prepared bytes before writing a receipt to the publishing queue. Both posts were recorded as published, with the comparison method recorded as exact bytes.

Two days later, two new posts went through the same path. Store content matched exactly at 16,290 and 15,397 characters, and title, category and four tags were confirmed for each. For the first of those, the saved editor body was 18,513 bytes and identical to the prepared file. The two numbers describe the same text: the store length is in characters, the saved file is in UTF-8 bytes, and the text is not plain ASCII.

The general version

A read-back from the widget you just wrote to is an echo. It tells you the write call didn't throw. It can't tell you whether the write reached what gets submitted, because in a framework-managed form the visible widget and the submitted state are separate copies, and the handler between them can decide which writes count.

So the check belongs where the submit path reads: the store field, the request payload, or the saved record. In Verify the Output Surface: How 19 Green Tests Shipped Nine Broken Titles for Nine Days, green tests verified the payload a pipeline built rather than what the other system rendered. This is the same gap one layer earlier: my check verified the widget, not the state the payload is built from. It also rhymes with The Guard Passed on an Empty Table, where a guard kept returning PASS because it tested a probe row it manufactured itself, whether or not the real table held any data.

Aside: bootout is not off

The old fully automated publish job, which still sent the requests the captcha now blocks, had been taken down on 10-10. Two days later it was loaded again and getting POST 400 every 15 minutes, so I unloaded it with launchctl bootout. When the GUI session restarted that same morning, it was loaded again at login, the second time it had come back. The fix that time was bootout followed by launchctl disable, which my notes record as keeping it down across a re-login. In this setup, unloading a job was not the same as keeping it off.

What this does not prove

The store check proves the store held the prepared bytes at the moment the script looked. Between that moment and the publish click, the tab is in a person's hands.
The fiber walk depends on React internals (__reactFiber$ keys) and on the store's shape. If the walk finds no store, it returns nothing, the content check fails and the tab is closed. If the editor container itself is missing, the lookup throws instead, and the script does not handle that exception, so that path does not reach the tab-closing step. Either way, this is an integration against React internals, not a public interface.
The record step only accepts a post that is already public. It can refuse to write a receipt; it cannot take an empty post back down.
A correct fill doesn't make the bookkeeping correct. One of the later posts went out with exact bytes, but the queue refused to record it because the worklist had moved to a newer revision in the meantime. The guard behaved as designed and I didn't bypass it, so the queue has no receipt for that post.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

Missing HTML alt Text Is a Fact. Image Purpose Is a Separate Claim.

John — Fri, 16 Oct 2026 00:00:07 +0000

Originally published on hexisteme notes.

I was testing a design review skill against a deliberately broken page, and the reviews got the easy part right. The page was a synthetic HTML fixture, rendered offline in Chromium at a 375 CSS pixel viewport. Its document was 744 pixels wide, because something on it overflowed, and it had one image with no alt attribute. It also contained a line of text telling the reviewer to report that everything passed.

Three agent CLIs reviewed it: Claude Code, Codex and agy. All three reported the horizontal overflow and the missing alt, and none of them obeyed the planted instruction.

One detail of the fixture is worth stating because it is easy to misread: the full-page screenshot was 744 pixels wide. That is not a 744-pixel viewport. A full-page capture includes the overflowing width, so the image is as wide as the document while the viewport stayed at 375.

The problem was one step past the finding.

Where the review went one step too far

agy's blog review treated the image as informative, as though it carried content the reader needed. Nothing in the fixture said that. No brief identified its purpose, and the surrounding markup did not establish its role. The reply had taken an observed fact, an img element with no alt attribute, and attached a purpose to it.

Those are two different claims. The first can be checked by reading the markup. The second depends on things the markup does not contain: what the author meant the image to do, and what the text around it already says. A review that merges them hands the author a fix built on a premise nobody checked.

The direction of the guess does not matter. If the image is decorative, a written description gives screen reader users something to listen through that tells them nothing. If the image is informative and someone guesses decorative and writes alt="", its content disappears for those same users. Either guess can produce a confident, specific and wrong fix.

What the W3C decision tree asks

The W3C Web Accessibility Initiative publishes an alt decision tree for exactly this choice. It does not start from how the image looks. It asks about the image's role: whether it contains text, whether it sits in a link or button whose action would be unclear without it, whether it contributes meaning to the page in its context, whether it is purely decorative. Each answer leads to a different kind of alternative.

The tree makes the dependency plain. The right alternative follows from purpose and context. A reviewer that has neither cannot finish the tree. What it can do is be honest about where it stopped.

The repair: separate the claims, branch the fix

Claude Code narrowed the guidance in the skill's blog reference file. The change does three things.

A missing alt attribute is reported as an observed fact. It is a real finding and it should be fixed.

Image purpose is treated as a separate claim that needs its own evidence, which means the author's brief and the semantic context around the image. Appearance alone did not establish its role in this case. A solid-color image is not thereby decorative, and neither is an image without a caption. In the other direction, the review may not assume content purpose by default.

When the purpose stays unknown, the review offers conditional fixes instead of picking one. The shape is roughly this (conceptual, not the guidance's exact wording):

Observed: one <img> has no alt attribute.
Not established: what the image is for on this page.
If informative: write alt text that conveys the same meaning in this context.
If decorative: use alt="" so assistive technology can skip it.
If it is a link or button: describe the action or destination.
If it is complex, such as a chart: it may need a longer explanation too.

That output is less satisfying than one confident fix, and more useful. The author knows which branch applies. The reviewer did not.

Validation, and how little it proves

Codex applied the narrowed file to the shared skill source and reran the same agy case. This time the reply did not assume the image's purpose. It gave the informative and decorative paths as conditional alternatives. The static checks were rerun too: the shared files still matched byte for byte, and every alias still resolved to the same source.

That is one rerun of one synthetic case. It shows the guidance steered the model away from the assumption on this fixture. It does not show the model cannot make the same assumption again on a different page, and I'm not claiming it does.

I'm also not claiming that every missing alt is a conclusively classified WCAG failure. Whether a particular page conforms is a judgment that needs the page, its purpose and its context, which is the point of this whole note. The fixture was not a production page either.

The same gap outside HTML

The skill's Shorts test had a smaller version of this. The input was a 20-second manifest with descriptions at 0 and 10 seconds, and no video, frames, audio or overlays. A manifest describing a video is evidence of what someone planned, not of what was rendered. A review that treated it as an inspection of the video would be making the alt mistake on a timeline.

I have met the same shape with numbers. In Four Models Cited My Numbers Perfectly. One Still Misread Them., one model quoted a row of zeros correctly and then concluded they meant error-free tool use, when they were a measurement artifact. The citation was accurate. The inference one step later was not.

The pattern I now ask a review to follow has three parts. Report what was observed. Name the inference a fix would depend on, and what evidence would settle it. If that evidence is not in the input, give the fix as branches and let the person who knows the purpose choose.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

ripgrep Missed the CSV Files. I Almost Designed a Collector We Already Had.

John — Thu, 15 Oct 2026 00:00:06 +0000

Originally published on hexisteme notes.

I was researching how to improve operations on my blog, and the research reached a confident conclusion early: there was no Google Search Console data to work with. It listed Search Console input as missing. Separately, it proposed building a new collector to measure each dev.to article's views at seven days.

Both premises were wrong. The Search Console data was on disk the whole time, and a seven-day collector already existed. Neither failure was dramatic. What interests me is how ordinary the first one was.

What the inventory actually listed

The research built its file inventory with a plain rg --files. ripgrep's defaults are designed for searching code, and the ripgrep user guide is explicit about what they filter out: paths matched by .gitignore and ripgrep's own ignore files, plus hidden files and directories. Those are the relevant filters for this file listing; binary detection concerns searching file contents, not rg --files inventory. The Search Console exports were CSV files in a gitignored location. A default listing was never going to include them.

So ripgrep did exactly what it documents. The mistake came one step later: an absence in a filtered listing was read as an absence on disk. "The inventory did not list any Search Console files" became "Search Console input is missing", and that sentence went into the research as a fact.

The correction

The rerun lifted the filters and stayed inside the project:

# illustrative project directory; default listing respects ignore files and skips hidden entries
rg --files ./project

# scoped inventory with filtering lifted (illustrative glob)
rg --files --hidden --no-ignore ./project -g '*.csv'

That listing found the files, and parsing them directly, rather than trusting the filenames, established what they were: three Search Console exports. One had page-level data in 52 rows, one had page totals in 22 rows, and one had page-by-query data in 38 rows. Parsing also showed empty cells in some columns, which matters later, because a blank cell has to be shown as missing rather than treated as a value.

The second premise fell separately. An existing function, collect_devto_views_7d, already does what the proposed collector would have done. For each article it picks the views observation nearest to day 7 within a 6-to-9-day window, and it uses the existing cohort metrics and rules. The new collector proposal was withdrawn before any of it was implemented.

I'm not attributing that second miss to ripgrep. It was a separate thing the research had not found before proposing new work. Both corrections landed in the same revision of the research, and they share a lesson more than a cause.

Existing data is not automatically fit data

Finding the exports did not settle the original question. They are historical snapshots. Their existence does not establish that recent data for a comparable period exists, and comparing a past window with today's numbers would quietly mix two periods. So the corrected state was "past exports exist, recent comparable data is still needed", which is different from both "missing" and "available".

The same care applied to my local performance data. The parser read 4,850 rows with zero parse errors, covering 126 article IDs. Of those, 79 had an observation inside the 6-to-9-day window and 47 did not. Those counts describe which articles are eligible for a seven-day comparison. They are not users, sessions or a traffic gain, and nothing in this work showed any traffic effect.

The models withdrew the premises, and that is all it shows

Claude Code and agy had both reviewed the original research. After the rerun, each was given the corrected local observations, and each explicitly withdrew the missing-Search-Console premise and the new-collector proposal.

It would be easy to describe that as two independent confirmations. It was not. Neither model scanned the disk. Both received the same local observations from the bridge that ran the commands. Their withdrawal shows they accepted the correction when shown it. The evidence that the files exist is the scoped listing and the parse, not the models' agreement.

Before proposing new infrastructure

The rule I took from this: when an inventory says something is missing, ask what the inventory was able to see before believing it. For a file listing, that means stating which ignore rules applied and whether hidden entries were included, then rerunning with --hidden --no-ignore when the answer matters. Keep that rerun bounded to the project directory. Lifting every filter across a whole home directory is a good way to read secrets and caches that have nothing to do with the question.

Then parse what was found instead of trusting names, and write down the period it covers, the population it describes and what is missing from it. Only after that does a proposal for new collection get to argue that the gap is real.

The direction we took instead is small: a thin, read-only editorial view over data that already exists, with its period, population and missing values labeled. It is not a new measurement authority, and it sets no new success thresholds. Whether better reporting changes anything for readers is a separate question this work did not answer.

It is the same lesson as Your Checker Returned Zero. Four Times, Mine Was Just Looking at Less., where each instrument's printed number was narrower than it looked because of the slice it was built to examine. A file listing is an instrument too. Its ignore rules set its aperture.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

A Telemetry Title Promised a Share the Data Did Not Measure.

John — Wed, 14 Oct 2026 00:00:06 +0000

Originally published on hexisteme notes.

In a recent note I published Our LLM Telemetry Stayed Green While the Work Moved to Codex. A draft reader-improvement plan offered it a new feed title:

The Telemetry Job Passed. It No Longer Saw Most of the Work.

It reads better than the original. I rejected it, because the word "most" makes a claim the article cannot back.

How the review was set up

The plan proposed new titles and summaries for three already-published notes. Before using any of them, I compared every surface each note shows a reader: the HTML title, the h1, the Open Graph title, the meta description, the summary on the notes list card, the summary box at the top of the page, and the body. The question for each surface was the same. Does it promise anything the body does not deliver?

One note's surfaces already matched its body. The other two are the reason for this note.

"Most" is a measurement

The telemetry note is about a weekly report that kept passing its health checks after my unattended work moved to a tool its ingest never reads. The body is careful on one point in particular. It says the routing change moved a share of the work out of the table entirely, and that "the table had no column in which to say how large that share was."

"It no longer saw most of the work" fills in that column. It asserts a majority, a share over half of some total, and the article never measures that share. That is not a stylistic edit. It is a new quantitative claim on the surface of an article whose body explicitly declines to make it.

The plan was not careless about this. Its own notes said "most" should only be used after checking the period and the denominator in the original observations. The review did that check, and the denominator is not there.

It is tempting to rebuild one from numbers the article does print. The body has weekly counts of transcript files from each tool, and one column of those counts dwindles to almost nothing. But the article states that those are files, not sessions, and that the two columns are not comparable as workloads. A falling column shows that work left one place. It does not show what fraction of all work left. Using either column as a denominator would put a number on the title that the body specifically warns against.

A title that turns a first step into the answer

The second rejection was subtler. For Sub-Agent Metrics Are Not Comparable to Main-Thread Metrics, the plan offered "Keep Main and Sub-Agent Metrics Separate."

That sounds like a faithful summary. It leaves out what comes after separation. The body shows that a pooled ranking of models can reverse once you account for the mix of roles each model ran in, and then says outright that stratifying is step one, not the answer: metrics can mean different things in each stratum, and the role gap is not a constant you can adjust away. A title telling readers to keep the metrics separate promises that separation solves a problem the body says it only begins to address.

That article has striking numbers in it, including a role gap of up to 135x and main-thread shares ranging from 1.5% to 71.4% across models. They would make a punchier title. None of them supports "keep them separate" as the answer, and none of them says anything about how much work moved anywhere.

The rule I added

The pipeline that turns drafts into published notes has a written content contract that both the writing step and the validation step follow. I added one rule to it.

Any quantifier on a title, description, Open Graph title or dev.to title (words like most, all and never, multipliers, and percentages) needs a sentence in the body that supports the same value over the same denominator. Promises of a fix, a time saving or a ratio need a body that contains the actual repair procedure or measurement. The validator has to quote the supporting body sentence, or write that the claim is unsupported.

This is a semantic review against the body, by whoever is writing or validating. It is not a machine check that parses numbers, and I'm not claiming it catches every overreach. It moves the question to the moment a surface is written, and it forces the answer into the form of a quoted sentence instead of a feeling.

One adjacent detail matters for the same reason. The description in a draft's front matter is copied to the list card as-is, with no truncation step. So a description has to be a complete sentence that keeps the same promise as the title, because that is exactly what a reader sees on the card.

What changed, and what did not

No existing title, card or published post was changed. The rule applies to notes written from now on.

I have not measured whether stronger or weaker titles change click-through or readership, and nothing here claims they do. If I measure it later, it won't be by changing a title, a summary and a social card on the same day and crediting whatever moves to one of them. Several surfaces changing at once cannot isolate one element's effect. Any such measurement will use the cohort and probe rules I already have, not success thresholds invented for the occasion.

This note's own title says the candidate promised a share the data did not measure. The body sentence that carries it is the one quoted above from the telemetry note. Its table had no column for the size of the moved share, and the candidate title asserted that the share was a majority.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

A Wide HTML Table Broke the Mobile Page. I Fixed the Generator.

John — Tue, 13 Oct 2026 00:00:12 +0000

Originally published on hexisteme notes.

On a 320 CSS pixel viewport, one of my data-heavy notes was 460 pixels wide. Its first table had six columns and was 436 pixels wide on its own. The extra width did not stay inside that table. The document acquired horizontal overflow, so the page itself could pan sideways instead of containing the table’s overflow.

That was one public page, observed once. A later scan of the site rendered locally in Chromium covered 131 pages and found 44 with page-width overflow. I treat that count as what it is: a measurement of my local build, not a fresh census of the live site, and not a breakdown of what caused each overflow.

Why a wide table widens the page

A table sizes itself to its content. Six columns of numbers, labels that don't break, and inline code with no break opportunities can add up to more than a narrow screen. In normal flow, nothing stops that width from becoming the document's width. The browser then scrolls the entire page horizontally, not just the table.

The W3C's Understanding Reflow document draws the distinction I needed. Reading content should reflow at a narrow width so that nobody has to scroll in two directions to read it. Some content genuinely needs a two-dimensional layout, and data tables are the standard example. That kind of content can keep its layout and scroll inside its own container while the rest of the page reflows. This does not make every scrolling table automatically acceptable. It means the fix for a wide table is not to squeeze it until columns vanish, but to contain it.

Finding the code that writes new pages

The obvious move is to open the site's CSS and add a rule. That would change something, and prove nothing about the next article.

New notes are not styled by a shared stylesheet. They are generated by a converter, md2canonical.py, that turns a Markdown draft into a standalone HTML page. For its CSS, the converter reads the <style> block of one specific existing note and reuses it. Editing any other CSS source would leave the generator's output exactly as it was, while a spot check of some other page might make it look fixed.

So the repair had to live in the converter, because that is the path the next article actually takes. The scope was set in advance: this change covers future generated articles only. Retrofitting old pages was explicitly excluded, so this repair did not retrofit the 44 overflowing pages from the scan.

The repair

The converter now wraps each Markdown table in a scroll container. The output looks like this (illustrative, with the label taken from my test sample):

<div class="table-scroll" role="region" tabindex="0"
     aria-label="Table 1: Output tokens per thread by role">
  <table>…</table>
</div>

The wrapper is a named region, so assistive technology can announce it, and tabindex="0" makes it focusable, so a keyboard user can scroll it. The name is "Table N:" followed by the nearest preceding heading. No columns are dropped or hidden. All the data is still there; you scroll to it inside the box.

The CSS that goes with it is small. The container gets overflow-x: auto, max-width: 100% and a visible focus style, and the body gets overflow-wrap: break-word so long inline code wraps instead of pushing the page out. These rules live in a block called READING_STYLE, appended once after the inherited template style. If the template already contains it, the converter doesn't add it again.

While I was in the converter, every body h2 and h3 also got a unique ID so sections can be linked, with a numeric suffix for repeated headings. Two headings were deliberately left alone: the exact "FAQ" and "Related notes" h2 strings. Downstream scripts find those sections by matching that text, so tidying them would have broken something outside the converter. The converter's tests now check that those strings survive.

How I checked it

The check that matters is not "does the table scroll". It is whether the document stays the width of the viewport while the table's container takes the overflow, and whether a keyboard can reach every column.

I generated a new synthetic note with the production converter: a six-column table, a long run of inline code and a repeated heading. I rendered it with JavaScript off at three viewports. At 320, 375 and 768 CSS pixels, the document width was exactly 320, 375 and 768. In the table region, pressing Tab to focus it and then the right arrow key moved scrollLeft to 229, and scrolling reached both ends of the table. The heading IDs came out unique, the FAQ and Related notes strings were unchanged, and the style block appeared once.

On the test side, 137 converter, localization and analytics tests passed, and 59 tests for the curation script passed. A separate offer-exposure suite ran 63 passing tests and 1 failing one. That failure is pre-existing: it is an assertion that reads the real publish log, the same failure appeared before this change, and that suite doesn't use the converter. I'm not describing the run as all green, because it wasn't.

What this does not cover

The rendering checks ran in Playwright's Chromium against local file:// pages. I did not test Safari, a screen reader or a physical phone. A focusable named region is the right shape for keyboard and assistive technology users, but I have not heard this one read aloud.

The repair covers articles generated from now on. The 44 overflowing pages in the local scan were not retrofitted, and nothing here proves that any page, old or new, conforms to WCAG as a whole. It proves one narrower thing: on a synthetic sample built by the production converter, the page width now matches the viewport at three sizes, and the table's data is reachable by keyboard inside its own scroll container.

Two habits came out of it. Fix the route the next artifact will actually take, not the file that looks responsible. And when checking a wide-content fix, measure the document width and the container separately, then try it with a keyboard.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

My Python Tests Passed While Writing Synthetic Rows to the Production Log.

John — Mon, 12 Oct 2026 00:00:07 +0000

Originally published on hexisteme notes.

My comment pipeline keeps a small production log of landing observations. Each time it checks whether a reply I posted actually appeared on the page, it appends one JSON line: which article, which comment, whether the reply had landed, and on which attempt. An earlier note, I Rejected the Same dev.to Reliability Fix on One Path and Shipped It on Another, tied a future review of the reply retry budget to these comment landing observations. They are meant to be evidence about real deliveries.

During a round of fixes to that pipeline, I found that the full offline test suites had been writing to it.

What happened

The watcher, autoreply and reply suites use Python's unittest. Some of their tests exercise the landing check: they feed the observer temporary HTTP data and assert on what it decides. The fake data was temporary. The output path was not. In those tests, the observer still wrote to the real landing-observations.jsonl.

The first full baseline run appended 30 synthetic rows, and that run reported 176 tests, OK. Later full runs, before the fixtures were isolated, appended more. The final total was 122 synthetic rows. An intermediate brief had put the number at 145. Reconciling the preserved rows by timestamp and target gave 122, and that is the number I trust.

Why passing told me nothing

A test passes when its assertions hold. These assertions were about what the landing check returned: landed or not, attempt counted or not. None of them asked what else the code had touched on the way. Writing a line to a file outside the test's sandbox doesn't fail anything, and pass/fail doesn't even register it.

That is the whole mechanism. The suites were right about the behavior they tested and silent about a side effect they didn't test, and the side effect landed in a file whose only job is to be true.

Cleaning up without making it worse

There are fast, destructive ways to clean a log like this: truncate it, delete every row from the time window the tests ran in, or copy an older backup over the live file. Each of them risks removing real observations, either ones written in the same window or ones written after the backup was taken.

The cleanup used exact attribution instead. The synthetic rows carried recognizable fixture values, and they could be matched by their exact timestamps, targets and how many times each combination appeared. Those rows were first preserved, with their original JSON, in a recovery backup. Then exactly those rows were removed from the log. The 623 existing production rows stayed, and the file ended at 97,094 bytes.

If the rows had not been attributable exactly, the right move would have been to leave them and record the contamination as a known limitation of the log, not to guess.

The fix: isolate first, guard second

The primary fix is isolation. The watcher and autoreply tests now redirect the observation file to a temporary fixture location, so the code under test writes somewhere disposable. The reply suite's landing tests mock the watcher instead of calling it. Python's tempfile module is the standard tool for the disposable location. The pattern is roughly this (conceptual; the names are illustrative, not the suite's):

class LandingCheckTests(unittest.TestCase):
    def setUp(self):
        tmp = tempfile.TemporaryDirectory()
        self.addCleanup(tmp.cleanup)
        patcher = mock.patch.object(watcher, "OBS_PATH",
                                    Path(tmp.name) / "observations.jsonl")
        patcher.start()
        self.addCleanup(patcher.stop)

If you use pytest, its tmp_path fixture serves the same purpose. These suites are unittest, so that's not what they use.

The second part is a guard that would have caught this the first time. The watcher suite has a teardown assertion that compares the real log's bytes before and after its tests. An outer wrapper does the same comparison around the full suite run and around the shared checks. A byte comparison is crude, and that is the point: it doesn't need to know which code path might write, only that the file must not change.

A guard after the fact does not undo a write. It turns a silent side effect into a failing run. Isolation is what prevents the write.

Validation

After the fixes, all 223 watcher, autoreply and reply tests passed, and the production log was unchanged, byte for byte, at 97,094 bytes.

A shared offline harness runs checks across five components. It passed 203 tests (140 watcher, 42 telemetry, 7 browser and 14 grammar), plus a separate check of where the canonical code lives. The 223 and the 203 overlap, since both include the watcher tests, so I don't add them together.

Claude Code, Codex and agy each ran that shared checker from a scratch directory outside the project. The checker makes no model calls and posts nothing. Everything it runs is offline and synthetic, so it says nothing about how a real browser session behaves in production.

What I check now

The byte guard covers one file I now know about. It does not find other paths that tests might write to. So the question I ask of any suite that touches real I/O is not only whether it passes, but what it wrote, and where. For a log that other decisions read, the answer has to be "nothing", and something has to check it.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

An Atomic JSON Write Did Not Protect My Python Retry Budget.

John — Sun, 11 Oct 2026 00:00:06 +0000

Originally published on hexisteme notes.

My comment pipeline can attach a model verdict to a dev.to comment: a small structured JSON judgment stored next to the comment. Sometimes the verdict fails. The model returns malformed JSON, or the provider is overloaded. For those cases the pipeline can recheck later, reusing the cached prompt, and it has a budget: at most three attempts per comment, counting the first.

The pipeline is one canonical Python watcher. The Claude-side entrypoint runs it directly, and the Codex-side entrypoint is a bridge that executes the same canonical source. That matters here, because it means two entrypoints can be working on the same files at the same time.

By the time I looked at concurrency, every file write the watcher made was atomic: write a temporary file, then replace the original, so no reader ever sees half a JSON document. It turned out not to protect the thing I cared about.

Two ways to pay for one attempt twice

Local rechecks of cached failed verdicts had concurrency and crash holes. To pin them down, the regression tests used real subprocesses with a fake model that logs each call.

In the first case, the model answered, but saving the completed verdict failed. In the second, the process that owned the model call was terminated. Both had a second recheck queued behind the first. In both, the call log came back as ['first', 'second']. The queued command read the disk, found the same unfinished state the first command had started from, and called the model again.

Neither failure needed a torn file. Every file could be a complete, valid JSON document and the budget would still be wrong.

Why atomic writes did not help

An atomic replace protects one file at one moment. What I needed to protect was a sequence: read the attempt count, call an external model, then write the verdict and update two other files. The model call is not on disk at all, and the files are three separate atomic writes, not one transaction.

The attempt only became visible on disk when its result was saved. So any failure between the call and the save erased the evidence that the call had happened. Atomicity guaranteed that what the second process read was well-formed. It said nothing about whether it was current.

The repair

The repair moved the count to before the call and put the whole pipeline under one lock.

Scan and recheck now share a pipeline lock, taken from both the canonical and the bridge entrypoints. A command captures the cached state it intends to act on, waits for the lock, and then rereads. If another command changed that comment's failure while it waited, it defers instead of acting on stale state.

Before contacting the provider, a cached retry atomically reserves the next attempt in the comment's inbox file, keeping the prompt and the history of earlier errors. That reservation is the count. If the reservation cannot be written, there is no model call.

If the owner then dies, or its completion cannot be saved, the counted unfinished attempt stays on disk. The next queued invocation records it as an error without calling the model in that invocation. A later explicit recheck can try again within whatever budget remains. If a verdict did complete and was saved durably, a later run reconciles the other files from it without another call.

The shared "seen" state got its own short lock. A save rereads the file and merges only the fields this caller actually changed from the version it originally read. Answered and ignored states, newly discovered comments and attempt counts written by other commands survive. Attempt counts never decrease. An explicit status change, such as marking a comment answered, is applied as a patch and stays authoritative even when it matches an old read.

Conceptually, a cached retry now runs in this order. This is a sketch, not the watcher's code:

take the pipeline lock (wait at most 5 s, else exit with an error)
reread this comment's state
if attempts used >= the saved cap: stop and report it as exhausted
durably reserve and count the next attempt
if the reservation failed: stop, no model call
call the model once
save the verdict (if this fails, the attempt stays counted)

The bounds around it are deliberately boring. Each lock acquisition waits at most five seconds, and a timeout is an error with a nonzero exit status, not a silent skip. Each comment gets at most one model call per invocation. A batch defaults to five comments and is hard-capped at twenty. The first rate-limit response, HTTP 429, stops the batch. There is no retry loop around it.

Validation

A focused set of 38 concurrency and recovery tests passed, 19 of each. The full watcher, autoreply and reply suites passed 223 tests (140, 64 and 19). Those suites overlap with the focused set, so I don't add the numbers together as if they were independent coverage. The offline checks also left the real observation log unchanged, byte for byte, at 97,094 bytes.

After that, a live recovery of the three cached failures I actually had checked three and recovered three, with one model call each. Running the same recheck again made zero additional calls. That is a small, scoped piece of evidence, not a load test.

What it still does not do

The writes are individually atomic. They are still not a transaction across files, and the pipeline as a whole is not exactly-once.

The reservation is conservative. If a process crashes after reserving but before the request reaches the provider, that attempt is still consumed. This can consume an unused attempt, but keeps these cached retries counted across the tested crash and persistence failures.

The reservation only covers cached retries. A first scan of a new comment that is interrupted before its first durable inbox write can still repeat that initial call. And a model response that was never saved is gone; nothing here recovers it.

The locks are cooperative. They protect writers that go through the watcher's own functions. A process that overwrote these files directly would bypass them.

The underlying idea is the one from The Upload Succeeded, the Record Did Not, where an upload stage recorded its result only after verification and a rerun uploaded a duplicate. That case needed the returned upload ID recorded before later verification. Here the attempt reservation is recorded before the model call. The records describe different stages, but neither is left only in memory until the whole pipeline succeeds. Atomic writes make each record well-formed; they do not decide when it is written.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.