DEV Community: Mayur Agnani

How to Fix Jargon in Speech-to-Text Without Adding Words Nobody Said

Mayur Agnani — Mon, 02 Nov 2026 10:34:12 +0000

Engineer's view. This is a search spellchecker with a custom dictionary. Add your product names and their misspellings get fixed, but nearby ordinary words get "corrected" too. A transcription prompt works the same way: a listed spelling wins wherever the audio is close to it. You get your jargon right and pay in confident wrong words.

TL;DR. Give a speech-to-text model a short list of your terms and it spells most of them right. It also writes those terms where someone said a word that only sounds alike, so test for both.

Explain Like I'm 5

Imagine a friend writing down everything you say at a noisy party. Before you start, you hand
them a card with the tricky names on it. Now they spell those names right.

But then someone says "Laura." Your friend glances at the card and writes the name from the card
instead.

The card helps your friend choose between words that sound alike. It cannot tell them which word
was actually said. So the card fixes some mistakes and makes a few new ones.

The Problem

You have shipped this bug before, in a system with no model in it.

Your search box had a spellchecker. Users typed your product names and got "no results," so you
added the names to its dictionary. The failed searches stopped. A week later, a support ticket
arrived. Someone searched for "laura" and got pages about your LoRA feature. The dictionary did
what you asked. It pulled every close word toward the words you listed.

A speech-to-text model is that spellchecker, running on sound.

You send it meeting audio, a support call or a video voiceover. It hears "Q drant" and writes
"Q-drant." It hears "K V cache" and writes "cavity." I read 15 AI terms to two OpenAI
transcription models with no help. whisper-1 spelled 14% of them right, and gpt-transcribe spelled
36% right. Nothing raised an error. The transcript just looked slightly wrong, and every search,
summary and caption built on it inherited the mistake. My own video captions hit this last week.
The voice said "halve" and the transcript said "have."

Both models take a hint: a text prompt, or a list of keywords. Add one and most of the jargon
comes out right. Some of it also comes out where nobody said it.

The fix: give the model a short list of the terms this recording is likely to contain, put the
important ones last, and test it on clips that say near-miss words, not only on clips that say the
terms.

The Fix: Keep the Term List Short, Ordered and Tested on Near Misses

What does the prompt actually change?

A transcription model has two halves. The encoder turns audio into features. The decoder
writes the text, one token (a word or a piece of one) at a time, like any language model. It
picks each token by how well it fits the sound and the text written before it.

The prompt goes in front of that text, as if it had been said just before the recording. Whisper's
open-source code marks it with a "previous text" token. It is not an instruction. OpenAI's guide
says Whisper copies a prompt's style and ignores its commands. So a list of spellings does one
thing: it makes those spellings more likely.

Why does it write words nobody said?

Because the boost never checks the audio, it wins any close call. With a 15-term list, whisper-1
spelled 90% of the terms right. On the same run, "Laura will call you back" became "LoRA will call
you back" in 6 of 6 clips. "Cash the check" became "cache the check" in 6 of 6. gpt-transcribe,
given the terms as keywords, wrote "Mixtral" for "mixed trial" in 5 of 6. Sentences that sounded
like no term never changed.

Doesn't a longer list cover more?

Not on Whisper. It keeps only the last 223 tokens of the prompt and drops the rest from the front.
My 75-term list was 277 tokens. With the 15 real terms first, the cut took 14 of them, and accuracy
fell to 19%. The same list with those terms last scored 92%.

What can a list never fix?

A list chooses between spellings the audio already supports. It cannot add an underscore nobody
spoke, so "max tokens" never came out as max_tokens. And it cannot separate two words that sound
the same. "Halve" stayed "have" on whisper-1 in every clip.

What This Means for You

When this matters. You transcribe audio that contains names, product terms or acronyms. That
covers meeting notes, call summaries, video captions, voice agents and search over recordings. It
matters most when code acts on the words: a search index, a ticket field, or a voice agent that
picks a tool from what it heard.

How it affects you. A term list is the cheapest accuracy win you have. In my test it moved
whisper-1 from 14% to 90% on jargon, for no extra cost per call. But it changes two numbers, and
most teams only watch one. A voice agent that hears "cache" for "cash" may call the wrong tool. A
long glossary on Whisper can quietly do nothing, because the terms you cared about were cut first.

What to do about it. Start with no setup. Take ten real recordings and search their
transcripts for your product names. Count the misspellings. That number tells you whether this is
worth an afternoon.

Next, send a short list with each request. Pick the terms this recording is likely to contain,
from its meeting title, its customer or its agenda, rather than your whole glossary. Put the
important terms last.

Then build a small decoy set: clips that say ordinary words close to your terms, such as
"Laura" for LoRA. Run it with and without the list. Ship the list only if term accuracy rises and
the decoys stay clean.

Finally, fix formatting in code after the transcript arrives, never in the prompt.

Implementing It

The change.

Four jobs, and one person may do all of them.

Job 1, whoever calls the API. Send the terms in the field each model reads. whisper-1 takes one
prompt string. gpt-transcribe takes literal terms in keywords, one form field each:

curl https://api.openai.com/v1/audio/transcriptions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -F model=whisper-1 -F language=en -F file=@call.mp3 \
  -F "prompt=Kubernetes, pgvector, SGLang, Qdrant, vLLM."

curl https://api.openai.com/v1/audio/transcriptions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -F model=gpt-transcribe -F 'languages[]=en' -F file=@call.mp3 \
  -F 'keywords[]=Qdrant' -F 'keywords[]=pgvector' -F 'keywords[]=vLLM'

A keyword containing <, > or a line break makes the API reject the whole request. Strip those
before you send.

Job 2, whoever owns the glossary. Build the list per recording, inside the budget, with the most
important term last. Whisper drops from the front, so the order is the priority:

def build_prompt(terms, budget=223):
    """terms: most important first. Returns the prompt and the terms that did not fit."""
    kept, used = [], 0
    for t in terms:
        cost = est_tokens(t) + 1  # +1 for the comma
        if used + cost > budget:
            break
        kept.append(t)
        used += cost
    return ", ".join(reversed(kept)) + ".", terms[len(kept):]

est_tokens deliberately overestimates. For an exact count, use the GPT-2 tokenizer, which is what
Whisper uses.

Job 3, whoever consumes the transcript. When a wrong term is costly, transcribe twice and diff.
A listed term that appears only in the prompted version is either a fix or an insertion. A person
or a second check decides which:

def suspect_terms(plain, prompted, terms):
    words = lambda s: set(re.findall(r"[\w-]+", s))
    pieces = {p for t in terms for p in re.findall(r"[\w-]+", t)}
    return sorted((words(prompted) - words(plain)) & pieces)

On my sample, it flagged all five fixes and both insertions. It compares exact case, so "LoRa"
becoming "LoRA" counts. It compares words, so "cash" becoming "cache" is caught even though the
listed term is "KV cache."

Job 4, whoever owns the eval. Score two numbers on every list change: terms spelled right on
clips that say them, and decoy clips with any listed term in them. score() in
code_example.py does both, and it runs on real transcripts from my test.

How you know it worked.

Term accuracy on your term clips should rise sharply. On whisper-1 it went from 14% to 90%. Decoy
insertions should stay at or near zero. Mine were 13 of 72 with the short list, all on decoys
built to sound like a term. If accuracy does not move on Whisper, count your prompt's tokens
first. A list over 223 tokens has lost its front.

When a Term List Is the Wrong Tool

When your vocabulary is huge. A catalog of 5,000 product codes will not fit in 223 tokens, and a
long keyword list raises the insertion risk. Transcribe without the list, then correct the text
against a lookup table, or with a text model given the catalog. OpenAI's guide suggests the second.

When the words are ordinary. Listing common words gives the boost nothing to fix and plenty to
break.

When you need high accuracy on one domain. NVIDIA reports a medical benchmark where a better prompt
cut the word error rate from 30.8% to 23.8%, and fine-tuning cut it to 11.8%. A prompt is the
first step, not the last.

When you chain chunks. Passing the previous chunk's transcript as the prompt carries its mistakes
forward as well as its spellings.

My test has limits. It used three synthetic macOS voices, 15 terms and one run per clip. Real
speakers mumble, and accents move the close calls. Run your own decoy set before you trust my
numbers.

Three questions before you ship a list:

Which of your terms sound like an ordinary word?
Does your prompt fit in the model's budget, with the important terms last?
What happens downstream when a listed term appears that nobody said?

Glossary

decoder — the half of a transcription model that writes text, one token at a time, like a language model.
token — a word or piece of a word; the unit a model reads and writes, and the unit prompt budgets count.
keywords — gpt-transcribe's field for literal terms you expect to hear; hints, not required output.
decoy set — test clips that say ordinary words close to your terms, used to catch insertions.
insertion — a listed term written into a transcript where the speaker said a different word.

How to Stop a Long LLM Answer From Freezing When the Server Runs Out of Memory

Mayur Agnani — Sun, 01 Nov 2026 13:34:10 +0000

Engineer's view. This is memory overcommit. Back every page up front and half your RAM sits idle. Overcommit, and an out-of-memory killer stops a process at peak. An LLM server faces the same choice, because each request's memory grows with its answer. You pay in idle GPU memory or in frozen streams.

TL;DR. An LLM server must decide how much memory to hold for an answer before the model has written it. Hold too much and you waste capacity. Hold too little and a running answer gets paused to make room.

Explain Like I'm 5

Picture a restaurant where nobody knows how long each family will stay. You could save a giant
table for every family, just in case. Most families leave early, so half your seats sit empty while
people wait outside.

Or you could seat everyone at small tables and hope. When the room gets full, you ask the newest
family to stand up and wait, even in the middle of dinner. They keep their food. They just can't
eat for a while.

An AI server runs that restaurant, and every answer is a family.

The Problem

You have shipped this bug already, in a system with no model in it.

You ran a service on Kubernetes. You set its memory request to the worst case you had ever seen.
Nothing crashed. But the cluster reported itself full while the real usage sat near 40%, and new
pods stayed pending.

So you lowered the request to the average. The cluster filled up nicely. Then a busy afternoon
came, several pods grew at once, and the kernel killed one in the middle of its work.

You were guessing a number before you knew the answer. An LLM server guesses that number for every
request.

The memory here is the KV cache: the model's saved work for every token so far. Without it, the
model would reread the whole conversation for each new token. It grows by one token per token
generated. The model only decides the length by stopping, so at the moment a request starts, nobody
knows its final size. The caller sends max_tokens, but that is a ceiling, not a forecast.

Under load, that guess shows up in two ways. Your GPU memory looks full while throughput sits far
below what you paid for. Or a streaming answer stops mid-sentence for many seconds, then carries on
as if nothing happened. Neither one logs an error.

The fix: find out which rule your server uses, measure how long your answers really are, then set
max_tokens and your concurrency limit from those lengths. New to the KV cache? Start at
How to Work Out How Much GPU Memory One Request Really Needs.

The Fix: Match the Limits to Your Server's Admission Rule

Admission is the moment a server lets a waiting request start. Three popular servers give three
different answers to one question: what must be free first?

What does each server hold back?

TGI counts the prompt plus the full max_new_tokens against its budget. It cannot run out mid-answer.

SGLang counts max_new_tokens times a ratio, clipped at 4,096 tokens. The ratio starts at 0.7 and
falls to about 0.1 while things go well. When memory runs out, it resets from what requests really
used.

vLLM counts only what exists now. Its --watermark setting can keep some memory free, but the default
is zero.

What happens when memory runs out?

The server evicts a running request. It frees that request's cache and sends it to the front of
the waiting line. vLLM picks the newest request. SGLang picks the one with the least output.

The answer isn't lost. Tokens already generated are kept. The request must rebuild its cache by
rereading the prompt and its own output, and only then does the stream move again. In vLLM, nothing
new starts during a step that evicted.

Is the redone work the expensive part?

I expected it to be. It wasn't. My first simulation was built to show a server thrashing, and
throughput never collapsed. Rereading is much faster than generating.

Eviction redid 80% of the prefill work and still delivered about five times the throughput. The
price landed elsewhere: 26% of finished requests were evicted, and their median freeze was 58
seconds. Throughput hid that completely.

What This Means for You

When this matters. You run your own model server, or you are choosing one. It matters most when
answers vary widely in length, as they do with reasoning models and agents. If you only call a hosted
API, this is the provider's problem. You can still detect it from your side, and your max_tokens
may still matter.

How it affects you. The same GPU can give very different results depending on which rule it runs.
On a reserving server, one team's careless limit costs everyone. In my chat simulation, moving
max_tokens from 8,192 to 4,096 raised throughput 29%, from 351 to 454 tokens per second. Answers
were short, so about 0.5% would have been cut. On an evicting server, average throughput looks
healthy while a few users stare at a stopped stream. Your dashboard's mean hides them.

What to do about it. Start with no setup. Log the largest gap between chunks on your streaming
calls. A gap of many seconds in the middle of an answer is your first sign.

Next, log how many tokens your answers really use. Compare that with the max_tokens you send.
Most teams send a default nobody chose.

Then cap concurrency from those lengths. On an evicting server, that cap removed more freezing than
any other setting I tried. Finally, check the eviction counter and alert on it.

Implementing It

The change.

Four jobs, and one person may do all of them.

Job 1, whoever owns the client. Measure the freeze from the outside. This works against any server.
In your streaming call:

gap, last = 0.0, None
for chunk in client.chat.completions.create(model=MODEL, messages=msgs, stream=True):
    now = time.monotonic()
    gap, last = max(gap, now - last) if last else 0.0, now
metrics.observe("stream_max_gap_seconds", gap)

Start timing after the first chunk, as this does, so that queue time isn't counted as a freeze.

Job 2, whoever sets max_tokens. Choose it from logged lengths, not from the model's limit. Take a
value a little above your longest normal answer, then count what it would cut:

limit = sorted(lengths)[int(0.999 * len(lengths))]
cut = sum(n > limit for n in lengths) / len(lengths)

A wrong limit shows up as finish_reason: "length". Alert on that rate.

Job 3, whoever runs the server. Work out two limits from the same logs. safe_concurrency() in
code_example.py does this:

never = pool / (mean_prompt + p95_answer)     # almost no evictions, low throughput
average = pool / (mean_prompt + mean_answer)  # the memory is full on an average mix

pool is your KV cache size in tokens. Those limits came out at 4 and 15 for the long-answer case. The upper
limit is the useful one here. It gave up 17% of throughput, cut evicted requests from 26% to 7%, and
cut the median freeze from 58 to 17 seconds.

Job 4, the setting for your server. On vLLM, lower admission without shrinking the batch buffers:

vllm serve "$MODEL" --max-num-active-seqs 15

--max-num-seqs also works, and the docs recommend it. Be careful with --watermark: at 0.2 it cut
redone work from 80% to 53%, but the median freeze rose to 247 seconds, because an evicted request
then waits for extra room.

On SGLang, watch for the pool-full warning. Its tuning guide says to raise
--schedule-conservativeness to about 1.3 if the warning fires often. Lower it to about 0.3 if token
usage stays low while requests queue. On TGI, set --max-total-tokens as low as your longest real
request. TGI is in maintenance mode, so weigh that for a new deployment.

How you know it worked.

On vLLM, this should sit near zero:

rate(vllm:num_preemptions_total[5m])

vllm:kv_cache_usage_perc should stay high, or your cap is too low. On any server, p99 of
stream_max_gap_seconds should fall while total tokens per second holds. If usage drops and waiting
requests climb, you set the cap too low. Raise it until evictions just reappear.

When Tuning Admission Is the Wrong Tool

When you never fill the pool. If vllm:kv_cache_usage_perc stays well under 1 at peak, no rule is
making a difference. Fix something else.

When batch work doesn't care. A nightly job nobody watches can freeze for a minute at no cost. Leave
concurrency high, and take the throughput.

When you are simply out of capacity. A cap moves waiting from the stream to the queue. It doesn't
make more GPU. In my runs, the queue still grew, and your time to first token will show that.

When a limit would cut real work. A low max_tokens on a reasoning model can end the answer before
it starts. Check finish_reason first.

Treat my numbers as a shape, not a measurement. They come from a simulation that follows each
server's source code, with invented lengths and an assumed prefill speed. Single runs also varied widely: the same setup froze for 26 to 170 seconds across nine seeds. It
ignores prefix caching, which can make a reread much cheaper. I did not run these servers on a GPU. If prefill is slow, or
contexts are long, eviction costs more than it did here.

Three questions before you change anything:

How much longer is your p95 answer than your mean answer?
Does your server hold memory for max_tokens, or only for what exists?
Would a user notice a ten-second pause, or only a slower batch?

Glossary

KV cache — the model's saved work for every token so far, which grows as the answer grows.
admission — the moment a server lets a waiting request start using memory.
evicts — frees a running request's cache and sends it back to wait.
prefill — processing input tokens to build the cache, much faster than generating.
max_tokens — the most tokens a caller allows for one answer; a ceiling, not a forecast.
watermark — a share of memory vLLM can keep free when admitting requests.

How to Tell What Claude Code and Codex Forget When They Compact a Session

Mayur Agnani — Sat, 31 Oct 2026 11:30:43 +0000

Engineer's view. This is a process restart with a handoff note. Whatever the process wrote down or saved to disk comes back. Whatever sat only in memory is gone, and nothing errors. A coding agent's memory is its conversation, so values it read but never wrote down vanish, and it must read them again.

TL;DR. A compacted agent keeps your instructions and its own edits, but loses most of what it only read. Whether a fact survives depends on how it entered the session, so you can predict the loss and tell the summary what to keep.

Explain Like I'm 5

You are building a huge model castle, and your table is full. So you ask a friend to write a
short note about the castle, then you clear the table.

The note says what you were told to build and which towers you finished. It does not list every
piece you looked at in the box. Those are still in the box, so you can look again.

But some pieces you borrowed for a minute, and they went back to a friend's house. If the note
does not mention them, you will not remember them, and you cannot go look.

So before you clear the table, tell your friend what to put in the note.

The Problem

You have shipped this bug already, in a system with no model in it.

Your service held a session cache in process memory. A deploy restarted it. The health check went
green and the error rate stayed flat. Then users started logging in again, because the sessions
never reached disk. Nothing crashed. The process simply came back without the state that had only
lived in memory.

A coding agent has the same split, and it restarts in the middle of your work.

Its working memory is the conversation. When that fills, the agent runs compaction: it
replaces the conversation with a summary and carries on. In my test session, 78,642 tokens became
2,632. That is 97% gone in about twenty seconds, and the session looks healthy afterward.

The agent then keeps working from the note. If a detail is missing, you see no error. You only see
it read the same file twice, or ask something you already answered. Worse, the detail may have
come from a test run or a web page that no longer exists.

You could treat this as bad luck. It is not. What survives follows a rule.

The fix: know which lane each fact is in, tell the summary what to keep, and write anything you
cannot regenerate to a file before compaction runs.

The Fix: Sort Facts by How They Entered the Session

Compaction is a rebuild, not a shortening. The new context comes from three lanes.

What reloads from disk?

Claude Code reloads the root CLAUDE.md, auto memory and the plan file. It also re-reads up to
five recent files, each under 5,000 tokens. Rules with paths: and nested CLAUDE.md files are
different: they entered as messages, so they get summarized. They load again only when the agent
touches a matching file.

What does a budget keep?

Codex's open-source compaction path keeps your recent messages word for word, up to 20,000 tokens,
walking newest to oldest. It adds a summary and re-injects its starting context. Older messages
depend on the summary.

Which facts does the summary drop?

I tested this rather than assume it. I planted facts in real Claude Code sessions, compacted them,
then asked from memory with tools off. Each of three sessions read 25 small config files.

Every instruction survived, and so did all ten edited files, even after a second compaction. The
agent said so plainly when a value was gone, even when told never to answer "unknown".

The summary kept what mattered to the work so far. In an earlier run, the last task counted timeouts
over 5000. Of six values I probed, only the two timeouts over 5000 survived.

What This Means for You

When this matters. You run long sessions in Claude Code, Codex or any agent that summarizes its
history. It matters most when the agent reads something it cannot read again: a failed test run, a
production log, a web page, or the output of a command that changes state.

How it affects you. A file on disk costs you one more read. The agent gets it back and you pay
tokens. A one-time output costs more, because the evidence is gone. The agent will then rerun the
command, or reason from a vague line in the note. Long sessions may lose more than my small test
did. Factory ran probe questions after compression on 36,611 production messages. All three methods
scored worst on tracking files, at 2.19 to 2.45 out of 5, even though each kept instructions well.

What to do about it. Start with no setup. Before you compact, name what matters:

/compact Keep the exact failing test names, error strings and ids.

A line like that, naming the values I needed, took my test from 0 to 36 of 36.

Next, watch for the moment that hurts. When the agent reads something you cannot regenerate, ask it
to save the facts to a notes file. A file sits in the disk lane.

Then fix the repeated cases. Move a rule that must always hold into your root rules file. Add a hook
that restores your working state after every compaction. Finally, audit a real session to see your
own numbers.

Implementing It

The change.

Four jobs, and one person may do all of them.

Job 1, whoever drives the session. Save anything you cannot regenerate, then compact at a task
boundary with a focus. Codex takes a compaction prompt in its config, so the same habit can be set
once there. In ~/.codex/config.toml:

compact_prompt = "Summarize this session. Keep exact error strings, ids, file paths and test names."

Job 2, whoever owns the repo's agent setup. Check which lane your rules are in. A rule with
paths: enters as a message, so its survival is up to the summary until the agent touches a
matching file again. My short test kept one, but the docs warn it can go. If it must hold across
compaction, move it to the root file. Then add a section the summarizer reads, in CLAUDE.md:

# Compact Instructions
Keep exact error strings, failing test names, ids and every file edited.

Add a hook for state that changes. A SessionStart hook with the compact matcher runs after
every compaction, and its output joins the new context. In .claude/settings.json:

{ "hooks": { "SessionStart": [ { "matcher": "compact", "hooks": [
  { "type": "command",
    "command": "git status --short; cat notes/session.md 2>/dev/null" }
] } ] } }

Job 3, whoever builds their own agent. Do not trust a summary with the file list. Keep it as data
in your loop, and put it back after you summarize.

def rebuild(history, pinned, ledger, summarize, keep_tokens=20_000):
    recent = newest_user_messages(history, keep_tokens)
    note = summarize(history, focus="exact errors, ids, decisions")
    edited = "Files edited: " + ", ".join(sorted(ledger.edited))
    return [*pinned, *recent, edited, note]

Job 4, whoever wants the numbers. code_example.py needs no model. It reads a session transcript,
finds each compaction, and checks which identifiers reached the summary, grouped by lane. Run it on
one of your own sessions:

python3 code_example.py ~/.claude/projects/<project>/<session>.jsonl

An identifier here means an exact string such as a port, path or error code. On 40 compactions
from my own sessions, 4% of the identifiers the agent only read reached the summary, against 19% of those the agent or I wrote. An exact-string match undercounts paraphrase, so
trust the gap more than the percentages.

How you know it worked.

Probe it. Fork a session right after a compaction, turn tools off, and ask for five facts you need.
Offer "UNKNOWN" as an answer so a miss is visible. In Claude Code:

claude -p --resume "$SESSION" --fork-session --tools "" \
  "From memory, no tools. Which files did you edit? What was the exact error? Reply UNKNOWN if unsure."

Before your change, count the misses. After it, the same probe should return your focus items
exactly. Also watch re-reads: the same file read twice right after a compaction means the note
missed it.

When Managing Compaction Is the Wrong Tool

When the task is short. A session that never fills its context never compacts. Spending effort on a
hook then buys nothing.

When the facts are on disk and cheap to read. Losing a value from a config file costs one more
read. Asking the summary to keep every value makes the note longer, and a long note costs tokens on
every later turn. Keep what is expensive to get back.

When you should start over. If the next task is unrelated, /clear is better than any summary. Old
context crowds the new work, however well it was compressed.

When the work belongs elsewhere. Send large research to a subagent, so the reads never enter your
main conversation at all.

My test has limits too. It used small synthetic sessions on one model, and I did not run the Codex
side. Run the probe on your own work before you trust these numbers.

Three questions before you add a hook:

Which fact, if lost, could the agent not get back?
Does that fact reach the model through a file, or only through a message?
Would a probe after compaction return it exactly?

Glossary

compaction — replacing a conversation with a shorter summary so the agent can keep working.
lane — the route a fact takes through compaction: reloaded, kept by budget, or summarized.
probe — a question asked after compaction to test whether a fact survived.
focus — an instruction telling the summary what to keep, as in /compact with text after it.
identifier — an exact string such as a port, id, file path or error code.

How to Tell Whether a Second Model Reviewing Your Agent's Work Pays for Itself

Mayur Agnani — Fri, 30 Oct 2026 11:39:40 +0000

Engineer's view. This is running two replicas of the same software for safety. Both crash on the same leap second, because they share the bug. A reviewer model trained like the drafter misses the same failures. Pick a reviewer that fails differently, and measure what it catches on your drafter's failures, not on its own.

TL;DR. A reviewer model can only fix errors it would not have made itself, and models share many blind spots. Measure what it fixes and what it breaks on your own tasks, then review only where the drafter often fails.

Explain Like I'm 5

You write a story for school. Before you hand it in, your twin reads it to look for mistakes.

Your twin is smart, but your twin learned spelling from the same teacher. The words you get
wrong are the words your twin gets wrong too. So those mistakes slip through.

A friend from another school spots them, because they learned differently. But sometimes the
friend says a right word is wrong. If you change it, your story gets worse.

So the best helper is someone who makes different mistakes than you do. And they should only
read the hard stories, where you make the most mistakes.

The Problem

You have shipped this bug already, in a system with no model in it.

Your service ran two replicas for safety. If one crashed, the other would take the traffic. Then
a leap second arrived. Both replicas crashed in the same minute, because both ran the same code
with the same bug. The backup was real, and it protected you against nothing that mattered.
Engineers call this a common-mode failure: one cause takes out every copy at once.

Redundancy only helps against failures the copies do not share.

A reviewer model is the same idea applied to an agent. One model, the drafter, writes a
plan or a patch. A second model reads it and lists concerns. The drafter then revises once.
Some coding tools now run this step for you. You can add it to your own agent in an afternoon.

It costs at least two extra model calls per task. Whether it pays depends on two things the
setup hides. The reviewer misses whatever blind spots it shares with the drafter. And when it
raises a false alarm on a correct draft, the revision can break it.

The fix: measure what the reviewer fixes and breaks on your drafter's own tasks, pick one
that fails differently, and call it only where the drafter often fails.

The Fix: Measure Fixed and Broken, Then Review Only the Hard Tasks

What does one review pass add?

Count the four outcomes above. The gain is the fixed tasks minus the broken ones. As rates:

net = fail × catch × fix − (1 − fail) × false alarm × break

Here fail is how often the drafter alone gets it wrong. Catch is how often the reviewer flags
those failures. Fix is how often the revision then gets it right.

Why isn't the reviewer's own accuracy enough?

Because the catch rate is measured on the drafter's failures, and those are the hard cases. A
study of over 350 models found that when two models both get a question wrong, they pick the
same wrong answer about 60% of the time. Same developer, same architecture and similar size all
raised that overlap.

When does review hurt?

On easy tasks. If the drafter fails 4% of the time, there is little to fix. A 10% false-alarm
rate with a 25% break rate then costs more than review gains.

What This Means for You

When this matters. Your agent writes code, plans or answers, and you are adding a second model
to check them. Or you turned on a tool's built-in critic and it now costs more. It matters most
on long tasks, where one missed error ruins the run. GitHub found Rubber Duck helped most on
problems spanning three or more files that take 70 or more steps.

How it affects you. A reviewer from the drafter's own family feels safe and catches less than
its benchmark score suggests. A reviewer on every task pays two extra calls on work that was
already right. Its false alarms then rewrite correct code. You will not see either in a demo,
because demos show the fixed case.

What to do about it. Start with no new code. Take 20 tasks your agent failed last month.
Paste each failing draft into a reviewer from another model family and ask for its top three
concerns. Count how many name the real bug. That is a first estimate of catch.

Then run the full check. Run a few hundred tasks twice from the same start, alone and with
review. Count fixed and broken. Split them by a signal you have before review runs, such as
files touched. Turn review on only for the buckets where fixed clearly beats broken.

Implementing It

The change.

Three people touch this. The second one decides whether it ships.

Role 1, whoever writes the agent loop. Add one review pass. Give the reviewer the task and the
draft only, with no tools and no write access. HydraFusion does the same: its critic is
read-only, runs in a tool-less context, and comes from a different model family. Then let the
drafter revise once.

REVIEW = ("You are reviewing another engineer's change. List at most three concerns, "
          "most important first: bugs, wrong assumptions, missed edge cases. "
          "If you find nothing serious, reply exactly: LGTM.")

def draft_with_review(task, drafter, reviewer):
    draft = drafter.run(task)                        # full agent loop, with tools
    notes = reviewer.complete(system=REVIEW,          # one call, no tools
                              user=f"Task:\n{task}\n\nChange:\n{draft.diff}")
    if notes.strip() == "LGTM":
        return draft, {"flagged": False}
    revised = drafter.run(task, extra=f"A reviewer raised:\n{notes}\n"
                                      "Fix what is real. Ignore what is wrong.")
    return revised, {"flagged": True, "notes": notes}

The "ignore what is wrong" line matters. It is your only defense against the broken case.

Role 2, whoever owns evals. Run paired trials. Every task runs twice from the same commit:
drafter alone, then drafter with review. Grade both with the task's own tests. Only the tasks
whose outcome changed carry evidence, so test those with McNemar's test.

fixed  = sum(1 for r in runs if not r.alone_ok and r.reviewed_ok)
broken = sum(1 for r in runs if r.alone_ok and not r.reviewed_ok)
helps  = fixed - broken > 2 * math.sqrt(fixed + broken)   # about 95% confidence

In the demo, 400 tasks give 27 fixed and 8 broken overall. Split by files touched, one-file tasks
give 2 fixed and 0 broken, too few to trust. Two-file tasks give 4 and 6. Tasks with three or more
files give 21 and 2, the only bucket that clears the test.

Role 3, whoever owns the routing. Gate review on that signal. Files touched is the one GitHub
reports. Plan length, step count or a past failure on the same module also work.

def needs_review(task, plan):
    return len(plan.files) >= 3 or plan.steps >= 70

If you drive Copilot CLI, Rubber Duck is behind /experimental and pairs a Claude drafter with a
GPT reviewer. The same paired check tells you whether it earns its cost on your repo.

How you know it worked.

Rerun the paired trials a week later on fresh tasks. Fixed minus broken should still clear the
test in the buckets you review. Your success rate should rise while total model spend rises
less than the review share would suggest. Log every false alarm. If broken climbs, tighten the
prompt before you widen the gate.

When a Reviewer Model Is the Wrong Tool

When the task has a test that runs. A failing test is a reviewer with no blind spots and no
false alarms. Run it, feed back the failure, and save the second model for what tests miss.

When the drafter rarely fails. On easy work there is little to catch. A few false alarms then
cost more than review gains, and every call costs the same.

When you can only get a reviewer from the drafter's family. It still catches some errors, but
fewer than its score suggests. A stronger single model may cost less than two weaker ones.

Latency is real too. HydraFusion holds the draft until review and revision finish. A reader
waiting on the answer feels every extra call.

Three questions before you add one:

What share of your agent's failed tasks does this reviewer actually flag?
How often does it flag a draft that was already right?
Which signal, known before review, separates the tasks where it helps?

Glossary

common-mode failure — one cause that takes out every redundant copy at once, because they share it.
reviewer model — a second model that reads another model's work and lists concerns, without changing it.
drafter — the model that does the task and revises after review.
catch — the share of the drafter's failures that the reviewer flags.
false alarm — the reviewer flagging a draft that was already correct.
McNemar's test — a check on paired runs that uses only the tasks whose outcome changed.

How Many Pull Requests a Merge Queue Can Land in an Hour

Mayur Agnani — Thu, 29 Oct 2026 09:22:28 +0000

Engineer's view. This is optimistic locking under contention. Each transaction validates against everything committed before it, and one abort throws away the work queued behind it. More workers only waste more work. Throughput is set by the abort rate and the transaction time. Agents raise the load and leave that ceiling where it was.

TL;DR. A merge queue's speed is capped by how often a queued change fails and how long CI takes, not by how many pipelines run at once. More pull requests from agents push it into long waits unless you cut one of those two.

Explain Like I'm 5

A school lunch line has one rule. Each kid must show their tray to the cook, and the cook
checks it against every tray that went before.

The cook can look at twenty trays at once. If a tray is wrong, that kid steps out. Every kid
behind them has to show their tray again, because the cook checked it next to the wrong one.

On a normal day the line moves fine. Then a class of fast kids joins. They bring three times
as many trays. The cook is not slower. The line still backs up, because each wrong tray sends
everyone behind it back to the start.

The Problem

You have shipped this bug already, in a system with no model in it.

Your service used optimistic locking. Each write read a row, changed it and committed only if
nobody else had changed it first. At low traffic it was fast. Then traffic tripled. Writes
started aborting and retrying, and each retry raced the next wave. Latency went from
milliseconds to seconds. Adding app servers made it worse, because they only added contenders.

The limit was never the servers. It was how often a commit failed, times how long it took.

A merge queue has the same shape. It is the gate that lands pull requests on main one
tested batch at a time. GitHub calls it the merge queue. GitLab calls it a merge train. Each
pull request is tested on top of main plus every pull request ahead of it, so main never
breaks. When one fails, everything behind it was tested on a tree that included the failure.
All of it reruns.

Coding agents change the arrival rate. At Anthropic, merges grew from about 2,500 a day in May
2026 to over 30,000 a day by September. They stopped using a merge queue. A team without that
option has to know where its queue's ceiling is.

The fix: measure your queue's failure rate and CI time, compute its ceiling, and lower the
failure rate before adding pipelines.

The Fix: Cut the Failure Rate Before You Add Pipelines

Why doesn't running more pipelines fix it?

Call the share of queued pull requests that fail p, and the CI time T. Between two
failures the queue lands about 1/p pull requests. Each failure throws away up to T of work
behind it. So no number of pipelines lands more than about 1/(p × T) an hour.

With a 30-minute CI and 5% failing, that is 40 an hour. With 20 pipelines it is about 24. Going
to 100 pipelines buys 38, at five times the CI bill.

Where does the failure rate come from?

Two places. Real breaks: a change that passed on its own branch and conflicts with one ahead of
it. And flaky tests, tests that fail now and then with no change at all. Flakes add up. Fifty
tests that each flake one run in a thousand fail about one run in twenty. That is p = 5%.

What happens to the wait near the ceiling?

It climbs fast. A queue at 8 pull requests an hour waits 34 minutes on average. Triple the
arrivals to 24, inside the ceiling on paper, and the wait passes 100 minutes.

What This Means for You

When this matters. Your repo uses a merge queue, and people or agents open more pull
requests than they did last quarter. It matters most when CI runs long and has a few tests
everyone retries. If you see "removed from the merge queue" often, you are paying for it.

How it affects you. The queue feels slow, and the obvious fix is more parallel pipelines.
That buys little once failures are common, and it raises the CI bill. Waits grow faster than
volume. Agents then rebase on a stale main and conflict with each other, which adds real
failures on top of the flaky ones. A flaky test used to be an annoyance. Under load it sets
your throughput.

What to do about it. Start with two numbers from last week, with no new tooling. Count the
pull requests the queue merged, and the ones it removed for failed checks. Divide removed by
the total, and that is p. Read T off any recent queue run. Then compute 1/(p × T) and compare it
with your busiest hour.

If the busy hour is within half of the ceiling, waits are already long. Lower p first: retry a
failing test once before ejecting, and quarantine the tests that flake. Then shorten T by moving
slow checks out of the queue. Add pipelines last.

Implementing It

The change.

Four people touch this. Each change is small, and they compound.

Role 1, whoever owns CI. Run only fast required checks on the merge group, and make a flake
fail twice before it ejects a pull request. In GitHub Actions the queue fires the merge_group
event. The retry here uses the pytest-rerunfailures plugin.

on:
  pull_request:
  merge_group:
jobs:
  required:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pytest -q --reruns 1   # a flaky test must fail twice to eject

On GitLab, the same idea is retry: { max: 1, when: script_failure } on the job. A retry turns
a 5% flake rate into about 0.25%, since both runs have to fail. It also hides flakes, so pair it
with Role 3.

Role 2, whoever administers the repo. Size the queue from the formula, not by feel. GitHub
sets the pipeline count with build concurrency (1 to 100). GitLab allows 20 parallel
pipelines by default.

def landed_per_hour(depth, p, ci_minutes):
    return 60 * depth / (ci_minutes * (1 + p * depth))

# depth 1/p reaches half the ceiling; past about 3/p you mostly pay for reruns

Plug in your own numbers before you touch the setting. With p at 5% and a 30-minute CI, twenty
pipelines land about 20 an hour by the formula and 24 in the simulation. Cut p to 1% and the same
twenty land about 33. That is the cheaper change.

Role 3, whoever owns the test suite. Find the flakes. A test that fails and then passes on
the same commit is flaky. Log those, rank tests by count, and quarantine the top of the list
until they are fixed.

flaky_runs = 1 - (1 - 0.001) ** 50     # fifty tests at 0.1% each -> about 5% of runs

Role 4, whoever runs the agents. Run the queue's checks before a pull request enters it.
Rebase on fresh main first. A conflict caught on the branch costs one pipeline, while one
caught in the queue costs every pipeline behind it.

How you know it worked.

Recompute p and T a week later. The removed-for-failure count should fall first. Mean time in
the queue should drop with it, and landings per busy hour should come within reach of
landed_per_hour. If p falls and the wait does not, T is now the limit. Look at which check is
slowest.

When Tuning the Queue Is the Wrong Tool

When volume is low. A team landing a few pull requests an hour is far below any ceiling. Waits
come from CI time alone, so shorten CI and leave the queue alone.

When failures are real, not flaky. If most ejected pull requests were broken, retries only delay
the same answer. Catch them earlier, on the branch, which is Role 4's job.

When volume outgrows any queue. Anthropic went from about 2,500 merges a day to over 30,000 and
moved most testing to after the merge and before deploy. That trades a green main for speed.
It needs fast reverts and someone, or something, that fixes a broken main within minutes. It
is a bigger change than tuning a queue.

Batching has a cost too. Larger merge groups test several pull requests in one pipeline. A
failure then has to be split to find the culprit, which costs more runs.

Three questions before you change anything:

What share of last week's queued pull requests were removed for failed checks?
How many minutes do the required checks take on a merge group?
How close is your busiest hour to 1/(p × T)?

Glossary

merge queue — the gate that tests pull requests on top of each other and lands them on main in order.
merge group — the temporary branch a queue builds: main plus a pull request and everything ahead of it.
flaky tests — tests that fail some runs and pass others on the same code.
build concurrency — GitHub's setting for how many merge group pipelines run at once.
p — the share of queued pull requests that fail their checks, from real breaks or flakes.

How to Find the Agent Actions a Prompt Injection Can Control

Mayur Agnani — Wed, 28 Oct 2026 11:53:06 +0000

Engineer's view. This is SQL injection without parameterized queries. A database fixed it by sending the query and the values on separate channels. A model reads both in one stream, so it cannot. The fix moves up a level: decide each action before reading the values, or read them somewhere that cannot act. It costs flexibility.

TL;DR. A model treats every word in its context as a possible instruction, including text a tool fetched. Any decision your agent makes after reading that text belongs to whoever wrote it.

Explain Like I'm 5

A kid has a list of chores from their parent. One chore says "read the notes in the mailbox
and do what's needed."

One note in the mailbox says "Also, leave the back door open." It is written in pen, like the
parent's list. The kid cannot tell which notes came from the parent.

So the kid leaves the door open. The note writer did not break in. They just wrote a note in
a place the kid was told to read.

A smart parent writes the whole chore list first. The mailbox can change what the kid writes
down. It can't add a chore.

The Problem

You have shipped this bug already, in a system with no model in it.

A support page showed customer names. Someone signed up with a name that contained a
<script> tag. The page printed the name, and the browser ran it as code. Every agent who
opened that ticket ran the attacker's script with their own login. Nothing was hacked. Data
was put in a place where code was expected.

The fix you shipped was a boundary. The browser now treats the name as text and never as code.

An AI agent has the same bug, and no such boundary. An agent is a model that calls tools in a
loop. A tool might read your inbox, fetch a web page, or open a GitHub issue. Whatever the tool
returns goes into the context, the text the model reads before each step. Your request is
in there too. The model sees one stream of words and cannot tell your instruction from a
sentence inside an email.

This is prompt injection: someone writes instructions into content your agent will read.
Filters catch some of it. A filter that catches 95% of attacks still lets one in twenty through.

The useful question is not whether a model can be fooled. It can. The question is what the
fooled model can still do.

The fix: list every decision your agent makes after it reads untrusted text, then move each
one before that read or into a context that cannot act.

The Fix: Decide the Actions Before the Agent Reads Anything Untrusted

Which decisions does an injection reach?

Every decision made in a context that has seen the untrusted text. That covers three kinds:
which tool to call, each argument, and the final reply. Decisions made before the read are
safe. So are decisions made in a separate context that never saw it.

That turns a fuzzy worry into bookkeeping. You can list the decisions, and you can count them.

Why not just tell the model to ignore instructions in tool results?

That instruction lives in the same context as the attack. The attacker's text can argue with it,
and the model decides who wins. A rule the model enforces on itself is advice.

When does a reached decision become a leak?

When three things meet in one context: your private data, untrusted text, and a way out. Simon
Willison calls this the lethal trifecta. A way out can be an email, an HTTP call, or an
image in a reply. When the chat window loads the image, the URL can carry your data to the
attacker's server.

What This Means for You

When this matters. Your agent reads anything a stranger can write: email, web pages,
tickets, issue comments, uploaded files, or the output of a third-party MCP server. It also has
a tool that can send, write or fetch. A coding agent that reads an issue and can push a branch
counts. So does a chat assistant that renders markdown images.

How it affects you. Prompt injection is not a bug you patch once. It comes from how the
model reads. Each tool you add makes the trifecta easier to complete, because MCP lets anyone
combine tools from different servers. The real risk is often a tool you would call harmless. A
summary rendered in your chat window can carry an image URL. A calendar invite can carry a
title. Neither looks like "sending data", and both can.

What to do about it. Start with paper and no code. Write down every tool your agent can
call. Mark each one: does it read text a stranger wrote, does it read private data, and can it
send anything out? Then walk one real task in order. After the first untrusted read, every
later decision on that path is exposed. If private data and a way out sit downstream of it,
you have a leak path.

Then change the design, not the prompt. Fix the tool sequence before the read. Have a separate
model read untrusted text and hand back only values. Make the chat window stop loading remote
images. ## Implementing It shows each one.

Implementing It

The change.

Four people touch this. The design work belongs to the first.

Role 1, whoever writes the agent loop. Label every tool, then gate the loop. After the first
untrusted result, anything consequential needs a person. This is the paper's rule in code: once
untrusted input is in, it must be impossible for it to trigger a consequential action.

TOOLS = {
    "read_inbox":  Tool(reads=UNTRUSTED),
    "search_docs": Tool(reads=PRIVATE, payload="query"),
    "send_email":  Tool(dest="to", payload="body",
                        renders_remote=True, consequential=True),
}
gate = InjectionGate(TOOLS)

for call in model_turns(messages):
    if not gate.allow(call.name) and not ask_user(call):
        messages.append(refused(call)); continue
    result = run_tool(call)
    gate.saw_result(call.name)          # flips once untrusted text arrives
    messages.append(result)

The gate is the floor. The bigger win is the design. Have a planner fix the tool list and the
recipients from the user's request alone. That closes five of the ten exposed decisions in the demo.
Better still, let a quarantined model read the email and return values, such as $VAR1 for a
summary. The model that calls tools sees only the variable name. Your code swaps the real text
in when it builds the final message, so no model that can call tools ever reads it.

Role 2, whoever writes the MCP server. Say what each tool touches. MCP has
openWorldHint for tools that reach outside entities, and readOnlyHint for tools that change
nothing. A client can use them to label tools. The spec warns clients to ignore annotations
from servers they do not trust, so set them honestly.

{ "name": "fetch_page",
  "annotations": { "readOnlyHint": true, "openWorldHint": true } }

Role 3, whoever owns the chat window. Stop the reply from making requests. Strip remote images
from model output, or block them with a content security policy: img-src 'self'. Show full
URLs on links, so a person sees where a click goes. This one change removes the reply from the
list of ways out.

Role 4, whoever owns CI. Plant a canary. Put an instruction in a fixture email that asks for
the canary to be sent somewhere. Then assert it never leaves.

def test_inbox_cannot_exfiltrate():
    calls, reply = run_agent("Summarize my inbox", inbox=[CANARY_EMAIL])
    outward = [c for c in calls if TOOLS[c.name].consequential]
    assert not any("CANARY-7f3" in str(c.args) for c in outward)
    assert "evil.example" not in strip_remote_images(reply)

How you know it worked.

Run the audit in code_example.py on your agent's description. The leak count should read zero.
The exposed count will not, and that is expected. A quarantined reader still writes the text
you see. The canary test should pass on every model you ship. Re-run it when you add a tool.

When Changing the Design Is the Wrong Tool

When your agent never reads text a stranger wrote. A coding agent working only on your own repo
has no untrusted input. The gate would only add prompts.

When nothing downstream can act or leak. An agent that summarizes web pages into a chat with no
images, links or tools can be fooled. A fooled summary is a wrong answer, not a breach. Keep
the stronger designs for agents that hold private data or can send things.

When the task needs to change course after reading. A plan fixed in advance cannot follow a
link the email mentions. The paper says it plainly: general-purpose agents are unlikely to give
reliable safety guarantees. Sometimes the honest answer is a narrower agent.

The cost is real. A quarantined reader doubles the model calls. Every approval prompt costs a
person's attention, and people learn to click yes.

Three questions before you change anything:

Which tools can read text that someone outside your team wrote?
After that read, which tools can still send, write or fetch?
Does your chat window load images from URLs the model wrote?

Glossary

context — the text the model reads before each step: your request, its own replies, and every tool result.
prompt injection — instructions written into content an agent will read, so the model follows them.
lethal trifecta — private data, untrusted text and a way to send data out, all reachable from one context.
quarantined model — a model that reads untrusted text but has no tools, and returns only values.
canary — a unique marker string planted in a test, so you can detect if it ever leaves.

How Claude Code and Codex Decide Which Skill Descriptions the Model Reads

Mayur Agnani — Tue, 27 Oct 2026 11:44:35 +0000

Engineer's view. This is a cache that evicts the least-used entry. A key you just added has the lowest hit count, so it goes first, and it never earns the hits that would keep it. Claude Code trims its skill list the same way. A skill you installed today is the first to lose its description.

TL;DR. A coding agent shows the model only a short list of your skills, and that list has a size limit. Past the limit, descriptions get cut, and a skill the model cannot read about is a skill it rarely picks.

Explain Like I'm 5

A school has one small board by the door. Every club pins a card with its name and what it
does. Kids read the board to decide where to go after lunch.

Then too many clubs sign up. The teacher keeps every name on the board but crosses out the
descriptions of the clubs that got the fewest visitors lately. A brand-new club has had no
visitors yet, so its description is crossed out first.

A kid who loves chess walks past a card that just says "Knights". Nothing tells them it is the
chess club. So the new club stays empty, and its card stays crossed out.

The Problem

You have shipped this bug already, in a system with no model in it.

Your service wrote one log line per request. Someone added a field near the front, and the line
crossed the log shipper's size limit. The shipper did not fail. It cut each line at the limit and
sent the rest. The request ID sat at the end of the line, so searching by request ID stopped
finding anything. No alert fired, because every line still arrived.

The bug was not in any one field. It was a shared size limit, plus a rule for what gets cut when
the limit is hit. Adding something new quietly took words away from something old.

A skill in a coding agent has the same bug. A skill is a folder with a SKILL.md file: a
name, a description, and instructions for one kind of task. Coding agents that support the open
Agent Skills format do not load your skills up front. They show the model
a skill listing: each skill's name and description, and nothing else. The model reads that
listing and decides which skill fits your request.

The listing has a size limit, a fixed share of the context window, the model's working
memory for the session. Install enough skills, and the listing overflows. Each tool then cuts
it by its own rule. A skill whose description was cut can still be listed and still never be
chosen.

The fix: keep the listing under its budget, put each skill's use case in its first sentence
and its name, and check what the model actually sees.

The Fix: Keep the Listing Under Budget and Put the Use Case First

What does the model actually see?

Agent Skills loads in three steps, which the spec calls progressive disclosure. At the start
of a session the model sees only the listing. When a request matches a description, it reads
that skill's full SKILL.md. Scripts and reference files load only when the instructions point
at them.

So the description does all of the choosing. There is no keyword matcher behind it. The model
reads the words and makes a judgment.

How big is the listing allowed to be?

Claude Code gives it 1% of the context window. Codex gives it 2%, or 8,000 characters when the
window size is unknown. On a 200K-token model, 1% is about 8,000 characters, at roughly four
characters per token.

That is not much. The 41 skills on the machine this article was written on have a median
description of 408 characters. About eighteen such skills fill Claude Code's listing.

What gets cut when it overflows?

The two tools use different rules. Claude Code keeps every name. It drops whole descriptions,
starting with the skills you invoke least. Codex shortens descriptions first and omits whole
skills only for very large sets.

Why does a brand-new skill go quiet first?

Under Claude Code's rule, a skill you installed today has zero invocations. It is first in line
to lose its description. The model then sees only its name, picks it less, and its count stays
at zero. Unless you invoke it by hand, it never climbs out.

What This Means for You

When this matters. You have more than about twenty skills, counting the ones that plugins
bring along. Or you switch between a 1M-token model and a smaller one. The budget is a share of
the window, so a smaller window shrinks it. A skill that worked yesterday can lose its
description when you change models.

How it affects you. The failure is quiet. The skill still shows up in the / menu, and it
still works when you type its name. What stops is the model choosing it on its own. You notice
weeks later, when a review skips the checklist you wrote for it. In Claude Code the newest skills
are hit first. In Codex every description gets shorter, so a description that saves its trigger
words for the last sentence loses exactly those words.

What to do about it. Start with one minute and no new tooling. In Claude Code, run
/context and read the Skills row. It shows the listing's size after the budget is applied. If
it sits near 1% of your window, you are at the edge. Then open your three newest skills and read
the first sentence of each description. If it does not say when to use the skill, rewrite it so
it does.

Then make it a habit. Turn off skills you never use, set manual-only skills so they leave the
listing, and add a size check to the repo that holds your team's skills. ## Implementing It
shows all three.

Implementing It

The change.

Three people touch this, and each change is small.

Role 1, whoever writes the skill. Put the use case first, and make the name carry it too. A
name survives every cut that a description does not. Keep the whole description short enough
that a cut at about 250 characters still leaves the trigger.

# before: the trigger is in the last sentence, and the name says nothing
name: notes
description: Collects merged pull requests, groups them by area, and drafts a
  summary for customers in the house style. ... Use when the user asks for
  release notes.

# after: the trigger leads, and the name alone still says what it is for
name: release-notes
description: Use when the user asks for release notes or a changelog for
  customers. Groups merged pull requests by area in the house style.

In the demo, the Codex cut keeps 244 characters per description once a plugin doubles the
skill count. The "before" description loses the words "release notes". The "after" keeps them.

Role 2, whoever owns the agent setup. Remove what the model never needs to choose. A skill you
only run by hand should leave the listing entirely. In its frontmatter, Claude Code reads
disable-model-invocation: true. Codex reads allow_implicit_invocation: false under policy
in agents/openai.yaml. For skills you cannot edit, use .claude/settings.json in Claude Code and
~/.codex/config.toml in Codex:

{ "skillOverrides": { "legacy-context": "name-only", "old-deploy": "off" } }

# ~/.codex/config.toml
[[skills.config]]
path = "/Users/you/.agents/skills/old-deploy"
enabled = false

You can also raise the budget. In Claude Code, set skillListingBudgetFraction (for example
0.02). In Codex, set skills.max_context_tokens, up to 10,000 tokens. Every character you add
is paid on every turn, so prune first.

Role 3, whoever owns the repo that ships the team's skills. Fail the build when the listing
would overflow on the smallest model your team uses. read_skills and claude_code_listing
in code_example.py do the parsing and the arithmetic:

skills = read_skills([".claude/skills", ".agents/skills"])
shown, budget = claude_code_listing(skills, context_tokens=200_000)
dropped = [name for name, desc in shown.items() if not desc]
assert not dropped, f"over the {budget}-char listing budget: {dropped}"

How you know it worked.

In Claude Code, the Skills row in /context should drop below the budget, and --debug should
stop logging the overflow warning. /skill-doctor lists skills that were never invoked, with
what each costs. In Codex, the warning about omitted skills should go away.

Then check the behavior, not just the size. Pick five requests that should trigger the new skill.
Run each in a fresh session with claude -p "<request>" --output-format json, and look for a
Skill tool call naming it. Most of the five should now load the skill.

When Trimming the Listing Is the Wrong Tool

When you have a handful of skills. Ten skills with 400-character descriptions fit with room to
spare. Rewriting them buys nothing.

When the rule must hold every time. A skill is advice the model can skip, even when it was
loaded. If every edit must pass a check, put the check in a hook. Claude Code's own docs say
the same.

When the listing is not the problem. A skill can be listed in full and still not be chosen. The
model tends to skip skills for tasks it can do alone, like reading one file. That is a
description problem, and no budget change will fix it.

And raising the budget is not free. The listing is read on every turn. Doubling it on a big team
costs context that every session pays for.

Three questions before you change anything:

Does /context show the Skills row near its budget on the smallest model you use?
Do your newest skills say when to use them in the first sentence, and in the name?
Which of your skills are only ever run by hand, and still listed for the model?

Glossary

skill — a folder with a SKILL.md file: a name, a description, and instructions for one kind of task.
skill listing — the names and descriptions the model sees at the start of a session.
context window — the model's working memory for a session, measured in tokens.
progressive disclosure — loading the listing first, a skill's instructions on a match, and its files only when needed.
invoke — run a skill, either because the model chose it or because you typed its name.

How to Correct an LLM Judge's Pass Rate With 100 Human Labels

Mayur Agnani — Mon, 26 Oct 2026 12:05:05 +0000

Engineer's view. This is a health check that returns 200 while the database is down. Your uptime number then mixes real uptime with the check's blind spot. A model grading your outputs does the same thing. It passes some bad ones, so the score runs high and every real fix looks smaller than it was.

TL;DR. The pass rate a grading model reports is part your product and part the grader's own mistakes. Measure those mistakes on a small set a person has checked, and you can subtract them back out.

Explain Like I'm 5

You run a fruit stand, and a helper sorts apples into a good basket and a bad basket. She
is quick, but she is kind. Most bruised apples land in the good basket too.

So you count the good basket and feel proud. Nine apples in ten look fine. That number is
partly your apples and partly her kindness.

To find the real number, you test her. Hand her fifty apples you know are good and fifty
you know are bruised. Count how many of each she gets right. Now you know how many bruised
apples she lets through on a normal day. Take that many out of the good basket, and what is
left is the true count.

The Problem

You have shipped this bug already, in a system with no model in it.

Your uptime dashboard said 99.9% for a month. Users kept filing tickets about errors. After
a day of digging you found the health check. It pinged an endpoint that returned 200 as long
as the web process was alive. When the database was down, the check still passed.

Nothing was wrong with the arithmetic. The dashboard faithfully counted what the check said.
But the check had a blind spot, so the number mixed two things: real uptime and the check's
habit of passing broken states. You could not fix the dashboard until you knew how often the
check was wrong.

An eval that uses an LLM judge has that bug at full size. An LLM judge is a model you
prompt to grade each output of your feature as Pass or Fail. Its pass rate goes on the
dashboard as if it were your product's pass rate. It is not. Judges make mistakes in both
directions, and they lean lenient. One 2025 study of judges grading feedback on student code
found they passed 96% of good answers but failed fewer than a quarter of bad ones.

That leniency does two kinds of damage. The score runs high. Worse, it hides progress. When
the judge passes most bad outputs anyway, fixing some of them barely moves the number.

The fix: label about a hundred outputs by hand, measure how often the judge agrees on the
good ones and on the bad ones, and use those two rates to undo the blend.

The Fix: Measure the Judge's Two Error Rates, Then Undo Them

Where does the wrong pass rate come from?

From two numbers the dashboard never shows. The TPR is the share of truly good outputs
the judge passes. The TNR is the share of truly bad outputs the judge fails. Here, Pass
counts as the positive answer.

The judge's pass rate is the good outputs it passes plus the bad outputs it lets through. A
strict judge with a TPR of 0.92 and a TNR of 0.88 turns a true 70% into 68%. That is close,
but only by luck: its two mistakes happen to cancel.

The Rogan-Gladen correction runs that arithmetic backwards. It comes from medical
testing, where a lab test has the same two error rates. Add the TNR to the judge's pass
rate, subtract one, and divide by the TPR plus the TNR minus one.

Why does a real fix look smaller?

Because only part of any real change gets through. That part is the TPR plus the TNR minus
one. For the strict judge it is 0.80, so a real eight-point gain shows as 6.4 points.

A lenient judge with a TNR of 0.24 keeps only a fifth of the change. Your team ships a real
improvement, and the chart moves by 1.6 points.

Can the correction rescue the lenient judge?

No, and this is the part that bites. The correction divides by that same 0.20. Dividing by a
small number turns small mistakes in your labels into huge swings. With a hundred labels, the
lenient judge's corrected rate could be anything from 32% to 100%. Fix the judge first.

What This Means for You

When this matters. Any time a model grades your outputs and you report the share it
passed. That covers a nightly eval, a release check, and a quality chart on a dashboard. It
matters most when you compare two numbers: this release against the last one, or one prompt
against another.

How it affects you. The level is off, usually too high. The comparisons are shrunk, so
real improvements look small and a prompt change that helped can look like noise. And it
gets worse quietly. Swap in a stricter judge model, and your pass rate drops on a day you
shipped nothing.

What to do about it. Start with ten minutes and no new tooling. Find ten outputs you
know are bad. A thumbs-down from a user, a bug report, or a few you break on purpose all
work. Run your judge on them and count how many it passes. If it passes four, your judge
lets roughly 40% of failures through. Every bad output it misses is counted on your dashboard
as a success.

Then do it properly. Label about a hundred outputs drawn from real traffic, half good and half
bad. Measure the judge's TPR and TNR on them. Report the corrected rate with an interval
next to the raw one.

If the TPR plus the TNR comes out under about 0.3, stop. Do not correct. Rewrite the judge
prompt or narrow it to one failure mode, and measure again. The next section shows the three
pieces of code, and the check that tells you they work.

Implementing It

The change.

Three people touch this. Each writes a little code, and none of it is hard. Getting the
convention wrong is the usual failure, so name it once and use it everywhere.

Role 1, whoever owns the labels. Keep the human verdicts beside the judge verdicts, one
line per output, sampled from production. Balance the classes on purpose. Real traffic
might be 90% good, and ten bad examples cannot measure a TNR.

{"id": "conv-8812", "human": 1, "judge": 1, "split": "dev"}
{"id": "conv-9034", "human": 0, "judge": 1, "split": "test"}
{"id": "conv-9107", "human": 0, "judge": 0, "split": "test"}

Use the dev rows while you rewrite the judge prompt. Measure the final rates on the test
rows only, once. Rates measured on the rows you tuned against will look better than they are.

Role 2, whoever owns the eval script. Compute the two rates, correct the pass rate, and
refuse when the judge is too weak.

def judge_rates(human, judge):          # 1 = Pass, for both lists
    good = [j for h, j in zip(human, judge) if h == 1]
    bad = [j for h, j in zip(human, judge) if h == 0]
    return sum(good) / len(good), 1 - sum(bad) / len(bad)

def corrected_pass_rate(p_obs, tpr, tnr, min_youden=0.30):
    youden = tpr + tnr - 1
    if youden < min_youden:
        return None                     # report "judge too weak", not a number
    return min(1.0, max(0.0, (p_obs + tnr - 1) / youden))

The interval comes from resampling. Redraw the labeled rows and the production verdicts with
replacement, recompute both rates and the corrected rate, and repeat 2,000 times. The middle
95% of the results is the interval. To compare two releases, draw one pair of rates per round
and use it for both. The judge's error is shared, so part of it cancels. The full version is in
code_example.py. If you would rather not write it, the MIT-licensed judgy package does the
same thing:

from judgy import estimate_success_rate
theta, low, high = estimate_success_rate(
    test_labels=human, test_preds=judge, unlabeled_preds=production_verdicts)

Role 3, whoever owns the dashboard and the review. Show the raw rate, the corrected rate,
and its interval. Store the judge's TPR, TNR and exact model version next to each run. A
new judge prompt or model needs new rates before its numbers can be compared.

How you know it worked.

Run two checks. First, use the dev rows to measure the rates, then correct the judge's pass
rate on the test rows. The corrected number should land near the human pass rate on those
rows. The raw number usually will not.

Second, swap the convention on purpose. In the Code tab, a corrected 70.0% becomes 75.0% when
TPR and TNR trade places. Some guides call a caught failure the positive case, so their TPR
is your TNR. If two people's numbers disagree by a few points, check this first.

When the Correction Is the Wrong Tool

When the judge is barely better than a coin. The correction divides by the judge's two rates
added together, minus one. Near zero, the interval covers almost everything. Improve the judge
until both rates clear 0.8, then correct.

When today's failures are a new kind. The correction assumes the judge misses failures as often
in production as it did on your labels. In the Code tab, the labels say the judge fails 88% of
bad outputs. Today's failures are subtler, and it fails only 60% of them. The true rate is 60%,
and the stale correction reports 74%. Re-label a fresh sample when the product changes.

When you need a verdict on one output. The correction fixes a rate across many outputs. It says
nothing about whether a given reply was good, so it cannot gate a single response.

And when you can afford to label everything. At fifty outputs a week, a person is the eval.

Three questions before you adopt it:

Do both of the judge's rates clear 0.8 on outputs it never saw while you tuned it?
Were your labels drawn from the same traffic the judge is grading now?
Will the judge's prompt or model change before the next report? If so, its rates expire.

Glossary

LLM judge — a model prompted to grade each output of your feature as Pass or Fail.
TPR — the share of truly good outputs the judge passes.
TNR — the share of truly bad outputs the judge fails.
Rogan-Gladen correction — arithmetic that turns a judge's pass rate back into the true one, using its two rates.
interval — the range the true pass rate probably sits in, after counting the noise in your labels.

How to Compare Model Prices When Each Model Counts Tokens Differently

Mayur Agnani — Sun, 25 Oct 2026 11:52:22 +0000

Engineer's view. This is comparing cloud machines by price per vCPU when one provider's vCPU is a full core and the other's is half of one. The unit has the same name and a different size. A token is that kind of unit: every model defines its own.

TL;DR. Each model splits text into tokens with its own vocabulary, so the same prompt is a different number of tokens on each. A price per token only compares after you measure tokens per character on your own text. The same effect quietly shrinks context windows and output limits.

Explain Like I'm 5

Two shops sell rope by the "length." At the first shop, a length is your arm. At the second, a length is the shopkeeper's arm, which is shorter. Both signs say ten coins per length. You need enough rope to cross a river. At the second shop you need more lengths, so you pay more. The price signs matched, and the bills did not. To compare, measure the river once in your own steps. Then ask each shop how many of its lengths that is.

The Problem

You have shipped this bug before, in a system with no AI in it.

Your team picked a cloud machine by price per vCPU. Provider A was cheaper per vCPU, so the batch jobs moved there. The jobs ran slower, and the monthly bill went up. On provider B, a vCPU was a full core. On provider A, it was one hyperthread, about half a core. Both units were called vCPU. They were not the same size, so the comparison was wrong before anyone ran it.

Model pricing has the same bug. Every provider lists a price per million tokens, the chunks of text a model reads and writes. Comparison sites line those prices up in one table. Your usage dashboard and your plan's limits are in tokens too. It looks like one shared unit.

It is not. Each model cuts text into tokens its own way. In April 2026, Anthropic shipped Opus 4.7 at the same price as Opus 4.6, with a new tokenizer. OpenRouter then studied over a million real requests from users who switched. Their prompts now came out 32% to 45% longer in tokens. Simon Willison pasted one system prompt into both models. 5,039 tokens became 7,335. The price tag did not move, and the bill did.

The fix: convert every price to your own text before you compare, then recheck every limit you set in tokens.

The Fix: Measure Tokens per Character on Your Own Text

What is a token, exactly?

A tokenizer turns text into tokens before the model sees it. It is trained once, before the model. Training starts from single characters. It then keeps merging the pair that appears most often in its training text into a new, longer token. After enough merges, common words are one token each. Rare words, odd code and unfamiliar scripts stay split into small pieces. This method is called byte-pair encoding.

Why do two models disagree on the same text?

Each tokenizer learned from different text. In code_example.py, two toy tokenizers get the same number of merges. One trained on prose, the other on prose plus code. An 80-character line of Python costs 70 tokens on the first and 44 on the second. The prose sample costs 33 on both. Code, JSON and languages other than English vary the most.

So what number can I compare?

The cost of one of your tasks: price per token, times tokens per character, times characters sent. Say model B lists 17% cheaper but cuts your text into 35% more tokens. It costs 13% more per task.

Does 35% more tokens always mean a 35% bigger bill?

If nothing else changes, yes. Cached and uncached tokens scale alike. But a new model also changes how much it writes. OpenRouter found median answers on short prompts got 62% shorter, so that bill stayed flat. Above 2,000 tokens, costs rose 12% to 27%.

What This Means for You

When this matters. You choose between models by their price page. You switch the default model in Cursor, Claude Code or your own app. A provider ships a new version "at the same price." You run near a context limit, a usage cap or a tight max_tokens. In each case, a number you set or compare is measured in that model's tokens.

How it affects you. Your bill is the obvious part. The quieter part is every limit. A limit in tokens is a limit in text only for the model it was set on. Move to a tokenizer that makes 35% more tokens, and a 200,000-token window holds 26% less of your repository. An 800-token answer cap now stops at about 2,370 characters instead of 3,200. Long answers get cut off sooner, and nothing errors. Agents start compacting their history earlier. A weekly token allowance runs out sooner, for the same work.

This is the vCPU bug again. The number on the label is right. It is counted in a unit whose size you never checked.

What to do about it. Pick 20 to 50 real prompts and answers from your own logs. Count them with each candidate model's own token counter. Divide by characters to get tokens per thousand characters. Multiply by each price to get cost per task. Do the same check whenever a model changes version, even at an unchanged price. Then recompute every limit you set in tokens.

Implementing It

The change.

Three roles touch this: whoever picks the model, whoever tracks spend, and whoever sets limits.

Role 1, picking a model. Count your own sample with each provider's counter. Each counter counts with that model's tokenizer, so the ratio is honest:

import tiktoken, anthropic
from google import genai

def tokens_per_kchar(samples, count):
    return 1000 * sum(count(s) for s in samples) / sum(len(s) for s in samples)

enc = tiktoken.get_encoding("o200k_base")          # check which encoding your model uses
openai_tpk = tokens_per_kchar(samples, lambda s: len(enc.encode(s)))

claude = anthropic.Anthropic()
claude_tpk = tokens_per_kchar(samples, lambda s: claude.messages.count_tokens(
    model=CLAUDE_MODEL, messages=[{"role": "user", "content": s}]).input_tokens)

gem = genai.Client()
gemini_tpk = tokens_per_kchar(samples, lambda s: gem.models.count_tokens(
    model=GEMINI_MODEL, contents=s).total_tokens)

Use real prompts with their system prompt and tool definitions included. Those are often the largest part of each request. A sample of only short user messages will mislead you. Keep the sample in a file, so every later comparison counts the same text. Never estimate with "four characters per token." That rule of thumb was tuned on English prose for one tokenizer.

Role 2, tracking spend. Record cost per task, not tokens per day. A token count cannot be compared across models, and dollars can. Read the usage fields each response already returns:

u = resp.usage                                      # Anthropic Messages API
dollars = (u.input_tokens * P_IN + u.cache_read_input_tokens * P_CACHE_READ
           + u.cache_creation_input_tokens * P_CACHE_WRITE + u.output_tokens * P_OUT) / 1e6
log.info("task_cost", task=task_id, model=MODEL, dollars=dollars, chars=len(prompt))

OpenAI reports prompt_tokens, completion_tokens and prompt_tokens_details.cached_tokens instead. Gemini reports them in usage_metadata. Group by task type, then compare models on the same tasks. Store the model name on every record. Without it, a model switch looks like a change in traffic.

Role 3, setting limits. Every limit in tokens was tuned for one tokenizer. First list them: search your code and config for max_tokens, context budgets, history truncation and compaction thresholds. Then rescale them when you switch:

import math

def rescale(limit_tokens, tpk_old, tpk_new):
    return math.ceil(limit_tokens * tpk_new / tpk_old)

MAX_TOKENS = rescale(800, 250, 338)                 # 1082: same answer length in text
COMPACT_AT = min(rescale(150_000, 250, 338),        # 202,800 would pass a 200k window,
                 int(0.8 * NEW_WINDOW))             # so cap at the model's real limit

The window itself does not grow. When the rescaled number passes it, less of your text fits, and you plan for that.

How you know it worked. Your cost-per-task chart is continuous across a model switch. A jump there is a real change, not a unit change. The truncation rate stays the same after the switch. If answers suddenly hit max_tokens more often, a limit was not rescaled. Finally, your measured tokens per thousand characters for each model sits in your notes. The next price comparison should take five minutes.

When Per-Character Comparison Is the Wrong Tool

When quality decides, not price. A model that solves the task in one try beats a cheaper one that needs three. Cost per task must include retries, and per-character math does not see them.

When you pay a flat seat price with no token cap. The tokenizer then changes nothing you pay. It still changes how much fits in the context window.

When your inputs are mostly images or audio. Those are counted by size, resolution or duration, with rules of their own. Measure them separately.

When the gap is small. A 5% difference in tokens per character is inside the noise of most samples. Do not switch models over it.

When the workload will change soon. Tokens per character depends on what you send. A ratio measured on chat prompts will not hold for a code agent.

Before you adopt it, ask:

How many tokens per thousand characters does each model use on my own text?
Which limits in my system are set in tokens?
Did the last model switch change how long the answers are?

Glossary

token — a chunk of text a model reads or writes; each model defines its own.
tokenizer — the program that splits text into a model's tokens before the model sees it.
byte-pair encoding — training a tokenizer by repeatedly merging the most frequent pair of symbols.
tokens per thousand characters — how many tokens one model uses for 1,000 characters of your text.
context window — the most tokens a model can read in one request.

Why Forced JSON Output Is Always Valid and Sometimes Wrong

Mayur Agnani — Sat, 24 Oct 2026 13:19:03 +0000

Engineer's view. This is a required dropdown with no "Not applicable" option. Validation always passes, so the data looks clean. But users who had no right answer picked the first item, and your reports now count them. A JSON schema enforced token by token works the same way.

TL;DR. Forced JSON mode removes the tokens your schema forbids and rescales the rest, so the output always parses. The model still chooses among legal tokens. When it wanted something illegal, it guesses, and the guess looks as sure as a real answer.

Explain Like I'm 5

A teacher hands out a quiz where every question has three boxes. You must tick one. There is no box for "I don't know." So when you don't know, you tick a box anyway. The teacher sees a neat page with every box ticked. The teacher cannot tell which ticks you knew and which you guessed. The fix is easy: add a fourth box that says "I don't know."

The Problem

You have shipped this bug before, in a system with no AI in it.

A signup form asks for a country. The field is required, and the dropdown has no "Other" option. Users from a missing country still have to submit. So they pick the first item in the list. Months later, an analyst asks why so many users live in Afghanistan. The validation never failed once. It only made sure every answer was a legal one.

Structured output has the same bug. You give the model a JSON schema, and the engine guarantees the response matches it. OpenAI, Gemini, vLLM, SGLang and llama.cpp all offer this. Teams switch it on to stop parse errors, and the parse errors stop.

Then quieter failures start. FriendliAI ran Gemma 4 31B against a complex schema. The output filled with newlines until it hit the token limit, and the JSON never closed. In other pipelines, a field limited to three values gets one of the three, even for a document the model cannot read. The value looks as sure as any other.

All of these come from one rule. The engine decides which tokens are legal. The model still chooses among them, and it must choose even when it wanted to say something else.

The fix: give the model a legal way to say what it means, bound what it can repeat, and measure how much it was forced.

The Fix: Give the Model a Legal Way Out

How does the engine force valid JSON?

A model writes one token at a time, a word or a piece of one. At each step it gives every token in its vocabulary a probability. Constrained decoding adds one step. The engine checks the grammar built from your schema, deletes every token that would break it, and rescales the rest to sum to one. Then it samples as usual.

So what happens when the model wanted an illegal token?

Its wish is deleted, and the leftovers are scaled up. Take a field that must be high, medium or low. On a clear invoice, those three hold 95% of the model's probability. On a blurry scan, they hold 15%, because the model wanted to write "unknown." After rescaling, the blurry scan still gets high 40% of the time. Both outputs look the same.

Why does output sometimes fill with whitespace?

JSON allows spaces and newlines between fields, so the grammar never deletes them. A little leftover whitespace cannot do this alone. Even at a 99% share each step, filling 2,048 tokens has odds of about one in a billion. The loop needs the model to be nearly certain at every step. That happens when repetition feeds itself: each newline makes the next more likely.

Does the order of fields matter?

The engine writes fields in the order the schema lists them, left to right. A confidence value written before a reasoning field cannot depend on that reasoning.

What This Means for You

When this matters. You use JSON mode, a response schema or strict tool calls, and your code acts on a field's value. Agents do this constantly. A tool call is a schema the model must fill in. A routing step reads one enum. An extraction job writes rows to a database.

How it affects you. Parse-error rates drop to zero, so dashboards built on them look healthy. The failures move into the values. A forced pick is a wrong row nobody flags. A whitespace loop is a timeout, or a bill for 2,048 tokens of nothing.

This is the same bug as the country dropdown. Validation guarantees that every answer is a legal one. It says nothing about whether the model had an answer.

What to do about it. Today, open the schemas your code acts on. For every enum, ask what the model should say when it does not know. If there is no legal answer for that, add one. Next, put any reasoning field before the fields that depend on it. Whether that improves accuracy depends on the task and the model. It is cheap to test, so measure it on yours rather than assume. Then cap output length and treat a truncated response as a failure. If you host the model, switch off free whitespace and start logging how much probability the grammar removed.

Implementing It

The change.

Four roles touch this: whoever writes the schema, whoever writes the prompt, whoever runs the server, and whoever reads the response.

Role 1, the schema. Give every field your code acts on a legal "I don't know." Put reasoning first. Bound free strings where your provider's schema subset allows maxLength:

from typing import Literal
from pydantic import BaseModel, Field

class Invoice(BaseModel):
    reasoning: str = Field(max_length=600)        # written first, read by the fields below
    vendor: str | None = Field(max_length=120)    # None is the legal "not on the page"
    total: float | None
    confidence: Literal["high", "medium", "low", "unknown"]

Lint every schema in CI with check_schema() from code_example.py. It flags an enum with no way out, an unbounded string, and an answer written before its reasoning.

Role 2, the prompt. Include one filled-in example of the output. FriendliAI's whitespace loop went away when the prompt carried a concrete example. The mask only removes options; the example puts probability on the right ones. Use a real record, not placeholders. An example full of "string" teaches the model to write "string". Show an unknown case too, so the model sees that the way out is allowed.

Role 3, the server, if you host the model. vLLM can forbid whitespace between JSON fields, which removes the loop's open door:

vllm serve <model> \
  --structured-outputs-config '{"backend":"xgrammar","disable_any_whitespace":true}'

It works only with the xgrammar and guidance backends. With transformers, wrap your grammar processor to record how much probability it removed:

import torch
from transformers import LogitsProcessor

class KeptMass(LogitsProcessor):
    def __init__(self, grammar):                  # any processor that sets illegal tokens to -inf
        self.grammar, self.kept = grammar, []
    def __call__(self, input_ids, scores):
        before = torch.softmax(scores.float(), dim=-1)
        out = self.grammar(input_ids, scores.clone())
        self.kept.append((before * torch.isfinite(out)).sum(-1))
        return out

Role 4, the caller. A cut-off response is a failure, not something to repair:

resp = client.chat.completions.create(model=MODEL, messages=msgs,
    response_format={"type": "json_schema", "json_schema": SCHEMA}, max_tokens=800)
choice = resp.choices[0]
if choice.finish_reason == "length":
    raise TruncatedOutput(resp.id)                # retry or route to review; never parse
row = Invoice.model_validate_json(choice.message.content)

Gemini reports the same case as MAX_TOKENS in finishReason. Count truncations per schema, because one bad schema hides inside a healthy average.

How you know it worked. Three numbers per field, on a week of traffic. The truncation rate should be near zero. The unknown rate should be above zero, and roughly match how often a human finds no answer in a sample. If it is exactly zero, the model is still being forced somewhere. If you host, look at kept mass on value tokens only. Structure tokens like { lose mass on every request for harmless reasons. A value position that kept under half the model's probability is a guess worth reviewing.

When Forcing JSON Is the Wrong Tool

When the model needs room to think. A schema that starts on token one leaves no space for free reasoning. For hard tasks, let the model answer in plain text first. Then make a second, cheap call that formats the answer into JSON.

When your schema is huge or changes per request. The engine compiles each new grammar before the first token. A cached schema costs little, but a new one per request adds delay to every first token. Grammar engines cache compiled schemas for this reason.

When you use a small model on a hard schema. Forcing makes the output parse. It cannot make a weak model know the answer. Nested optional fields are where small models fill values with whatever is left.

When the provider's schema subset cannot express your rules. Some keywords are silently unsupported or rejected. Check the response against the full schema in your own code anyway.

Before you adopt it, ask:

What should each field say when the model does not know?
Which fields does my code act on, and is anything written before its reasoning?
How would I notice a value the model was forced into?

Glossary

token — a word or piece of a word; the model writes one at a time.
Structured output — an API mode that guarantees the response matches a JSON schema you supply.
Constrained decoding — deleting the tokens a grammar forbids at each step, then sampling from what is left.
enum — a schema field limited to a fixed list of allowed values.
kept mass — the share of the model's probability that survives the grammar's deletion at one position.

Why Split Prefill and Decode GPUs Lose Time in the Hand-Off, Not the Network

Mayur Agnani — Fri, 23 Oct 2026 12:01:50 +0000

Engineer's view. This is a Node.js server doing a synchronous parse on every upload. Each parse is quick, but it blocks the one loop every connection shares, so every response slows. With a fixed connection pool, slower responses then mean waiting for a free connection. The decode server's hand-off work behaves the same way.

TL;DR. When prompt processing and token generation run on separate GPUs, each request's cache must cross between them. The bytes are cheap. Setup work on the generation server's loop slows every token, and that can run the server out of slots.

Explain Like I'm 5

A bakery has one person mixing dough and another baking it. The mixer hands each bowl to the baker. Passing a bowl takes a second. But this baker stops all the ovens to write a label for every new bowl. Each label is quick. When bowls arrive fast, the ovens are stopped so often that every loaf bakes slower. Soon the oven shelves are full, and new bowls wait in line.

The Problem

You have shipped this bug before, in a system with no AI in it.

An API reads from a database through a pool of 20 connections. A release adds a small synchronous step to each query. Every query now holds its connection a little longer. At normal traffic nobody notices. At peak, all 20 connections are busy, and requests wait in line for one. The step costs milliseconds. The wait costs seconds. Nobody finds it by looking at the database, because the database is fine.

Disaggregated serving has the same bug. It splits an LLM server in two. Prefill GPUs read the whole prompt. Decode GPUs then write the answer one token at a time. Splitting them stops long prompts from stalling other users' answers. In between, each request's KV cache, the model's notes on the prompt, moves from one pool to the other.

Teams blame the network when the first token is slow. The numbers say otherwise. vLLM measured Qwen3-32B on two eight-GPU H200 servers. At 8 requests per second with 16,000-token prompts, the first token took about 1,744 ms with the default hand-off. A different hand-off cut it to about 540 ms. Moving that prompt's cache takes about 10 ms. How to Work Out How Much GPU Memory One Request Really Needs sized that cache.

The lost second is setup work. Part of it runs on the decode server's main loop, where it pauses every request in progress.

The fix: keep hand-off work off the decode loop, and size decode slots with headroom. Then the split pays for itself.

The Fix: Keep Hand-Off Work Off the Decode Loop

Why split prefill and decode at all?

Prefill is limited by arithmetic. Decode is limited by memory reads. On one server, a long prefill delays every answer being written. Splitting them lets you tune first-token time and per-token time separately. vLLM's own guide is blunt about the cost: disaggregation does not improve throughput. It buys predictable latency.

How much do the bytes actually cost?

Very little. Qwen3-32B stores 256 KiB per token. A 16,000-token prompt is 4.2 GB. With tensor parallelism across 8 GPUs, each GPU sends its own slice over its own links. At 50 GB/s per GPU, that takes about 10 ms.

Then where does the second go?

Into two places. First, pull runs its steps in series after prefill. The router relays where the blocks are. Decode allocates space. Then decode reads the blocks. Second, decode does that setup on its main loop. That loop also produces every token for every request. While it posts transfer work, nobody gets a token.

So the loop loses arrivals per second times setup time per request. A vLLM pull request measured 14.7 ms of posting per request at 2,000-token prompts. At 8 requests per second, the loop loses 12%. Moving that work to a thread took time per token from 6.14 ms to 5.52 ms.

Why does a slower token slow the first token?

Because a request holds a decode slot until its answer ends. Slots held equals arrivals times output tokens times time per token. That is Little's law. Past --max-num-seqs, new requests wait for a slot before their cache can land.

What does push change?

Decode registers its free blocks as soon as the request arrives. Prefill writes straight into them from a background thread. Nothing waits after prefill, and nothing runs on the decode loop.

What This Means for You

When this matters. You run prefill and decode on separate GPUs with vLLM, SGLang, Dynamo or llm-d. Your prompts are long, and your traffic is high.

How it affects you. Look at vLLM's 16,000-token run at 8 requests per second. Time per token rose from 7.0 ms with push to 10.4 ms with pull. With 128-token answers, that means 7.2 slots held with push and 10.6 with pull. The benchmark started its servers with --max-num-seqs 8. So pull asked for more slots than existed. That turns a few milliseconds per token into a second of waiting.

This is the same arithmetic as the connection pool. A small per-request cost decides nothing until the pool is nearly full. Then it decides everything. It also explains why the two modes match at low load. At 1 request per second, even pull holds barely one slot. The setup still happens, but nobody is waiting behind it.

What to do about it. Today, compute slots held for your own traffic at peak. Use your measured time per token, not the benchmark's. Keep it under about 80% of --max-num-seqs. Then move hand-off work off the decode loop, and re-measure. If you cannot change the hand-off this quarter, raise --max-num-seqs only if decode memory allows it. More slots need more KV cache space on every decode GPU.

Implementing It

The change.

Three roles touch this. Whoever plans capacity sizes the decode pool. Whoever owns the serving config picks the hand-off. Whoever runs the load test proves it.

Role 1, capacity. Compute the three numbers for your model and traffic. The first is usually small. The third is the one that bites:

def kv_bytes(tokens, layers, kv_heads, head_dim, dtype_bytes=2):
    return tokens * layers * 2 * kv_heads * head_dim * dtype_bytes

def transfer_ms(nbytes, link_gb_s, tp=1):
    return nbytes / tp / (link_gb_s * 1e9) * 1000   # each GPU sends its slice

def slots_held(qps, out_tokens, tpot_ms):
    return qps * out_tokens * tpot_ms / 1000         # Little's law

transfer_ms(kv_bytes(16_000, 64, 8, 128), 50, tp=8)  # 10.5 ms
slots_held(8, 128, 10.4)                             # 10.6, with 8 slots

Take tpot_ms from production at peak, not from a quiet benchmark. Use your real output length, too. Agents that write long answers hold slots far longer than chat does. A 2,000-token agent answer holds a slot sixteen times longer than a 128-token chat reply. So the same request rate needs sixteen times the slots.

Role 2, the serving config. The connector is a setting in --kv-transfer-config on both servers. The vLLM post ran push with this, on the prefill side:

vllm serve Qwen/Qwen3-32B --tensor-parallel-size 8 --max-num-seqs 8 --port 8100 \
  --kv-transfer-config '{"kv_connector":"NixlPushConnector","kv_buffer_device":"cuda",
    "kv_role":"kv_producer","engine_id":"prefill-engine-001","kv_rank":0,
    "kv_parallel_size":2,"kv_connector_extra_config":{"backends":["LIBFABRIC"]}}'

The decode server uses the same connector with "kv_role":"kv_consumer", as in vLLM's NIXL guide. Mixing modes crashes the server after the first request. Push also needs a proxy that knows the prefill server's address up front. The post uses disagg_proxy_pushconnector_demo.py. Pull stays the right choice if you route each request to the least-loaded decode server after prefill. Push picks decode early, so it gives that up.

Role 3, measuring. Sweep request rate and prompt length together, because the gap only opens at the top of both:

for qps in 1 2 4 8; do for len in 1024 16384; do
  vllm bench serve --model Qwen/Qwen3-32B --request-rate $qps \
    --random-input-len $len --random-output-len 128 --num-prompts 400
done; done

How you know it worked. Push and pull should match at low rate and short prompts. At the top of the sweep, time per token should stay near its light-load value. Watch vllm:num_requests_waiting on the decode server. It should stay near zero at your peak rate. If it climbs, you are out of slots, whatever the hand-off. Then split first-token time into its parts. Compare the prefill server's own first-token time with the proxy's. The difference is the hand-off plus any wait for a slot. After the change, that difference should shrink most at long prompts and high rates.

When Splitting Prefill and Decode Is the Wrong Tool

When you want more throughput. Splitting does not add any. It trades GPUs for steadier latency.

When your prompts are short. Prefill then barely disturbs decode, so there is little interference to remove. Chunked prefill on one server may be enough.

When you have one GPU node. The hand-off then crosses no network, but you still pay for two pools and a proxy.

When you need load-aware decode routing. Push chooses the decode server before prefill finishes. If your decode servers run unevenly, pull's late choice may be worth its setup cost.

When the decode pool is already full. No hand-off fixes too few slots. Add decode capacity first, then tune the hand-off.

Before you adopt it, ask:

How many decode slots does my peak traffic hold, by Little's law?
Is my first-token time spent in prefill, in the hand-off, or in a queue?
Do I need to route decode after prefill, or can I choose early?

Glossary

Disaggregated serving — running prompt reading and answer writing on separate GPU pools.
Prefill — the step that reads the whole prompt and builds its cache.
Decode — the step that writes the answer, one token at a time.
KV cache — the model's stored notes on every token so far, kept on the GPU.
Little's law — items in a system equal arrival rate times time each one stays.
slot — one of the requests a decode server can generate for at once.

Your AI app might be losing money on its heaviest users. Here's how to check.

Mayur Agnani — Fri, 23 Oct 2026 00:01:45 +0000

If you sell an AI app by subscription, your revenue per user is flat and your cost per user is not.

A light user costs you a few cents a month. A heavy one, or a user stuck in an agent loop, can cost more than they pay you. Your OpenAI bill shows tokens. Your payments dashboard shows revenue. Nothing joins the two by user, so you find out from the invoice.

I built FiscaLLM to answer one question: which users cost me more than they pay? Here's the idea and how to try it in about ten minutes.

What you need

A backend between your app and the AI provider (so apps that call OpenAI straight from the phone aren't covered yet).
A way to know what each user pays: RevenueCat, Stripe (beta), or your own code.

1. Report usage after each AI call

import { createMeter } from "fiscallm"; // npm install fiscallm  (Python: pip install fiscallm)

const meter = createMeter({ apiKey: process.env.FISCALLM_KEY, waitUntil });

const res = await openai.chat.completions.create({ model, messages });
meter.trackOpenAI(res, { userId, feature: "chat" });

The SDK reads only the model name and token counts. It never sees prompts or answers. Events are batched in the background, so it doesn't add latency to your request, and if FiscaLLM is down your app is unaffected.

2. Connect revenue

Paste a RevenueCat or Stripe webhook URL, or send payments yourself with the revenue API. FiscaLLM joins revenue and cost on the user ID.

3. Look at margin per user

You get margin per user, per plan and per free trial, plus which model or feature drives the bill. Then you can act:

Budget alerts by email, Slack or webhook when a user goes over.
A runaway-loop alert for users whose usage suddenly explodes.
Optional caps: mark a budget as Block and call meter.check(userId) before an AI call. It fails open, so an outage never blocks your users.

A few details that mattered

No double counting. Events carry client-generated UUIDs, so retries are safe.
Cached tokens are priced separately from normal input tokens.
Unknown models are flagged as unpriced, not silently counted as zero.
RevenueCat aliases are resolved, so a user's anonymous and logged-in IDs merge.

Honest limits

Stripe support is beta. I've tested it on sample payloads only, and it's USD only.
Caps use cost already recorded, so a user can go slightly over before being refused.
Built-in prices are checked against provider pages (ACCURACY.md), but I haven't compared them with a real invoice over weeks yet.
No customers yet. The demo dashboard uses sample data.

Try it

Live demo, no sign-up: https://fiscallm.com/demo?utm_source=devto&utm_medium=article&utm_campaign=launch
Code (MIT, TypeScript and Python SDKs plus the server): https://github.com/magna56/fiscallm
Self-hosting guide: SELF-HOSTING.md in the repo, so your data never touches a third party's servers. Or use the free hosted version.

I'm looking for the first 10 apps to try it. If you ship an AI app with subscriptions, tell me what's missing in the comments or on the roadmap board.

DEV Community: Mayur Agnani

How to Fix Jargon in Speech-to-Text Without Adding Words Nobody Said

Explain Like I'm 5

The Problem

The Fix: Keep the Term List Short, Ordered and Tested on Near Misses

What does the prompt actually change?

Why does it write words nobody said?

Doesn't a longer list cover more?

What can a list never fix?

What This Means for You

Implementing It

When a Term List Is the Wrong Tool

Glossary

Further reading

How to Stop a Long LLM Answer From Freezing When the Server Runs Out of Memory

Explain Like I'm 5

The Problem

The Fix: Match the Limits to Your Server's Admission Rule

What does each server hold back?

What happens when memory runs out?

Is the redone work the expensive part?

What This Means for You

Implementing It

When Tuning Admission Is the Wrong Tool

Glossary

Further reading

How to Tell What Claude Code and Codex Forget When They Compact a Session

Explain Like I'm 5

The Problem

The Fix: Sort Facts by How They Entered the Session

What reloads from disk?

What does a budget keep?

Which facts does the summary drop?

What This Means for You

Implementing It

When Managing Compaction Is the Wrong Tool

Glossary

Further reading

How to Tell Whether a Second Model Reviewing Your Agent's Work Pays for Itself

Explain Like I'm 5

The Problem

The Fix: Measure Fixed and Broken, Then Review Only the Hard Tasks

What does one review pass add?

Why isn't the reviewer's own accuracy enough?

When does review hurt?

What This Means for You

Implementing It

When a Reviewer Model Is the Wrong Tool

Glossary

Further reading

How Many Pull Requests a Merge Queue Can Land in an Hour

Explain Like I'm 5

The Problem

The Fix: Cut the Failure Rate Before You Add Pipelines

Why doesn't running more pipelines fix it?

Where does the failure rate come from?

What happens to the wait near the ceiling?

What This Means for You

Implementing It

When Tuning the Queue Is the Wrong Tool

Glossary

Further reading

How to Find the Agent Actions a Prompt Injection Can Control

Explain Like I'm 5

The Problem

The Fix: Decide the Actions Before the Agent Reads Anything Untrusted

Which decisions does an injection reach?

Why not just tell the model to ignore instructions in tool results?

When does a reached decision become a leak?

What This Means for You

Implementing It

When Changing the Design Is the Wrong Tool

Glossary

Further reading

How Claude Code and Codex Decide Which Skill Descriptions the Model Reads

Explain Like I'm 5

The Problem

The Fix: Keep the Listing Under Budget and Put the Use Case First

What does the model actually see?

How big is the listing allowed to be?