DEV Community: jidonglab

Python requests timeout=60 Let My LLM Stream Run for 38 Minutes

jidonglab — Mon, 02 Nov 2026 00:16:44 +0000

My nightly summarization job had timeout=60 on every request. One call ran for 38 minutes and 12 seconds anyway. Nothing crashed and nothing logged an error. The 6:00 a.m. report just didn't show up.

The Python requests timeout is not a deadline for the whole call. It caps how long you wait for the next bytes. Point it at a streaming LLM endpoint that emits a token every 70 milliseconds and that timeout never fires.

This post covers what I found over 14 nights and 33,600 calls, how I tracked it down, and the 20 lines that fixed it.

TL;DR

requests.post(..., timeout=60) sets a connect timeout and a read timeout. The read timeout is the longest allowed gap between bytes, not the total duration.
A streaming LLM response sends bytes constantly, so it never trips the read timeout, even when the model is stuck in a repetition loop for half an hour.
httpx works the same way: Timeout(60) covers connect, read, write and pool separately. There is no total-time cap by default.
Fix: set max_tokens, keep a short read timeout, and enforce a wall-clock deadline yourself (a check inside the stream loop, or asyncio.timeout() around the whole call).
After the fix: 0 missed reports in the next 14 nights. The longest call dropped from 38m12s to 1m51s.

The setup

Nothing exotic here:

A 7B instruct model served by vLLM through its OpenAI-compatible endpoint
A Python batch job with 6 worker threads
About 2,400 support tickets per night, each summarized into a short JSON blob
Streaming on, so I could log time-to-first-token

The call looked like this, and I'd guess half the LLM scripts on GitHub look the same:

r = requests.post(URL, json=payload, stream=True, timeout=60)
for line in r.iter_lines():
    if line.startswith(b"data: ") and line != b"data: [DONE]":
        chunk = json.loads(line[6:])
        parts.append(chunk["choices"][0]["delta"].get("content") or "")

I read timeout=60 as "this call dies after a minute." That was wrong.

What does the timeout parameter in Python requests actually do?

The Python requests timeout sets two limits: how long to wait for the TCP connection, and how long to wait between bytes on the socket. Passing one number sets both to that value. Passing a tuple like (5, 30) sets them separately. Neither one limits total request time.

The requests docs say this outright. The read timeout is the number of seconds the client will wait for the server to send a byte. Every byte that arrives resets that clock.

For a normal JSON API this almost never matters. The server is silent until it has an answer, so "time between bytes" and "total time" are basically the same number.

Streaming changes that. A server streaming tokens is never silent for long. My model produced a chunk roughly every 70 ms under load, so the read timeout restarted about 14 times a second. A 60-second read timeout couldn't fire unless the model stopped completely.

Why would an LLM stream run for 38 minutes?

It ran that long because the model got stuck repeating itself, and nothing told it to stop. I hadn't set max_tokens. vLLM's OpenAI-compatible server then allows generation up to whatever context window is left after the prompt. My context was 32k tokens, the prompts were about 1.5k, and that left roughly 30,000 tokens of room.

At around 15 tokens per second per request under batch load, 30,000 tokens takes about 33 minutes. My worst call ran 38. The numbers line up.

The outputs from those calls looked like what you'd expect:

"summary": "Customer reports login failure. Customer reports login failure.
Customer reports login failure. Customer reports login failure. ...

That repeated a few thousand times. Some tickets (very long ones, pasted stack traces, logs full of near-identical lines) push small models into these loops. It's rare, but over 2,400 calls a night it happens.

How rare, and how bad?

Here's what 14 nights of logs showed:

Metric	Value
Total calls	33,600
p50 duration	4.1 s
Calls over 5 minutes	61 (0.18%)
Longest call	38m 12s
GPU time spent on those 61 calls	~9.2 hours
Nights the 6:00 a.m. report was late	3

0.18% sounds harmless until you remember there were only 6 workers. When 2 or 3 runaway calls landed on the same night, a third to half of my capacity spent half an hour generating "Customer reports login failure." The other workers also slowed down, because every runaway sequence took a slot in vLLM's batch and a chunk of KV cache.

On the 3 late nights, 4 or more workers were stuck at once.

How I found it

The job wasn't failing, so I had no errors to read. What I had:

py-spy dump on the stuck process. All 6 threads were sitting in iter_lines() → urllib3 → socket.recv. So they weren't deadlocked. They were busy receiving data.
vLLM's /metrics endpoint. vllm:num_requests_running sat at 6 with nothing waiting. The server thought it was doing useful work.
Logging output length. Normal summaries came out at 150 to 400 tokens. The 61 slow calls averaged about 21,000.

Then I wrote a 15-line test server that sent one SSE line every 30 seconds and pointed requests.post(..., timeout=60) at it. It sat there for an hour until I killed it. That settled it.

How do you set a total timeout for a streaming request in Python?

Set a total timeout by enforcing a wall-clock deadline in your own code. Neither requests nor httpx gives you one. Check elapsed time on every chunk, cap generation with max_tokens, and close the connection when you give up so the server stops generating.

Here's what I run now:

import json
import time
import requests

def stream_completion(payload, deadline_s=90, read_timeout=20):
    payload = {**payload, "max_tokens": 1024, "stream": True}
    start = time.monotonic()
    parts, finish = [], None

    with requests.post(URL, json=payload, stream=True,
                       timeout=(5, read_timeout)) as r:
        r.raise_for_status()
        for line in r.iter_lines():
            if time.monotonic() - start > deadline_s:
                raise TimeoutError(f"exceeded {deadline_s}s wall clock")
            if not line.startswith(b"data: ") or line == b"data: [DONE]":
                continue
            choice = json.loads(line[6:])["choices"][0]
            parts.append(choice["delta"].get("content") or "")
            finish = choice.get("finish_reason") or finish

    return "".join(parts), finish

Four things changed, and each one matters:

max_tokens=1024. My longest legitimate summary was about 600 tokens. This alone puts a ceiling on the runaway case at roughly 70 seconds.
timeout=(5, 20). Short connect timeout, moderate read timeout. This still catches a server that goes completely silent.
The deadline check inside the loop. This is what actually caps total time. It only runs when a chunk arrives, so the true worst case is deadline_s + read_timeout, here 110 seconds. Know that number before you promise anyone an SLA.
The with block. Leaving it closes the socket. vLLM notices the disconnect and aborts the request, so the GPU stops generating junk for nobody. My server logs showed the aborts.

Is there a hard cap that doesn't depend on chunks arriving?

Yes. With async code, wrap the entire call in asyncio.timeout() (Python 3.11+). It cancels the task at the deadline whether or not bytes are flowing:

import asyncio
import httpx

async def stream_completion(client: httpx.AsyncClient, payload):
    async with asyncio.timeout(90):
        async with client.stream("POST", URL, json=payload) as r:
            async for line in r.aiter_lines():
                ...

Cancellation unwinds the async with and closes the stream. This is my preferred version. I kept the sync one above because the batch job is threaded and I didn't want to rewrite it at 2 a.m.

If you use an SDK built on httpx, like the OpenAI Python client, read what its timeout argument means for streams in the version you have installed. Don't assume it's a total deadline. Wrap the call in your own deadline either way. It costs one line.

What about the 58 calls that hit max_tokens?

Capping generation doesn't fix the loop. It just makes the loop cheap. In the 14 nights after the fix, 58 calls ended with finish_reason == "length", which in this job always meant a repetition loop.

I handle those explicitly now:

Never parse them as valid output. A truncated JSON blob full of repeated sentences is garbage.
Retry once at a slightly higher temperature (0.3 instead of 0). 49 of 58 came back clean.
Send the other 9 to a manual review queue with the ticket ID.

That's 9 tickets a fortnight for a person to look at, instead of 9 GPU-hours of "Customer reports login failure."

Results after the fix

Same job, same model, next 14 nights:

Longest call: 1m 51s (it hit max_tokens, then went through the retry path)
Calls over 5 minutes: 0
Late reports: 0
Total nightly runtime dropped by about 11%, mostly because runaway sequences stopped taking batch slots from healthy requests

The checklist I now apply to every LLM HTTP call

Always set max_tokens. A server default of "whatever context is left" is not a limit.
Use a tuple timeout: short connect, moderate read.
Add a wall-clock deadline that you enforce yourself.
Close the stream when you give up so the server stops generating.
Treat finish_reason == "length" as a failure until proven otherwise.
Log output token counts. That's how the 21,000-token outliers showed up.

So why didn't timeout=60 stop my request?

Because the Python requests timeout only limits connecting and the gap between received bytes. It never limits total request duration. A streaming LLM response sends a byte every few milliseconds, so the read timeout keeps resetting and a call can run as long as the model keeps generating: 38 minutes in my case, with no max_tokens set. To actually bound a streaming call, set max_tokens, keep a short read timeout, enforce a wall-clock deadline in your own loop or with asyncio.timeout(), and close the connection when the deadline hits so the server stops generating.

Written by the developer behind Preterview, an interview prep platform.

My Claude Code Sign-in Fix Needed Two Fixes of Its Own

jidonglab — Sun, 01 Nov 2026 18:29:43 +0000

On September 29 I removed a sign-in refusal from my app. The next day a review pass whose only job was to break that change found that it left some sign-ins reported as ready that then failed, and my repair of that opened a second hole. Both were caught before the first public release on October 1.

Refmade is a desktop app that wraps the Claude Code and Codex command-line tools and draws what they are doing as an office floor. It is an independent app, not affiliated with Anthropic or OpenAI. The UI is Korean right now; that changes later in this series. Claude Code and Codex agents wrote most of the code, working from specs I gave them. I planned the work, made the calls and reviewed the results. This post is about one call that turned out to be half right.

The problem: the app refused a sign-in the CLI accepts

The app has a "team" mode, where several agents plan, write and check a task. Until September 29 it only worked when Claude Code was signed in with a Claude account (the plan login). If you had an API key instead, the app answered with a reason code, AUTH_MODE_UNSUPPORTED, and stopped.

I read Anthropic's documentation around that refusal. My conclusion was that an app wrapping the CLI should not refuse a sign-in method the CLI itself supports. I'm a developer, not a lawyer, and this is not legal advice. It is simply how I decided to design it, so I asked for the refusal to be removed.

The app also tells people what it does with their data. Here is the first screen as it looked in a capture from the end of September:

Preview-mode first screen, late September (Korean UI, demo state). Title: "First, let's get an AI ready." Cards: Claude Code "ready", Codex "login needed". Notes below, translated: the app itself sends nothing to the internet; when you hand it a task, the Claude Code or Codex on your computer sends your request and needed file contents to Anthropic, OpenAI or wherever you connected, and checking that the AI is ready can also reach Anthropic; sign-in is managed by each AI tool and the app only checks whether and how you are signed in; tasks use up your plan's usage; AI answers can be wrong. A fifth note, cropped here, says the app is not made or endorsed by Anthropic or OpenAI.

One correction to that first note. It was true when I took the capture. Since version 0.3.0 the app also asks for a version number once a day, and since 0.3.3 it fetches banners for my other services in the office view. Neither request carries task content, but "sends nothing" is no longer an accurate sentence.

Review round one: the status check and the run disagree

Removing the refusal was a small change. I asked for an adversarial review (September 30), a few agents whose only task is to find how the change fails, and it found this.

Team mode starts Claude Code with --restricted, which keeps settings files out of the run. That includes your own user settings file. But before a run, the app asks claude auth status --json whether you are ready, and that command does read the user settings file. Some sign-ins live only there: an apiKeyHelper, or the Bedrock and Google Cloud login wizards.

Step	Reads user settings file?	What it sees
Readiness check (`auth status`)	yes	"signed in, ready"
Team run (`--restricted`)	no	no sign-in at all
Result		run fails, or quietly uses another stored login

I reproduced it with a fake API key and a local test server standing in for the API. The run either failed after the screen said "ready" or went out under a different Claude account that happened to be stored on the machine. For those people the screen now showed a green light, followed by a failure or a silent account switch.

The fix, and the second problem inside it

--settings still applies under --restricted, so the app can hand the file back by path. I never read its contents. The app checks that it exists, is at most 2 MB and is readable, and Claude Code reads the rest. This is the condition from providers/claude.mjs (one branch for third-party providers trimmed):

function signInFromSettings(signIn, env, platform) {
  if (!signIn) return false;
  const has = name => Boolean(envValue(env, name, platform));
  if (signIn.method === 'api_key_helper' || signIn.keySource === 'apiKeyHelper') return true;
  if (signIn.keySource === 'ANTHROPIC_API_KEY') return !has('ANTHROPIC_API_KEY');
  if (signIn.method === 'oauth_token') return !has('ANTHROPIC_AUTH_TOKEN') && !has('CLAUDE_CODE_OAUTH_TOKEN');
  return false;
}

My first version passed the file whenever a sign-in existed. A second review pass, one agent this time, found that the same file can contain permissions.additionalDirectories. Handing it back would let a run reach folders outside the one you picked. The restriction I was relying on was gone. So the file is passed only when the sign-in actually comes from it.

That condition has a trap. auth status reports a key from the environment and a key from the settings file with the same label, api_key and ANTHROPIC_API_KEY. The only way I found to tell them apart was to check whether the run's own environment holds the key, which is what has(...) does above. A related lesson from the Codex side: its -c override values are type-checked, so a value that looks right can still be rejected.

Other things the same review turned up

Team runs with Codex forced the provider to OpenAI. People using Azure or another provider had their tasks sent to OpenAI. I removed the forcing.
Claude Code and Codex could each receive the other's API key through the environment. Each tool now gets only its own family of variables (ENV_FAMILIES in the code).
The label "Claude account" appeared for any login. It now appears only for a plan login.

What passed, and what that proves

After the fixes: app tests 405, CLI tests 323, engine tests 401 plus one. I also ran mutation checks, which deliberately break the code and confirm a test fails. Real team runs through the actual CLIs finished in 6.1 seconds for Claude and 14.3 seconds for Codex.

Those numbers say the code does what my specs say. They say nothing about whether real users find sign-in easy. I have not run a usability test with real users.

A correction that belongs here

On September 29 I reported that the app's macOS signature check had zero fatal findings. That was wrong. When I ran syspolicy_check again, it listed three: two about the signature and one about missing notarization. Version 0.2.3 (September 30, 08:07) went out ad-hoc signed with 1,042 files sealed, and the count dropped from three to one. The remaining one is notarization, which I chose not to do. The next post covers that choice.

Takeaways

A status check and a run must read the same inputs. If they differ, "ready" is a promise the run can't keep.
Review the fix, not only the original change. Both regressions here came from the repairs, not from the code that was already there.
Reproduce with fake credentials and a local server before you trust a login fix.
Say what a test count measures. 405 passing tests is a statement about the specs.

The app link comes in the last post of the series.

If you wrap a CLI that has several ways to sign in, how do you keep your readiness check and your actual run reading the same configuration?

LLM-as-Judge Position Bias: Swapping A/B Flipped 87 of 300 Verdicts

jidonglab — Sun, 01 Nov 2026 12:15:13 +0000

In a head-to-head eval, my new system prompt beat the old one in 66% of comparisons. I already had the "we're shipping v2" message typed.

Then I swapped which answer went first and reran the exact same 300 pairs. The new prompt won 43%.

The judge, the answers, the rubric and temperature 0 were all the same. The only change was which answer sat in the slot labeled "A". That is LLM-as-judge position bias, and if your eval script calls judge(new, old) in a fixed order, your headline number is probably measuring it too.

TL;DR

LLM-as-judge position bias means a pairwise judge prefers whichever answer sits in a particular slot (usually the first) regardless of quality.
In my run, swapping A and B flipped 87 of 300 verdicts (29%), and 78 of those flips went to whichever answer was in slot A.
The bias shows up in close calls. When two answers are near-equal, slot position acts as the tiebreaker.
Fix: judge every pair in both orders. Count a win only when both runs agree, and treat disagreement as a tie.
Before trusting a judge, run the mirror test: put the identical answer in both slots. A fair judge has no reason to prefer either one.

What is LLM-as-judge position bias?

LLM-as-judge position bias is a judge model's tendency to pick an answer because of where it appears in the prompt, not because of what it says. It's one of the first biases documented for LLM judges (the MT-Bench / Chatbot Arena paper from Zheng et al. called it out back in 2023), and it's still easy to walk straight into.

It hits pairwise evals hardest, the "here are Answer A and Answer B, which is better?" setup. That's the setup most of us use to compare prompts, models or RAG configs, because it's easier than defining an absolute 1-10 scale.

How I caught it

My eval script looked like every eval script:

for q in questions:
    verdict = ask_judge(q.text, new_answer[q.id], old_answer[q.id])
    wins += verdict == "A"

The new answer always went in slot A. I didn't pick that order for any reason. It was just the order of the function arguments.

A coworker asked "did you swap them?" and I had no good answer. So I ran it again with the old answer in slot A. Here is what the 300 pairs did across both runs:

Outcome across both orders	Pairs
New prompt won both times	120
Old prompt won both times	93
Slot A won both times (verdict flipped)	78
Slot B won both times (verdict flipped)	9

Count the slot picks: slot A won 369 of 600 judgments, or 61.5%. A judge with no position preference would land somewhere near 50%.

The 66% headline was really 120 consistent wins plus 78 pairs where the new prompt just happened to sit in the favored seat.

Why does an LLM judge prefer the first answer?

An LLM judge prefers the first answer mostly because of how it reads: one token at a time, left to right. Answer A gets evaluated with nothing to compare it to. By the time the model reaches Answer B, it has already formed a picture of what "good" looks like for this question, and that picture came from A.

A few things stack on top of that:

Close calls get decided by the seat. I read the 87 flipped pairs myself. In most of them I couldn't pick a winner either: both answers were correct, worded differently, and about the same length. With no real quality signal to go on, the position prior decides.
Temperature 0 doesn't help. It makes the bias reproducible, which is worse in a way, because rerunning the same order gives you the same wrong answer and feels like confirmation.
The direction isn't universal. Some judge and prompt combinations favor slot B, especially with long answers where the most recent text is freshest. My judge favored A. Yours might favor B. Measure it before you assume a direction.
Format priors. "A" is the first option in countless multiple-choice examples the model has seen. Label tokens carry their own baggage before the model reads a word of the answers.

None of this means the judge is useless. On the 213 pairs where it agreed with itself, its picks matched mine on almost every one I spot-checked. The judge is fine at clear differences. On near-ties it isn't measuring quality at all.

How do you test an LLM judge for position bias?

Run the mirror test: build pairs where Answer A and Answer B are the exact same text, then ask the judge to pick. A fair judge has no basis to prefer either, so it should say TIE or split close to 50/50.

I ran 50 mirror pairs. My judge picked A 34 times, B 9 times and TIE 7 times. It was confidently picking a winner between two copies of the same text, and it usually picked the first one.

This takes ten minutes and tells you more about your judge than a week of reading verdicts. I now run it every time I change the judge model or the judge prompt.

How do you fix position bias in pairwise LLM evals?

Judge every pair twice, once in each order, and count a win only when both runs agree. If the judge contradicts itself, call it a tie. That's the whole fix:

JUDGE_PROMPT = """You are comparing two answers to the same question.

Question:
{question}

Answer A:
{a}

Answer B:
{b}

Briefly list the strengths and weaknesses of each answer.
Then end with exactly one line: VERDICT: A, VERDICT: B, or VERDICT: TIE"""


def ask_judge(question, a, b):
    out = call_llm(JUDGE_PROMPT.format(question=question, a=a, b=b), temperature=0)
    return parse_verdict(out)  # "A" | "B" | "TIE"


def judge_pair(question, new, old):
    run1 = ask_judge(question, new, old)  # new in slot A
    run2 = ask_judge(question, old, new)  # new in slot B

    if run1 == "A" and run2 == "B":
        return "new"
    if run1 == "B" and run2 == "A":
        return "old"
    # Self-contradiction, or a TIE in either run: no signal.
    return "tie"

Then report three numbers instead of one:

results = [judge_pair(q.text, new[q.id], old[q.id]) for q in questions]
n = len(results)

new_wins = results.count("new")
old_wins = results.count("old")
ties = results.count("tie")

print(f"new {new_wins/n:.0%}  old {old_wins/n:.0%}  tie {ties/n:.0%}")
print(f"new win rate on decisive pairs: {new_wins/(new_wins+old_wins):.0%}")

The tie rate is the important number here. It tells you how much of your eval set the judge can't actually separate. If it's 30%, a 5-point swing in your headline doesn't mean much.

Some other things that helped:

Give the judge a TIE option. Without it, you're forcing a coin flip on every near-equal pair, and the position prior is what decides the flip.
Ask for reasoning before the verdict. Put the verdict line last so the model writes its comparison before it commits to a letter. It doesn't remove the bias, but it cut my flip rate.
Use logprobs if your API exposes them. Take P("A") in one order and P("B") in the swapped order, then average. You get a soft score that's symmetric by construction.

Isn't randomizing the A/B order enough?

Randomizing the order per pair is better than a fixed order, but it doesn't fix the problem. Random assignment spreads the position bias evenly across both systems, so your aggregate win rate stops being skewed in one direction. That's real progress.

What it hides is the noise. Those 87 coin-flip pairs are still in your data, now split randomly between "new won" and "old won." Your headline drifts toward 50% and looks like an honest close race, when actually a third of your eval set says nothing at all. Rerun it with a different random seed and the number moves.

Running both orders costs twice the judge calls. On 300 pairs, that's 300 extra API calls. Shipping a prompt that was never better costs more.

Which other LLM judge biases show up alongside position bias?

Position bias rarely travels alone. Watch for these two:

Verbosity bias. Judges tend to favor the longer answer, even when the extra length is padding. If your new prompt makes answers 40% longer, some of its "wins" are word count. Check the win rate split by length difference.
Self-preference. A judge can favor outputs that sound like its own writing. If you're comparing Model X against Model Y and Model X is also the judge, be suspicious of a big X win.

Swapping the order fixes neither of these. They need their own checks.

So was my new prompt actually better?

Yes, a little. On the 213 pairs where the judge agreed with itself, the new prompt won 120 to 93, which is 56%. That's a real but modest improvement. It isn't the 66% blowout I nearly announced. I shipped it with a much quieter message and a note that 29% of the eval set was a toss-up.

How does LLM-as-judge position bias distort pairwise evals?

LLM-as-judge position bias makes a pairwise judge favor whichever answer occupies a particular slot, usually the first. It mostly decides near-tie comparisons, which can quietly inflate or deflate a win rate by double digits. In my eval, swapping A and B flipped 87 of 300 verdicts and turned a 66% win into 43%. The fix is to judge every pair in both orders, count only verdicts that agree, treat contradictions as ties, and report the tie rate next to the win rate. Run the mirror test, with the identical answer in both slots, before you trust any judge.

Written by the developer behind Preterview, an interview prep platform.

git push --force-with-lease Is Unsafe If Your Editor Auto-Fetches

jidonglab — Sun, 01 Nov 2026 00:13:30 +0000

I have typed git push --force-with-lease for years with the smug calm of someone wearing a seatbelt. Then one afternoon it overwrote two of my teammate's commits on our shared feature branch, and git didn't print a single warning.

No typo, no stale alias, no plain --force. The lease checked out fine. The problem was my editor, quietly running git fetch in the background every few minutes, which turned my seatbelt into a decorative strap.

This post explains exactly why git push --force-with-lease is unsafe when something auto-fetches, how to reproduce it in two minutes, and the flag that actually closes the hole.

TL;DR

git push --force-with-lease with no argument compares the remote branch to your remote-tracking ref (origin/feature), not to the commit you actually rebased on.
Any background git fetch (VS Code's git.autofetch, GitHub Desktop, a cron job, a prompt script) updates origin/feature to include your teammate's new commits. The lease now matches, and your push deletes their work.
The fix is --force-if-includes (Git 2.30+), which refuses the push if the remote tip was never part of your local branch's history.
Set it once: git config --global push.useForceIfIncludes true. It only kicks in alongside --force-with-lease, so plain --force stays dangerous.
Lost commits are usually recoverable from git reflog show origin/<branch>.

What does git push --force-with-lease actually check?

git push --force-with-lease checks one thing: that the branch on the remote still points to the same commit your local remote-tracking ref points to. If they match, the force push goes through. If they differ, git rejects it with "stale info".

That's it. It doesn't know which commit you started your rebase from. It doesn't know what you've looked at. It reads refs/remotes/origin/feature and assumes that's the last state you saw.

That assumption holds when you are the only thing that runs git fetch. It quietly breaks the moment anything else does.

Why does a background fetch break the lease?

A background fetch breaks the lease because it updates the exact ref the lease compares against, without touching your branch. Your view of the world goes stale, but git's record of "what you last saw" gets refreshed behind your back.

Here's the timeline from my afternoon:

14:02. I start an interactive rebase on feature/billing to squash some fixup commits. My local branch and origin/feature/billing both point at commit A.
14:09. My teammate pushes two commits, B and C, to the same branch.
14:11. VS Code's autofetch runs. origin/feature/billing now points at C. My local branch is still my rewritten history on top of the old base.
14:15. I finish the rebase and run git push --force-with-lease.
Git compares: remote says C, my tracking ref says C. Lease matches. Push accepted. B and C are gone from the remote.

The lease worked perfectly. It protected me against a state I had technically "seen". I hadn't seen anything. My editor had.

The git documentation actually warns about this in the --force-with-lease section: if something updates your remote-tracking refs in the background, the default lease form gives you no protection. Almost nobody reads that paragraph, because the flag's name sounds like it already handled it.

How do I reproduce the force-with-lease overwrite?

You can reproduce it locally with a bare repo and two clones, no network needed. Paste this into a scratch directory:

git init --bare origin.git
git clone origin.git me
git clone origin.git them

# me: create a branch with one commit
cd me
echo a > f && git add f && git commit -m "base"
git push origin HEAD:main
git branch -u origin/main

# them: pull it, add a fix, push
cd ../them
git pull origin main
echo fix > g && git add g && git commit -m "teammate fix"
git push origin HEAD:main

# me: rewrite my history, then a "background" fetch happens
cd ../me
git commit --amend -m "base (amended)"
git fetch            # pretend this was your editor

git push --force-with-lease
git log --oneline origin/main   # "teammate fix" is gone

The push succeeds. Now repeat the last section with the fix:

git push --force-with-lease --force-if-includes

This time git rejects the push, because the remote tip (teammate fix) never appeared in the history of your local branch. The rejection reason says the remote ref was updated since checkout, which is precisely the truth.

What does --force-if-includes do?

--force-if-includes makes git verify that the commit your remote-tracking ref points to is something your local branch has actually included at some point. It checks the reflog of your local branch: if the remote tip isn't reachable from any entry, git assumes you never integrated it and refuses the force push.

That's the missing check. The lease asks "did the remote change since my last fetch?" The include check asks "did I ever incorporate what's on the remote?" A background fetch can fake the first answer. It can't fake the second, because it never touches your local branch.

A few rules worth knowing:

It only works together with --force-with-lease. With plain --force it does nothing.
It's ignored if you pass an explicit lease value, like --force-with-lease=feature:abc123, since you've already told git exactly what to expect.
It requires Git 2.30 or newer. Run git --version before you rely on it.

How do I make --force-if-includes the default?

Set one config option and keep typing what you already type:

git config --global push.useForceIfIncludes true

Now every git push --force-with-lease also runs the include check. I also added an alias so my muscle memory never reaches for plain --force:

git config --global alias.pushf "push --force-with-lease --force-if-includes"

The explicit flag in the alias is redundant with the config, but it means the alias still works on a machine where I forgot to set the config. Cheap insurance.

When does --force-if-includes still let you overwrite work?

--force-if-includes still lets you overwrite work if your teammate's commits were in your local branch at any point, even briefly. Because the check looks at the reflog, "I pulled it once" counts as "I included it".

The classic case: you git pull --rebase, see your teammate's commits, then start an interactive rebase and drop them (on purpose or by fat-fingering d instead of p). The tip was in your reflog, so the check passes, and the push removes their work. Git is trusting that you made a decision. It can't tell the difference between a decision and an accident.

If you want the strongest guarantee, use an explicit lease captured before you start rewriting:

BASE=$(git rev-parse origin/feature/billing)
git rebase -i main
git push --force-with-lease=feature/billing:$BASE

Now the push only succeeds if the remote is exactly the commit you started from. Background fetches can't touch a shell variable. This is what I use for long-lived shared branches. For my own short branches, the config option is enough.

How do I recover commits lost to a force push?

You can usually recover them from the reflog of your remote-tracking ref, because the background fetch that caused the problem also recorded your teammate's commits locally. Run:

git reflog show origin/feature/billing

You'll see an entry from the fetch pointing at the old remote tip (in my case, C). Cherry-pick the lost commits back onto your branch:

git cherry-pick <sha-of-B> <sha-of-C>
git push --force-with-lease --force-if-includes

If your reflog doesn't have them, your teammate's local clone almost certainly does. That's how we recovered: a slightly awkward "hey, can you push again?" message and five minutes of cleanup. Lucky. On a branch with three people and a CI bot, it would have been a much longer afternoon.

Should I stop auto-fetching instead?

No. Auto-fetch is genuinely useful: it keeps your branch status accurate and shows incoming changes before you start a rebase. Turning it off trades a precise fix for a vague one, and you'd still get bitten by the next tool that fetches on your behalf.

The better move is to stop trusting the default lease form. Fetching should never be what decides whether a destructive push is safe.

So is git push --force-with-lease safe?

git push --force-with-lease is only safe if nothing else runs git fetch in your repo, because by default it compares the remote against your remote-tracking ref, and a background fetch silently updates that ref to include commits you never saw. With VS Code autofetch, GitHub Desktop, or any scheduled fetch, the lease will happily approve a push that deletes your teammate's work. Add --force-if-includes (or set git config --global push.useForceIfIncludes true on Git 2.30+) so git also checks that the remote tip was ever part of your local branch, and use an explicit --force-with-lease=branch:sha when you need a hard guarantee on shared branches.

Written by the developer behind Preterview, an interview prep platform.

Python ThreadPoolExecutor Swallowed 212 Claude API Errors Silently

jidonglab — Sat, 31 Oct 2026 12:11:49 +0000

My Claude labeling job finished in 14 minutes, printed done, and exited with code 0. Every log line was green. Three days later a dashboard said I had 3,788 labeled documents. I had sent 4,000.

Nothing crashed. Nothing warned. Python's ThreadPoolExecutor swallowed exceptions from 212 API calls and kept them in objects I never looked at. This is the autopsy, the two wrong fixes I tried first, and the 15 lines that ended it.

TL;DR

ThreadPoolExecutor.submit() stores any exception inside the returned Future. If you never call .result(), the exception is never raised and never logged.
The with ThreadPoolExecutor() block waits for all tasks, then exits cleanly even if every task failed.
executor.map() is not the fix: it raises the first error and cancels every task that hasn't started yet. Mine killed 2,906 documents.
The fix: keep a {future: input} dict, loop with as_completed(), call .result() inside try/except, retry failures at lower concurrency, and exit non-zero if anything is still missing.
Always reconcile counts. saved == sent would have caught this on night one.

What did the broken code look like?

It was the kind of code you write in five minutes and trust forever:

from concurrent.futures import ThreadPoolExecutor

def label_one(doc):
    resp = client.messages.create(
        model=MODEL,
        max_tokens=300,
        messages=[{"role": "user", "content": PROMPT.format(text=doc.text)}],
    )
    data = json.loads(resp.content[0].text)
    db.save(doc.id, data["label"])

with ThreadPoolExecutor(max_workers=8) as pool:
    for doc in docs:
        pool.submit(label_one, doc)

print("done")

Eight threads, 4,000 short documents, one label each. The worker writes its own result to the database, so the main thread has nothing to collect. That's the trap. Because I didn't need the return value, I threw away the Future, and the Future was the only thing holding the error.

Why does ThreadPoolExecutor swallow exceptions?

ThreadPoolExecutor swallows exceptions because a worker's exception is captured and stored on its Future object, and it only re-raises when someone calls future.result() or future.exception(). No caller, no raise. There's no "exception was never retrieved" warning when the Future is garbage collected.

That last part surprised me most. asyncio does warn you: an unawaited failed task prints Task exception was never retrieved when it gets collected. concurrent.futures has no such safety net. The exception sits on an object with zero references and quietly disappears.

The with block makes it feel safer than it is. Leaving the block calls shutdown(wait=True), which waits for every task to finish. It does not check how they finished. Waiting looks like supervision. It isn't.

What were the 212 errors?

I added error capture (more on how below), replayed the same 4,000 documents against a fresh table, and grouped what came back:

Failure	Count	Cause
429 rate limit	171	Burst exceeded my tokens-per-minute limit, survived the SDK's 2 default retries
529 overloaded	29	Transient server overload, also out of retries
`KeyError: 'label'`	12	Model returned `{"category": ...}` instead of `{"label": ...}`
Total	212	5.3% of the run

The 429s were my fault. Eight threads start at the same moment, all hit the limit at the same moment, and the SDK's backoff retries them at roughly the same moment. Two retries weren't enough to spread them out. The Anthropic Python client defaults to max_retries=2, which is reasonable for one call and thin for a burst of eight.

The 12 KeyErrors were the scariest. Rate limits are noisy in the API console. A parser bug on a model's slightly different JSON shows up nowhere except your own missing rows.

Is executor.map safer than submit?

executor.map() is louder than submit() but not safer: it re-raises the first exception when your loop reaches that item, and then cancels every task that hasn't started yet. You trade silent data loss for loud data loss.

That was my first "fix":

with ThreadPoolExecutor(max_workers=8) as pool:
    for _ in pool.map(label_one, docs):
        pass

The next run crashed at document 517 with a RateLimitError traceback. Great, I thought, now I see errors. Then I counted rows. map() submits everything upfront, yields results in input order, and when the iterator dies it cancels all pending futures in a finally block. 2,906 documents were cancelled before they ever started. One rate limit took out 73% of the job.

My second "fix" was wrapping label_one in a giant try/except that logged and returned. That works, technically, but it moves error handling into every worker function I'll ever write and gives the main thread no idea how many things failed. The job still exited 0. I'd just made the silence more verbose.

How do I catch exceptions from ThreadPoolExecutor properly?

Map each future back to its input, iterate with as_completed(), call .result() inside try/except, and collect failures in the main thread. Then retry the failures and exit with a non-zero code if any remain.

import sys
from concurrent.futures import ThreadPoolExecutor, as_completed

def run(docs, workers):
    failures = []
    with ThreadPoolExecutor(max_workers=workers) as pool:
        futures = {pool.submit(label_one, d): d for d in docs}
        for fut in as_completed(futures):
            doc = futures[fut]
            try:
                fut.result()
            except Exception as e:
                failures.append(doc)
                log.warning("label failed doc=%s err=%s", doc.id, type(e).__name__)
    return failures

failed = run(docs, workers=8)
if failed:
    log.info("retrying %d docs at low concurrency", len(failed))
    failed = run(failed, workers=2)

saved = db.count_labeled(batch_id)
if failed or saved != len(docs):
    sys.exit(f"{len(failed)} failed, saved {saved}/{len(docs)}")
print("done")

Four things are doing the work here:

The dict. futures[fut] tells you which document failed. A bare list of futures gives you a traceback with no ID.
as_completed(). Errors surface as they happen, and one failure doesn't cancel anything else.
The retry pass at 2 workers. Most 429s and 529s are about burst shape, not volume. Fewer threads on the second pass means fewer collisions.
The count check. Even if my error handling has a bug, saved != len(docs) catches it. This is the line I wish I'd written on day one.

I also bumped the client to Anthropic(max_retries=5) and replaced the free-text JSON with a forced tool schema so the model can't rename label to category. Those fixed the causes. The loop above fixes the blindness, which matters more, because next month's cause will be something I haven't thought of.

What happened after the fix?

The next full run of 4,000 documents:

First pass, 8 workers: 3,959 succeeded, 41 failed (all 429 or 529, zero KeyErrors after the schema change)
Retry pass, 2 workers: 41 of 41 succeeded
Count check: 4,000 of 4,000 saved
Wall time: 14 minutes to 16 minutes

Two extra minutes for a job that tells the truth. The failure rate in the first pass dropped from 5.3% to 1%, mostly from the extra SDK retries. But the number I care about is the one that went from "unknown" to zero: rows lost without a log line.

What should a batch job check before printing "done"?

A batch job should verify that the number of outputs it wrote equals the number of inputs it was given, and exit non-zero if not. Exit code 0 is a claim. Make the code earn it.

My checklist for any threaded API job now:

Never call pool.submit() without keeping the returned Future.
Never use pool.map() on calls that can fail transiently.
Log the input ID with every exception, not just the traceback.
Retry failures in a separate pass with lower concurrency.
Reconcile counts against the source of truth, not against your own success counter.
Make cron or CI treat a non-zero exit as an alert.

The original bug ran for nine nights before I noticed. Nobody ignored an alert. There was no alert to ignore.

So why did ThreadPoolExecutor swallow my Claude API errors?

Python's ThreadPoolExecutor swallowed 212 of my Claude API errors because submit() stores each worker's exception on the returned Future, and I never kept or checked those futures. Without a call to future.result(), the exception is never raised and never logged, and the with block exits cleanly once all threads finish, so the script printed done with exit code 0. Switching to executor.map() made errors visible but cancelled every unstarted task after the first failure. The reliable pattern is a {future: input} dict, an as_completed() loop with .result() inside try/except, a low-concurrency retry pass, and a final saved == sent count check that fails the job loudly when the numbers don't match.

Written by the developer behind Preterview, an interview prep platform.

AWS ALB Idle Timeout Killed 14% of My Voice WebSockets at 60s

jidonglab — Sat, 31 Oct 2026 00:10:05 +0000

A user emailed me a screen recording. The AI voice agent asks a hard question. The user goes quiet for about a minute to think. Then they start talking, and nothing happens. A second later the app reconnects on its own and the agent cheerfully asks the first question again.

That loop came from the AWS ALB idle timeout on WebSocket connections. No exceptions, no 5xx in the dashboards, just a load balancer doing exactly what it was configured to do. It dropped 58 of my 412 voice sessions in one week. That's 14%.

TL;DR

AWS Application Load Balancer closes any connection with no data in either direction for 60 seconds. That default applies to WebSockets too.
Voice apps that only stream mic audio while the user is speaking (VAD-gated) go fully silent during long pauses, and the ALB cuts them.
The browser sees close code 1006 with no close frame, which looks like a random network error.
Fix: send a WebSocket ping from the server every 25 seconds, raise the ALB idle_timeout, and make reconnects resume the session instead of restarting it.
After the fix: 2 drops in 436 sessions, both real network loss.

What does the ALB idle timeout actually do to a WebSocket?

The ALB idle timeout is a timer on the connection that resets whenever a byte moves in either direction. If 60 seconds pass with zero traffic, the ALB closes both sides. It doesn't care that the HTTP request upgraded to a WebSocket. A quiet socket is a quiet socket.

Most WebSocket apps never hit this because something is always chatting: a chat app's typing indicators, a dashboard's price ticks. My app had long, legitimate silences. That was the whole product.

The setup that made it fail

The system is a voice interview simulator. Browser mic goes over a WebSocket to a Node server running the ws library behind an ALB. The server transcribes, calls the LLM, and streams TTS audio back.

To save bandwidth and transcription cost, the client runs voice activity detection locally and only sends audio frames while the user is speaking. Silence sends nothing.

So the traffic looks like this:

Agent asks a question. Server streams TTS audio. Bytes flowing.
User answers. Client streams mic frames. Bytes flowing.
User pauses to think. Client sends nothing. Server is waiting. Zero bytes.

Step 3 is where the timer runs. On easy questions people pause for 5 seconds. On a hard system design question, they think for 70.

Why did only some sessions drop?

Only sessions with a silent gap longer than 60 seconds dropped, and one interviewer style accidentally protected itself. That's the detail that took me longest to spot.

The product has 3 interviewer styles. The "friendly" style sends a gentle "take your time" nudge after 45 seconds of silence. That nudge is TTS audio, which is traffic, which resets the ALB timer. The "strict" style sits in silence on purpose, because that's what a strict interviewer does.

From one week of logs:

Metric	Count
Total sessions	412
Sessions with a silent gap over 60s	71
Of those, sessions using the friendly style	13
Dropped sessions	58
Dropped sessions with a silent gap over 60s	58

All 58 drops happened during a gap over 60 seconds. All 13 sessions that survived a long gap were friendly-style ones where the 45-second nudge kept the pipe warm. I had shipped a load balancer workaround as a personality trait and didn't know it.

How I found it: subtract two timestamps

I was logging the close event, but I wasn't logging when the last frame moved. Once I added lastFrameAt (updated on every message in either direction) the pattern fell out of one query:

ws.on('close', (code) => {
  const idleMs = Date.now() - ws.lastFrameAt;
  log.info({ sessionId, code, idleMs }, 'ws closed');
});

Every abnormal close had code: 1006 and idleMs between 60,000 and 61,400. Not "around a minute". A cliff at exactly 60 seconds with about a second of jitter. When your failure distribution has a cliff at a round number, look for a config default before you look for a bug.

Why does the browser report WebSocket close code 1006?

Code 1006 means the connection closed without a close frame. The ALB drops the TCP connection rather than performing a WebSocket close handshake, so neither the browser nor the ws server ever receives a proper close frame. Both sides report 1006, the same code you get when Wi-Fi dies.

That's why this hid in my error tracking for weeks. It was bucketed with "flaky user networks", a category everyone learns to ignore.

The second bug: reconnecting made it worse

My client had auto-reconnect with backoff, which felt responsible. But the server treated every new socket as a new session. So the user experience was: you think hard, you start answering, and the agent restarts from question 1 like nothing happened.

That's worse than a visible error. A visible error at least tells you something broke.

Full disclosure: I built this. It's Preterview, a voice interview practice app with 3 interviewer styles that scores a resume or portfolio and returns a written report. Long thinking pauses are normal there, which is exactly why this timeout bit so hard.

How do you keep a WebSocket alive behind an AWS ALB?

Send traffic more often than the idle timeout, from the server, using WebSocket ping frames. The browser answers pings with pongs automatically, both frames count as traffic for the ALB, and you get dead-connection detection for free.

One trap: the browser WebSocket API cannot send protocol-level ping frames. If you want a client-side heartbeat, you have to send an application message. Server-side ping is simpler and covers every client.

The ws library doesn't ping by default. Here's the heartbeat I added:

const HEARTBEAT_MS = 25_000;

wss.on('connection', (ws) => {
  ws.isAlive = true;
  ws.on('pong', () => { ws.isAlive = true; });
});

setInterval(() => {
  for (const ws of wss.clients) {
    if (!ws.isAlive) { ws.terminate(); continue; }
    ws.isAlive = false;
    ws.ping();
  }
}, HEARTBEAT_MS);

25 seconds gives two chances to land a ping inside one 60-second window, so a single delayed frame doesn't kill the session.

Should you also raise the ALB idle timeout?

Yes, but as a safety margin, not as the fix. The ALB idle timeout can go up to 4,000 seconds. I set it to 300:

resource "aws_lb" "voice" {
  name               = "voice-ws"
  load_balancer_type = "application"
  idle_timeout       = 300
  # ...
}

Relying on the timeout alone is fragile. There may be other hops in the path (a corporate proxy, a CDN, a mobile carrier NAT) with their own idle timers you don't control. The heartbeat survives all of them. The bigger timeout just means one missed ping isn't fatal.

Making reconnects resume instead of restart

The third change: the server now issues a resume token on connect. The client stores it in memory and sends it on reconnect. If the session is still in the server's state store, the client rejoins at the same question with the transcript intact. Tokens expire after 5 minutes.

Now a real network blip costs the user a second of audio, not their whole session.

Results after one week

Metric	Before	After
Sessions	412	436
Dropped sessions	58 (14.1%)	2 (0.5%)
Drops with `idleMs` near 60,000	58	0
Drops from real network loss	0 logged	2

The heartbeat cost is a ping and a pong every 25 seconds, a few bytes each. Against a stream of audio frames, it doesn't show up in any bill I can read.

Caveat: two weeks of traffic at this volume is a small sample. The 0 after-fix timeout drops is convincing because the before-fix failures were so mechanical, not because 436 is a big number.

What I'd check in any voice or long-lived WebSocket app

Log lastFrameAt for every socket and compute idle time at close. This one number would have found the bug on day one.
Treat close code 1006 as "investigate", not "user's Wi-Fi".
Never let "silence" mean "no bytes". VAD-gated audio is great for cost, but pair it with a heartbeat.
Make reconnect resume state. Auto-reconnect into a fresh session hides failures from you and makes them worse for users.
Test with a deliberate 90-second pause. My QA runs never paused that long, so they never hit the timer.

So why does the AWS ALB idle timeout kill WebSocket connections?

The AWS ALB idle timeout closes any connection, including an upgraded WebSocket, after 60 seconds with no data in either direction. Apps that go genuinely quiet, like VAD-gated voice streams during a user's pause, cross that line and get dropped with close code 1006 and no close frame. The fix is a server-side WebSocket ping every 20 to 30 seconds, an idle_timeout raised as a margin, and reconnect logic that resumes the session. For me that took drops from 58 of 412 sessions to 2 of 436.

Written by the developer behind Preterview, an interview prep platform.

Measuring What My Electron App Sends: 94 Sockets, 0 Out

jidonglab — Fri, 30 Oct 2026 18:46:27 +0000

On September 29, my Electron app opened 94 sockets across four runs, and every one of them pointed at 127.0.0.1. The same runs also caused 20 connections to the outside world from a single one-line task. Both numbers are true.

This is part 10 of a series about Refmade, a desktop app I'm building alone. It lets people who don't use a terminal hand work to the Claude Code or Codex they already have installed, and shows the agents as an office you can watch. It is an independent app, not affiliated with Anthropic or OpenAI. Claude Code and Codex agents wrote most of the code at my direction; I planned, decided and reviewed. The UI is Korean right now; that changes later in this series.

The sentence I refused to write

From the first product plan on September 21, I had a rule for the copy: never say "your data never leaves your PC." Refmade starts a CLI on your machine, and that CLI sends your prompt and the file contents it needs to a model provider. A claim that sounds like "nothing leaves" would be false on the first task.

What I could say was narrower. The app itself should send nothing, and the gap between that and the CLI's traffic is what I measured. Whatever the CLI sends is the CLI's business, and the app should say so. That is a claim you can measure, so on September 29 I measured it, against the 0.2.x Mac build.

How I measured

Four runs: two plain smoke launches, one Claude task, one Codex task. During each run I took a snapshot of the socket table every 0.25 seconds, ran nettop next to it, and collected the DNS records. A snapshot only sees sockets that are open at that instant, so a short connection can fall between two samples. The short interval, nettop and the DNS records are three views of the same run, and each covers a gap in the others.

What I looked at	Result
Sockets opened by the app's own processes	94, all on 127.0.0.1
DNS lookups from the app's own processes	0
Connections during a one-line Claude task	20, all from the CLI the app started
Packet contents	not inspected (needs admin rights)
Login flow, long-running tasks	not measured

I take the 94 local sockets to be the app's own pieces talking to each other, such as the window and its office server. I didn't trace each one. The 20 outside connections come from Claude Code. My own Claude setup had MCP servers enabled, and Claude Code also sends usage statistics. A person without those servers would probably see fewer connections, but I didn't measure that.

I'd skip "0 external sends" as a headline. The accurate version is "0 from the app's processes, 20 from the tool it launched, content not inspected."

The guard in the window

The renderer has its own rule. Every request a page in the app's window makes goes through one function, and only four kinds get through: the app's own files, DevTools, inline data, and the office server on 127.0.0.1 at its one port.

case `${APP_SCHEME}:`: return isAppUrl(parsed) ? {allow: true, reason: 'app'} : {allow: false, reason: 'app-host'};
case 'devtools:':
case 'chrome-devtools:': return {allow: true, reason: 'devtools'};
case 'data:':
case 'blob:': return {allow: true, reason: 'inline'};
case 'http:': return isOfficeUrl(parsed, officePort) ? {allow: true, reason: 'office'} : {allow: false, reason: 'network'};
default: return {allow: false, reason: 'network'};

A Content-Security-Policy with connect-src 'self' sits behind it. A page script that tries to reach another host is cancelled by the filter and refused again by the policy.

What the first screen says

I wrote the measurement into the first screen of the app, in plain Korean.

The first screen as I prepared 0.2.3, captured at 01:20 on Sept 30 in preview mode (Korean UI, nothing from real work). Heading: "Let's get an AI ready first". Two cards: Claude Code "Ready", Codex "Login needed". The first note under them reads: "The Refmade app itself sends nothing over the internet. When you hand it work, the Claude Code or Codex on this computer sends the request and the file contents it needs, under your account, to Anthropic, OpenAI, or wherever you connected the AI. Even when checking that the AI is ready, Claude Code sometimes connects to Anthropic." Below the crop, two more notes say that logins are managed by each AI and that usage counts against your plan.

A reader who sees only "the app sends nothing" might assume a one-line task is silent. It isn't: the capture showed 20 connections from the CLI, which is why the note names the CLI's own traffic in its second and third sentences.

Then the 0 stopped being true

The first note in that screenshot stopped being accurate less than 24 hours after I captured it. I built 0.3.0 the next night, and it went live at 00:35 on October 1, with a new-version notice. About ten seconds after launch, and once a day after that, the app makes a GET request for a small version file on my own site. A menu switch turns it off.

On October 8, the Mac build 0.3.3 added billboards in the office view. The app fetches banners for my other services from my site about every 60 seconds, up to six at a time, each image capped at 350 KB. There is no switch to turn this off. The request carries no work data or identifiers, and I don't count impressions or clicks. A request to a web server still shows your IP address to that server, and I haven't checked how long my hosting provider keeps request logs.

Here is the whole history in one table.

Version	Requests the app makes to Refmade's site	Basis
0.2.x, Sept 29	none	measured, 4 runs
0.3.0, Oct 1 (built Sept 30)	version file, 10 s after launch then daily	code reading
0.3.3, Oct 8 (Mac)	the same, plus the billboard feed every 60 s	code reading

Both calls run in the main process through Node's https module, which the window's request filter above never sees. A comment at the top of that filter said "the app sends nothing to any server." That stopped being true when 0.3.0 went out, which is a good reminder that comments don't get tested.

The first note on the screen changed. This is the current Korean text, translated (the original names the site's domain, which I left out here): "The Refmade app fetches the new version number and the office ad banners from Refmade's site. These requests don't send your work or files. You can turn off update notices from the menu."

I haven't repeated the socket capture on 0.3.x. What I say about those two requests comes from reading the code, not from a new measurement.

What I'd keep

A measurement belongs to a build number and a date. "0" was the right answer for one version and the wrong one for the next, and nothing in the app failed when that happened. The only thing that went stale was the sentence next to the number.

If you ship a desktop app that talks to your own server, the part to automate is the comparison between what the screen promises and what the code requests.

How do you verify what an Electron app sends: a proxy and packet capture, OS tools like nettop, or reading the code? And does anything fail in your CI when the privacy copy and the network calls drift apart?

The app link comes in the last post of the series.

vLLM Speculative Decoding: 1.8x Faster Alone, Slower at 32 Users

jidonglab — Fri, 30 Oct 2026 12:08:39 +0000

I turned on vLLM speculative decoding on a Friday afternoon, ran my usual one-request benchmark, and watched tokens per second jump from about 48 to about 88. I wrote "1.8x free speedup" in the team channel. On Monday, with real traffic hitting the box at around 32 concurrent requests, aggregate throughput was lower than before I touched anything, and p50 inter-token latency had gone up.

Nothing was broken. Speculative decoding did exactly what it is designed to do. I just measured it in the one regime where it always looks good.

TL;DR

Speculative decoding has a small draft model guess the next k tokens, then the big model checks all k in one forward pass and keeps the prefix it agrees with.
It is fast at low concurrency because decoding is memory-bandwidth-bound: checking 5 tokens costs roughly the same as generating 1.
The whole win depends on the acceptance rate. Below roughly 0.5 per token (k=4 with a 1B draft on an 8B target), the draft work eats the gain.
Under load the GPU drifts toward compute-bound, so rejected draft tokens become real wasted FLOPs, and the draft model's weights and KV cache take memory from other requests.
Measure acceptance on production traffic, use n-gram (prompt lookup) drafting for copy-heavy workloads, lower k, and keep speculation off for high-concurrency deployments.

What actually happened on my server?

The setup: Llama 3.1 8B Instruct as the target, Llama 3.2 1B Instruct as the draft (same tokenizer, which is required), one 48GB GPU, num_speculative_tokens set to 4.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --speculative-config '{"model": "meta-llama/Llama-3.2-1B-Instruct", "num_speculative_tokens": 4}'

Two workloads share this box: a JSON extraction service (temperature 0, lots of text copied from the input) and an internal chat assistant (temperature 0.8). My Friday benchmark was a single extraction prompt. Monday was both workloads mixed at around 32 concurrent sequences.

Roughly, on my hardware:

Scenario	Without spec decoding	With spec decoding
1 request, extraction	~48 tok/s	~88 tok/s
1 request, chat (temp 0.8)	~48 tok/s	~47 tok/s
~32 concurrent, mixed	baseline	~15% lower aggregate throughput

The second row was the first clue. Even with a single user, chat gained nothing. To see why, you need the mechanism.

How does speculative decoding actually work?

Normal decoding runs the big model once per token. Speculative decoding splits each step in two:

The draft model (small and cheap) generates k tokens autoregressively: t1, t2, t3, t4.
The target model runs one forward pass over all k positions at once and computes its own distribution at each.
Walk left to right. Accept t1 if the target agrees, then t2, and so on. At the first rejection, throw away that token and everything after it, and take the target's own token at that position instead.

So every step yields at least 1 token (the target's correction) and at most k+1 (all drafts accepted, plus one bonus token from the last position).

At temperature 0, "agrees" means the target's argmax matches the draft token. At temperature above 0, vLLM uses rejection sampling: accept with probability min(1, p_target / p_draft), and on rejection resample from the leftover distribution. This keeps the output distribution identical to the target model alone. Speculative decoding changes speed, never quality. That guarantee is what makes it easy to turn on without thinking, which is how I ended up here.

Why is speculative decoding faster at batch size 1?

Because generating one token for one user barely uses the GPU's math units. Every decode step has to stream all of the model's weights from memory. An 8B model in fp16 is about 16GB. On a GPU with around 860 GB/s of memory bandwidth, that is about 19ms per step as a floor, no matter how little math you do.

Verifying 5 positions instead of 1 reuses the same weight read. The extra matrix math is tiny next to the memory traffic, so the verification pass costs close to the same 19ms. The draft model reads its own 2.5GB of weights per token, about 3ms each.

That gives a simple cost model for one step with k=4:

4 draft forwards: ~12ms
1 target verification: ~19ms
Total: ~31ms, versus 19ms for one plain token

You pay 31ms per step. You break even only if you get more than 31/19 ≈ 1.6 tokens per step.

What is the acceptance rate, and why does it decide everything?

Acceptance rate (α) is the probability the target accepts any given draft token. If you treat each position as independent, the expected tokens per step is:

E[tokens per step] = (1 - α^(k+1)) / (1 - α)

Plug in k=4 and the timings above:

def tokens_per_step(alpha, k):
    return (1 - alpha ** (k + 1)) / (1 - alpha)

def speedup(alpha, k, t_target=19.0, t_draft=3.0):
    step_ms = t_target + k * t_draft
    return tokens_per_step(alpha, k) * t_target / step_ms

for a in (0.3, 0.5, 0.7, 0.9):
    print(a, round(tokens_per_step(a, 4), 2), round(speedup(a, 4), 2))

α	tokens per step	speedup
0.3	1.42	0.87x
0.5	1.94	1.19x
0.7	2.77	1.70x
0.9	4.10	2.51x

Below about 0.45, you lose. My extraction traffic sat near 0.7: the output is mostly field values copied from the input, and a 1B model guesses those easily. My chat traffic at temperature 0.8 sat near 0.4. Higher temperature flattens both distributions, the draft and target disagree more often, and the sampled acceptance drops. That explains row two of my table without any load at all.

One trap in reading vLLM's own numbers: the /metrics endpoint exposes spec_decode_* counters (exact names shift between versions, so grep for spec_decode). The ratio accepted tokens / draft tokens is not α. With k=4 and α=0.7, you accept about 1.77 of 4 drafts on average, so that ratio reads 0.44 and looks terrible even though the speedup is 1.7x. Compute mean accepted length instead:

curl -s localhost:8000/metrics | grep spec_decode
# mean accepted per step = num_accepted_tokens / num_drafts
# tokens per step        = that + 1

Then compare tokens per step against your own break-even (step cost divided by plain token cost).

Why does vLLM speculative decoding get slower under load?

Under load, the memory-bound assumption stops holding, and three costs show up at once.

1. Verification is no longer free. With 32 sequences and k=4, the target's verification pass processes 32 × 5 = 160 token positions per forward instead of 32. In fp16, each token does about 2 FLOPs per parameter while the weights cost 2 bytes each, so arithmetic intensity is roughly 1 FLOP per byte per token in the batch. A modern GPU's ridge point (peak FLOPs divided by bandwidth) sits in the hundreds. Real kernels hit the compute wall well before the theoretical ridge, so 160 positions per pass is already in the zone where extra positions cost real time. Every rejected draft token is now math you paid for and threw away, and that math was taken from other users.

2. The draft model scales too. Four sequential draft forwards per step don't disappear at batch 32. They're small, but they're four extra kernel rounds per step in the critical path for every sequence.

3. Memory goes to the draft. The draft's weights and its own KV cache come out of the same GPU memory pool. Fewer KV cache blocks means fewer sequences fit at once, so under real traffic more requests wait in the queue. Throughput can drop even before any compute math enters the picture.

Without speculation, batching already solved the memory-bandwidth problem. Thirty-two users share one weight read per step. Speculative decoding attacks the same inefficiency, and once batching has eaten it, you're left with only the overhead.

When should you use speculative decoding in vLLM?

Use it when concurrency is low and acceptance is high. In practice:

Measure α on production traffic, not on your favorite prompt. My benchmark prompt was the best case I could have chosen by accident.
Try n-gram drafting first for copy-heavy work. Extraction, code edits, and RAG answers that quote the context repeat input spans. Prompt lookup proposes tokens by matching n-grams from the prompt, so it needs no draft model, no extra weights, and no extra KV cache:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 4, "prompt_lookup_max": 4}'

Lower k when α is mediocre. Each additional draft position is worth α^i, which decays fast. At α=0.5, the 4th draft token adds about 0.06 expected tokens for a full draft forward of cost.
Split deployments by traffic shape. I moved extraction to a small low-concurrency instance with n-gram speculation on, and kept the chat pool on plain decoding with a bigger batch. Some older vLLM versions had a setting to disable speculation above a batch size threshold; check what your version supports before relying on it.
Look at trained draft heads like EAGLE or Medusa if you need speculation on chat-style traffic. They predict from the target's own hidden states and usually agree with it more often than a separate small model. Measure them the same way.

So, does vLLM speculative decoding make inference faster?

vLLM speculative decoding makes inference faster only when the GPU is memory-bound and the acceptance rate is high: low concurrency, low temperature, and outputs that repeat or closely follow the input. In that regime, verifying k draft tokens costs about the same as one normal step, so a per-token acceptance of 0.7 with k=4 gives roughly 1.7x. Under load, batching has already used up the idle memory bandwidth, verification turns into real compute, rejected drafts waste it, and the draft model takes KV cache memory from other requests, so aggregate throughput can drop. Measure mean accepted tokens per step from /metrics on real traffic, compare it to your break-even, and turn speculation on only where the numbers clear it.

Written by the developer behind Preterview, an interview prep platform.

Claude Code Deny Rules Block Read(.env) But Not cat .env

jidonglab — Fri, 30 Oct 2026 00:06:44 +0000

I watched my production database password scroll past in a Claude Code transcript. I had a deny rule for .env. I had copied it straight from the docs. Claude never touched the Read tool. It ran grep -r DATABASE_URL . because I asked it why the app couldn't connect, and grep did exactly what grep does.

Claude Code deny rules are real and they work. They just guard a narrower door than most people think. This post covers what Read(./.env) actually blocks, the 7 commands I tested against it, and the setup that closed the gap on my machine.

TL;DR

Claude Code deny rules like Read(./.env) apply to Claude's built-in file tools. They are not a filesystem lock.
The Bash tool is matched by its own rules. cat .env or grep -r gets through if your Bash allow list lets those commands run.
Bash deny patterns like Bash(cat .env) are plain prefix matches, so cat ./.env or head .env gets past them.
A PreToolUse hook that exits with code 2 blocks the obvious Bash routes. The sandbox or keeping plaintext secrets out of the project is the only real boundary.
Secrets exported into the shell you launch claude from are inherited by every Bash command it runs.

What does a Claude Code deny rule actually block?

A Claude Code deny rule blocks a specific tool call pattern, not a file. Read(./.env) means "refuse when the Read tool targets this path." The rule says nothing about what a shell process does with that path.

This is the snippet almost everyone has in .claude/settings.json:

{
  "permissions": {
    "deny": [
      "Read(./.env)",
      "Read(./.env.*)",
      "Read(./secrets/**)"
    ]
  }
}

It does its job. Ask Claude to open .env with the Read tool and it gets refused, every time. The trouble is the mental model. I read that config as "Claude cannot see .env." What it really says is "this one tool cannot open .env." Those are very different promises.

Why does Claude Code's Bash tool ignore Read deny rules?

Bash is a separate tool with separate rules. Permission checks happen per tool: a Bash call is compared against Bash(...) patterns, and a Read call is compared against Read(...) patterns. A Read deny rule never sees a Bash call.

Once a shell command runs, it is just a process on your machine with your user's permissions. cat, grep, python3, node, sed: none of them know a JSON file somewhere said .env was off limits.

So the question becomes: what does your Bash allow list look like? Mine had grown the way everyone's does. I got tired of approving searches, so I clicked "don't ask again" a few times. Here's what had piled up:

{
  "permissions": {
    "allow": [
      "Bash(grep:*)",
      "Bash(cat:*)",
      "Bash(ls:*)",
      "Bash(npm run test:*)"
    ],
    "deny": [
      "Read(./.env)",
      "Read(./.env.*)"
    ]
  }
}

Every one of those allows felt harmless when I added it.

I tested 7 ways to read .env. The deny rule stopped 2.

I put a fake .env and .env.local in a scratch repo, with the settings above, and asked Claude Code to run each of these as a separate step. You can repeat this in two minutes on your own install, and you should, because your allow list is not my allow list.

#	What ran	Result
1	Read tool on `.env`	Denied
2	Read tool on `.env.local`	Denied
3	`cat .env`	Ran, no prompt
4	`grep -r DATABASE_URL .`	Ran, no prompt, printed the secret line
5	`grep KEY .env.local`	Ran, no prompt
6	`head -3 .env`	Asked me for permission
7	`python3 -c "print(open('.env').read())"`	Asked me for permission

Two blocked. Three ran silently. Two depended on me reading a permission prompt carefully at 11pm.

Number 4 is the scary one, because nobody asked for .env. Claude was doing reasonable debugging. Plain grep -r does not care about .gitignore, so it walks straight into the file your .gitignore was hiding from git. The secret lands in the tool output, which lands in the conversation context, which lands in your session log on disk.

Can I fix it with a Bash deny rule like Bash(cat .env)?

Partially, and not reliably. Bash permission patterns are prefix string matches on the command. Bash(cat .env) stops exactly cat .env. It does not stop:

cat ./.env
cat .env with two spaces
head .env, tail .env, less .env, awk 1 .env
grep -r from the project root
cp .env /tmp/x && cat /tmp/x

Claude Code is smart about shell operators. An allow rule for Bash(safe-cmd:*) won't approve safe-cmd && something-else. But no string pattern can enumerate every program that can open a file. The docs themselves warn that Bash patterns trying to constrain arguments are fragile, and this is why.

The honest takeaway: deny rules are for steering, not for security. Use them, but don't stop there.

How do I block .env reads in Claude Code Bash commands?

Use a PreToolUse hook on the Bash tool that inspects the command and exits with code 2. Exit 2 blocks the call and sends your stderr message back to Claude, which then picks a different approach. Exit 1 only logs an error and lets the command run, which surprises a lot of people.

Save this as .claude/hooks/no-dotenv.py:

#!/usr/bin/env python3
import json
import re
import sys

data = json.load(sys.stdin)
cmd = data.get("tool_input", {}).get("command", "")

touches_env = re.search(r"\.env\b", cmd)
blind_grep = re.search(r"\bgrep\s+(-\w*r|--recursive)", cmd) and "--exclude" not in cmd

if touches_env or blind_grep:
    print(
        "Blocked: this command can read .env. "
        "Ask the user for the specific value you need, "
        "or use grep with --exclude='.env*'.",
        file=sys.stderr,
    )
    sys.exit(2)

sys.exit(0)

Wire it up in .claude/settings.json:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "python3 \"$CLAUDE_PROJECT_DIR\"/.claude/hooks/no-dotenv.py"
          }
        ]
      }
    ]
  }
}

I reran the 7 tests with the hook in place. All five Bash variants were blocked before they ran, including the grep that only reached .env by accident. Claude's next move on the grep case was to rerun it with --exclude='.env*', which is exactly the behaviour I wanted.

Better still, the stderr message is a nudge. Claude reads it and adjusts, so the hook teaches as it blocks.

Is a hook enough to protect secrets from Claude Code?

No. A hook is still string matching, and string matching loses to creativity. cat .e* slips past my regex. So does python3 -c "open('.'+'env')". Claude isn't trying to break in, so in practice the hook catches the accidents, which are the real risk. But "catches accidents" is not a security boundary.

Here's the layering I ended up with, from weakest to strongest:

Read deny rules. Keep them. They stop the built-in tools and signal intent.
Trim the Bash allow list. I removed Bash(cat:*) and Bash(grep:*). Searches go through Claude's built-in search tools, which are covered by the Read rules. Approving the occasional cat costs me a second.
The PreToolUse hook. Catches accidental reads and redirects Claude.
The sandbox. Claude Code's sandbox enforces filesystem and network rules at the OS level for Bash and every child process it spawns. That is the first layer where python3, awk and cat all hit the same wall. Check the sandbox docs for how your version maps deny paths.
No plaintext secrets in the project at all. This is the only layer that can't be bypassed. A secrets manager that injects values at process start means there is no file to leak.

The leak nobody checks: your shell environment

One more trap. The Bash tool inherits the environment of the shell you started claude from. If you ran export OPENAI_API_KEY=... in that terminal, or your .zshrc exports production credentials, then printenv or env prints them, no file involved. Not a single rule above helps.

I now start Claude Code from a terminal with nothing sensitive exported. Inject credentials into the command that needs them, not into the parent shell.

So do Claude Code deny rules protect my .env file?

Claude Code deny rules protect .env from Claude's built-in file tools only. Read(./.env) is matched against Read tool calls, so the Bash tool can still read the file with cat, grep -r, head or any script whenever your Bash allow list or a hurried approval lets the command run. Bash deny patterns are prefix matches and easy to sidestep. To actually keep secrets out of the transcript, trim broad Bash allows, add a PreToolUse hook that exits with code 2 on commands touching .env, turn on the sandbox for OS-level enforcement, and keep plaintext secrets out of the project directory and out of the shell environment you launch claude from.

Written by the developer behind Preterview, an interview prep platform.

pgvector HNSW Filtering Returned 0 Rows for 52 of My 400 Queries

jidonglab — Thu, 29 Oct 2026 12:04:57 +0000

A customer asked our docs assistant how to rotate an API key. Claude answered politely and with total confidence: "I couldn't find anything about key rotation in your documentation."

The answer was on page two of their docs. The chunk was in my database. The embedding was fine. Postgres just never handed it over.

This post is about pgvector HNSW filtering: one innocent WHERE tenant_id = $1 that made my retrieval return zero rows for 52 of 400 eval queries, with no error, no warning, and nothing on any dashboard.

TL;DR

pgvector's HNSW index collects hnsw.ef_search candidates first (default 40), and Postgres applies your WHERE filter afterwards. If 2% of rows match the filter, you get about 0.8 rows back, not 10.
Nothing errors. LIMIT 10 just quietly returns fewer rows.
In my multi-tenant RAG (1.2M chunks, 40 tenants), 147 of 400 queries got fewer than 10 chunks. 52 got zero.
Raising ef_search to 400 cut the short queries to 19 but tripled p95 latency. pgvector 0.8's hnsw.iterative_scan = relaxed_order cut them to 3 at almost no cost.
The last 3 came from a 600-row tenant. For tenants that small, skip the vector index entirely: an exact scan took 4ms.

The setup

A docs assistant for 40 customer workspaces. One chunks table, 1.2M rows, embedding vector(1024), an HNSW index with vector_cosine_ops, and a plain btree on tenant_id. The biggest tenant had 410,000 chunks (34% of the table). The smallest had 600.

Retrieval was the query every pgvector tutorial shows you:

SELECT id, content
FROM chunks
WHERE tenant_id = $1
ORDER BY embedding <=> $2
LIMIT 10;

The top 10 went into Claude with the usual instruction: answer only from the provided context, and if the answer isn't there, say so.

That instruction is why this bug lived for weeks. Claude did exactly what I asked. Empty context in, honest refusal out. A grounded "I don't know" looks like correct behavior on every metric I was tracking.

Why does pgvector HNSW filtering return fewer rows than LIMIT?

Because the filter runs after the index scan, not inside it. HNSW walks its graph and keeps a candidate list of size hnsw.ef_search, which defaults to 40. Those 40 nearest neighbors come from the entire table. Postgres then discards every candidate where tenant_id doesn't match, and whatever survives is your result set.

So the expected number of rows you get back is roughly ef_search × p, where p is the share of the table your filter matches:

Tenant share of rows	Expected matches in 40 candidates
34% (410k rows)	13.6
10%	4.0
2%	0.8
0.05% (600 rows)	0.02

My biggest tenant was fine. Everyone else was eating crumbs.

It was worse than the table suggests, because chunks aren't spread uniformly. Generic questions like "how do I reset my password" land in a region of the graph packed with the big tenant's auth docs. A small tenant asking that question was competing with hundreds of near-identical chunks from someone else's workspace, all of them filtered out after winning the race.

Here's the rule of thumb I wish I'd known on day one: with default settings and LIMIT 10, any filter matching less than 25% of your table can come back short. That's just 40 × 0.25 = 10.

How I measured the damage

I had been evaluating answer quality, never retrieval count. So I built a retrieval-only eval: 400 queries, 10 per tenant, each paired with a gold chunk I knew existed. Then I added the one log line that should have been there from the start:

rows = cur.execute(RETRIEVE_SQL, (tenant_id, qvec)).fetchall()
if len(rows) < k:
    log.warning("short_retrieval", tenant=tenant_id, got=len(rows), want=k)

The results with defaults:

253 of 400 queries returned the full 10 rows
147 returned fewer than 10
52 returned zero
Every zero-row query came from a tenant holding under 3% of the table

For those 52 empty contexts, Claude said "not in your docs" 49 times. The other 3 times it ignored my instruction and answered from general knowledge, inventing a settings page that didn't exist. Honestly, I'm not sure which failure is worse. The refusals were wrong but at least looked like a safe system.

Does raising hnsw.ef_search fix it?

Partly, and you pay for it in latency. A bigger candidate list means more survivors after filtering, but every query walks a bigger chunk of the graph, including queries from the big tenant that never needed help.

`ef_search`	Short queries (of 400)	Zero-row queries	p95 latency
40 (default)	147	52	18ms
100	81	21	29ms
400	19	4	61ms
1000 (max)	9	1	140ms

Even at the maximum value pgvector allows, 9 queries still came back short. You can't buy your way out of a 0.05% filter with a bigger list.

There was also a dumber problem. My first "fix" was SET hnsw.ef_search = 400 at the start of the request, and the numbers didn't move at all. We run PgBouncer in transaction mode, so the SET ran on one backend and the query ran on another. Session settings and transaction pooling don't mix. Use SET LOCAL inside the same transaction as the query, every time.

What is a pgvector iterative index scan?

An iterative index scan, added in pgvector 0.8.0, keeps walking the HNSW graph when filtering leaves too few rows, instead of stopping after the first ef_search candidates. It continues until it has enough rows to satisfy the LIMIT or hits a cap set by hnsw.max_scan_tuples (default 20,000).

BEGIN;
SET LOCAL hnsw.iterative_scan = relaxed_order;
WITH candidates AS MATERIALIZED (
  SELECT id, content, embedding <=> $2 AS distance
  FROM chunks
  WHERE tenant_id = $1
  ORDER BY distance
  LIMIT 10
)
SELECT * FROM candidates ORDER BY distance;
COMMIT;

relaxed_order means results can come back slightly out of distance order, which is why the materialized CTE re-sorts them. There's also strict_order, which keeps exact ordering and is slower. For a top-10 that gets stuffed into a prompt anyway, relaxed plus a re-sort is fine.

Results with ef_search back at 40 and iterative scan on:

Short queries: 147 down to 3
Zero-row queries: 52 down to 0
p95 latency: 24ms, up from 18ms

The big tenant's queries didn't get slower at all, because they fill 10 rows on the first pass and never iterate. The cost lands only on the queries that needed the extra work. That's the shape you want from a fix.

When should you skip the HNSW index entirely?

When the filter is so selective that an approximate index is pointless. All 3 remaining short queries came from my 600-row tenant. Do the math: scanning the 20,000-tuple cap at 0.05% selectivity yields about 10 matching rows on average, so roughly half the time you land just under the limit.

But computing 600 exact cosine distances is nothing. So small tenants now skip the vector index:

EXACT_SCAN_MAX_ROWS = 5_000

def retrieve(conn, tenant, qvec, k=10):
    with conn.transaction():
        cur = conn.cursor()
        if tenant.chunk_count <= EXACT_SCAN_MAX_ROWS:
            # tenant_id btree via bitmap scan, then exact sort
            cur.execute("SET LOCAL enable_indexscan = off")
        else:
            cur.execute("SET LOCAL hnsw.iterative_scan = relaxed_order")
        return cur.execute(RETRIEVE_SQL, (tenant.id, qvec, k)).fetchall()

Turning off enable_indexscan stops the planner from picking HNSW, but bitmap scans are a separate setting, so Postgres still uses the tenant_id btree to fetch that tenant's rows and sorts them exactly. I confirmed the plan with EXPLAIN ANALYZE before trusting it. Exact scan for the 600-row tenant: 4ms, 100% recall.

If you have a few huge tenants and many tiny ones, partitioning by tenant or partial HNSW indexes on the largest tenants are the heavier, cleaner versions of the same idea. My routing hack was 6 lines and solved the problem I actually had.

Final scoreboard on the same 400 queries:

Config	Short	Zero-row	Gold chunk in top 10
Defaults	147	52	241
Iterative scan + exact for small tenants	0	0	368

127 more questions where the right chunk actually reached Claude. No model change, no prompt change, no reranker.

What should you check in any pgvector RAG pipeline?

Log rows returned vs. rows requested. If you only log answer quality, a retrieval shortfall looks like a careful model.
Compute your filter selectivity. If ef_search × share_of_rows < k, that filter value will come back short.
Use SET LOCAL, not SET, if anything sits between your app and Postgres.
Re-sort after relaxed_order. It's one CTE.
Eval retrieval separately from generation. A grounded refusal on an empty context is a retrieval bug dressed as correct behavior.

So why does pgvector HNSW filtering return fewer rows than your LIMIT? Because HNSW grabs only hnsw.ef_search nearest candidates (40 by default) from the whole table, and Postgres applies your WHERE clause afterwards, so a filter matching a small share of rows leaves almost nothing behind. Fix it by upgrading to pgvector 0.8+ and enabling hnsw.iterative_scan = relaxed_order with SET LOCAL, re-sorting the results, and routing very selective filters to an exact scan. Then add a log line for len(rows) < k, because Postgres will never tell you on its own.

Written by the developer behind Preterview, an interview prep platform.

frequency_penalty Broke My JSON: 47 of 88 Long Invoices Lost Items

jidonglab — Thu, 29 Oct 2026 00:02:58 +0000

The invoice had 34 line items. The JSON my extractor returned had 19. It parsed. It passed schema validation. Nobody noticed for two weeks.

The cause was one line in a shared client config: frequency_penalty=0.3. I had added it to stop a summarizer from looping. It was also quietly applied to my invoice extraction endpoint, and frequency_penalty breaks JSON output in a way that gets worse with every item in the list. Short invoices were fine. Long ones fell apart.

Here's the mechanism, the math, and what I changed.

TL;DR

frequency_penalty subtracts alpha * count from the logit of every token that has already appeared in the output. The penalty has no idea what JSON is.
JSON repeats the same tokens constantly: quotes, colons, commas, braces, and every key name once per array item. Those counts grow linearly with array length.
Eventually the penalized "continue" token (,) loses to the never-used "stop" token (]). The model closes the array early. The output is valid JSON with missing items.
Constrained decoding (JSON schema / grammar mode) fixes renamed keys but does not fix truncation, because ] is a legal token at every item boundary.
Fix: set frequency_penalty=0, presence_penalty=0, repetition_penalty=1.0 for any structured output request, and validate item counts against something independent.

What actually happened?

Setup: a vLLM server running an open-weights 7B instruct model, called through the OpenAI-compatible API. One endpoint summarized support threads. Another extracted line items from invoices into this shape:

{"items": [{"sku": "A-1001", "description": "Hex bolt M8", "qty": 200, "unit_price": 0.14}]}

The summarizer had a habit of repeating the same sentence three times at the end of long threads. I added this to the shared client wrapper:

DEFAULT_SAMPLING = {
    "temperature": 0.2,
    "frequency_penalty": 0.3,
    "presence_penalty": 0.1,
}

The looping stopped. I moved on.

Two weeks later, accounting flagged an invoice where our extracted total was way below the printed total. I pulled every extraction from that window and compared item counts against a separate OCR line count. These are my numbers from that batch:

Invoice size	Invoices	Wrong item count or invalid JSON
1-10 items	290	1
11-25 items	122	9
26+ items	88	47

Of the 47 long-invoice failures, most were valid JSON with the array closed early. A smaller group had keys that drifted mid-array: "sku" became "SKU" or "sku_id", or "unit_price" turned into "price". Those failed Pydantic validation, so at least they were loud. The truncated ones were silent.

The failure rate tracking array length was the clue. Prompt problems don't usually care how long your output is. A per-token counter does.

How does frequency_penalty actually work?

frequency_penalty is a logit adjustment applied at every decoding step: for each token already in the generated output, subtract alpha times the number of times it has appeared. presence_penalty subtracts a flat amount once a token has appeared at all.

The OpenAI-style formula, which vLLM implements for these two parameters:

logit[t] = logit[t] - frequency_penalty * count[t] - presence_penalty * (count[t] > 0)

Two details matter here:

It operates on tokens, not words or meaning. It doesn't know , is syntax. It just sees a token with a high count.
The count never resets. Item 30 is penalized for every comma in items 1 through 29.

For prose that's the point. You want the model to stop reaching for the same phrase. For JSON it's poison, because JSON is supposed to repeat. An array of 30 objects with 4 keys repeats each key 30 times and each separator over a hundred times.

Why does the model close the array early?

At the end of every array item, the model makes one choice: emit , and keep going, or emit ] and stop. The penalty tilts that choice further toward ] on every item.

, has been emitted many times. ] has been emitted zero times, because the array hasn't closed yet. So , carries a large penalty and ] carries none. At some item, the gap the model "wanted" between them gets erased.

Here's a toy simulation. These logits are made up to show the shape of the curve. They are not measured from any model:

import math

def p_continue(n_items, base_gap=9.0, alpha=0.3, commas_per_item=4):
    """Probability of ',' vs ']' after n_items, two-token toy model."""
    comma_count = n_items * commas_per_item
    gap = base_gap - alpha * comma_count  # ']' has count 0, so no penalty
    return 1 / (1 + math.exp(-gap))

for n in [1, 5, 8, 10, 15, 20]:
    print(n, round(p_continue(n), 3))

Look at the shape. Nothing happens for a while, then the probability drops off a cliff over two or three items. That matches what I saw: invoices were either complete or badly truncated, rarely off by one.

Real models have larger logit gaps for syntax than my toy, and tokenizers often merge ", or }, into single tokens, which spreads the count across several token IDs. That's why my real failures started in the 20s instead of at item 8. The curve is the same. Only where it starts moves.

Why do the keys get renamed?

Key drift is the same math applied to key tokens. By item 25, the token for sku has a count of 25. With alpha=0.3 that's a 7.5 logit penalty, which is roughly a 1,800x hit to its relative probability. Meanwhile SKU, sku, and sku_id are different token IDs with a count of zero.

The model still "knows" the key is sku. It just picks the closest unpenalized spelling. You get a valid-looking object with a key your schema has never seen.

Values suffer too. A column where every row says "EA" or "USD" gets penalized into "each" or "usd". Digits get counted, so long numbers with repeated digits are under pressure as well.

Does structured output or guided decoding fix this?

Partly. Constrained decoding (vLLM's guided JSON, response_format with a JSON schema, grammar-based samplers) masks out any token the schema doesn't allow. That kills key renaming, because "SKU" isn't a legal token sequence when the schema says "sku".

It does not fix truncation. At the end of every array item, both , and ] are legal under the grammar. The mask lets both through, and the penalty still decides between them. If your schema has no minItems, a 19-item array is just as valid as a 34-item one.

So the order of fixes is: penalty off first, constraints second.

What about repetition_penalty in vLLM and Hugging Face?

repetition_penalty is a different knob with a different failure mode, and it's arguably worse for structured output. Instead of subtracting a count, it scales any token that has already appeared: positive logits get divided by the penalty, negative logits get multiplied by it. It's binary per token, not count-based.

The catch: in both Hugging Face generate and vLLM, repetition_penalty looks at the prompt tokens too, not just the output. If your system prompt contains an example JSON object, every key and every brace in it is penalized before the model writes its first token. A value of 1.15 sounds tiny. On a token with logit 20, it's a drop to about 17.4, a 2.6 logit hit on the exact tokens your schema needs.

If you put a schema example in your prompt and run with repetition_penalty > 1, you are penalizing the model for following your example.

How do I fix frequency_penalty breaking JSON?

Stop sharing sampling parameters between prose and structured endpoints. Here's what I run now:

PROSE_SAMPLING = {
    "temperature": 0.7,
    "frequency_penalty": 0.3,
    "presence_penalty": 0.1,
}

STRUCTURED_SAMPLING = {
    "temperature": 0.0,
    "frequency_penalty": 0.0,
    "presence_penalty": 0.0,
    "extra_body": {"repetition_penalty": 1.0},  # vLLM-specific, explicit on purpose
}

I set the zeros explicitly instead of omitting them. If someone changes server-side defaults or a wrapper adds a penalty later, the explicit value wins.

Then, the checks that would have caught this in a day instead of two weeks:

Validate counts against an independent signal. For invoices: sum of qty * unit_price should match the printed total within rounding. For any list extraction, compare against a cheap heuristic count (OCR rows, regex hits, bullet count).
Add minItems or an expected count to the schema when you know it. If the upstream system knows there are 34 rows, say so.
Log finish_reason and output token count per request. Truncation from the penalty shows finish_reason: "stop", not "length", which is exactly why it hides. A sudden drop in tokens per item is the tell.
Write a length sweep test. Generate synthetic inputs with 5, 10, 20, 40, 80 items and assert the output count matches. Run it whenever sampling config changes. My failures only showed up past 25 items; a test suite of 3-item fixtures would never catch them.

The summarizer looping problem I originally added the penalty for? It went away after I lowered max_tokens for that endpoint and added a stop sequence. The penalty was the wrong tool even there.

Why does frequency_penalty break JSON output?

frequency_penalty breaks JSON output because it subtracts alpha * count from every token already generated, and JSON is built from tokens that must repeat: quotes, colons, commas, braces, and the same key names in every array item. As an array grows, the continue token , piles up penalty while the closing token ] has none, so the model ends the array early and returns valid but incomplete JSON. Key tokens get the same treatment and drift to unpenalized spellings like SKU or sku_id. Constrained decoding blocks the renamed keys but not the early ]. Set frequency_penalty, presence_penalty, and repetition_penalty to neutral values on every structured output request, and check extracted item counts against a source that doesn't come from the model.

Written by the developer behind Preterview, an interview prep platform.

Wrapping Claude Code and Codex in Electron: A Stop Bug

jidonglab — Wed, 28 Oct 2026 19:01:46 +0000

I pressed Stop on a task. When I looked 35 seconds later, a file in my test folder was gone.

The file was keep.txt, the folder was throwaway, and I had told the agent to set the trap itself: nohup sh -c 'sleep 20; rm keep.txt' > /dev/null 2>&1 &. Stop was supposed to end everything the task had started. It ended Claude Code, and the rm ran anyway. Of the bugs my end-to-end runs found in the wrapper app, this is the one that deleted a file.

This is part 9 of a series about Refmade, a desktop app I'm building alone. Refmade lets people who don't live in a terminal hand work to the Claude Code or Codex they have already installed, and shows the agents as an office you can watch. It is an independent app, not affiliated with Anthropic or OpenAI. Claude Code and Codex agents wrote most of the code at my direction; I planned, decided and reviewed. The UI is Korean right now; that changes later in this series. This part covers the evening of September 27 through September 29, when the app went from a web dashboard to an Electron shell around two command-line tools.

Spawn the CLI, don't call the model

A wrapper that calls the model API itself needs a key, and then it is a different product with a different bill. Refmade starts the claude or codex binary you installed, in the folder you picked, under your login. The header of the Codex runner in the current source says it plainly: the user's own binary and config are used as-is, and the app never reads auth or tokens.

Both tools turned out to have a machine-readable mode with a channel for permission questions. That channel is the whole trick.

Claude Code: JSON lines in both directions

The runner starts one long-lived claude -p per task. Everything on stdin and stdout is one JSON object per line.

claude -p --input-format stream-json --output-format stream-json \
  --verbose --include-partial-messages --replay-user-messages \
  --permission-prompt-tool stdio
# stdin, first line:
{"type":"control_request","request_id":"init_1","request":{"subtype":"initialize","hooks":null}}
# answers to the CLI's questions reuse the question's id:
{"type":"control_response","response":{"subtype":"success","request_id":"<id>","response":{...}}}

--permission-prompt-tool stdio is what replaces the terminal prompt. When Claude Code wants to run a tool that its settings don't already allow, it sends a can_use_tool request on stdout. The app shows a card, waits for a click, and sends back a control response. --replay-user-messages makes the CLI echo each message it takes up, which is how the app knows whose turn it is when a background job finishes on its own. I pass no permission mode. Your settings.json rules apply, and a tool you allowed there runs without a card.

Codex: JSON-RPC over stdio

Codex goes through codex app-server, newline-delimited JSON-RPC on stdio. My runner was written against the schema of Codex CLI 0.156.0.

 Refmade window
      |  IPC
 Refmade main process ---- JSON lines ----> claude -p            (your install, your login)
      |                 \-- JSON-RPC  ----> codex app-server     (your install, your login)
      '-- plain-words card <---- permission request

	Claude Code	Codex
Process	`claude -p`, stream-json	`codex app-server`
Permission asks	`can_use_tool` control request	`item/commandExecution/requestApproval`, `item/fileChange/requestApproval`, `item/permissions/requestApproval`
Questions to the person	`AskUserQuestion` through the same channel	`item/tool/requestUserInput`

A request the runner doesn't know gets a JSON-RPC error back, -32601, with the text "Refmade does not handle" and the method name. I would rather fail loudly on a request I don't understand than guess an answer.

Cards in words, not command lines

A raw rm -rf prompt is useless to a person who has never used a shell. main/plain.mjs turns a request into a title, one plain sentence, a place (inside or outside the folder you picked), and chips for what changes. It is over 5,000 lines now.

For shell commands it splits on |, &&, ;, || and newlines, reads each part by its first word, skips prefixes like sudo, env and nohup, and reads bash -c '...' from the inside. The sentence is always the app's own reading of the command. The AI's description is shown separately as "AI explanation", so a soft description can never hide what the command does. A program it can't read with confidence gets the maybe chip, and "don't ask again" is not offered for it.

A Codex permission card from my Sept 29 test run, in a throwaway test folder (Korean UI). Title: "May I create a new file?" What: "Creates the file c2.txt in the w-codex-outside folder." Where, in red: "Outside the folder you picked (your user folder)". Changes: "A new file appears". Buttons: "Yes, go ahead", "No, don't", "No, do it differently".

The two runners never answer a permission request by themselves. "Don't ask again" only forwards the CLI's own suggestion, scoped to the session. Later versions add an auto mode that answers light multiple-choice questions for you. It is not a safety guarantee, and a later post covers where I got that wrong in my own copy.

None of this is a sandbox. It is a reading of requests, and the reading can be wrong.

The nohup bug

Claude Code 2.1.283 runs each Bash command in a detached shell of its own. When I stopped a task, the app ended the CLI and its process group, and that did not reach a process the command had pushed into the background. nohup sh -c 'sleep 20; rm keep.txt' & outlived the Stop button and deleted the file.

The fix is in main/procs.mjs, and it is a heuristic. Linux can read another process's environment, so every CLI session gets a REFMADE_TASK variable and leftovers are found by it. macOS ps will not show another process's environment. There the app looks for processes that were orphaned (parent pid 1), started while that task's CLI was running, and whose working folder is the task folder or inside it. It leaves apps and system programs alone.

Known limit, written in the file: a dev server that you or another program started in the same folder while the task ran, and left running, looks identical. On Windows the function returns an empty list. Stop ends only the CLI's own process tree there, and the public Windows build is older than the Mac one. After the fix, the same test left keep.txt in place 35 seconds after Stop.

The same test after the fix, Sept 28, 10:05 to 10:08 PM (Korean UI). My message asks for the nohup command and then a sleep 40. The row says "Thought it through, ran 1 command on the computer, may have deleted 1 file". The status lines read "Task stopped" and "Work running in the background stopped too".

A race I found by launching in the wrong order

Start a task right after launching the app, and the team did not appear in the mini office within 60 seconds. If I opened the office first, it appeared in about a second. The collector's first read of the session logs takes around 4 seconds, and a session folder created during that read was never picked up. After the fix, a first task showed up in 6 seconds.

An end-to-end run against the real app found it, because the run sent a task in the first seconds after launch.

How it was checked

Three rounds of adversarial review by separate agents went over this code. Round 1 ended with 36 fixes and round 2 with 35 more. After round 3 the app tests passed 394 of 394, three runs in a row. Those numbers say the tests agree with the code. They say nothing about whether anyone finds the app easy to use, and I haven't run a usability test with a non-developer.

The app link comes in the last post of the series.

If you wrap an agent CLI, how do you decide that a stopped session is really stopped: process-group kill, an environment tag like mine, or something else?