DEV Community: Ken Imoto

5 PR Metrics on 303 Merged PRs from pytest and pydantic: 2 Point at Review Bottlenecks

Ken Imoto — Thu, 22 Oct 2026 13:00:01 +0000

Engineering-productivity dashboards tend to show the same five pull-request metrics: Time to First Review, Time to Merge, Comments per PR, Review Cycles, and AI Resolution Rate. Each gets a card and a trend line.

I wanted to know which of them can answer one narrow question: "is review the slow part of shipping a change?" My own repositories are no help here. I work alone, and my busiest repo merged four PRs in the last 90 days. So I pointed a script at two public projects with real review traffic:

pytest-dev/pytest: 107 merged PRs from humans (166 merged, 59 from bots), 46 authors
pydantic/pydantic: 196 merged PRs from humans (221 merged, 25 from bots), 40 authors

The window is PRs merged between 2026-07-21 and 2026-10-19 (UTC). All numbers below come from the GitHub REST API, and you can rerun the script to check them.

The script

# review_metrics.py
import json, os, sys
from datetime import datetime, timedelta, timezone
from statistics import mean, median
from github import Auth, Github

REPO = sys.argv[1] if len(sys.argv) > 1 else "pytest-dev/pytest"
END = datetime(2026, 10, 19, tzinfo=timezone.utc)
START = END - timedelta(days=90)

g = Github(auth=Auth.Token(os.environ["GITHUB_TOKEN"]))
repo = g.get_repo(REPO)

rows = []
for pr in repo.get_pulls(state="closed", sort="updated", direction="desc"):
    if pr.updated_at < START:
        break  # sorted by updated: nothing older can have merged in the window
    if not pr.merged_at or not (START <= pr.merged_at < END):
        continue
    if pr.user.type == "Bot" or pr.user.login.endswith("[bot]"):
        continue

    author = pr.user.login
    reviews = [r for r in pr.get_reviews()
               if r.user and r.user.login != author and r.submitted_at]
    first = min((r.submitted_at for r in reviews), default=None)
    approvals = [r.submitted_at for r in reviews if r.state == "APPROVED"]
    hours = lambda a, b: (b - a).total_seconds() / 3600

    rows.append({
        "pr": pr.number,
        "author": author,
        "created_at": pr.created_at.isoformat(),
        "merged_at": pr.merged_at.isoformat(),
        "first_review_h": hours(pr.created_at, first) if first else None,
        "merge_h": hours(pr.created_at, pr.merged_at),
        "last_approval_to_merge_h": hours(max(approvals), pr.merged_at) if approvals else None,
        "review_comments": pr.review_comments,
        "cycles": sum(1 for r in reviews if r.state in ("CHANGES_REQUESTED", "APPROVED")),
        "changed_lines": pr.additions + pr.deletions,
    })

json.dump(rows, open(f"{REPO.replace('/', '__')}.json", "w"), indent=1)

def summary(key):
    v = [r[key] for r in rows if r[key] is not None]
    return f"n={len(v):3d}  median={median(v):7.1f}  mean={mean(v):7.1f}"

print(f"{REPO}  merged {START:%Y-%m-%d}..{END:%Y-%m-%d}  human PRs: {len(rows)}")
print(f"PRs with no review from anyone but the author: {sum(r['first_review_h'] is None for r in rows)}")
for k in ("first_review_h", "merge_h", "last_approval_to_merge_h", "review_comments", "cycles"):
    print(f"{k:26s} {summary(k)}")

Three definitions matter. A "review" is any submitted GitHub review (approve, request changes, or comment) from someone other than the PR author. review_comments counts inline code comments only, not the conversation tab. A "cycle" is one approve or request-changes event.

Raw output:

pytest-dev/pytest  merged 2026-07-21..2026-10-19  human PRs: 107
PRs with no review from anyone but the author: 0
first_review_h             n=107  median=    9.2  mean=  284.6
merge_h                    n=107  median=  143.4  mean=  806.0
last_approval_to_merge_h   n=107  median=    1.8  mean=  128.3
review_comments            n=107  median=    0.0  mean=    3.3
cycles                     n=107  median=    1.0  mean=    1.5

pydantic/pydantic  merged 2026-07-21..2026-10-19  human PRs: 196
PRs with no review from anyone but the author: 100
first_review_h             n= 96  median=    0.2  mean=   38.2
merge_h                    n=196  median=    1.6  mean=   63.4
last_approval_to_merge_h   n= 41  median=    0.3  mean=   28.7
review_comments            n=196  median=    0.0  mean=    0.8
cycles                     n=196  median=    0.0  mean=    0.2

Two things in that output are worth more than any single metric.

Problem 1: the average describes five PRs

pytest's mean Time to First Review is 284.6 hours. The median is 9.2 hours. A dashboard that shows the mean says "nearly 12 days"; nearly two thirds of the PRs got one within 24 hours.

The gap comes from a handful of old PRs. Five of the 107 were opened between January and early July 2026 and only merged inside the window. Their first-review waits (5,568h, 5,415h, 4,261h, 3,305h, 1,650h) add up to 66% of all first-review hours in the sample. Drop those five and the mean falls to 100.5 hours.

The spread is wide even without them:

pytest, Time to First Review	Hours
25th percentile	0.6
median	9.2
75th percentile	95.4
90th percentile	507.5

43 of 107 PRs (40%) got a first review within 4 hours, 69 (64%) within 24 hours, and 20 waited more than a week. "Share of PRs reviewed within N hours" plus the 90th percentile says far more than an average.

Problem 2: the metric quietly skips half the PRs

In pydantic, 100 of 196 human PRs were merged without any GitHub review from someone other than the author. 75 of those 100 came from a single maintainer. Unreviewed PRs merged in a median of 0.3 hours, reviewed ones in 7.5 hours.

Time to First Review cannot be computed for a PR that never got a review, so the 0.2-hour median above describes only 96 of the 196 PRs. Whether merging without a review is fine depends on the project's rules, and I am not judging pydantic's. The dashboard problem is that the number looks complete when it covers 49% of merges. Put the coverage next to the metric: "median 0.2h, 96 of 196 PRs reviewed".

The two metrics that point at review

Time to First Review, reported as a median and percentiles with coverage, measures the stretch where a PR waits and no reviewer has acted. That wait is a review-process delay by definition.

Published research gives a reference point. Google's 2018 case study of its own code review, based on logs of about nine million reviewed changes, reports a median wait for initial feedback of under an hour for small changes and about 5 hours for very large ones, with an overall median under 4 hours for the whole review process (Sadowski et al., ICSE-SEIP 2018). That is a large company with dedicated tooling, so treat it as a reference point rather than a target for a volunteer project.

The lever with the strongest evidence is explicit assignment. Meta suspected that diffs assigned only to a team were slowed by the bystander effect ("someone else will probably review this") and tested it. In an A/B test on 12.5k authors in spring 2023, it randomly assigned one of the top three recommended reviewers on team-only diffs. TimeInReview fell 11.6%, with no significant change in the time reviewers spent on each diff (Improving Code Reviewer Recommendation, ACM TOSEM 2026, arXiv:2312.17169). You do not need Meta's recommender to try the idea. GitHub can request code owners automatically, and team review requests can be routed to individual members by round robin or load balance.

Review Cycles measures whether review converges. In pytest, 68 of 107 PRs merged after one approve-or-request-changes event, 26 after two, and 13 after three to five. Cycles barely tracked PR size (Spearman 0.19 against changed lines), so a high count is less about big diffs and more about how a review gets run, for example comments arriving in several rounds. That makes it a review-process signal you can act on.

The three that answer other questions

Time to Merge mixes three intervals: waiting for a first review, the back-and-forth until the last approval, and the wait from last approval to merge. I split every pytest PR into those three segments and looked at which one was longest:

Longest segment of Time to Merge (pytest)	PRs
Waiting for first review	44
Approval to merge	34
First review to last approval	29

Summed over all PRs, the split was 35% waiting, 49% back-and-forth, 16% after approval. One number covers three different problems, and the approval-to-merge part (CI, merge queues, release timing) is outside review entirely. Time to Merge is still a fine throughput metric. It just cannot tell you review is the bottleneck.

Comments per PR mostly tracks size. In pytest, inline comment count correlated with changed lines at Spearman 0.53, and 67 of 107 PRs (63%) had zero inline comments. In pydantic, 135 of 196 had none. A month-over-month rise in the average usually reflects a shift in PR mix. What it can flag is a sustained zero on large PRs, which is worth a look.

AI Resolution Rate cannot be computed from the GitHub API alone, because GitHub does not record whether a bot comment was "resolved" in the sense the metric needs. You need data from the AI review tool itself, so I did not measure it here. One structural caution applies anyway: the rate depends on what the bot chooses to flag, so changing the tool or its rules moves the number without any change in how your team responds. Compare it only within one tool and one configuration.

What I would put on the dashboard

Time to First Review: median, 90th percentile, share reviewed within 4h and 24h, and how many merged PRs had a review at all
Review Cycles: distribution (1 / 2 / 3+) alongside the mean
Time to Merge, Comments per PR, AI Resolution Rate: on a separate throughput or tooling page

Rerun the script on your own repo before you trust any of these. The two projects here behaved so differently (pytest reviews every PR, pydantic merged half without one) that a single benchmark would have misled both.

The longer version, with the three-layer review model (hooks, AI review, human review), the CodeRabbit configuration, and a GitHub Actions workflow that writes a weekly metrics report, is in my book Systematizing AI Code Review. The English edition is on Kindle.

Sources:

pytest-dev/pytest and pydantic/pydantic, GitHub REST API, PRs merged 2026-07-21 to 2026-10-19
Sadowski, Söderberg, Church, Sipko, Bacchelli. Modern Code Review: A Case Study at Google. ICSE-SEIP 2018
Improving Code Reviewer Recommendation: Accuracy, Latency, Workload, and Bystanders. ACM TOSEM 35(1), 2026
GitHub Docs: About code owners, Managing code review settings for your team

RTX 4070 + logprobs: 96 of 100 Claude Code Requests Needed the Cloud. 97 of 200 Steps Could Go Local

Ken Imoto — Wed, 21 Oct 2026 13:00:00 +0000

Routing whole requests to a local LLM did not work for my Claude Code workload. I took 100 jobs from my work log (March to September 2026), rewrote each one as the instruction I would have typed, and labelled where it could run. 96 needed a frontier model. The other 4 had to stay local because they contained personal or financial details, not because a local model could do them. Zero were simply "a local model is good enough."

Then the unit went down to single steps inside agent sessions. Out of 200 tool-call steps pulled from my Claude Code logs, 97 were labelled "a 4B to 9B local model is enough if you hand it the material." That is 48.5%.

The step-level router that came out of this is on GitHub under MIT: kenimo49/local-step-router. This post covers the numbers behind it, what the router code actually looks like, and where a logprobs judge on an RTX 4070 helped and where it did not.

Why request-level routing failed

A request like "fix the flaky test" is a label, not a payload. Nothing in the text tells a local model what to do until something reads the repo. In the 100 reconstructed requests, the task mix was mostly agent work (74 of 100), and agent work is exactly what you hand Claude Code in the first place.

Raw instructions told the same story. Of 150 instructions sampled from interactive sessions, 138 only made sense with the earlier conversation ("send it", "do the daily log"), and only 5 were labelled local-capable. Of 100 fixed prompts that my cron pipeline sends to claude -p, also 5.

Unit	n	Local is enough	Must stay local (sensitive)	Frontier
Requests rebuilt from the work log	100	0	4	96
Raw instructions	150	5	2	143
Fixed pipeline prompts	100	5	0	95
Single steps	200	97	4	99

Which steps go local

The 200 steps came from three tool types in the logs. The split is lopsided:

Step type	n	Local is enough	Must stay local	Frontier
`WebFetch` (extract something from a page)	90	86	0	4
`git commit` message	50	10	4	36
Sub-agent delegation	60	1	0	59
Total	200	97	4	99

Almost all of the local share is WebFetch. A fetch prompt usually says what to pull out ("the price", "the release date"), and once the page text is in hand, that is extraction. Commit messages go local only when they are one-line summaries of a diff. Sub-agent delegation is research and multi-step tool use, and it stays frontier.

Two caveats matter here. The labels are judgments, not runs: "local is enough" means "given the page body or the diff." And they were assigned by my AI assistant, with my review planned only for disagreements.

So I ran a sample. 20 of the 86 local-labelled WebFetch steps went to qwen3.5:9b on Ollama (think=false, temperature 0), with the page re-fetched and extracted by trafilatura. 11 could be compared with the original output: 8 matched, 2 were partly wrong (a merged-cell spec table read into the wrong column, a title reported as missing), and 1 was wrong (it quoted "about 1 to 3 months" as evidence and still answered yes to "does it say 2 to 3 months?"). The other 9 never reached a fair comparison: JS-rendered pages with no extractable text, pages that had disappeared, or an original fetch that had already failed. Local processing took 6 to 66 seconds per item, median about 18. Most of the loss happened before the model, in fetching and extracting the page.

The router: rules first

The repo's Router checks rules top to bottom, first match wins. A probability judge only runs for steps no rule matched, and it is off by default. This is the shipped examples/routing.yaml, rules part:

rules:                      # step rules, top to bottom, first match wins
  - match: {tool: WebFetch}
    route: local
  - match: {tool: [Agent, Task]}
    route: frontier
  - match: {kind: commit_message}
    route: frontier         # frontier until a probability judge is added

probability: null           # off until measured with the same judge model (chapter 14)
default_route: frontier

And from Python:

from steprouter import config
from steprouter.router import Router

router = Router.from_config(config.load("examples/routing.yaml"))

router.route({"tool": "WebFetch", "prompt": "Extract the price table"})
# Decision(route='local', by='rule', p_local=None)
router.route({"tool": "Edit"})
# Decision(route='frontier', by='default', p_local=None)

Decision.by tells you whether a rule, the probability judge, or the default made the call.

Why rules first: on the 444 non-sensitive items across all three sets (107 labelled local-capable), I compared a single rule against asking a local model for P(local is enough):

Router	Local-capable items caught (of 107)	Frontier items sent local by mistake (of 337)
qwen3.5:9b probability, threshold set for at most 5% wrong-locals (AUROC 0.809)	59	17
gemma4:e4b probability, same calibration (AUROC 0.618)	8	19
One rule: `WebFetch` goes local	86	4

The 9b judge's AUROC of 0.809 was mostly the judge recognising the step type. Within one type it could barely separate items: 0.095 on the pipeline prompts, 0.411 on raw instructions. If the step type decides, write a rule. Keep the probability for what the rules leave open.

I also tried qwen3.5:4b as the routing judge. It timed out under GPU contention at 414 of 450 items, so it has no routing result.

Reading logprobs from Ollama

Where the local model does earn its place is the entrance gate: "does this request contain actual sensitive data?" The judge asks for a one-letter answer (Y or N), generates exactly one token, and reads the candidate letters from top_logprobs. In the repo this is steprouter/judge.py, using the native /api/chat with think: false, num_predict: 1 and top_logprobs: 20. It returns the probabilities renormalised over the candidates, plus mass: how much probability landed on the candidates at all.

mass is the field to watch, because two things make it collapse without any error:

The first token is not the answer. A model that reasons before answering spends token 0 on its reasoning. With think: false, qwen3.5 answers in the first token. gpt-oss:20b did not put a label there and gave mass 0 on my setup, so I could not use it this way. The other models I tried stayed between 0.89 (qwen3.5:0.8b) and 1.00.
A candidate outside the top 20 reads as 0. Ollama rejects top_logprobs above 20 (the check in server/routes.go; a PR to raise it to 100 was closed without merging). If a candidate is not in that list, its probability is simply missing, and the others share 100%.

The repo's judge marks a result unusable when mass is below 0.5. A first test on any new judge model: send one known question and check that mass is close to 1.

One correction to something you may have read, including in an earlier version of my own notes: the OpenAI-compatible /v1/chat/completions endpoint does return logprobs. The v0.12.11 release notes say both APIs support them, and in issue #16117 a maintainer showed it working before the reporter closed it as their own client bug. The compatibility page in the docs still marks logprobs unsupported. The repo uses the native endpoint, so it does not depend on which of those is current.

The threshold is not 0.5

qwen3.5:4b ranked the 60 English test requests almost perfectly for sensitive content (AUROC 0.999), but its P(Y) stayed low. With a fixed 0.5 line it missed all 20 sensitive requests. Drawing the line from labelled examples with 5-fold cross-validation brought that to 2 misses and 1 false alarm. On the Japanese version of the same set, the fixed line missed 9 of 20 and the calibrated one missed 1.

The repo's CLI does this from your own judged examples:

python3 scripts/judge_set.py qwen3.5:4b en > judged.jsonl
steprouter-threshold judged.jsonl --margin 0.3

It prints JSON with auroc, the line (lowest P among your positive examples times the margin), and 5-fold cv counts of misses and false alarms, and it warns when AUROC is under 0.95. The data set is 60 made-up requests, so treat these as small-sample numbers, and redraw the line on your own labels.

Where the RTX 4070 fits

All of the judging above ran through Ollama on one RTX 4070 12 GB (version 0.20.2 for the 60-request tests). Small judges were not faster on that box: qwen3.5:0.8b through 9b all took about 0.5 to 0.7 seconds for two questions, mostly network round trip, while qwen3.5:35b-a3b took about 3.3 seconds because it does not fit in 12 GB. If you want a 35B MoE model on the same card for heavier local work, the --cpu-moe flag in llama.cpp took it from 12.2 to 34.6 tok/s in a separate benchmark. That benchmark is about speed; the routing numbers above did not use it.

What I would copy

Count your own steps before tuning anything. My 48.5% is almost entirely WebFetch. A workload with fewer fetches will get a smaller share.
Decide by step type first. One WebFetch rule beat both probability judges in my data.
Check mass, then draw the threshold from your own labels. A fixed 0.5 missed every sensitive English request with qwen3.5:4b.

The code: kenimo49/local-step-router. MIT, Python 3.10+, PyYAML as the only dependency, Ollama 0.12.11 or later for the judge. The tests run without Ollama.

The full write-up, with the step labelling criteria, the sensitive-data gate, and the typed return values, is the book this repo accompanies: Local LLM or Claude Code? Kindle, included in Kindle Unlimited.

GitHub Stars Lied: 3 Tech-Selection Traps Senior Devs Still Walk Into

Ken Imoto — Tue, 20 Oct 2026 13:00:00 +0000

In the Stack Overflow Developer Survey 2025, 17.6% of respondents said they had done extensive work in Vue.js over the past year. On W3Techs (checked 2026-10-19), Vue.js runs on 0.6% of all websites. That is a 29x spread.

Angular: 18.2% in the survey, 0.2% of websites. About 90x. React, the safe pick: 44.7% against 6.0%, about 7x.

Neither number is wrong. The survey counts developers who answered a question about frameworks. W3Techs counts every website it can detect, and most of those are not hand-built single-page apps. The spread mostly says that the two denominators have nothing to do with each other.

That is the useful part. Every popularity number answers a question about some population, and in a tech-selection meeting nobody asks which population. GitHub stars are the most quoted popularity number of all, and they have a population problem of their own.

Trap 1: the bandwagon effect, dressed as "community size"

The bandwagon effect is the tendency to judge something as right because many people have adopted it. Part of that is rational. A widely used tool has more documentation, a more mature ecosystem, and a bigger hiring pool. The trouble starts when that advantage outweighs every other axis.

GitHub stars, npm download counts, Stack Overflow answers. All three signal community size. None of them signal fit for your project. As of 2026-10-19 (GitHub API), apache/echarts has 67.5k stars, chartjs/Chart.js has 67.7k, and d3/d3 has 113.8k. ECharts and Chart.js look like twins on a comparison table. D3 looks like the winner. D3 is a low-level toolkit for building visualizations; the other two hand you finished chart types. The star column cannot show that.

Some of the stars are not people

A study by He et al., published at ICSE 2026, built a detector called StarScout and ran it over all GitHub event data from July 2019 to December 2024. It flagged 6.0 million suspected fake stars and 18,617 repositories with fake-star campaigns. Fake stars stayed at or below 1% of all stars in any month, but campaigns grew from 2022 and surged in 2024. Most of them promoted short-lived phishing malware repositories. The rest went mostly to AI/LLM, blockchain, tool, and tutorial repos.

The paper also measured whether buying stars works. The effect on real attention lasts under two months, is about 5x smaller than the effect of real stars, and turns negative over the long term.

So for a big, old project, fake stars are a rounding error. For a young repo in a hyped category, the star count can be partly bought. Either way, stars are a weak input, and the bandwagon effect makes them the loudest one.

When the effect gets stronger

The bandwagon effect strengthens under three conditions, and all three describe a normal tech-selection meeting:

Your own expertise in the domain is shallow (so you lean on "what everyone picks")
The decision carries visible blame if it fails (so "we picked the popular one" is the safe story)
You are under time pressure (so detailed comparison gets compressed into "which has more stars")

Fifty thousand stars on one framework and five hundred on a competitor is the setup that triggers all three, even when the smaller one fits your use case better.

What catches it: a requirements matrix

Telling the room to "be careful about stars" does nothing. Changing the table does.

List the project's requirements before you look at candidates: performance targets, the team's existing skills, licensing, long-term support, learning cost. Then add community size as one more row, weighted like the others. Score every candidate on every row. Stars now contribute one row out of six, and the five rows that actually describe your project get filled in before anyone opens a GitHub tab.

Trap 2: the IKEA effect on "the thing we built ourselves"

The IKEA effect is the tendency to overvalue things you put labor into. Norton, Mochon, and Ariely described it after experiments in which people assembled IKEA boxes, folded origami, and built Lego sets, then valued their own results above equivalent ready-made ones. Code is labor you can see, so the effect carries over:

An in-house framework that two engineers wrote in a hackathon five years ago, defended with "it fits our needs exactly"
A custom build config that only one person can modify, kept because "replacing it is risky"
A fork of a library that has drifted three major versions, maintained because "we invested a lot of time understanding it"

Not Invented Here syndrome is the acute version. The chronic version is quieter: a steady preference for the thing that has your team's hours in it.

Market numbers will not prove the effect for you. npm counted 76.6 million Vitest downloads and 26.0 million Jest downloads in the week of 2026-10-10 to 10-16. Jest is still a very large installed base. Some of those projects stay because Jest fits them; some stay because of years of local config. The download count cannot tell you which, and neither can you, until you ask the question below about your own stack.

The point to hold on to: "we built it" is a fact about past labor. It says nothing about quality. The question is what the next twelve months of maintenance will cost.

The question that defuses it

One line, on a sticky note by the monitor:

If this system did not exist and we were choosing today, would we pick this?

A no does not mean "migrate now". Migration costs are real. A no means the current choice may be held in place by the IKEA effect plus the next trap, and deserves an actual comparison.

Trap 3: sunk cost, also known as the Concorde fallacy

Everyone can define the sunk cost fallacy, and almost nobody applies it. "We have already invested three months in this refactor" is the shape. The three months are gone. They do not come back if you continue, and they do not vanish if you stop. The only live comparison is the cost of continuing from here against the cost of stopping and switching from here.

The nickname comes from Concorde. Britain and France signed the development treaty in 1962, and costs overran the original estimate many times over. The British government privately regarded the project as a commercial disaster that should never have been started, yet both governments kept paying. Political and legal commitments made withdrawal hard, which is a lesson in itself: sunk cost rarely acts alone. Twenty aircraft were built, fourteen entered airline service, and the fleet retired in 2003.

In engineering, sunk cost shows up as:

Deferred refactors ("throwing this code away wastes three months of work")
A struggling project that keeps going ("we are 60% of the way there")
Unused features kept alive ("we went to the trouble of building it")

Each of those sentences is a sunk-cost claim in disguise.

The forward-only decision frame

The frame that works gives "already spent" zero weight. Three questions:

Set all past investment to zero. Looking only at future cost and return, which option wins?
What does staying on this choice cost over the next year (maintenance hours, learning cost for new hires, effect on hiring)?
What is the one-time cost of switching, and how much does it save per year afterward? Does the saving pay back the switch within a horizon that matters?

"Already spent" is the loudest voice in a technical meeting. It is also the one voice that cannot change the future.

The hype cycle chains all three

Gartner's hype cycle describes technology maturity in five phases, and a different bias dominates each part of it:

Innovation Trigger to Peak of Inflated Expectations: the bandwagon effect at maximum. "Do not get left behind" overrides every other signal.
Trough of Disillusionment: negativity bias takes over, and the technology gets written off below its real value.
Slope of Enlightenment to Plateau of Productivity: calm evaluation finally becomes possible, but whatever was adopted at the peak is now defended by sunk cost ("we cannot throw it away now") and the IKEA effect ("we worked hard to make it run").

Whatever you adopted during the current AI tooling peak will meet this chain on schedule. When the plateau arrives, the choice will be defended by two biases at once, and the forward-only frame is the tool for that review.

Three sentences for the wall

GitHub stars are one row out of six in the requirements matrix. Not a tiebreaker.
"If this did not exist, would we pick it today?" Ask it for every stack you maintain, once a quarter.
Money already spent is zero on the future-cost line. The only live comparison is forward.

Tech selection is a judgment you keep revisiting, and the three biases keep running the whole time. The sentences do not switch them off. They make the room notice when a bias is making the decision, and that is usually enough to get the actual comparison onto the whiteboard.

These three traps and the hype-cycle chain are Chapter 5 of Cognitive Bias for Software Engineers. Chapter 6 applies the same lens to debugging, starting with confirmation bias and the habit of holding several hypotheses at once. It is on Kindle in English, Japanese, and Portuguese.

Sources (all checked 2026-10-19):

Stack Overflow Developer Survey 2025: Technology: web frameworks, all respondents (React 44.7%, Angular 18.2%, Vue.js 17.6%)
W3Techs: React vs. Vue.js vs. Angular: share of all websites (React 6.0%, Vue.js 0.6%, Angular 0.2%), W3Techs data of 19 October 2026
GitHub REST API, gh api repos/<owner>/<repo>, 2026-10-19: apache/echarts 67,543 stars, chartjs/Chart.js 67,746, d3/d3 113,767
He, Yang, Burckhardt, Kapravelos, Vasilescu, Kästner, Six Million (Suspected) Fake Stars in GitHub: A Growing Spiral of Popularity Contests, Spams, and Malware, ICSE 2026 (doi:10.1145/3744916.3764531)
npm downloads API, last week (2026-10-10 to 2026-10-16): vitest 76,554,353, jest 26,009,195
Norton, Mochon, Ariely, "The IKEA effect: When labor leads to love", Journal of Consumer Psychology (2012)
Wikipedia: Concorde and Sunk cost

Claude, Codex, Kimi K3, Qwen: I graded their web search 120 times

Ken Imoto — Fri, 16 Oct 2026 13:00:02 +0000

The Kimi API usage in this post was paid for with credits Moonshot AI provided for evaluation. Moonshot AI had no part in designing, running, or concluding the test.

Four months ago I wrote that my own AI agents search Brave, not Google. Like most people, I thought about AI search in terms of which search engine sits underneath.

This time I reread the official docs to compare web search in Claude, Codex, Kimi K3 and Qwen. That framing no longer fits. None of the four vendors names its search engine. OpenAI makes its own maintained index the default in Codex. Kimi now sells search itself as a standalone API. The question has moved from "which engine?" to "where does search run, and who calls it?" I did not expect that much change in four months.

If the mechanics changed, the results should too. Before measuring, I assumed live search would beat a cache whenever the question needs fresh information.

Codex CLI's web search is not live by default. It runs in "cached" mode against an index OpenAI maintains. The docs give the reason: it lowers prompt injection risk (Codex: Web search). I read that as trading some freshness for safety.

The measurements said the opposite. I asked the same 8 questions 3 times under 5 setups: Claude, Codex live, Codex cached, Kimi K3 and a local Qwen. Only cached search got every answer right. It even found the title of an article that had been online for about three hours.

The other surprise: the same search API produced 21 correct answers with one model and 9 with another. I gave Kimi K3 and a local Qwen identical access to Kimi's search API. The results were the same. How each model read them was not.

Where web search runs differs by vendor

Before comparing anything, I checked the mechanics in public docs. The biggest difference is where the search actually runs.

	Claude	Codex / OpenAI	Kimi (Moonshot AI)	Qwen
Where search runs	Anthropic's side (server tool)	OpenAI's side (hosted tool)	New API: your code calls it and passes results to the model	Alibaba's side (DashScope). A local Qwen has no search
Freshness control	None	Codex defaults to cached; `--search` for live	Time window on Pro search	7-365 day window
Price per search	$0.01	$0.01	$0.002-0.003	Varies by region
Search engine	Not disclosed	Not disclosed ("OpenAI-maintained index")	Not disclosed	Not disclosed

None of the four names its search engine officially. For Claude, a March 2025 report said Brave Search had been added to Anthropic's list of subprocessors (Simon Willison). Today's docs only say "Anthropic's web search backend."

Claude and OpenAI run the search on their own servers and hand the results to the model. With Claude, results come back encrypted, and you must send them back unchanged to continue the conversation (Claude: Web search tool).

Kimi is mid-transition. The built-in $web_search tool retires on 2026-10-20. Its replacements are standalone REST APIs for search and page fetch, such as /v1/tools/search (Kimi API: Web Search). With the new APIs, you decide when and how often to search. They cost a third to a fifth of the others, billed only when a call succeeds and returns results (Kimi: pricing).

Qwen the model has no search of its own. Alibaba Cloud's API offers search, but a Qwen running locally can only look things up if you wire in an external search tool.

Two details in the docs you will want to know

Claude Code's WebSearch returns only titles and URLs. It does not read the pages. Reading happens in WebFetch, which summarizes the page with a separate model call first. The docs call this "lossy by design" (Claude Code: Tools reference)
Qwen's search on Alibaba Cloud silently skips searching above 15 requests per second. There is no error; the search just does not happen. The FAQ also says English questions often fail to trigger search, and recommends asking in Chinese or forcing search. Both notes appear only in the Chinese docs (阿里云: 联网搜索)

I hit the "lossy" part myself while researching this post. I had Claude Code's WebFetch read the Qwen docs, and it reported finding neither the 15 RPS note nor the English FAQ. Running grep over the raw HTML (about 460 KB) found both. Once a summary sits in the middle, "it doesn't say that" may only mean "the summary dropped it."

How I measured

Five setups, the same 8 questions, 3 runs each: 120 graded answers in total, collected on the evening of 2026-10-14.

Setup	What it is
Claude	Claude Code (`claude -p`) with WebSearch / WebFetch, Opus 5.5
OpenAI live	Responses API `web_search`, gpt-5.6-terra (my Codex CLI's default model)
OpenAI cached	Same, plus `external_web_access: false`
Kimi K3	kimi-k3 with my own search function backed by Kimi's `/v1/tools/search`
Local Qwen	qwen3.5:35b-a3b on Ollama with the same search function

For Codex, I called the API the CLI uses under the hood. In Codex's source, cached mode maps to external_web_access=false and live to true (codex hosted_spec.rs). So the API calls reproduce the CLI's two modes.

These were the 8 questions. I only used questions whose answers I could verify from primary sources or my own site.

#	Question	Answer	What it tests
q1	Kimi K3 API release date	2026-07-16	News after the training cutoff
q2	Kimi K3 open-weights release date (asked in English with "release date")	2026-07-27	The "date" trap
q3	Release date of a Japanese book, MCP実践セキュリティ (Impress)	2026-10-16	Japanese-language sources
q4	Japanese title of a Kimi K3 article posted on kenimoto.dev that day	Kimi K3の動画レビュー、Claudeと互角	Freshness: a page a few hours old
q5	Date the MCP 2026-07-28 spec release candidate came out	2026-05-21	A dated fact
q6	Retirement date of Kimi's `$web_search`	2026-10-20	A detail buried in docs
q7	Kimi K3 total and activated parameters	2.8T / 104B	Numbers
q8	Kimi K4 release date	Not announced	Does it invent something that doesn't exist?

q3 is my own book and q4 is my own article, chosen because I know the answers for certain. That choice carries my own bias, and I'd rather say so. Both are in Japanese, so they also test how well each setup reaches Japanese-language sources.

Results

Setup	Correct (of 24)	Partial	Wrong or no answer	Median time	Searches (avg)
Claude	21	1	2	22.0 s	3.5
OpenAI live	22	1	1	5.7 s	2.3
OpenAI cached	24	0	0	6.2 s	2.7
Kimi K3	21	0	3	25.9 s	4.0
Local Qwen	9	2	13	31.1 s	1.7

Correct answers per question, out of 3 runs:

Setup	q1	q2	q3	q4	q5	q6	q7	q8
Claude	3	2	1	3	3	3	3	3
OpenAI live	3	2	2	3	3	3	3	3
OpenAI cached	3	3	3	3	3	3	3	3
Kimi K3	3	3	3	0	3	3	3	3
Local Qwen	0	2	0	0	0	2	3	2

On the nonexistent Kimi K4 (q8), every setup said it hasn't been announced. Qwen once added, without a source, that K4 had been "mentioned as part of plans for 2027 or later."

Finding 1: the cache wasn't stale

Cached search answered all 24 correctly. The q4 article went live a little after 5 p.m. that day, and I measured about three hours later. Cached search returned its title in all three runs. Honestly, that surprised me. At least for my site, "an index OpenAI maintains" was refreshed within hours.

Source quality also favored the cache at times. On q6 (when $web_search retires), cached search cited Kimi's official docs. Live search got the same answer but, in one run, cited an unofficial roundup site. A wider reach also means picking up more unofficial pages.

Live search missed q3, the Japanese book, once, saying it couldn't confirm a book by that title.

Finding 2: same results, different answers

Kimi K3 and the local Qwen got exactly the same search API in exactly the same way. They still scored 21 and 9. That's more than double.

Qwen's mistakes followed a few patterns:

Its belief that it's 2024 beat the search results. On q4 it said: "As of now (May 2024), an article dated October 14, 2026 does not exist. That is a future date." On q1 it found the July 16 date, then concluded it was "not released as of 2025"
Its sources and its answers disagreed. On q5, all three runs cited the release-candidate announcement dated May 21, then answered with the final release date, July 28
It invented a source. On q3 it cited an Impress press release URL that doesn't exist (404) and gave a wrong release date. Each run gave a different wrong date
It barely searched. 1.7 searches on average, less than half of K3's 4.0

Model size differs, so this isn't a verdict on Qwen as a whole. What it does show is that web search accuracy is not decided by the search API alone.

Finding 3: Kimi's search API has two gaps

All three of K3's misses were q4. It rephrased the query 12 times and never found the same-day article. The page simply wasn't in Kimi's search index yet. The cause wasn't the model. It was index freshness.

The second gap: queries containing the site: operator return HTTP 502. Both K3 and Qwen tried site:kenimoto.dev when hunting for a specific page. Each time, the search API returned a non-JSON error page, and on the first run it crashed my script. My tool now returns the failure to the model so it can rephrase and try again.

Hitting the search API directly also showed how much wording matters. Searching "Kimi K3 release date Moonshot AI" let "date" take over: I got a time-zone site and JavaScript Date docs. Searching just "Kimi K3" put the official site, the paper and GitHub on top. None of the models fell into this trap during the actual runs, though. Both K3 and Qwen rephrased their own queries around it.

By the way, the soon-to-retire $web_search tool didn't work with K3. The second call, the one that returns search results, failed every time with 400 tokenization failed. Passing search through regular function calling, as in this test, works fine with K3 (more in Kimi K3 vs Claude: Video Review, Same Findings).

Claude struggled with a Japanese book

On q3, Claude said in 2 of 3 runs that it couldn't find the book and that it may not exist. The book is listed, with its release date, on the publisher's NextPublishing page and on Rakuten Books.

It was persistent elsewhere, though. In one q4 run, WebSearch didn't surface the article, so Claude found it through the sitemap and opened the page to confirm the title and date. Because WebSearch returns only titles and URLs, fetching pages when needed is part of the design, and here it paid off. Claude also often noted that it had only checked secondary sources, and avoided firm claims.

The 22-second median includes Claude Code's agent loop. Calling the search tool directly through the API should be faster.

How I'll use them

Here's what I'm taking from this:

Leave cached search as the default for research. If you use Codex, there's no need to pass --search every time. The safer setting didn't lose on accuracy
Don't trust "not found" for non-English sources. Not found is not the same as not existing. When Claude says so, I check another way
If you wire in search yourself, the model matters more. With the same search API, K3 and Qwen differed by more than 2x. Pick a model that can read search results before picking the cheapest search API
Make your own search tool return failures. Kimi's search API returns 502 on site:. Pass the failure back instead of crashing, and the model will rephrase and search again

Limits

This is a small test: 8 questions, 3 runs. Topics lean toward Kimi K3 and MCP, and q3 and q4 are my own book and article
Claude ran through Claude Code; Codex results come from calling the API the CLI uses
The Qwen setup is a local 35B model wired to Kimi's search, not Qwen's official search
Each setup works differently, so times aren't directly comparable. I didn't total the costs in this test

Next, I want to call Claude's search tool directly through the API and try Qwen's official search on Alibaba Cloud, to close the gaps between setups.

References

Claude: Web search tool: pricing, citations, encrypted results, versions
Claude Code: Tools reference: how WebSearch and WebFetch behave
Codex: Web search: cached / live modes
OpenAI: Web search tool: external_web_access
openai/codex hosted_spec.rs: how the modes map to external_web_access
Kimi API: Web Search / best practices / pricing
阿里云百炼: 联网搜索 (Chinese; the 15 RPS note and the English-question FAQ appear only here)

Kimi K3 reads video natively. Claude still caught the same issues

Ken Imoto — Thu, 15 Oct 2026 13:00:02 +0000

Moonshot AI provided the Kimi API credits used here ($50) for evaluation. They had no part in designing, running, or concluding the test. Everything K3 couldn't do, and the features that didn't work, are in this post too.

Moonshot AI's Kimi K3 takes video files straight through the API. Claude has no video input, so when I want Claude to look at a video, I cut it into frames first. Codex CLI doesn't have a video channel either, even when you point it at K3 (Kimi API: Use K3 in Codex).

I wanted to know how much that difference matters in real work. My test material was a pair of 40-50 second hands-on videos I made for the SSH series on kenimoto.dev /learn/ (Japanese).

The short version:

K3 and Claude agreed on what was wrong. Where viewers would drop off, the six fixes, where to insert diagrams, even which video to post first: all the same
The differences were timestamp precision and effort. K3 pinpointed frozen-screen intervals to fractions of a second, and all I did was hand it the file. Claude estimated from frames sampled every 2 seconds, so its timestamps drift by about ±1 second
K3 doesn't understand audio. It can't hear the narration or catch audio sync problems
Claude was faster. The same request took K3 363 seconds and Claude 185

The test material: auto-generated hands-on videos

The videos come from handson-cast, a CLI I'm building. You give it a JSON file of steps. It runs each command for real inside Docker containers while vhs records the terminal, then adds narration from OpenAI's text-to-speech and burns in subtitles. It's Python and ffmpeg underneath. The narration and subtitles are in Japanese.

I made two videos:

Key authentication (55 s): create a key, install the public key, confirm key-only login works, then disable password auth
Bastion host (47 s): the same key gets rejected by the bastion with ssh -J, but works through ~/.ssh/config

Every line of terminal output was produced by OpenSSH 10.0p2 running in a container. Nothing is staged. When the rejection message changes from (publickey,password) to (publickey), that really happened on camera.

The first cut (v1) was not watchable, though. The first second shows me typing an internal wrapper script, bash /lab/ssh-config.step-01.sh. You can't tell what the video is about. (I made it, and I couldn't tell either.)

Here's v1 (47 s, with audio):

Click the image to open the mp4 (with Japanese narration).

I showed v1 to K3 and Claude and asked what was wrong with it.

Sending video to K3, and what it costs

The K3 API is OpenAI-compatible. You upload the video as a file first, then reference the returned ID with an ms:// prefix.

from openai import OpenAI
client = OpenAI(api_key=KEY, base_url="https://api.moonshot.ai/v1")

f = client.files.create(file=Path("learn-ssh-key-auth.mp4"), purpose="video")
r = client.chat.completions.create(
    model="kimi-k3",
    max_tokens=32000,
    messages=[{"role": "user", "content": [
        {"type": "video_url", "video_url": {"url": f"ms://{f.id}"}},
        {"type": "text", "text": "Why would viewers drop off in the first 3 seconds?"},
    ]}],
)

Public URLs aren't accepted. The official docs describe video as a set of key frames, and higher resolution means more tokens (Kimi API: Vision).

My first measurement used one 55-second, 1920×1080 video:

Item	Measured
Input tokens	40,659 (almost all video)
Output tokens	2,500 (1,912 of them reasoning)
Cost	about $0.16
Time	83.5 s

Costs use $3 per million input tokens and $15 per million output tokens.

Two practical tips from actually running it:

Assume it can't hear anything

For this run I made a version without burned-in subtitles and asked K3 to transcribe the last sentence of the narration. It answered that it couldn't recognize any audio. Not pretending to hear something it can't is exactly what you want from a reviewer. But it means K3 can't see narration content or subtitle-audio drift. It sees the screen and any subtitles burned into it, nothing else.

It reads the screen well, and accurately. It listed the commands and output for all four steps in order, including the 600 / 644 permissions, the change in the rejection message, and an on-screen comment. The only misread was the wrapper script's filename.

Give it a generous output budget

K3 always reasons before it answers. With max_tokens at 300, the reasoning used everything up and the answer came back empty. It thought hard and then left without saying a word. At 8,000, a full video review got cut off in the middle of the first question. For long answers like reviews, I use 32,000.

The comparison setup

K3 and Claude each reviewed the same two v1 videos, under matched conditions:

	K3	Claude
How the video goes in	The file, once	Frames extracted and read as images
Audio	Not understood	Not used (transcription turned off)
Narration script	Step JSON attached	Same JSON attached
Questions	Same wording (5 questions)	Same wording (5 questions)

The Claude side ran as a subagent that never saw K3's answers. I had already read them, so if I'd done the review myself, I would have been anchored. That's not a comparison.

For frame extraction I used /watch from claude-video, and left the frame count and resolution up to Claude. It ended up re-extracting 4 times. For the opening it pulled frames at 0 / 0.5 / 1 / 2 / 3 seconds, then one every 2 seconds for the rest: 34 frames in total.

The five questions:

Where would viewers drop off in the first 3 seconds?
Up to six improvements, biggest impact first (what to change / why it works / which pipeline stage implements it)
If you combined the article's diagrams with the terminal recording, which diagram goes where, for how long?
Which video should go to social media first?
What's already good?

Result: nearly identical findings

They mostly matched. Lined up side by side, the answers overlap almost completely:

Finding	K3	Claude
Seconds 0-2 show the wrapper script being typed; the topic is unclear	✓	✓
The first narration line introduces the site, not a reason to watch	✓	✓
`Could not resolve hostname` in the bastion video is a strong hook, buried	✓	✓
Open with the conclusion or a question (cold open)	✓	✓
Cut the typing and the waits between steps	✓	✓
Bigger font, highlight the key output lines	✓	✓
Make a vertical version	✓	✓
Insert the article's diagrams: permissions at `ls -l`, the before/after at the climax	✓	✓
End on the article's OG image	✓	✓
Post the bastion video first	✓	✓
Show a progress bar along the top	✓
Rewritten narration lines, as actual sentences		✓
The four `ssh-copy-id` INFO lines are noise		✓

The fixes themselves are standard short-video advice: open with the conclusion, cut silence, keep subtitles short, put the call to action at the end. Either model will give you that. This comparison didn't surface anything only K3 could find. I expected a model that reads video directly to catch something Claude couldn't see. It didn't.

Where they differed: timestamps

The biggest gap was in how precisely each one placed things in time.

K3 wrote things like "the first meaningful output appears at 3.7 seconds" and "the error at 00:03.2 in the bastion video gets buried," sub-second throughout. When I later had it review an improved version, it came back with intervals: "from 30.5 to 36.1 seconds the screen is frozen after the output appears."

Claude's timestamps were estimates from frames 2 seconds apart, and Claude said so at the end of its review: interval boundaries could be off by about ±1 second. You can sample more densely, but then there are more images to read. It's a trade-off.

"Where exactly is the screen frozen?" is hard to answer from a handful of stills. That's where reading the video directly paid off.

Effort and speed

	K3	Claude
My effort	Hand over the file	Decide how to sample frames (Claude redid it 4 times)
Time	363 s	185 s
Size	84,461 input / 10,863 output tokens, about $0.20 (part of the input billed at the cache rate)	111,313 tokens, 40 tool calls

K3 took less effort; Claude was faster. Each has its edge. Because K3 spends time reasoning, each review took 6-8 minutes. Enough time to make a coffee and come back.

The videos really did get better

I made the fixes. Following the reviews, I added six changes to handson-cast:

Open with a card: "Same key, but ssh -J gets rejected. Why?"
Cut the wrapper-script typing and the waits between steps
Type commands out character by character, with a larger font
Right after the rejection, show the article's diagram full-screen for 4.5 seconds
Once the output is complete, slowly zoom into the top-left of the terminal
End with the article's OG image and URL

Here's the final cut (v4, 41 s, with audio):

Click the image to open the mp4 (with Japanese narration).

Frames from the same timestamps, side by side:

The bastion video went from 46.8 to 41.0 seconds, about 6 seconds shorter. I had K3 grade the improved cut against a 10-item checklist for technical short videos. It covers things like a hook in the first 3 seconds, something changing on screen every 1-3 seconds, and a short call to action only at the end. It scored 7 pass, 2 borderline, 1 pass (estimated). The two borderline items were a frozen stretch after the output appears and an overlong subtitle. Both are fixed now.

I'd meant to have K3 research that checklist too, but that didn't work. The Kimi API has a built-in web search tool ($web_search). With K3, though, the second call, the one that returns the search results, failed every time with 400 tokenization failed. With kimi-k2.6 the first search went through, but after that it stopped searching and wrote text that only looked like search calls into its answer. The docs say this feature is scheduled for retirement on 2026-10-20 (Kimi API: Web Search). I gave up. Claude's web search did the research, and I passed the results to K3.

How I'll split the work

Based on this, here's how I plan to use them:

K3 when I need to know where in a video something happens. Frozen intervals, the moment the screen gets cut off, how long a caption stays up: questions whose answer is a span of time. I don't have to think about how to sample frames
Either one for deciding how to fix it. The improvement ideas didn't differ
Not K3 for anything involving audio. For narration content or sync, I prepare a transcript separately and compare against it
No confidential screens. The video goes to Moonshot AI's servers. These videos were going public anyway, so that was fine; internal screen recordings need different handling

It's a small comparison: two videos, one set of questions, nothing statistical. Still, it gave me a working answer. Reading video directly didn't change the judgment; it changed timestamp precision and effort. Next I want to plant deliberate defects in a video and count how many each model finds.

References

MoonshotAI/Kimi-K3 (GitHub): model card. 2.8T total parameters, MoE with 104B active per token, 1M context
Kimi API: Vision models: how video input works
Kimi API: Web Search: $web_search and its retirement date
bradautomates/claude-video: the /watch tool used for Claude's frame extraction
charmbracelet/vhs: terminal recording
SSH: five ways to reach a remote machine, compared on seven axes (Japanese): the /learn/ series these videos were made for

Shai-Hulud: 796 Packages, 1 Yes-Mash, and the Claude Code Auto-Approve Risk

Ken Imoto — Wed, 14 Oct 2026 13:00:01 +0000

On November 24, 2025, JFrog Security Research disclosed that a self-replicating npm worm had quietly backdoored 796 unique packages. The industry called the second wave Shai-Hulud 2.0, or "The Second Coming." It followed the September 16, 2025 original wave, which had already poisoned over 1,150 packages (JFrog, Datadog Security Labs).

Over 25,000 GitHub repositories had credentials leaked by the campaign (Sysdig). The 796 is the count of poisoned npm packages. The 25,000 is the count of victim repositories whose tokens and secrets the worm harvested; the attacker-side drop points were separate public repos (random 18-character names, description Sha1-Hulud: The Second Coming) created on victim accounts.

The infection vector was the same thing every developer does fifty times a day. Run a package install. Hit Enter.

Now add Claude Code, or any other coding agent, with "always allow" turned on for npm install. The fifty-times-a-day moves to 500-times-a-day. The attacker now needs one of those 500 to land on a compromised version.

That is the article.

What Shai-Hulud 2.0 actually did

The worm's loop was simple:

Infect. A victim ran npm install on a poisoned version of a popular package. The payload fired as a pre-install hook. (V1 used post-install; V2 moved it earlier, so it ran even on failed installs.)
Steal. The payload scanned the victim's machine for ~/.npmrc tokens, GitHub tokens from ~/.config/gh/, SSH keys, AWS and GCP credentials, and loose .env files. Credentials were exfiltrated by creating a public GitHub repository on the victim's account (description stamped Sha1-Hulud: The Second Coming) that acted as the drop point.
Self-replicate. Using the stolen npm token, the worm published poisoned versions of every package the victim had publish rights to. The victim became the next publisher.
Persist. In some cases it enrolled the victim's host as a self-hosted GitHub Actions runner named SHA1HULUD, so the attacker could execute code on the machine later (Netskope, Zscaler).

Three details matter for the agent-usage angle:

The payload ran on install, not on first use. There was no runtime signal.
The exfiltration channel was public GitHub, not an exotic webhook. Egress filters that block weird DNS wouldn't help.
The runtime was Bun, not Node. Signature-based scanners trained on Node payloads lagged (Endor Labs).

By the time npm removed the compromised versions, the token mesh was already seeded. The affected packages had roughly 132 million monthly downloads between them.

Why "always allow" is the exploited behavior

Claude Code's permission prompt is the exact friction Shai-Hulud needs you to remove.

Default behavior: Claude Code asks before running npm install <pkg>. You read the package name, you decide. One prompt per install. Annoying when you're installing twelve dependencies for a Next.js app, which is why /permissions exists and why most Claude Code users have Bash(npm install:*) on "always allow" by their third week. Mine was on by week two. I know this because I checked after reading the JFrog post.

The attacker's threat model relies on this. The payload does not need to defeat your judgment. It needs you to never be asked.

The equivalent risk on other agents:

MCP servers with auto-approve: true for all tools, so a server you installed last month can shell out and you would never see it.
Codex, Aider, or Cursor running under --yes or an auto-run shell wrapper.
CI: a bot account running npm install without lockfile integrity, auto-publishing on merge.

In every case the pattern is the same. A human decided once, early, then stopped looking. The worm does not need to be smarter than Claude Code. It needs the human in the loop to have left the loop.

A defensive checklist that isn't "write a CLAUDE.md"

CLAUDE.md helps against prompt-injection and model behavior. It does not stop a signed, published, poisoned package from running exec in a pre-install hook. By the time install scripts fire, the model is no longer in the loop.

The checklist below targets the install surface itself.

Per-session, before you say "always allow":

Run /permissions and read what is on always-allow right now. If Bash(npm:*) or Bash(*) is on the list, that is your exposure.
Scope allow rules to specific commands, not globs. Bash(npm ci) or Bash(npm install --package-lock-only) is survivable; Bash(npm install:*) is not.
For MCP: review auto-approve arrays per server. New MCP servers get a sandbox project first, not production.

At the install layer:

Pin everything. package-lock.json + npm ci + --ignore-scripts on CI where you can. Install scripts are the exact Shai-Hulud entrypoint.
Flip --ignore-scripts on globally in local dev (npm config set ignore-scripts true), then whitelist the handful of packages that legitimately need scripts.
Verify the registry you resolve from (npm config get registry) and the integrity hashes in your lockfile. The worm published to the real npm registry, so this does not save you from the attack itself, but it does catch tampering between your lockfile and your node_modules.

At the detection layer:

Socket, Snyk, or GitHub Dependabot wired into every repo. Socket in particular flags behavioral risks like install scripts and unusual network calls, and was fast on both Shai-Hulud waves.
Token hygiene. ~/.npmrc and ~/.config/gh/ are what the worm wants. Rotate npm tokens quarterly, use npm token create --read-only for CI, scope GitHub PATs down to the minimum.
Watch your account for the Nov 2025 exfiltration markers: public repos with randomly-generated 18-character names and description Sha1-Hulud: The Second Coming, workflow files at .github/workflows/discussion.yaml or formatter_*.yml, and self-hosted runners registered as SHA1HULUD. (The V1 wave in Sept 2025 used repos literally named Shai-Hulud and a workflow at .github/workflows/shai-hulud-workflow.yml.)

At the agent layer:

Do not leave always allow on across restarts for Bash, WebFetch, or anything that writes files outside the project. Per-session is survivable, forever is not.
Have the agent list the package name and the latest publish date before any install. A fresh version of a popular package deserves a human look, even when you are tired.

None of these are novel. All of them are standard. What Shai-Hulud 2.0 showed is that the "I'll just hit Enter" discount was always being subsidized by someone else's attention budget. The second wave took the subsidy away.

Trust, but verify

Reagan used it on arms control. It fits here. Claude Code is in the loop. Your other agents are in the loop. Both are useful. Neither replaces the one-second pause before you widen a trust boundary.

The industry has, so far, caught Shai-Hulud waves inside 48 hours of publish (CISA alert, Trellix). The vulnerability window is small, but it is the exact window your "always allow" paves over.

For a longer walk-through of Claude Code security patterns, including CLAUDE.md templates and data-retention policy per plan, I pulled these together in Practical Claude Code on Kindle.

Deepgram Nova-3 + Cartesia Sonic 3 + GPT-4o: Where 750ms of Voice AI TTFB Actually Hides

Ken Imoto — Tue, 13 Oct 2026 13:00:00 +0000

Voice-AI conversations die the moment the agent takes too long to say the first word. Not the full answer. The first byte of audio.

The industry calls this TTFB (time to first byte), or TTFA (time to first audio). I measure it from the moment the user stops speaking to the moment the first audio frame plays. Everything else, including total answer length and follow-up quality, happens after the user has already decided whether the agent feels alive.

Below is a ~750ms streaming budget on a Deepgram Nova-3 + GPT-4o + Cartesia Sonic 3 stack, broken into what each hop actually costs. And where almost every vendor-quoted number stops matching the number you will measure in production.

1,200ms to 750ms is the easy part

Batch pipelines wait for the full response before TTS starts:

[User stops speaking] → STT → LLM (full answer) → TTS (full) → [play]
                                                                 ^ >1,200ms

Streaming pipelines hand sentence 1 to TTS the moment the LLM emits it, while sentences 2 and 3 are still generating:

[User stops speaking] → STT → LLM (sentence 1) → TTS (sentence 1) → [play]
                               LLM (sentence 2) → TTS (sentence 2) → [continue]
                               LLM (sentence 3) → TTS (sentence 3) → [continue]

On the stack above, the streaming budget breaks down roughly like this:

Hop	Time	Notes
STT (Deepgram Nova-3 or Flux)	350ms	includes end-of-turn detection on Flux
LLM first sentence (GPT-4o)	150ms	first-token latency, best-case
TTS first chunk (Cartesia Sonic 3)	170ms	first audio frame
Network round-trips	80ms	user ↔ server + server ↔ vendor
Total (streaming TTFB)	~750ms

The 1,200ms to 750ms move pays for itself: no code change beyond "do not wait for the full response." The hard part is everything below 750ms.

STT: Nova-3 is 300ms, Flux adds end-of-turn for about 260ms

Deepgram's public numbers, cross-referenced with Softcery's 2026 comparison:

Deepgram Nova-3: streaming latency under 300ms, 36 languages. General ASR; does not detect end-of-turn.
Deepgram Flux: around 260ms end-of-turn, English only. Flux folds ASR together with turn detection into a single model and emits StartOfTurn, EagerEndOfTurn, EndOfTurn events (Deepgram launch post).

The combination matters. Nova-3 transcribes, but you still need a separate endpointing step to know when the user is done. Flux folds endpointing in, which cuts a 400-500ms round-trip, but costs you multi-language support.

For reference across other vendors: OpenAI gpt-4o-transcribe is around 320ms with 100+ languages. AssemblyAI Universal-3 Pro Streaming is under 600ms. ElevenLabs Scribe v2 Realtime is around 150ms with 90 languages.

Each of these is a vendor-quoted number. None of them is what you will measure from Tokyo over LTE to a US-East endpoint.

LLM: GPT-4o claims 200-400ms, measured 719-1,608ms

This is the hop where the vendor-vs-real gap gets ugly.

Deepgram's own blog post listing reasons they chose GPT-4o for voice agents cites a first-token latency of 200-400ms. The same post's measured table, on a MacBook M1 Pro over Wi-Fi, reports GPT-4o first-token times of 719ms to 1,608ms. That is 2x to 5x the vendor's number, on a laptop, on consumer Wi-Fi.

A separate Dev.to benchmark from CloudX tested 30+ stacks and found GPT-4.1-nano landing at 0.30-0.40 seconds on 2nd-turn-onward, where the KV cache is warm. First turn is always worse.

What the numbers teach: first-token latency is dominated by TTFT variance, which is dominated by model size, GPU type, quantization, input length, KV cache hit rate, inference backend load, and cold starts. A table of "model X equals N ms" is strictly less useful than one measurement from your own region to your own endpoint.

Running a small LLM locally removes the LLM-side network round-trip entirely. On a tuned setup, that is where sub-500ms TTFB actually lives.

TTS: Cartesia Sonic 3 says sub-90ms, measured median is 166-190ms

Cartesia's two models get swapped in half the voice-AI blog posts:

Cartesia Sonic 3: sub-90ms (vendor claim, measurement definition undisclosed), 40+ languages, emotion support.
Cartesia Sonic Turbo: 40ms (vendor claim, speed-specialized model), 9 languages.

The widely-circulated 40ms number is Sonic Turbo, not Sonic 3. Real Sonic 3 measurements:

Cartesia's own launch disclosure: median around 190ms.
Vapi's Humanness Index (50-run test, round-trip included): median 166ms.

Both numbers are roughly 2x the vendor-quoted sub-90ms. Neither 166 nor 190 is "wrong." They measure different things: vendor model-inference vs client-side round-trip.

ElevenLabs Flash v2.5 quotes around 75ms, with their own docs noting that 75ms refers to model inference only, and that end-to-end latency will vary with your location and endpoint type (ElevenLabs latency docs).

Research on streaming TTS reaches sub-50ms TTFA (arXiv 2508.04721). But the same paper's own end-to-end telecommunications voice agent, with the LLM in the loop, measured 678ms average TTFA. The TTS was fast. The stack still was not.

The 3-level optimization ladder

Treating these numbers as a budget, the ceiling moves in three steps.

Level 1, 500ms target: fast TTS + sentence-level streaming + streaming STT. This is the "do not wait for the full response" move above; gets you from 1,200ms+ to roughly 500-750ms.

Level 2, 300ms target: Level 1 plus Flux-style end-of-turn detection, speculative LLM invocation on partial ASR, a smaller LLM, and a TTS whose measured (not quoted) first byte is under 100ms. The emphasis is on measured.

Level 3, 200ms target: edge inference, a unified STT+LLM+TTS model, hardware tuning. At this point you stop composing vendor APIs and start building.

The 300ms figure is not arbitrary. It is the UX cliff below which a voice agent starts feeling like a human conversation partner, as opposed to a very fast IVR. Everything above 500ms feels like an echo.

Three rules before you trust any TTFB number in a blog post

Was the network included? ElevenLabs says the quiet part out loud (75ms is model inference only). Most vendor numbers do not.
Which model? Cartesia's 40ms is Sonic Turbo; the Sonic 3 you probably want is 2x that.
Who measured it? Vendor, third-party, or user-run on their own hardware? Each of the three routinely disagrees by a factor of 2.

Build your own measurement. Pick the stack above, or three alternatives on each row of the table, point at your actual region, record a 50-run distribution. The p95 is the number you should budget against. The vendor's median is marketing.

The full chapter this article compresses, including the TTS vendor-vs-measurement table and the 3-level ladder in more detail, is in The 300ms Threshold on Kindle. The book is built around voice-AI UX cliffs at 300, 500, and 800ms, and how to budget inside them.

Kernighan, Fowler, Torvalds: 7 Debugging Quotes Every Senior Still Repeats

Ken Imoto — Mon, 12 Oct 2026 13:00:00 +0000

Some engineering lines refuse to fade.

They survive because every senior ends up repeating them in code review, year after year, usually on a Thursday afternoon when a junior ships something that works and shouldn't.

I spent the last few months decoding a hundred of these for a short Kindle book. Below are seven, from three names that keep surfacing when the topic is debugging: Brian Kernighan, Martin Fowler, Linus Torvalds. Each quote has its original public source. Each one has a modern code-review move I use when I quote it.

1. "Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?"

Brian Kernighan & P. J. Plauger, The Elements of Programming Style, 2nd ed., ch. 2, 1978.

Half a century later, this is the single question I ask when a PR shows up with a one-line regex, a nested ternary, or a reduce that collapses six operations into one line. Not "is it correct." How will you ever debug it?

The AI-pair-programming era rediscovers this one every quarter. Give Claude Code or Copilot a prompt like "make this shorter" and it will happily compress four lines into one with a currying trick. The code works. Six months later, nobody (human or model) can rewrite the same function from scratch, because the compressed form hides the steps.

My review rule: if the author needs more than one sentence to explain what the line does, split it. I know, because I've been that author more than once.

2. "The most effective debugging tool is still careful thought, coupled with judiciously placed print statements."

Brian Kernighan, Unix for Beginners, 1979.

I used to be embarrassed by println debugging. Then I watched enough senior engineers reach for it before a debugger that I stopped apologizing.

The quote still holds, with one update. The tool today is console.log, fmt.Println, print(), or whatever your runtime offers, plus a tail of structured logs. Debuggers help when you can reproduce the bug and the state is static. Print statements help when the bug lives in a flow, in production, or at a boundary the debugger can't easily cross: gRPC, serverless cold start, worker queues, flaky async tests.

The actual skill the quote names is careful thought. The prints are only useful once you've guessed where to put them.

3. "Controlling complexity is the essence of computer programming."

Brian Kernighan & P. J. Plauger, Software Tools, p. 319, 1976.

A junior and a senior will both call code "complex." The junior usually means "I don't understand this yet." The senior usually means "nobody will ever understand this again, including the person who wrote it."

I keep this quote pinned when reviewing refactors. The question isn't "is this clever." It's "does the next on-call engineer, at 2 a.m., with no context, have a chance?" Most production bugs I've chased started as a cleverness-driven refactor that added coupling the diff didn't name.

Complexity accrues. Most PRs make it worse. The ones that make it better are rare and should be celebrated.

4. "Any fool can write code that a computer can understand. Good programmers write code that humans can understand."

Martin Fowler, Refactoring: Improving the Design of Existing Code, p. 15, Addison-Wesley, 1999.

This is the one line of Fowler's that engineers who've never read Refactoring can still recite. Fair, because it compresses the whole book: tests exist so humans can change code; names exist so humans can read it; small methods exist so a human can hold one in their head.

The habit I lifted from this: whenever I'm tempted to leave a comment explaining what a block does, I rename or split instead. If the code reads without the comment, the comment was a smell.

LLM code generators love long comments. Rename the function.

5. "If it hurts, do it more often."

Martin Fowler, martinfowler.com, FrequencyReducesDifficulty.

Fowler surfaces this as one of his favorite soundbites. The surface reading sounds like masochism. The real reading: if deploys are painful, deploy daily; if merges are painful, merge every hour; if integration tests are painful, run them on every push.

The pattern survives the current stack. Painful AI-code reviews? Review each PR as soon as it opens, not once a day in a batch. Painful dependency bumps? Automate them nightly instead of quarterly. Painful incident post-mortems? Hold them within 48 hours while people still remember.

Every time I've watched a team avoid a painful activity, the pain compounded. Fowler's rule flips the compounding.

6. "Talk is cheap. Show me the code."

Linus Torvalds, Linux Kernel Mailing List, 25 Aug 2000.

Linus wrote this in the middle of a kernel discussion where someone was proposing changes without an implementation. The adage has outlived its original thread.

Modern application: architecture debates on Slack where no participant has opened the file in question. Design reviews on a tool nobody has prototyped. "We should migrate to X" threads with no benchmark attached. The quote doesn't ban design. It bans debating before anyone has sampled the code. Open a branch. Commit a 30-line proof of concept. Attach the diff to the thread.

I keep a text expander snippet with this quote. I pull it out maybe once a month, and every time it ends the thread in under an hour.

7. "Bad programmers worry about the code. Good programmers worry about data structures and their relationships."

Linus Torvalds, Git mailing list, 2006. (git mailing list via LWN, 2006-07-27)

The underrated debugging quote on this list. Most bugs that reach production aren't in the control flow. They're in the shape of the data: a nullable field someone assumed was present, a one-to-many relationship someone modeled one-to-one, a cache that lies about freshness, a queue that quietly drops ordering guarantees.

When a bug surfaces, the first question I ask is what the data actually looks like right now, and what the code assumed it would look like. More than half the time, the fix is a schema change or a type refinement, and the logic was never wrong. I lost a weekend last year "fixing" logic that was fine — the schema was lying about a nullable column.

Modern echo: in LLM pipelines, the hardest bugs don't live in the prompt. They live in the context layout the prompt silently depends on: what the retriever returned, in what order, with what metadata.

Why these seven, out of a hundred

The engineering world has more quotes than it has reviewed PRs. These seven survive translation across stacks: a 1978 line is still live in a 2026 Claude-Code-generated diff.

If this slice landed, the full set (100 quotes across 10 chapters — code aesthetics, bugs and failure, design, teams, innovation, learning, craftsmanship, open source, the AI era, and an engineer's life) is in 100 Engineering Quotes Decoded. One decoded quote at a time, so you can read a decoding on a coffee break and bring it into code review the same day.

5 Code Review Biases: Why Senior PRs Pass Faster Even When They Shouldn't

Ken Imoto — Mon, 05 Oct 2026 13:00:00 +0000

Meta researchers suspected the bystander effect was slowing their code reviews when a group — rather than one person — got asked to look at a change. They tested it, confirmed it, and shipped BystanderRecRnd, a recommender that assigns a single reviewer so the "someone else will take it" delay stops happening. The paper is worth reading on its own. It is also one of five biases that quietly decide which PRs pass fast and which ones get picked apart, independent of code quality.

The other four — halo, authority, framing, ownership — show up every day in reviews that look technical from the outside. This piece walks through each one with where it fires, and the mechanism that stops it. Not vigilance. A mechanism. Reminders to "be careful" survive about a week.

1. Halo effect: the senior PR passes faster

The halo effect is the oldest finding in applied social psychology, replicated so many times it stopped being news. In code review it shows up as this: the same diff, submitted by a senior and by a new hire, gets different scrutiny. The senior's code is "they probably have a reason." The new hire's is "they haven't learned yet."

The inverse also fires. An engineer who shipped a famous bug last year gets careful reviews on code that is objectively fine. The halo rots in both directions.

I have been on both sides of this. The senior PR with a design flaw that got waved through, and the junior-era PR with a reasonable choice that got picked apart line by line. Neither reaction was really about the code.

The amplifiers in a modern codebase are visible:

GitHub commit count, PR frequency, OSS contributions on the sidebar
Prior employer (ex-FAANG, maintainer of a known OSS project)
How carefully written the PR description is — people assume the writing style predicts the code

The mechanism

Hide the author from yourself for the first pass. Open the diff tab, not the PR summary. Form a technical read of the code before you look at who wrote it. The vigilance version — "I will try to be fair" — has been measured and does not work. The structural version — "I read the diff before I see the name" — does.

Pair it with a fixed checklist (security, performance, test coverage, readability). A checklist does not stop you from being charitable to the senior; it stops you from skipping items for the senior and inventing items for the new hire.

2. Authority bias: tech leads get LGTM on auto-pilot

Authority bias is subtle because it does not look like a bias. The tech lead's PR gets approved because, in most cases, the tech lead's PR is actually correct. The bias is the margin. It is the fraction of approvals where the reviewer thought "this looks off" and did not comment, because the reviewer assumed the lead had a reason.

Over a year that margin compounds. The architectural choices that quietly went unchallenged are the ones that survive into legacy debt.

How to use it as a reviewee

The flip side is a tool. When you want a proposal to land, borrow authority deliberately:

Cite the official docs or RFC for your design choice in the PR description
Reference industry patterns or a specific paper for an architectural change
Ask a trusted senior for a pre-review, then mention the feedback was already applied

The ceiling of this technique is low. Borrowed authority does not improve the code; it only lowers the friction of the review. Use it where you are confident the proposal is right and the friction is costing you time.

The mechanism against it

For reviewers: when the author is a senior, switch to question-form comments. "Is this intentional?" lowers the social cost of a reply from the author, so you get the clarification instead of eating the uncertainty. "This is wrong" from a junior to a staff engineer rarely gets sent. "Why this instead of X?" gets sent.

3. Bystander effect: reviews that never start

This is the Meta finding. When a review request fans out to a team instead of a person, the review starts later. Everyone assumes someone else will pick it up. Nobody does. The PR sits.

Meta's team at Facebook rebuilt reviewer recommendation specifically to kill this. One reviewer, named at request time, with the group as fallback. Review start time dropped.

The mechanism

Default to single-reviewer requests. Group requests are a fallback, not the primary mode.
When you must request a group, name a point person in the first comment. "@alice primary, @team as backup" removes the ambiguity even if the UI does not.
A review older than 24 hours gets a direct ping, not a reshare into the group. Resharing distributes the "someone else" button to more people.

4. Framing effect: same bug, different reaction

Dr. Michaela Greiler, who studied code review feedback at Microsoft and now consults on code review culture, notes that aggressive feedback tends to slow down fixes and drag down the quality of the author's next PR, even when the technical content of the comment is identical to a constructive version.

The same N+1 bug, two framings:

Negative frame: "This N+1 query will cause performance problems. Fix it."
Positive frame: "If you switch this to eager loading, the query count drops from N+1 to 2. On production traffic the difference should be noticeable."

The technical content is identical. The receiver's reaction is not. One sounds like a problem report. The other sounds like a tip. Michael Lynch, formerly of Google, has a longer set of guidelines for code review comments; three of them do most of the work:

Talk about the code, not the author. "This line can produce a NullPointerException" is reviewable. "You messed up here" is a feelings reply.
Attach a why. "Change this" without a reason triggers a defensive reply. "Change this because X returns null in the pagination edge case" triggers a fix.
Use questions where possible. "Could foo return null here?" opens the conversation. "This is a bug" closes it.

The mechanism

A review comment template with three fields — "what", "why", "suggestion" — forces the framing. The field with "suggestion" at the end makes the comment hard to leave hostile without actively choosing to.

5. Ownership bias: taking the review as criticism

The last one lives on the receiving side. If you wrote the code five minutes ago, your sense of ownership is at its peak, and every comment reads as a judgment of you, not of the diff. The defensive-reply reflex fires. You start arguing with suggestions that, read a day later, you would just accept.

I have the Slack history to prove it: suggestions I defended at 11pm, merged at 9am the next morning without changing a single argument against them. The code did not improve overnight. I did.

The mechanism

Submit the PR a few hours after you finish, or the next morning. Ownership fades with distance. Comments read differently after one night of sleep.
Reframe the artifact. The codebase is a team's shared product, not a portfolio of personal creations. The reframe sounds abstract and feels silly, until the first time you accept a suggestion that an hour ago you would have fought.
Reviewers: name one thing you liked in the PR before any comments. "The test naming is clean" is not politeness; it signals that the review is about the diff, not an attack on the author. Reviews that are 100% suggestions trip the ownership reflex.

Why mechanisms, not reminders

Across all five, the pattern is the same. Reminders to "be aware" of a bias do not survive the next sprint. Mechanisms do. Hiding the author on the first pass is a mechanism. A single-reviewer default is a mechanism. A comment template with a "suggestion" field is a mechanism. Submitting a PR the next morning is a mechanism.

Code review is a technical filter and a social situation at the same time, and the social half is the one that decides whether the technical half ever lands. If you want the full set — 20 biases that distort estimates, meetings, and hiring, each with a mechanism rather than a reminder — the book collects them in one place:

Cognitive Bias for Software Engineers

7 Steps to Ship a Knowledge Graph on Neo4j: Step 3 Ate Both My Weekends

Ken Imoto — Wed, 30 Sep 2026 13:00:00 +0000

Neo4j's official framework for building a knowledge graph is seven steps long. It reads like a well-behaved playbook: define the use case, pick your sources, design the ontology, model the data, ingest, query, operate. If you were reviewing it on a slide it would look completely reasonable.

Then you sit down on a Friday night to actually build one, and Step 3 quietly reaches out and eats the whole weekend.

Twice.

The first time I built a knowledge graph on Neo4j, the ontology-design step ate two full weekends before I could even start writing the first Cypher CREATE statement. Steps 1, 2, 4, 5, 6, and 7 combined took less time than Step 3 by itself. That is not because the other six steps are trivial. It is because Step 3 is where every unresolved assumption from Steps 1 and 2 comes home to be reconciled, and you cannot fake your way past it.

Below is the 7-step arc as Neo4j teaches it, with what actually happens between the diagrams.

The 7 steps, at a glance

Step	What Neo4j calls it	What actually happens
1	Define the use case	You pick one query you want fast
2	Identify data sources	You classify structured / semi-structured / unstructured
3	Design the ontology	You loop back to Step 1 three times
4	Model the data	You write Cypher and remember indexes
5	Ingest and transform	LOAD CSV, APOC, LLM extraction
6	Build queries and APIs	Cypher becomes your business logic
7	Operate and extend	Freshness, quality monitoring, schema evolution

Steps 1 and 2 look easy on paper. Step 3 is the interesting one. Steps 4-7 are where the real technology sits.

Step 1: define the use case (or watch the whole project rot)

The bad version of a use case:

"We want to put all our internal data into a knowledge graph."

The good version:

"A new engineer needs to know, in 30 seconds, which services depend on /api/users before they change it."

What separates the two versions is not scope but specificity. A specific query pins down three things that would otherwise stay loose for months: which entities you need as nodes, which relationships you need as edges, and which query pattern has to be fast.

The industry has a graveyard full of KG projects that started with "let us model everything." Neo4j's own guidance is blunt about this: if the use case is not concrete, the graph is decoration.

Step 2: identify data sources (structured, semi-structured, unstructured)

The classification is the whole step:

Structured — CSVs, relational tables, customer masters, product catalogs.
Semi-structured — JSON API responses, config files, OpenAPI specs.
Unstructured — PDFs, meeting notes, source code, chat logs.

The reason this classification matters is that it decides how you get into Step 5. Structured data walks into Neo4j through LOAD CSV. Semi-structured comes in via APOC. Unstructured needs an LLM extractor between the source and the graph.

In September 2026 the LLM-extractor side of this has become a solved product problem: Neo4j's own LLM Knowledge Graph Builder requires Neo4j 5.23+ and now supports OpenAI, Gemini, Claude, Llama 3, Amazon Nova, Diffbot, and Qwen as the extraction model. The neo4j-graphrag-python SimpleKGPipeline turns on structured-output extraction when the LLM supports it, which cuts the "the model returned invalid JSON" retries you would otherwise write yourself. If you are starting today, you build the extractor last and use these tools.

Step 3: ontology design (the one that ate the weekends)

This is where the trouble happens. On paper, an ontology looks like a tiny document:

Node labels:
  - Service (name, version, team)
  - API (path, method, status)
  - Developer (name, email, team)
  - Repository (name, url, language)

Edge types:
  - EXPOSES:    Service -> API
  - CALLS:      API -> API
  - MAINTAINS:  Developer -> Service
  - HOSTED_IN:  Service -> Repository

That is maybe 15 lines. How did that eat two weekends?

Because writing those 15 lines forces you to answer questions Step 1 quietly deferred. Is a deployed instance of a service a Service or a Deployment? Is a Repository really one node, or one per branch? Does a MAINTAINS edge point from the developer to the service or the other way around, and does the direction change what queries become expensive?

The moment you start writing Cypher against a draft ontology, you find edges that need properties, node labels that need to split in two, and relationships whose direction you had wrong. You go back to Step 1, reread the use case, and change Step 3. Then you go back to Step 3 and change it again. The rough pattern I ended up with: Step 3 makes you loop back to Step 1 about three times before it settles.

The lesson I wish I had absorbed earlier:

Do not design the ontology in isolation. Sketch it in the same file as three sample Cypher queries you actually plan to run. If any of the three cannot be written cleanly, the ontology is wrong, not the query.
Start small. Two node labels and one edge type is a legitimate first iteration. Expanding is easy; unpicking is a weekend.
Read edges as sentences. "Service EXPOSES API." "Developer MAINTAINS Service." If you cannot say the edge out loud without it sounding weird, the direction is off.

I would also, in retrospect, have used Neo4j's Graph Consolidation feature earlier. It rolls a sprawling schema into fewer, more meaningful node labels and relationship types automatically. On a first attempt this is exactly the kind of judgement I did not yet have.

Step 4: data modelling in Cypher

Once the ontology settles, Step 4 is mechanical:

CREATE (:Service {name: "UserAPI", version: "2.1", team: "Platform"});
CREATE (:Service {name: "AuthService", version: "1.5", team: "Security"});
CREATE (:API {path: "/api/users", method: "GET", status: "active"});

MATCH (s:Service {name: "UserAPI"}),
      (a:API     {path: "/api/users"})
CREATE (s)-[:EXPOSES {since: "2024-01-15"}]->(a);

CREATE INDEX FOR (s:Service) ON (s.name);
CREATE INDEX FOR (a:API)     ON (a.path);

The single most common mistake at this step: forgetting the indexes. A KG with millions of nodes and no index on the query's start-node property will take seconds per query, and you will spend a week thinking your ontology is wrong when the real problem is a CREATE INDEX line you never wrote.

Step 5: ingestion and transformation

For structured data, LOAD CSV WITH HEADERS is the whole story:

LOAD CSV WITH HEADERS FROM 'file:///services.csv' AS row
CREATE (:Service {
  name:    row.name,
  version: row.version,
  team:    row.team
});

For unstructured data, the modern move is to hand the LLM a schema-constrained extraction prompt rather than a free-form one. SimpleKGPipeline in neo4j-graphrag-python does this for you when the underlying model supports structured output; you get JSON that conforms to your ontology rather than JSON that hopes to. The retries you save at this step add up.

Step 6: queries and APIs

This is where the graph starts paying rent. The three query shapes I ended up using most:

Blast radius — what breaks if I change this API?

MATCH (target:API {path: "/api/users"})<-[:CALLS]-(caller:API)<-[:EXPOSES]-(s:Service)
RETURN s.name AS affected_service,
       caller.path AS via_api;

Shortest path between two services.

MATCH path = shortestPath(
  (a:Service {name: "UserAPI"})-[*]-(b:Service {name: "PaymentService"})
)
RETURN path;

Community detection — which services cluster together?

CALL gds.louvain.stream('service-graph')
YIELD nodeId, communityId
RETURN gds.util.asNode(nodeId).name AS service, communityId
ORDER BY communityId;

Cypher is where a knowledge graph stops being a nice diagram and starts being business logic. Those three shapes cover most of what you actually reach for once the graph is live.

Step 7: operate and extend

The step nobody wants to write about, because it does not screenshot well.

Freshness. The source data changes; your graph does not, unless you have a sync pipeline.
Quality. Orphan nodes and duplicate entities creep in. Schedule a job to find and merge them.
Schema evolution. New use cases mean new node labels and new edges. Version your ontology.
Access control. Not every team should see every subgraph.

Neo4j AuraDB shoulders most of the infrastructure side of this, which is why a lot of small teams end up on Aura even if they started on a self-hosted instance. The trade is real: less flexibility, less ops.

What I would do differently

If I were starting over today with the same use case:

Write the three killer queries first, before the ontology. The queries constrain the ontology far better than the ontology constrains the queries.
Cap the first ontology at two node types and one edge type. Expand only when a query forces the expansion.
Use SimpleKGPipeline (or the LLM Knowledge Graph Builder if you want a UI) instead of hand-writing extraction prompts. The structured-output layer alone saves days.
Do not skip the indexes. Ever.
Budget for Step 3 to take longer than Steps 1, 2, 4, 5 combined. If it does not, either your use case is unusually narrow or you are about to find out you missed something.

The 7-step arc is honest. It is the shape of the effort inside each step that the diagram does not capture. Once you know Step 3 is going to be the weekend eater, you can plan around it — you sketch queries alongside the ontology, you keep the first pass small, and you accept that "the graph is designed" is a claim you cannot make until you have written Cypher against it.

If you want the longer version, my Zenn book Knowledge Graph Practical Guide walks the full arc, and Chapter 4 is exactly this 7-step build — including the ontology-loop story that turned my weekends into a lesson.

References

Neo4j — LLM Knowledge Graph Builder (Neo4j Labs)
Neo4j — LLM knowledge graph builder — First release of 2025
Neo4j — User Guide: Knowledge Graph Builder (neo4j-graphrag-python)
Neo4j Labs — llm-graph-builder GitHub releases

Local LLM vs Claude Code: 96% of My Requests Failed Locally. Half the Steps Didn't.

Ken Imoto — Tue, 29 Sep 2026 19:53:48 +0000

The pitch for running Claude Code on a local model goes like this: point it at Ollama, stop paying per token, keep your code on your machine. I have an RTX 4070 with qwen3.5:4b on it, so I wanted that to be true.

Before swapping anything, I counted. I took 100 requests I had actually sent to Claude Code and asked, for each one, whether a local model could have done it end to end.

96 could not. So much for cancelling the subscription.

Then I counted a different way, and the answer flipped. This post is about the second count and the small config file that came out of it.

Counting whole requests: 96 of 100 go to the frontier

A request to Claude Code is usually one line. "Fix the failing test in the billing module." "Why is this deploy slow?" The request itself contains very little context. The work happens after that, when the agent reads files, runs commands, fetches docs, and decides what to try next.

So when you judge the request as a whole, you are judging the hardest part of it: the planning. That part needs the frontier model almost every time. In my set, 96 of the 100 had to go there.

If you route at the request level, a local model gets almost nothing. Swapping the whole agent onto a local model asks it to do exactly that planning.

Counting single steps: 97 of 200 run fine locally

The agent's work is a chain of steps: tool calls, fetches, summaries, commit messages. I broke Claude Code sessions into 200 of those steps and asked the same question for each one.

97 of 200 were fine on a local model.

The clearest case was WebFetch: fetch a page, pull out the part that answers a question. That is extraction, not planning. I tried it on 20 real fetches, and it was not perfect: some pages never loaded, and some answers came back partly wrong. That is why the example config sends a failed fetch to the frontier (on_fetch_error: frontier). But when the page arrived, the local model usually got the extraction right.

Claude Code already does a version of this. Its own WebFetch tool description says the fetched page is processed with a small, fast model. That makes the tool itself an example of choosing a model per step.

One rule beat a 9B router

My first router asked a local model for a probability: "should this step go local?" It worked, sort of. Then I wrote a one-line rule, "WebFetch goes local", and the rule beat the 9B model's probability.

That changed the design. Rules decide what rules can decide. When enabled, the probability judge only sees what is left. Here is the rules block from the companion repo's example config:

rules:                      # step rules, top to bottom, first match wins
  - match: {tool: WebFetch}
    route: local
  - match: {tool: [Agent, Task]}
    route: frontier
  - match: {kind: commit_message}
    route: frontier         # frontier until a probability judge is added

probability: null           # off until measured with the same judge model
default_route: frontier

The probability judge is off by default. The routing function is short:

def route(self, step: dict) -> Decision:
    for r in self.rules:
        if r.matches(step):
            return Decision(r.route, "rule")
    if self.probability and self.local_threshold is not None:
        p = self.probability(step)
        return Decision("local" if p >= self.local_threshold else "frontier", "probability", p)
    return Decision(self.default, "default")

In my config, sub-agents (Agent, Task) are pinned to the frontier.

Getting a probability out of Ollama

For the steps rules can't settle, you want a number, not a paragraph. The trick is to make the model answer with one letter (Y or N) and read that letter's probability from logprobs.

This is the request body I send to Ollama's native /api/chat:

{
  "model": "qwen3.5:4b",
  "stream": false,
  "think": false,
  "logprobs": true,
  "top_logprobs": 20,
  "options": {"temperature": 0, "num_predict": 1},
  "messages": [{"role": "user", "content": "... Answer with a single letter, Y or N."}]
}

Three things fail silently here, with no error:

The OpenAI-compatible endpoint drops the probabilities. Use the native API. The repo's scripts expect Ollama 0.12.11 or later.
Thinking models don't answer in the first token. The first token is the start of their reasoning, so the Y/N you are looking for isn't there. think: false and num_predict: 1 pin it to the answer.
A candidate outside the top 20 looks like probability 0. Also, " Y" and "Y" are different tokens.

The fix for the third one is to add up the probability that landed on your candidate letters and refuse to trust the judgment when that total is small:

raw = {lab: 0.0 for lab in labels}
for t in response["logprobs"][0]["top_logprobs"]:
    tok = t["token"].strip()  # " Y" and "Y" are the same answer
    if tok in raw:
        raw[tok] += math.exp(t["logprob"])
mass = sum(raw.values())      # below 0.5: the answer wasn't where we looked

0.5 is a line nobody chose

I used the same kind of judge as a gate in front of Claude Code: "does this request contain a secret?" It read a note with a production database password in it and said it was fine, with 69% confidence. The password was there with 100% confidence.

The ranking was fine. Secrets scored higher than non-secrets. The scale was off: real secrets were landing well below 0.5. Cut at 0.5, the gate missed 20 secrets. When I drew the line from my own labelled examples instead, it missed 2.

That is why the gate threshold in the example config looks strange:

judge:
  model: qwen3.5:4b
  threshold: 0.0071    # example only. lowest P among your confidential examples x 0.3
  at_or_above: block
on_timeout: block

0.0071 is only an example from my setup, so don't copy it. The repo has a small CLI (steprouter-threshold) that takes your labelled judgments and prints a line, AUROC, and 5-fold cross-validated misses and false alarms.

A regex pass (gitleaks plus personal-data patterns) runs before the model. It catches secrets that have a recognizable shape. The model is there for the ones that don't.

The free local worker cost the most

One more result surprised me. I set up a strong model to plan and a free local model to do the work. That setup was the most expensive one I tested. Free, the way a puppy is free.

The local model's tokens were free. The orchestrator re-reading everything the worker sent back was not. What comes back from a routed step is part of the routing, so the example config pins the return value to a small typed shape:

return:
  fields: [green, mypy, ruff, pytest_failed, files, summary]
  summary_max_lines: 2
  full_log: file

Pass/fail counts, the files touched, and one or two lines of what was done. The full log is written to a file instead of the reply.

Try it on your own logs

My numbers come from my work. To get yours:

Take a day of Claude Code sessions and list the steps, not the requests.
Mark which step types always go one way. Those become rules.
Leave the probability judge off until you have measured it on your own examples.
Before you add a local worker, decide what it is allowed to send back.

The companion code, including the gate, the logprobs judge, the threshold CLI, the router, and a made-up evaluation set, is on GitHub as local-step-router (MIT). Clone it, run pip install -e ., and start by editing examples/routing.yaml. The README has an English summary at the bottom.

The full measurement (550 of my own requests and steps on one RTX 4070, how big the judge model needs to be from 0.8B to 35B, why none of my six real secrets matched the evaluation set, and the final blueprint) is written up in Local LLM or Claude Code?. Chapters 2 and 3 cover the step count and the rules-first router; chapter 8 is the threshold worksheet.

200ms Humans vs 700ms Voice Agents: What ACL IWSDS 2025 Says About the Turn-Taking Gap

Ken Imoto — Wed, 23 Sep 2026 13:00:00 +0000

The complaint that keeps showing up in every voice-agent beta test I've watched is not about the wrong answer. It's about the timing of the right answer.

"The AI cut me off." "It answered while I was still thinking." "I said 'um' and it started talking." Latency looks like a stopwatch problem, but users experience it as a manners problem, which is a much harder thing to fix.

The 200ms vs 700ms gap the field is trying to close

The ACL IWSDS 2025 survey on turn-taking in spoken dialogue puts a concrete number on the mismatch. In natural human conversation, a listener starts their reply roughly 200ms after the speaker's turn ends. Current dialogue agents take 700 to 1000ms to do the same thing.

Three to five times slower. If human dialogue is a tennis rally, current voice AI is playing chess by post.

Closing that gap is not one problem, it's two. And they pull in opposite directions.

Too slow to reply: awkward silence, dropped rapport, users start repeating themselves.
Too fast to reply: the agent talks over the user, interrupts mid-word, kills trust.

Both fail modes come from the same root cause: silence is not the same as end-of-turn.

Why plain VAD is not enough

Most current systems still decide the user is done talking with a Voice Activity Detector. VAD is cheap and fast, and it answers exactly one question: is there voice in this audio frame, yes or no.

That question is not the one the agent actually needs answered.

The 0.3–0.5s pause a speaker takes mid-sentence looks identical to end-of-turn.
The silence after "uh" or "let me think" looks identical to end-of-turn.
A cough or a sigh looks like voice.

You can tune the silence threshold, but you cannot tune your way out of the fact that VAD does not know what a sentence is. VoiceInfra's production data lands on 300–500ms of trailing silence as the least-bad setting: shorter and the agent chops off natural pauses, longer and the delay becomes obvious. The comfortable range depends on the use case. Call-center flows tolerate 400–500ms because callers pause to think; command interfaces want 200–300ms because utterances are short; long-form narration wants 500–600ms because the pauses inside the story are longer.

The threshold is a compromise, not a fix. The real fix has to know what the words mean.

Semantic endpointing: Deepgram Flux and the "one model does both" bet

The interesting move in 2026 is folding transcription and turn detection into the same model. Deepgram's Flux, launched as their first ASR built for voice agents rather than for general transcription, does exactly that. The model outputs turn boundaries directly from audio, using the same joint architecture it uses to output text. Because the same weights see both the acoustic signal and the emerging transcript, the model has a shot at answering the harder question: "is this utterance semantically complete?"

Flux exposes three knobs that make the tradeoff explicit rather than hidden inside a silence threshold:

eot_threshold — how confident the model has to be before it commits to end-of-turn.
eager_eot_threshold — a lower bar for tentative end-of-turn, useful for starting inference speculatively.
eot_timeout_ms — the maximum silence duration before the model forces end-of-turn even if the confidence never crosses the threshold.

You can also switch modes per-turn with a Configure message, so a barge-friendly interaction ("agent, wait") and a monologue turn ("read me the terms") do not have to share a threshold.

The pattern is a hint at where the field is going. Turn-taking will stop being a pipeline stage after ASR and start being a joint output of the ASR itself.

Krisp's 6M-parameter turn model: same idea, edge-shaped

Not everyone can afford a cloud round-trip on every silence check. Krisp ships two small audio-only models trained to handle the acoustic cues a naive VAD collapses — a 9M-parameter turn prediction model (~30MB) that scores end-of-turn from prosody and pausing patterns, and a 6M-parameter interruption model (~24MB) that separates real barge-ins from backchannels like "yeah" and "mhm". Together they cover the same failure modes VAD misclassifies:

intentional speech vs a thinking pause
filler words (um, uh, well) mid-sentence
backchannels vs interruptions

At that size it runs on-device in real time, which matters for privacy-sensitive deployments and for wearables where the round-trip to the cloud is itself the biggest source of latency.

The Krisp and Flux paths look opposite. One shrinks the model, the other gives the ASR the extra job. But they are attacking the same VAD failure mode from opposite sides.

Graceful abort: what to do when you're wrong

No detector, semantic or acoustic, is going to be right every time. The interesting question is what your pipeline does when it decides "the user is done" and turns out to be wrong.

Twilio's graceful abort pattern is the cleanest version I've seen written up. When STT signals end-of-turn early, the LLM starts generating a response. If fresh audio arrives before the response reaches the speaker, the pipeline kills the generation in flight and swallows the partial output. The window for this is the few hundred milliseconds it takes STT → LLM → TTS to produce audible speech. If you can revoke the guess inside that window, the user never hears it.

The economics matter here: you are paying for LLM tokens on turns that get thrown away. In exchange, you get to be aggressive on end-of-turn detection without punishing the user when you're aggressive-and-wrong. For most conversational products that tradeoff is worth it.

Barge-in: hearing the user through your own voice

The other half of turn-taking is the reverse case: the user interrupts while the agent is still speaking. Detecting that seems trivial until you realize the AI's own audio is leaking into the microphone, and any naive VAD will happily flag that leak as a barge-in and abort the agent's own utterance.

Sparkco's write-up on duplex barge-in handling is the clearest description of the fix I've read. Three moving parts:

Full-duplex audio — keep the mic hot the entire time the agent is speaking, never gate it on TTS output.
Echo cancellation — subtract the speaker output (as a reference signal) from the mic input, so what remains is the user's voice minus the agent's.
Nuisance rejection — filter environmental noise and short transients that a bare VAD would misclassify as speech onsets.

[agent is speaking]
  speaker out ──► reference signal ─┐
                                    ▼
  mic in ─────► echo canceller ────► user voice only ──► VAD / turn model

Without echo cancellation you get the failure mode users describe as "the AI got startled by its own voice and stopped talking." The dog scared of its own reflection.

The bot-feel paradox and the 200-300ms delay trick

Now the counterintuitive move. Once you've closed the acoustic and semantic gaps and the agent can reply in 300ms, it turns out that replying in 300ms feels worse. The response arrives before the user has finished processing their own sentence, and it registers as robotic rather than sharp.

The fix is to inject 200–300ms of intentional delay before the agent starts speaking, while continuing to run the LLM in the background. You get the "thinking for a moment" cue humans read as attention, without paying real latency for it. The tokens are already streaming; you're just holding the TTS start.

It's a UX trick, not a technical one. But it's the piece of the puzzle that gets forgotten when a team spends a quarter shaving milliseconds off ASR and then wonders why user ratings didn't move.

What actually closes the 500ms gap

There is no single component that takes you from 700ms to 200ms. You get there by stacking:

Layer	What it buys you
Semantic endpointing (Flux-class)	Stops chopping mid-sentence, stops waiting for silence that already means end-of-turn.
Small on-device turn + interruption models (Krisp-class)	Score end-of-turn and separate backchannels from real barge-ins without a round-trip.
Graceful abort	Lets you be aggressive on early end-of-turn without punishing the user when wrong.
Duplex barge-in with AEC	Lets you leave the mic hot without the agent interrupting itself.
200–300ms intentional delay	Buys back the "thinking" cue that pure speed removes.

The move that unlocks the stack is admitting that end-of-turn is a language problem, not a silence problem. Once you accept that, VAD-only pipelines look the way pre-BERT NLP pipelines look now: perfectly reasonable for their era, obviously incomplete in retrospect.

The gap to 200ms is not going to close by tuning thresholds. It closes when the model that hears the audio is the same model that understands the sentence.

The full 300ms-UX playbook (latency budgets by pipeline stage, when to pick Whisper vs Deepgram vs Piper, and how to design the wait states so users forgive the last 200ms you can't remove) is written up in Voice AI 300ms UX. Chapter 9 covers turn-taking end-to-end; chapters 4 and 5 cover the latency anatomy that decides whether closing the gap is even possible for your stack.

References

ACL IWSDS 2025. "A Survey of Recent Advances on Turn-taking Modeling in Spoken Dialogue."
Krisp AI. "Audio-only, 6M weights Turn-Taking model for Voice AI Agents." 2025.
Twilio. "Core Latency in AI Voice Agents." 2025.
Deepgram. "Flux: End-of-Turn Detection Parameters." 2026.
Sparkco. "Optimizing Voice Agent Barge-in Detection." 2025.
VoiceInfra. "Voice AI Prompt Engineering: Complete Technical Guide." 2025.