My Linter Scored 26% on Its Home Corpus and 0% on Docs — the Calibration Rule I Ship With Now

I ship small validators to my own coding agent. Nothing fancy: each one takes a path, reads config or tests or license files, and returns findings with a rule id, a severity, and a one-line fix.

The catalogue is now 30 packages with 271 rules. Four of them, re-run this morning:

225 assertions, all green. That number means almost nothing. A green selftest proves the code does what I wrote down. It says nothing about what happens when the thing runs against code I did not write.

This post is about the part I got wrong for weeks: calibration.

The experiment that killed my confidence

My cron validator parses schedules out of five different formats: crontab, GitHub Actions schedule, systemd .service/.timer, and a JSON job file. Before I built the real parser, my first draft was the obvious one:

a line whose first five whitespace-separated tokens are mostly numeric-ish is a cron expression.

I re-implemented exactly that, as a ~15-line probe, and pointed it at the 17-file corpus I use for the validator's own fixtures.

files_scanned      17
naive_hits         97
parses_as_schedule 25
false_positives    72
precision          0.258

74% of hits were wrong. Then I classified the 72 false positives, expecting the classic "English prose matched the pattern" story. It never showed up:

false positives by cause:
  schedule string split at the wrong boundary (quotes / JSON / YAML)   72
  exotic schedule dialect I don't support                               0

Every single one was a string-boundary error. "expr": "15 8 * * — the naive splitter cut the value mid-string, then happily parsed the fragment 15 8 * * + *", as a schedule. The rule did not have a dialect problem. It had a lexing problem, and I had been about to fix it in the wrong place.

So I ran the same rule somewhere it had no business being: 182 markdown docs from a knowledge base.

md_files           182
naive_hits         328
parses_as_schedule 0
precision          0.000

328 hits. Zero real schedules. The rule is not "26% accurate with room to improve" — on any corpus that isn't the one it was tuned on, it is pure noise.

Rule I ship with now: precision belongs to a corpus, not to a rule

The mistake was reporting precision as a property of the rule. It is a property of the rule and the corpus you measured it on. My naive rule's real spec is "26% on cron fixtures, 0% on prose," and only the second number predicts what a user will see.

So the shipping gate stopped being "selftests pass." It became three checks, and a package that fails any of them does not get published:

1. Real corpus, HIGH = 0.
Not fixtures — files I did not author. If my scanner's top severity fires on code that a maintainer would call fine, the scanner is wrong, full stop. Here's this morning's reading on the 17-file cron corpus (63 schedule entries across crontab, GHA, systemd, JSON):

扫描: 17 文件 / 63 条目
      (crontab=27, gha_schedule=3, json_job=23, systemd_service=6, systemd_timer=4)
问题: HIGH=0 MEDIUM=28 LOW=19

HIGH=0 is the gate. The 28 MEDIUMs are acceptable only because check #2 passes:

2. Every MEDIUM collapses into one explainable rule class.
MEDIUMs are threshold judgements, and thresholds are where noise lives. If MEDIUMs smear across six rule ids, the thresholds are wrong even though nothing is "failing." Here's the histogram from the same run:

CRN005  fires >=48x/day with no mutual exclusion      11   MEDIUM
CRN006  `cd dir; cmd` — semicolon, not &&               10   MEDIUM
CRN007  "* * * * *" / "*/1" almost always a typo         4   MEDIUM
CRN010  schedule still runs, command is commented out    3   MEDIUM
CRN013  no output redirection                       17   LOW
CRN015  timer without Persistent=true                2   LOW

11 + 10 + 4 + 3 = 28. Four rule classes, and they share one theme: the line says one thing and does another. It fires too often to finish, it runs in the wrong directory, its frequency is a typo, its command is commented out. The 19 LOWs are a second theme — failure that nobody sees (no redirect, no catch-up after a missed window).

That is a calibrated scanner: it says one thing per severity band, consistently, and a user can decide in one read whether that thing applies to them.

3. Zero input gets its own exit code.
I had a real bug where a review tool, handed a path that did not exist, printed files scanned: 0 / health: 100/100 / exit 0. Green light for never having looked. Now: no input is NO_INPUT, exit code 3, and it is not a pass.

"No input" is not "clean input." If your gate cannot fail, it is not a gate.

The cron validator says it plainly on an empty path:

verdict=NO_INPUT
结论: 这些路径里没有任何可识别的调度条目。"没输入"不等于"输入干净"，退出码 3。

Three more false positives I had to buy back

These all landed as HIGH on real code before I fixed them, and all three are the same species of error: the scanner mistaking "I don't recognize this" for "you did this wrong."

Unknown licenses are unknown, not violations. Scanning ~1.9k dependency entries, valid SPDX identifiers I simply had not put in my table — OFL-1.1, BlueOak-1.0.0 — came back as license violations. Fix: unknown IDs pass silently and increment a separate unknown_id counter. A counter that goes up is a signal to add data; a HIGH is a signal that someone's build is broken. They must not share a channel.

GPLv3 §13 mentions "Affero," so GPLv3 was flagged as AGPL. Two different licenses with very different obligations. Fix: classify on the first heading line only, not on a substring window over the whole text. Corollary: the standard Apache/GPL appendices contain <year> and [yyyy] placeholders — stop scanning past the first ~1200 characters or every second repo is "an unfilled template."

all.json, pdf.json and res.json are not language packs. My i18n validator identified locale files by "filename looks like a language tag," so every generic JSON file in a locales/-adjacent folder qualified. On a real repository that produced 50 fake HIGH findings. Fix: validate the code itself against the ISO 639-1 list (184 codes) plus 639-2, and only then decide the file is a language pack. Filename shape is a hint; the standard is the oracle.

The same disease, non-code edition

I once shipped a "shape gate": reject a post if it is too short or too templated. On a 600-post corpus it fired 7 times. Hand-auditing those 7, one was a legitimate post whose title carried the content, five were tests, one was genuinely malformed.

Precision: 1/7 ≈ 14%. Ninety percent of the tax landed on cold-start users who had done nothing wrong, and the only thing the gate reliably caught was people testing the gate.

Gate on recomputability, not on shape. "Can a second party reproduce this claim from the sources provided" is checkable and cheap to argue about. "This is shorter than 40 characters" is a proxy that mostly measures who arrived five minutes ago.

Checklist for anyone building validators that an agent will run

1. Gate on a corpus you didn't author. Fixture precision is not a measurement; it is a rehearsal.
2. HIGH = 0 on real code, or don't ship. If it cries wolf on r/real, the fix is the rule, never the user.
3. One MEDIUM class, max. Threshold noise that scatters across rules is unactionable even when every individual finding is defensible.
4. Unknown ≠ invalid. Separate counters for "I don't recognize this" and "this is wrong." Never the same severity.
5. No input gets its own exit code. A validator that cannot fail is decoration.
6. Ship selftest in the package, and make the gate require selftest to exit non-zero and require a "work was actually done" assertion (files_scanned > 0), not just a green exit.

Restated in one line: a rule's precision is a property of the corpus you measured it on, and the corpus that matters is the one belonging to someone who will uninstall you if you're wrong.

The probes and the fuller write-ups (clock calibration, partial-batch safety, publish-path fallbacks, dead-gate audits) live in github.com/Felixwang007/awesome-content-tools — tools/ has the runnable versions with selftest subcommands.

The packaged versions of the validators themselves (cron reliability, license compliance, i18n, tool schemas, and ~26 more) are on xiaping.coze.com under my handle, free to run.

If you've shipped a linter that turned out to be 0% accurate off its home turf, I'd like to hear what class of false positive you had to buy back — that taxonomy is the useful artifact, not the rule count.