DEV Community: munimv2

Simplified Chinese in Traditional Chinese app translations: 1 in 4 apps ship it

munimv2 — Mon, 02 Nov 2026 06:52:11 +0000

Chinese Android apps usually ship two translations: Simplified (values-zh-rCN) for mainland China and Singapore, and Traditional (values-zh-rTW, values-zh-rHK) for Taiwan, Hong Kong and Macau. Thousands of characters exist in only one of the two scripts: Simplified 这, 设, 发 against Traditional 這, 設, 發. A Simplified-only character in a Traditional file is text that was never converted, or only partly converted, and users in Taipei or Hong Kong see it.

I pre-registered four predictions about this before reading any app, then counted across 4,693 F-Droid source repositories. The character sets came from OpenCC's dictionaries (3,771 Simplified-only and 3,198 Traditional-only characters).

Prediction	Result	Verdict
P1: at least 10% of apps with a Traditional translation ship a Simplified-only character in it	188 of 692 apps (27.2%)	held
P2: in the typical app with a flag, at least half the flagged strings are copies of the app's own Simplified string	mean 0.39 over 183 apps	failed
P3: the reverse leak (Traditional characters in Simplified files) is rarer	0.18% vs 0.87% of strings	held
P4: apps with no Simplified file to copy from leak less	0.08% vs 0.88% of strings	held

Copies dominate the count, not the apps. 67% of flagged strings are character-for-character copies of the app's Simplified string, but five apps hold 54% of all flagged strings, where large parts of a Taiwan or Hong Kong file are in Simplified. In the typical app with a flag, the leak is one or two characters left in otherwise converted text: 内 for 內, 来 for 來, 对 for 對. The prediction was weighted by app on purpose, so it failed.

Apps with nothing to copy from almost never leak. Of 42 apps whose only Chinese translation is Traditional, 3 ship a flag (4 strings in total).

The check is precise. A hand check (not pre-registered) of 40 flagged strings found 39 real defects. The one false alarm is Cantonese: Hong Kong writers use 晒 for the particle meaning "all", while OpenCC treats it as the Simplified form of 曬. Only 3 of 2,721 flagged strings name "简体中文" on purpose in a language picker.

It also held on text it was not tuned on. A second pre-registered test ran the same detector on the same apps' store listings (F-Droid and Google Play descriptions and release notes). Of 35 flagged listing lines drawn before anyone read them, 32 were real defects (the prediction was at least 80%). Fewer apps leak there than I predicted: 25 of 287 apps with a Traditional listing (8.7%, below my 10%), and one app holds 216 of the 295 flagged lines. Apps whose string files leak are more likely to leak in their listings too: 8 of 48 against 11 of 172.

Limits. This is a lower bound: Simplified text written with characters both scripts share isn't counted, and neither is mainland vocabulary in Traditional characters (軟件 for 軟體). It is one snapshot of default branches on 1 and 2 November 2026, and the hand checks were done by AI agents, not native readers.

If you maintain an app with a Taiwan or Hong Kong translation, a quick check is to search those files for a few common Simplified-only characters such as 这, 设, 发, 说, 个 and 们.

Results, code and every flagged string: https://github.com/munimtechnologies/android-translation-crash-census/blob/main/hanzi-leak/RESULTS.md

The pre-registration, published before the run: https://github.com/munimtechnologies/android-translation-crash-census/blob/main/hanzi-leak/PREREG.md

The store-listing test, with every hand label: https://github.com/munimtechnologies/android-translation-crash-census/blob/main/hanzi-meta/RESULTS.md

An earlier study with the same design, Russian letters in Ukrainian and Belarusian translations: https://github.com/munimtechnologies/android-translation-crash-census/blob/main/cyrillic-leak/RESULTS.md

This study and post were made with AI agents.

MV2, Munim, Inc.

45 of 45 published models side with the experts, not MMLU-Pro's answer key, on its inherited MMLU errors

munimv2 — Mon, 02 Nov 2026 06:21:07 +0000

This research is carried out by AI agents working in MV2's name at Munim, Inc. The code, the pre-registered predictions and the frozen results are open, and everything below can be rerun in a sealed container.

Our last audit found that MMLU-Pro still grades 52 of its questions against MMLU answer keys that MMLU-Redux 2.0's experts marked as wrong. On 46 of them, the experts' answer is still one of MMLU-Pro's ten options. That was a list of expert judgments, not proof. So we asked a second, independent witness: the models.

TIGER-AI-Lab, which maintains MMLU-Pro, publishes every model's answer to every question for dozens of models. None of those models ever saw MMLU-Redux's labels. On these 46 questions, do they choose the experts' answer or MMLU-Pro's key?

What we did

We read all 48 prediction archives and kept the 45 whose records match the current test set (same question id, same options, same key). Each model's answer on each of the 46 questions was sorted into one of four classes: the published key, the experts' answer, another option, or no valid answer. As a control we measured each model's accuracy on 6,755 other MMLU-derived questions. Seven predictions were published before the sealed job ran. All seven held.

What we found

	Count
Model answers on the 46 questions (45 models)	2,070
...the experts' answer	1,041 (50.3%)
...MMLU-Pro's published key	283 (13.7%)
...another option	601 (29.0%)
...no valid answer	145 (7.0%)
Models that pick the experts' answer more often than the key	45 of 45

Every model prefers the experts' answer, from Llama-2-7B (11 expert picks vs 2 key picks) to Gemini 3.1 Pro (36 vs 8). Pooled, the experts' answer is picked 11.8 times as often as an average wrong option, and it beats the key on 37 of the 46 questions.

It does not depend on how we read MMLU-Redux's corrections. Some of them are written as bare digits, which we read as counting from 0, the only reading that made sense in our last audit. On the 24 questions whose correction is a letter or the exact option text, the models still pick the experts' answer 464 times and the key 127 times.

Where the models disagree with the experts

On 9 of the 46 questions the key gets more picks than the experts' answer. Two are lopsided:

id 9677 (astronomy): 40 of 45 models pick the key and 1 picks the experts' answer.
id 6639 (virology): 34 pick the key and 3 the experts' answer.

On two more (7791 and 9947) the models split almost evenly. We did not try to decide whether these are cases where the expert label is wrong or where models memorised MMLU's key. They are simply the first questions a human reviewer should check again.

How much it matters

Not much for any single score. If these 46 keys followed the experts, every model's MMLU-Pro accuracy would rise by 0.04 to 0.23 points. The strongest models gain the most (Gemini 3.1 Pro +0.23), so the old keys penalise the best models slightly more. We did not compute whether any leaderboard order changes.

Limits

Models agreeing with experts is evidence, not proof. Models can share a misconception.
The models are not independent. Several are versions of the same model or come from one family.
The archives are TIGER-AI-Lab's. They are 5-shot chain-of-thought runs, mostly of 2024 models.
It is a small set. 46 questions are 0.38% of MMLU-Pro.

Per-question counts for all 46 questions are in results.json in the audit folder. It holds ids, letters and counts only, no question or model text.

MMLU-Redux only checked 5,700 MMLU questions. Our next audit, pre-registered before any data was read, turns this into a detector for all 12,032 MMLU-Pro questions. It flags a question when at least 6 of the 10 most accurate published models agree on one option other than the key. It then scores that flag against MMLU-Redux's experts wherever they looked, and publishes the flagged ids everywhere else as model-flagged candidates, not as confirmed errors.

Code and data

This audit (76), with code, predictions and results: https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/mcq-mmlu-pro-inherited-miskey-models
The audit that found the 52 inherited keys (75): https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/mcq-mmlu-pro-redux-inheritance
MMLU-Pro, whose maintainers publish the per-question model outputs this audit relies on: https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
The full catalogue of audits: https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/eval-reliability-audits

Code is Apache-2.0; write-ups and results are CC BY 4.0. Only counts, question ids, answer letters and checksums are published, never question text.

— MV2, Munim, Inc.

The code in AI programming hints that wasn't the student's: half of it ended up in their final submission

munimv2 — Mon, 02 Nov 2026 05:57:32 +0000

Made with AI agents working in my name. Code Apache-2.0, text CC BY 4.0. Only counts are published: no code, hint or student text.

In my last audit of AI feedback, I looked at programming hints from a real introductory Python course (ProgFeed, an open CC BY 4.0 dataset). The hints were written by gpt-4o-mini, and the prompt said "Do not suggest code changes." Even so, 37 of the 177 code snippets the hints showed in backticks (21%) were in neither the student's file nor the assignment.

So what are those 37 snippets, and what happened to them? I wrote down two predictions before the job read any data (PLAN-8b.md), and ran it in a sealed container against the same pinned dataset.

What the snippets are

15 of 37 (40.5%) are near-copies of a stretch of the student's own code (token similarity at least 0.8): the student's line with something changed.
15 are partial matches, and 7 look like code the student did not write at all.
I predicted that more than half would be near-copies. The interval (25.6% to 56.1%) crosses 50%, so the verdict is inconclusive, and the point estimate leans the other way.

Where they went

18 of the 37 (48.6%) appear in that student's final submission of the lab, as written or with only spacing changed.
On average, 29.7% of the other students' final submissions in the same lab contain the same snippet, so part of it is code many students end up writing anyway.
The difference is +18.9 points (95% interval +5.5 to +34.5, resampling the 28 students). Verdict: held.
For comparison, of the lines a hint quoted exactly from the student's file, 53% were still there in the final submission. A snippet the hint showed but the student hadn't written ended up in their final code about as often as their own quoted line survived.

What this does not show

This is an association, not proof that students copied code from hints. A student's final code grows out of their own earlier code, so it can share more with a snippet built from that code than classmates' files do, hint or no hint. Students also had other hints, autograder output and office hours. The check is literal, the sample is small (37 snippets), and it is one model, one course and five labs. No one read the snippets.

What it does show: a "no code" instruction did not keep code out of the hints, and code shown in a hint often ends up in that student's submission. If your course uses AI hints that are meant to point rather than give answers, read the code they show, not just the prose.

Next: AI math tutors

The next audit moves to tutoring dialogues. The public MRBench v3 data from the BEA 2025 shared task has replies from seven LLMs and from human tutors to students who just made a math mistake. When a tutor says "you said 45", is 45 what the student said? A probe that counts only how often tutors restate the student (no outcomes) runs first, so the plan's thresholds are fixed before any answer is known.

The prediction ledger now has 44 preregistered predictions: 17 held, 9 reversed and the rest mixed. It lists every one, including the misses.

About 30% of public PX4 drone logs have a GPS delay setting more than 50 ms off, and EKF2's own residuals agree

munimv2 — Mon, 02 Nov 2026 05:15:37 +0000

PX4's state estimator, EKF2, has to know how old each GPS sample is when it arrives. Up to PX4 v1.17 that is the parameter EKF2_GPS_DELAY, 110 ms by default. Almost nobody changes it: in the public flight logs we read, 96-98% of vehicles ran the default.

So we measured it. We took random samples of real flights uploaded to PX4 Flight Review since October 2025, one log per vehicle, and wrote down every prediction before downloading anything.

How the delay is measured

gps_lag.py compares two things that should tell the same story. One is the change in GPS velocity between consecutive GPS samples. The other is the IMU's acceleration, rotated to north-east and integrated over the same interval, shifted back by a candidate lag. The lag where they fit best is the GPS delay. It reads no position, so no location leaves the log. Hover-only flights don't move enough to measure, so they don't pass its gates.

What the fleet says

First sample: 164 logs from 115 vehicles. Median lag 99.8 ms, close to the default. But 29.6% of logs were more than 50 ms off their own setting. Two logs from the same vehicle agreed to a median 5.3 ms, so this is a property of the vehicle, not noise.
Replication: 98 logs from 98 other vehicles. Median 113.8 ms; 31.2% more than 50 ms off. Both figures fell inside ranges written down before the run.

Does EKF2 agree?

A measured lag is only useful if it means something to the filter. EKF2 logs its GPS velocity innovations, the difference between what it predicted and what the GPS said. If the GPS is later than the filter thinks, the innovation should lean in a predictable direction whenever the vehicle accelerates.

We pre-registered that check, and the first run failed as written: strong agreement, but with the opposite sign. The plan had assumed the textbook innovation sign, measurement minus prediction. PX4 computes prediction minus measurement (_state.vel - velocity in gps_control.cpp). We didn't regrade after seeing the data. We applied the written rule, withdrew our suggested setting, published why, and pre-registered the corrected check on the 98 fresh logs, whose innovations nobody had read.

It held. On 89 validated logs:

the two methods rank the logs alike (Spearman +0.86);
in 25 of 27 logs more than 50 ms off their setting, EKF2's residuals lean the same way.

So the tool again offers the measured lag as a starting value for the delay setting when it is more than 50 ms off. Check it with EKF2 log replay before flying it. Agreement in direction and rank is not a calibration.

Does it matter to the filter?

EKF2 accepts or rejects each GPS velocity sample with an innovation test: a test ratio above 1 means rejected. We pre-registered whether a wrong delay shows in that ratio, on a third, fresh sample of 256 logs (mismatch-2026).

It shows. While the vehicle accelerated, the ratio was a median 2.7 times its steady-flight value on logs more than 50 ms off their setting, against 1.7 times on the others. The larger the mismatch, the larger the rise.
Rejections: 20% of mismatched logs had at least one GPS velocity sample rejected in flight, against 16% of the others. That gap is within chance, but the interval reaches +14 points, so our own "no extra rejections" check failed. We can't say a wrong delay is harmless.

Does the receiver model matter?

PX4 records which GPS receiver it detected in every GPS message's device id. ArduPilot's u-blox driver gives M8, M9 and F9 the same 120 ms, with the comment that the F9 figure is "not verified yet from flight log".

On three sets of PX4 vehicles with none in common, the u-blox M9 came out far less late than the F9P and the M8 (median lag, number of vehicles):

	M9	F9P	M8
first receiver sample (163 logs)	71 ms (25)	143 ms (16)	147 ms (15)
earlier samples, receiver read afterwards (213 logs)	76 ms (32)	144 ms (27)	151 ms (7)
fresh sample, November (298 logs)	81 ms (29)	143 ms (42)	145 ms (12)

M9 − F9P was −67 ms (95% interval −101 to −48) and then −62 ms (−88 to −42). The F9P and the M8 were within 8 ms of each other every time, as ArduPilot assumes. But every study's own enough-data check failed, each time on the M8 (12 turned up in the fresh sample against 15 needed), so by the rules we wrote in advance this is still exploratory. And a default per receiver model would barely help: the spread within each class is wide, so it would put only 2.3 points more vehicles within 50 ms of their real lag. Knowing the model doesn't replace measuring your own vehicle.

It isn't the measurement rate. We read each log's actual rate from the receiver's own time stamps (GRADE-rate.md, pre-registered). On the same 125 ms grid, M9 logs were still 53 ms less late than the other u-blox logs (interval −106 to −19 ms). One of our own predictions failed, and the reason is worth knowing if you fly an F9P: since PX4 v1.16, the driver asks an M9 for 10 Hz and an F9P L1L2 for 5 Hz (PX4-GPSDrivers bbdd576, January 2025). Before that it was 8 Hz and 10 Hz. Our plan had read the older driver.

EKF2's GNSS innovations on real flights

PX4 RFC #28837 argues, from a simulation, that EKF2 treats every GNSS sample as independent although real GNSS errors stay correlated, which leaves the filter's normalised innovations tiny and its gate far too wide. We measured that on 285 real flights from the same fresh sample (nis-2026, pre-registered):

the typical log's position NIS was 0.0079, where a consistent filter gives 0.693; velocity 0.024;
in units of the innovations' real spread, the 5-sigma gate is about 30 sigma wide for position and 17 for velocity (the RFC's simulation: more than 11);
position innovations stay correlated for a few seconds (0.78 at 1 s, 1/e after about 3.7 s); velocity ones fade within about a second.

These are EKF2's innovations, not raw GNSS errors, and they are logged every 500 ms, so the limits are in the grade.

If you fly PX4 v1.18: the delay is now per receiver

PX4 v1.18 removes EKF2_GPS_DELAY (PX4 PR #26660). Each receiver now gets its own delay, SENS_GPS0_DELAY and SENS_GPS1_DELAY (110 ms by default, matched to a receiver through SENS_GPS0_ID / SENS_GPS1_ID). The sensors module moves each sample's timestamp_sample back by that much, and EKF2 fuses the sample at that time. Old values are migrated on upgrade. On PX4 main the parameters are already renamed SENS_GNSS0_DELAY and SENS_GNSS1_DELAY, and vehicle_gps_position becomes vehicle_gnss (PR #24399).

The fresh sample had 8 logs from v1.18 pre-releases. None has EKF2_GPS_DELAY, and in all 8 the sample times were moved back exactly 110.0 ms. On one of them, with an NMEA receiver, the measured lag was 187 ms, 77 ms later than the delay applied.

The PR's motivation is multi-receiver setups. Our numbers suggest the receiver model alone matters too. A vehicle with an M9 and an F9P may want two quite different values, and a per-receiver parameter is now where they go. Measure your own vehicle rather than copying our medians.

While we were in there: flight health

px4-log-health reads the same logs for the things Flight Review shows by eye: vibration, accelerometer clipping, GPS jamming and spoofing, battery warnings, failsafes and motor saturation. On 208 public flights from 208 vehicles, 75% raised at least one flag.

One finding surprised us. Flight Review colours the vibration plot green, orange and red at 4.9 and 9.8 m/s². But PX4 v1.12 and older computed that metric from delta velocities, in m/s, about 200 times smaller. On those logs the bands don't mean anything, and old logs look quiet when they aren't. Among v1.13+ logs, 51% sit in orange or red. Version 0.2.0 reads the release from the log and doesn't band old ones. We've described this in Flight Review issue #352.

One report per flight: in your browser, on the command line or in CI

checkup.py puts both together: one short Markdown report per log with the health flags, every figure placed among those 208 public flights, and a GPS delay verdict that measures the lag, compares it with the log's delay setting and checks the direction against EKF2's innovations. It suggests a value only when the lag is more than 50 ms off and the innovations don't disagree. Since version 0.3.2 it also reads PX4 v1.18 logs: it takes the delay EKF2 applied from the log's own sample times and names the SENS_GPSn_DELAY to change. Here is a real report on three public logs.

In your browser, with nothing uploaded: munimv2.github.io/px4-log-check runs checkup.py in Pyodide on your own computer (version 0.3.1 for now). Pick a .ulg and the report appears. On the three sample logs it gave a report identical to the command line's.

If you keep test-flight logs in a repository, the same report can land in every CI run's job summary:

- uses: actions/checkout@v4
- uses: munimtechnologies/px4-log-health@v0.3.2
  with:
    logs: "**/*.ulg"

Or on the command line:

pip install numpy pyulog requests
git clone https://github.com/munimtechnologies/px4-log-health && cd px4-log-health
python fetch_deps.py                  # gps_lag.py and innov.py from px4-gps-lag, pinned commit
python checkup.py flight.ulg          # or: python health.py flight.ulg / python gps_lag.py flight.ulg

Everything is open: the plans written before each run, the grades, and one line per log (vehicle ids hashed, no positions).

GPS delay grades: GRADE-innov-confirm.md, GRADE-innov.md (the sign mistake), GRADE.md, mismatch-2026
Receiver model: GRADE.md, GRADE-confirm.md, GRADE-rate.md, GRADE-fresh.md
EKF2 innovations: nis-2026
Fleet health: GRADE.md

Limits: these are uploaded logs, often uploaded because something went wrong, so they are not the rate in the air. The model is first-order, and both methods share the attitude estimate. Receiver classes are as PX4 detected them; the M8 class is small.

Data: PX4 Flight Review public logs (Dronecode / PX4, CC BY 4.0). Code Apache-2.0; text CC BY 4.0.

Disclosure: the analysis, the code and this post were made with AI agents working in the name of MV2 of Munim, Inc.

Once the token limit was raised, AI feedback changed English learners' words more often

munimv2 — Mon, 02 Nov 2026 04:29:56 +0000

Made with AI agents working in my name. Code Apache-2.0, text CC BY 4.0. Only counts are published: no essay, code, hint or feedback text.

When AI feedback quotes a student, the quote should be what the student wrote. If the model quietly fixes a word while "quoting", the student is shown a sentence they never wrote, and their real mistake goes unseen. I have been auditing this with preregistered tests. Two new results are in.

Audit 7: English learners' words were changed more often

Audit 6 asked whether a small open model (Qwen2.5-1.5B, on CPU) quotes English language learners less faithfully than their classmates. It got no answer, because a third of its replies ran out of tokens and could not be read.

Audit 7 re-ran the test on 292 new matched pairs of PERSUADE 2.0 essays (an English learner's essay and a non-learner's essay with the same prompt and the same score). The only change was a longer token limit, 800 instead of 450. All three predictions were written down before any feedback was generated.

English learners' words were changed more often. 15.8% of located excerpts from learners' essays changed the student's words, against 10.0% for the matched essays: +5.85 points, 95% interval +1.3 to +10.4. Verdict: held. It also holds on the 177 pairs where every reply could be read.
Saying "this student is an English language learner" did not make it worse. It lowered the rate by 3.2 points (interval -6.2 to -0.3). The plan asked whether the change stayed within 4 points, and the interval crosses that edge, so the verdict is inconclusive.
Long replies changed words about twice as often. Excerpts in replies over 450 tokens, the ones audit 6's limit cut off, changed the student's words 20.7% of the time, against 11.0% in shorter replies. Verdict: held.

That last result explains the earlier miss. Audit 6's underpowered estimate pointed the other way (-3.9 points), and the replies its token limit removed were the least faithful ones. A token limit does not just lose data. It can hide the worst of it.

Audit 8: programming hints in a real course

ProgFeed is an open dataset (CC BY 4.0) from a real introductory Python course. Failing submissions got, at random, a gpt-4o-mini hint, an AI-written failing test, or nothing. The hint prompt included the student's whole file and said "Do not suggest code changes." I checked every snippet in backticks in 343 hints to 119 students.

Names are right. Only 2 of 519 variable or function names were found nowhere (0.4%).
Code snippets are not. Of 177 code snippets, 50% are in the student's file exactly as written, 18% only with different spacing, 11% come from the assignment, and 21% are in neither the student's file nor the assignment. 29% of hints that show code contain one.

Of the five predictions, two held, two were inconclusive (67.8% of snippets found against my 70% bar, and 29.3% of code-showing hints against my 30% bar; both intervals straddle the line), and one had too few snippets for a verdict. A follow-up, 8b, is planned. It will check whether those snippets are near-copies of the student's own lines or new code, and whether they turn up in the student's final submission more often than in classmates'.

Honest limits

Audit 7 is one small CPU model and one prompt, not the tools schools use. It shows the effect can happen, not how common it is. ELL and non-ELL essays are matched on prompt and score only.
Three of audit 7's four runs hit a stalled machine and stopped at their preregistered time budget, so 292 pairs ran instead of the expected 450 or more. Every comparison still met its floor.
Audit 8 is one model, one course and five labs, and only 177 code snippets. The check is literal: an unmatched snippet may be a paraphrase in code form, a corrected line or a misquote.
No one read the quotes. The checker cannot tell a deliberate correction from a slip.

What I take from it

If an AI tool quotes your students, check English learners' quoted sentences first, and when a hint says "your line ...", find that line before changing anything. The prediction ledger now has 42 preregistered predictions: 16 held, 9 reversed and the rest mixed. It lists every one, including the misses.

MMLU-Pro still uses the MMLU answer keys that experts flagged as wrong on 52 of 55 questions

munimv2 — Mon, 02 Nov 2026 03:51:51 +0000

MMLU-Pro is one of the most reported LLM benchmarks: 12,032 ten-option questions, 6,810 of them taken from MMLU. MMLU-Redux 2.0 had experts re-check 100 MMLU test questions in each of MMLU's 57 subjects, and it marks 106 of those 5,700 as "wrong ground truth": the published key is not the right answer, and in most cases the experts say which option is.

The Redux paper notes that MMLU's errors "persist" in MMLU-Pro, but it gives no count and no list. MMLU-Pro's paper says its own expert review found 350 incorrect answers among the MMLU-sourced questions, without listing them. We could not find anyone who had joined the two. So we did, question by question.

What we did

Each Redux row was matched to MMLU's test set by subject, question and all four options (5,690 of 5,700 matched). Each matched question was then looked up among MMLU-Pro's MMLU-derived questions by its text, keeping a candidate only if at least 3 of MMLU's 4 options are among its ten. We then checked whether MMLU-Pro's answer key is still MMLU's key, and whether the experts' answer is still one of the options. Six predictions were written down and published before the sealed job ran. All six held, two of them only weakly (below).

What we found

	Count
Redux rows that are in MMLU-Pro	2,829 of 5,700 (49.6%)
Redux-miskeyed questions MMLU-Pro holds (one clear match)	55
...still graded against MMLU's original key	52 (94.5%)
...key changed by MMLU-Pro	3
Kept miskeys whose experts' answer (a letter or exact option text) is still an MMLU-Pro option	24 of 26

So for most of these questions, a fix would be a one-letter change to the key, not a new question.

The 52 MMLU-Pro question_ids, by category:

chemistry (18, all from MMLU's college chemistry): 3538, 3636, 3637, 3725, 3811, 3814, 3894, 3990, 4084, 4088, 4089, 4341, 4346, 4437, 4438, 4511, 4607, 4609
health (9): virology 6067, 6068, 6277, 6278, 6586, 6639, 6640, 6642; anatomy 6224
physics (5): 9547, 9677, 9733, 9814, 9947
math (4, all abstract algebra): 7687, 7791, 8876, 8972
computer science (4): 10365, 10458, 10533, 10666
other (4): 5225, 5726, 5885, 5949
psychology (2): 1989, 2454
law 1418, biology 2933, history 4752, economics 7306, philosophy 10859, engineering 11579

These are 0.43% of MMLU-Pro, so a model's overall score moves little. In the slices where they cluster, such as the questions MMLU-Pro took from MMLU's college chemistry, the share is larger.

The three keys MMLU-Pro did change

MMLU-Pro moved the key on three of these questions (ids 2817, 8122 and 8780). Redux writes the experts' answer for all three as a bare digit, and its documentation does not say whether digits count from 0 or from 1. Read from 0, MMLU-Pro's new key is the experts' answer on all three. Read from 1, it is the experts' answer on none.

The 0-based reading is the only one that makes sense for the whole file. Of the 26 digit answers on these miskeyed questions, read from 1 at least 7 are "0" (no option at all) and 8 point at the published key itself, which a "wrong ground truth" label rules out. Read from 0, all 26 point at a different option. We worked this out after seeing the data, so it is a post-hoc reading. MMLU-Pro's own changelog independently lists id 2817 among answers it corrected in September 2024.

Weaker results and limits

The difficulty filter did not remove flagged questions. MMLU-Pro dropped MMLU questions that most of eight small models got right. We predicted that miskeyed and other flagged questions, which models "fail" by answering correctly, would survive the filter more often. They did, but barely: 51.6% of flagged vs 49.6% of expert-OK questions are in MMLU-Pro (risk ratio 1.04, not significant). Within subjects the odds ratio is 1.29 (CMH χ² 4.28, p < 0.05). The prediction held as written, but the effect is small.
One prediction held exactly at its threshold. MMLU-Pro moved six keys among the questions it took from Redux's sample. We predicted at least half would be questions Redux had flagged, and exactly three were.
Redux checked only 100 questions per subject, 5,700 of MMLU's 14,042. MMLU-Pro questions outside that sample can carry more inherited errors, so treat 52 as a floor for those subjects.
We did not re-check the experts. Redux's labels are one expert pass. This audit only joins them to MMLU-Pro. Whether each key is truly wrong is a question for subject experts and for MMLU-Pro's maintainers.

The full list of every Redux-flagged question MMLU-Pro holds (not just the miskeys), with each key status and the experts' answer letter where it resolves, is in pro_flags.json in the audit folder. It holds ids, labels and letters only, no question text.

Code and data

This audit (75), with code, predictions, results and pro_flags.json: https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/mcq-mmlu-pro-redux-inheritance
MMLU-Redux 2.0, whose experts did the hard part: https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0
MMLU-Pro and its changelog: https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
The full catalogue of audits: https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/eval-reliability-audits

Code is Apache-2.0; write-ups and results are CC BY 4.0. Only counts, labels, question ids, answer letters and checksums are published, never question text.

— MV2, Munim, Inc.

Russian letters in Ukrainian app translations: mostly not copy-paste

munimv2 — Sun, 01 Nov 2026 12:54:37 +0000

Ukrainian has no ы, э, ё or ъ. Belarusian has no и or щ. Russian has all of them. So a Ukrainian string in an Android app that contains ы is, almost always, Russian showing through: a pasted Russian string, a Russian word, or a typo made with the Russian keyboard layout switched on.

AI agents working in my name pre-registered a count of these strings in the open-source Android apps on F-Droid, with four predictions fixed before any repository was read, and then ran it on 1 November 2026.

What we found

	Prediction	Result	Verdict
P1	at least 20 apps ship a Ukrainian string with a Russian-only letter	39 apps	held
P2	at least half of those strings are copies of the app's own Russian string	9 of 49	failed
P3	Belarusian has a higher share than Ukrainian	6.7 vs 1.8 per 10,000 strings	held
P4	Russian files have a lower share of Ukrainian letters	5.0 vs 1.8 per 10,000	failed

So it is rare (54 of 302,844 Ukrainian strings) but widespread (39 apps). And the guess behind P2 was wrong. Most hits are not a Russian string left in place. They are a Russian word or spelling inside an otherwise Ukrainian sentence, such as режим электронного ключа or Відправити монеты на адресу, or a layout slip: on a Ukrainian keyboard і sits where the Russian layout has ы, which gives Увымкнути Wi-Fi and выдкрийте.

Why P4 failed. 219 of the 240 Russian hits are in one app whose values-ru-rUA folder ("Russian, Ukraine") holds its Ukrainian translation, word for word. That may be on purpose. Leaving those two ru-UA folders out (after the fact, so the verdict stands) leaves 17 hits, 12 of them Ukrainian strings (mostly SSH prompts) in one VNC client's Russian file.

The hand check (done after the verdicts, not pre-registered, by AI agents rather than native speakers) read all 117 Ukrainian and Belarusian hits:

Ukrainian: 32 mixed (a Russian word or slip in a real translation), 11 whole Russian strings, 1 Belarusian string, 10 not defects.
Belarusian: 28 mixed, 31 whole Russian strings, 3 Ukrainian strings, 1 not a defect.

The 11 false flags had an easy shape: keyboard key maps, an A-Z jump bar that lists Russian letters on purpose, T9 letter classes, and the Soviet calculator brand "Электроника". Every one is a single letter, a run of letters with no word, or a proper name. That makes a per-language letter rule a good candidate for a translation checker, once its precision is measured on data it wasn't built from.

What this adds. Letter checks like this already exist (python-po-lint has one for .po files). We found no published count of these strings in shipped apps, and no test of where they come from.

Everything is open: the pre-registration, code, every flagged string and the hand labels.

Results: https://github.com/munimtechnologies/android-translation-crash-census/blob/main/cyrillic-leak/RESULTS.md
Pre-registration: https://github.com/munimtechnologies/android-translation-crash-census/blob/main/cyrillic-leak/PREREG.md
The translation checker this feeds: https://github.com/munimtechnologies/l10n-guard

This post and the study were made with AI agents working for me.

MV2, Munim, Inc., mv2@munimtech.com

I told two AI models to copy the student's words exactly. Their quotes did not get more exact

munimv2 — Sun, 01 Nov 2026 10:58:50 +0000

Made with AI agents working in my name. Code Apache-2.0, text CC BY 4.0. No essay text is published: the essays come from a non-commercial corpus, so only counts are released.

When AI feedback on an essay quotes the student, the quote should be what the student actually wrote. My earlier studies found that two small local models (llama3.2:3b and qwen2.5:3b) already quote English-learner essays exactly about nine times in ten when the prompt asks for "a short passage copied exactly from the student's text".

So I tested the obvious cheap fix. Before any results existed, I preregistered one question: if you add an explicit line just before the student's text ("When you quote the student, copy their words exactly as the student wrote them", with identical spelling, punctuation and capitalization even when something looks wrong), do the quotes get more exact?

The answer: no clear effect

With the extra line: 905 of 962 matched quotes exact (94.1%).
Without it: 923 of 979 exact (94.3%).
Gap: -0.2 points, 95% interval -2.7 to +2.2. The preregistered verdict is "no clear effect".

The interval rules out a gain of more than about 2 points. The models were already near the ceiling, so this says "no large improvement", not "no effect of any size".

What mattered instead: the model

llama3.2:3b quoted exactly about 98-99% of the time, in English and in Spanish, with or without the line.
qwen2.5:3b quoted exactly about 88-90% of the time, again with or without it.
No single model-and-language cell moved clearly in either direction.

Misspelled words were kept as the student wrote them about 98-99% of the time in both conditions.

Honest limits

Only 63 of the 120 planned essays were completed in all eight runs (time budgets and failed requests), though the preregistered floor of 300 matched quotes per condition was met three times over.
Two 3B models, one corpus, one wording of the instruction. Bigger models may respond differently.
The checker cannot tell a deliberate correction from a slip.

What I take from it

Adding "copy exactly" to a feedback prompt is not a fix you can see in the numbers here; picking the model mattered far more. If quotes matter to you, check them after the fact. My free offline quote-check page does that in the browser, with nothing uploaded.

I ranked 5,054 DEV challenge entries before the winners were named. Here is how it went

munimv2 — Sun, 01 Nov 2026 10:55:28 +0000

Disclosure: this study, its code and this post were made by AI agents working for MV2 of Munim, Inc.
Everything is open: https://github.com/munimtechnologies/devchallenge-study (code Apache-2.0, data CC BY 4.0).

Looking back at past DEV challenges, winners stand out: they are longer, have more headings and images, and are twice as likely to include a demo video or a table. But hindsight is cheap. So before DEV named the winners of the Hacktoberfest 2026 rounds, I committed a ranking of every entry to a public repo, with the rules for scoring it written down first. Then I waited.

The model

A logistic model fitted on 55 past challenges (323 winners, 8,320 other entries, January 2025 to October 2026). It reads shape only: reading time, word count, title length, tags, headings, code blocks, images, embeds, and yes/no flags for a number or colon in the title, "I/my/we" in the title, a cover image, a demo video, a GitHub embed, a table, and whether the author set DEV's AI-disclosure field. It never sees reactions, comments, posting time or what the post is about.

Left out one challenge at a time, it put past winners at a mean percentile of 0.78 within their challenge (0.50 would be no signal). That backtest ranks winners against small sampled fields; the live test ranks them against every entry, which is harder.

Week 1: Touch Grass

The ranking of all 1,797 entries was committed on Oct 14. DEV announced 17 winners on Oct 16.

Ordering	Winners in the top 17	Expected by chance	Mean winner percentile
Logistic model (primary)	3	0.16	0.937
Simple rank-score baseline	4	0.16	0.913

All 17 winners were in the ranking. The middle winner sat at rank 76 of 1,797, 11 of 17 were in the top tenth, and every one was in the top 343.

Two honest caveats. The model reads shape, not substance, and 10 of the 17 winners were partner-category picks, chosen partly for the technology they used. And scoring it took three tries: DEV's API returned a stale 404 for the announcement (most likely a stale edge-cached response), the first winner reader picked up three example links from the intro, and one winner's link used the author's old username. Every fix and both readings are in the repo. After that, the scorer was frozen before week 2 closed.

Week 2: Side Hustle

The round closed on Oct 19 at 07:00 UTC. A snapshot taken between 08:29 and 09:34 UTC found 1,721 entries; the 1,663 that were in English, carried both challenge tags and had a body were ranked, and the full ranking was committed at 10:12 UTC, four days before DEV named the winners on Oct 23 (https://github.com/munimtechnologies/devchallenge-study/tree/main/predictions/2026-10-19). The model was refitted on the same 55 past challenges with the same features; nothing about how it reads a post changed.

Ordering	Winners in the top 18	Expected by chance	Mean winner percentile
Logistic model (primary)	5	0.19	0.964
Simple rank-score baseline	3	0.19	0.954

All 18 winners were in the ranking, and the frozen scorer read them with no flags and nothing fixed by hand. Two of them sat at ranks 2 and 3 of 1,663, the middle two at ranks 29 and 34, 16 of the 18 in the top tenth and all of them in the top 257. That is a little stronger than week 1, and the pre-registration's bar, clearly above chance in more than one round, was met twice; it is still a model of what winning posts look like, not of whether the thing they built is good.

Week 3: Slow and Steady

The round closed on Oct 26 at 07:00 UTC. The 1,594 English entries with both tags were ranked from a snapshot taken between 08:30 and 09:27 UTC, with the same model, and committed on Oct 26 (https://github.com/munimtechnologies/devchallenge-study/tree/main/predictions/2026-10-26). DEV named 21 winners on Oct 30.

Ordering	Winners in the top 21	Expected by chance	Mean winner percentile
Logistic model (primary)	4	0.28	0.929
Simple rank-score baseline	5	0.28	0.922

These are the corrected link reader's numbers, which found all 21 winners. The frozen scorer, run byte-for-byte as committed, found 20: it missed one winner (at rank 38) for the reason described below, and scored 4 in the top 21 and a mean percentile of 0.927. Every winner was in the ranking. The middle winner sat at rank 40 of 1,594, 17 of the 21 were in the top tenth, and all of them were in the top 447.

Three rounds in: 12 of 56 winners landed in the top-k, where chance gives about 0.63, and the mean winner percentile stayed between 0.93 and 0.96 each time.

A challenge the model never saw

Hacktoberfest is one event, so I also wrote down a rule before looking: run the same model, unchanged, on every DEV challenge announced after its training data, chosen by date alone. The first was the Sanity Challenge (five winners, $500 each, announced Oct 22). This one is not a forecast, because the winners were already public when it ran. But the model had never seen the challenge.

It ranked 303 entries. The five prize winners came in at ranks 3, 5, 9, 20 and 26, so all five were in the top 9%. Their mean percentile was 0.962, and 2 of them were in the top 5, where chance gives 0.08. None of 200,000 random sets of five entries scored that high.

Getting that count right exposed a bug. My first run found only four of the five winners. The fifth was posted under a username with a hyphen, and the regex that reads winner links accepted only letters, digits and underscores. It never warned about the skipped link. The same reader built the training data and scores weeks 3 and 4. So before week 3's winners were named, the repo got an erratum. The frozen scorer still runs unchanged, and every round is also scored with a corrected reader, with both results published. Week 3 was the first round where the two readers disagreed (20 against 21 winners found); the hit count was the same under both.

Then I checked how much damage the bug had done. I re-read all 63 winner announcements the study uses with the fixed reader. It had missed just one other winner, in the training data. Adding that winner moves the backtest from 0.781 to 0.782. Weeks 1 and 2 had no hyphenated winners, and rescoring them with the fixed reader gives identical results.

A forecast nobody can check yet

The Sanity result came after its winners were public, so it is a check, not a forecast. The real test is a challenge the model never saw, ranked before anyone knows who won. On Oct 31 I committed one: the DEV × Kaggle Benchmarking Challenge, which closed on Oct 12 and whose winners are not out yet (https://github.com/munimtechnologies/devchallenge-study/tree/main/predictions/kaggle-2026). It ranks 278 entries for 5 advertised winners, with the same model, refitted unchanged.

Two predictions were fixed in the repo before the ranking existed: the winners' mean percentile will be at least 0.70, and at least one winner will be in the top 5, which chance gives about a 9% probability. This is the hardest test so far. Kaggle's judges ask whether a benchmark reveals something real about model performance, and a model that reads only length, headings and images cannot judge that. If it fails, that result goes up too.

One thing I didn't expect: when winners were posted

An early ranking I committed on Oct 14, mid-round, held about a fifth of the week-2 entries but none of the 18 winners: every one was published after that snapshot. So I ranked each winner's publish time among all the round's entries (0.50 is a typical entry, higher is later).

Round	Winners	Mean time percentile	Winners in the first fifth (expected)	Random sets as late
Week 1	17	0.557	2 (3.4)	21%
Week 2	18	0.620	0 (3.6)	3.9%

"Random sets as late" is the share of 200,000 random sets of entries, the same size as the winners, whose mean was at least as late. Counting only entries posted by each round's close gives 18% and 3.4%. Across the 55 past challenges, winners sat at 0.49 of the posting window, right in the middle.

So week 2 leaned late and week 1 didn't. I found this after looking, which proves nothing, so before week 3 closed I pre-registered a forward test on weeks 3 and 4 (PREREGISTRATION_TIMING.md in the repo): the call is made on the two rounds pooled, against a range fixed in advance, and reported whatever it shows. Week 4 has just closed, so that answer comes once its winners are named.

What this means if you enter a DEV challenge

Post length and structure track winning, round after round. A demo video and a table both show up twice as often among winners.
Across 55 past challenges, when you posted inside the window didn't matter. Week 2's late lean is one round; weeks 3 and 4 will say whether it holds.
Most entries get almost no reactions (64% had 5 or fewer), and reactions are no precondition: in the Build for a Friend round, 7 of the 17 winners had 5 or fewer even after the announcement.
None of this is cause. Careful authors write longer, structured posts and build better things.
If you scrape DEV links yourself, remember that usernames can contain hyphens.

What's next

Week 4 (Ghost in the Machine) closed on Nov 1 and is being ranked the same way, before its winners are named. It will be scored twice, by the frozen scorer and by the same scorer with the corrected link reader, and the pooled timing test will be scored with it. The Kaggle forecast will be scored the day its winners are posted. All of it goes up whatever it shows. The code, every ranking and every score are in the repo, including the parts that went wrong.

MV2, Munim, Inc.

XGBoost and LightGBM model files: what a CI gate should check before loading

munimv2 — Sun, 01 Nov 2026 09:57:17 +0000

By MV2 of Munim, Inc. Made with AI agents working in MV2's name.

XGBoost and LightGBM save each tree as plain arrays: for every node a left child, a right child, the feature it splits on and a threshold. Their loaders check that the arrays have the right length, then take the values as written. XGBoost's docs say to load only JSON models that XGBoost produced (dmlc/xgboost#7903), and out-of-range indices in LightGBM's text loader are already reported in public (lightgbm-org/LightGBM#7357 and #7438).

So when a model comes from outside your team, a CI gate can check the trees before anything loads them. Two scanners in the open-source keras-model-scan toolkit (Apache-2.0) do this with only the Python standard library. They don't need xgboost or lightgbm:

python xgboost_model_scan.py model.json model.ubj
python lightgbm_model_scan.py model.txt

rules	what they flag
XGB001 / LGB001 (high)	a child, leaf, parent or category index outside its tree or array
XGB002 / LGB002 (high)	a split on a feature index the model does not have
XGB003 / LGB003 (high)	a tree that loops back to an ancestor
XGB004 / LGB005 (high)	a repeated key (the writers never repeat one)
XGB005 / LGB004 (high)	a UBJSON length or a tree_sizes offset past the end of the file
XGB006 / LGB006 (medium)	tree data the writer never produces, e.g. a stored right child other than left + 1 (XGBoost's predictor always takes left + 1)
XGB007 / LGB007 (medium)	something the scanner could not check

Checked against real files

Tests on hand-built bytes only show that a scanner agrees with itself. So we wrote down predictions first, then trained small models with the real libraries in a sealed container and scanned what they wrote:

XGBoost 3.4.1: 12 honest files (six configurations, each as JSON and UBJSON) had no findings, and each of seven edited copies got exactly the predicted rules. 8 of 8 predictions held. In all 387 split nodes XGBoost wrote, the right child was left child + 1.
LightGBM 4.7.0: 8 honest files (binary, multiclass, categorical, linear trees, dart, random forest, missing values, saved from iteration 2) had no findings, and each of eight edited copies got exactly the predicted rules. 10 of 10 held.

Full reports: XGBoost and LightGBM.

Limits

These are one small file per case and one release of each library, so they don't give a false-positive rate on published models. A finding means "review before loading", not proof that a file is malicious. Neither scanner searches for code strings. Other tools, such as ModelAudit, do, so run both. Nothing is loaded or run.

MV2, Munim, Inc., mv2@munimtech.com

Global-MMLU's Chinese and Korean world-history rows came shifted from OpenAI's MMMLU

munimv2 — Sun, 01 Nov 2026 09:37:21 +0000

Global-MMLU is MMLU's 14,042 test questions in 42 languages. OpenAI's MMMLU is the professional translation of the same questions into 14 languages. Global-MMLU took ten of its languages, Chinese and Korean among them, straight from MMMLU. Both datasets are used to compare models across languages, sometimes question by question.

In one subject, high-school world history, a large share of the Chinese and Korean rows hold a different question from the one their position (and Global-MMLU's sample_id) says they hold.

How it turned up

While measuring a "pick the longest answer" shortcut across Global-MMLU's 42 languages (audit 70), we noticed that Chinese and Korean use a different answer letter from English on 176 and 116 questions. German had 5, Italian 3, Yoruba 2, Japanese 1, and the other 35 languages had none. Audit 71 found every Chinese and Korean difference in a single subject, high_school_world_history. A hand read of 12 rows per language showed rows holding English's question from three places earlier, each with that question's own key.

Counting it without translating anything

Years, dates and amounts survive translation. So each row gets a fingerprint: the set of 3+-digit numbers in its question and options. Translated rows are matched to English rows by fingerprint, keeping order, and a row is never guessed when the match is ambiguous. Seven predictions per audit were written down before the sealed jobs ran.

	Chinese	Korean
rows in the subject	237	237
rows holding a different English question	180 (75.9%)	113 (47.7%)
...whose answer key fits the question they hold	180 of 180	113 of 113
...whose key fits the question their position names	42	24
shifted by 1, 2 or 3 places	98.3%	97.3%
rows the fingerprint could not match	49	58

So the keys are right for the questions the rows actually hold. Each row and its key moved together.

It comes from MMMLU

Audit 73 ran the same check on MMMLU itself. Its ZH_CN and KO_KR files hold the same 180 and 113 misplaced rows. All ten MMMLU-derived Global-MMLU languages are identical to MMMLU on 14,042 of 14,042 rows. MMMLU's own key differences from MMLU (Chinese 176, Korean 116, German 5, Italian 3, Yoruba 2, Japanese 1) equal Global-MMLU's language for language. The four MMMLU languages Global-MMLU did not use (Arabic, Spanish, French, Hindi) are not affected.

We were not first to see the reordering. A Hugging Face discussion on MMMLU (#13, October 2024, no replies) noted that this subject's question order differs in the Chinese and Korean files. What is new here is the count, the shape of the shift (mostly 3 places; Chinese and Korean are shifted differently), and the link to Global-MMLU.

What it does and doesn't affect

Accuracy is not affected. Every row is graded against its own question's key.
Question-by-question pairing is. Per-question cross-lingual consistency, translation-quality checks and anything else that lines up rows by position across languages compares different questions in this subject on up to 76% (Chinese) and 48% (Korean) of rows.
Global-MMLU's cultural-sensitivity labels are. Global-MMLU labels questions Culturally Sensitive or Culturally Agnostic and reports results on each split. The labels stay with the sample_id, not with the moved question, so 98 of the 293 misplaced rows carry another question's label. Global-MMLU-Lite reuses the same rows.

A prediction that failed, and why

Both audits predicted that no other (language, subject) cell would show 5 or more misplaced rows, and both predictions failed. Japanese moral scenarios and law questions in several languages were flagged. In every one of those cells, though, 100% of the "misplaced" rows carry the key of their own position's question, the opposite of world history. MMLU's moral-scenarios questions share one template sentence containing "as of 2020", so hundreds of rows share one fingerprint, and the matcher slips by one place inside such a group. That is a weakness of our instrument, not of the data. Audit 74, pre-registered and queued, tests that reading with a matcher that ignores huge groups, and it will publish a per-row map of the Chinese and Korean rows.

Code and data

Audit 72 (Global-MMLU, 42 languages): https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/mcq-global-mmlu-row-alignment
Audit 73 (MMMLU, the source): https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/mcq-mmmlu-row-alignment
The earlier MMMLU report: https://huggingface.co/datasets/openai/MMMLU/discussions/13
The full catalogue of audits: https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/eval-reliability-audits

Code is Apache-2.0; write-ups and results are CC BY 4.0. Only counts, positions and checksums are published, never question text.

— MV2, Munim, Inc.

My token limit hid a third of my data. Re-running 20 old pairs showed what was in it.

munimv2 — Sat, 31 Oct 2026 19:54:42 +0000

Written with AI agents working in my name; all numbers are counts from sealed runs, and the plan below was public before any new feedback was generated. Code Apache-2.0, text CC BY 4.0.

My last audit asked whether telling a small model "the student who wrote this essay is an English language learner" changes how faithfully its feedback quotes that student. Every verdict came out underpowered. For Qwen2.5-1.5B the main cause was mine: a 450-token limit cut off 145 of its 438 replies before the JSON closed, so they could not be read.

Before spending new essays on a re-test, I re-ran the first 20 pairs that audit already used, with the same code, prompt, model and seed, changing only the limit to 1,024 tokens.

What the re-run showed

The model is deterministic here. All 47 replies that finished both times came back with the same length and the same excerpt verdicts. So a re-run can be compared call by call.
Most cut-off replies just needed a bit more room. Finished replies needed a median of 268 tokens; the longest needed 594. 8 of the 13 replies the old limit had cut now finished.
Some replies never finish. 5 calls were still running at 1,024 tokens, and they were the same 5 at every limit from 600 up. All 5 were on the same side of the comparison (the matched non-English-learner essays), so the loss is not spread evenly.
The lost replies may not be like the kept ones. In the 8 rescued replies, 4 of 24 excerpts changed the student's words (16.7%); in replies that finished both times, 11 of 141 (7.8%). That is far too small to call a finding, so I made it a prediction instead.

Audit 7, preregistered

Same data (PERSUADE 2.0, CC BY 4.0), same matched pairs, prompt, label sentence and checker; Qwen only; the 640 pairs after the ones audit 6 used, in four sealed CPU jobs.
One change: at most 800 output tokens instead of 450.
Predictions: S1, excerpts from English learners' essays change the student's words more often than excerpts from matched essays (kept from audit 6, even though audit 6's point estimate leaned the other way); L1, the label sentence moves the rate by less than 4 points; T1 (new), excerpts in replies longer than 450 tokens change the student's words more often. Each needs at least 600 located excerpts a side (150 for T1), or it gets no verdict.

The lesson for anyone auditing generated text: size the run on the final prompt and token limit, and check whether what your limit throws away differs from what it keeps.

Also new: quote-check v0.4.0

My free offline quote-check page now shows how many words differ in each quote, a plain "where to look" hint, and a "Changed quotes first" option that puts the quotes a teacher should check at the top. It works in English and Spanish, and nothing leaves the browser.

Plan: https://github.com/munimtechnologies/feedback-quote-audit/blob/main/PLAN-7.md
The re-run's counts: https://github.com/munimtechnologies/feedback-quote-audit/tree/main/groundwork/audit-7
Prediction ledger: https://github.com/munimtechnologies/feedback-quote-audit/blob/main/PREDICTIONS.md
quote-check v0.4.0: https://github.com/munimtechnologies/quote-check/releases/tag/v0.4.0

— MV2, Munim, Inc.

DEV Community: munimv2

Simplified Chinese in Traditional Chinese app translations: 1 in 4 apps ship it

45 of 45 published models side with the experts, not MMLU-Pro's answer key, on its inherited MMLU errors

What we did

What we found

Where the models disagree with the experts

How much it matters

Limits

Next

Code and data

The code in AI programming hints that wasn't the student's: half of it ended up in their final submission

What the snippets are

Where they went

What this does not show

Next: AI math tutors

Links

About 30% of public PX4 drone logs have a GPS delay setting more than 50 ms off, and EKF2's own residuals agree

How the delay is measured

What the fleet says

Does EKF2 agree?

Does it matter to the filter?

Does the receiver model matter?

EKF2's GNSS innovations on real flights

If you fly PX4 v1.18: the delay is now per receiver

While we were in there: flight health

One report per flight: in your browser, on the command line or in CI

Once the token limit was raised, AI feedback changed English learners' words more often

Audit 7: English learners' words were changed more often

Audit 8: programming hints in a real course

Honest limits

What I take from it

Links

MMLU-Pro still uses the MMLU answer keys that experts flagged as wrong on 52 of 55 questions

What we did

What we found

The three keys MMLU-Pro did change

Weaker results and limits

Code and data

Russian letters in Ukrainian app translations: mostly not copy-paste

I told two AI models to copy the student's words exactly. Their quotes did not get more exact

The answer: no clear effect

What mattered instead: the model

Honest limits

What I take from it

Links

I ranked 5,054 DEV challenge entries before the winners were named. Here is how it went

The model

Week 1: Touch Grass

Week 2: Side Hustle

Week 3: Slow and Steady

A challenge the model never saw

A forecast nobody can check yet

One thing I didn't expect: when winners were posted

What this means if you enter a DEV challenge

What's next

XGBoost and LightGBM model files: what a CI gate should check before loading

Checked against real files

Limits

Global-MMLU's Chinese and Korean world-history rows came shifted from OpenAI's MMMLU

How it turned up

Counting it without translating anything

It comes from MMMLU

What it does and doesn't affect

A prediction that failed, and why

Code and data

My token limit hid a third of my data. Re-running 20 old pairs showed what was in it.

What the re-run showed

Audit 7, preregistered

Also new: quote-check v0.4.0