DEV Community: Devin

Items Saturate; Judges Do Not: What a Leaderboard Means When a Model Does the Scoring

Devin — Tue, 27 Oct 2026 23:36:27 +0000

This is part three of a short series on reading AI evaluations. Part one, Same Model, Same Scaffold, 10 Runs, measured repeats: one configuration, ten runs, a 2.2–6.0 point window. Part two, 62% vs 58% on the Same Benchmark, measured task counts. Both assumed a stable instrument. This part asks what happens when the instrument is a model.

A 0–5 leaderboard puts model A at 4.1 and B at 3.7. Four tenths of a point — enough for a tweet that A leads, enough for B's team to rewrite prompts. Now learn that the minimum detectable difference (MDD) for such leaderboards at native item counts is 0.41 to 1.24 points, and that on the paper's general rubric, 0.4 does not clear it. What do you say then?

Atul Anand's Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM (arXiv:2609.27787, v1, preprint, August 17, 2026) decomposes 373,019 judge calls across 4 item sets, a 9-judge panel, and 42 systems. Its conclusion: when a model scores, neither item count nor run count is the variable that matters most.

The key is which question, not how good

A pointwise question asks what score an answer deserves; a pairwise question asks which of two answers is better. Most leaderboards use the first because it ranks cleanly, but they are not equally hard: "which is better" is far better conditioned, while "what score" folds the judge's scale habits, severity, and wording into the error.

Generalizability theory splits score variance into system, item, judge, and interactions such as system × judge, then computes Eρ², the share that is signal: rerun and the ranking holds, from 0 (noise) to 1 (repeatable). It is not accuracy; a judge can be stable and consistently wrong.

Why the ceiling does not rise with item count

This is the counterintuitive part:

"under a single judge, generalizability asymptotes to σ²s/(σ²s+σ²sj) regardless of item count, because the system-by-judge term carries no n_i. Items saturate; judges do not."

With one judge, the ceiling is σ²s/(σ²s+σ²sj) however many items you add: system × judge enters the error as σ²sj/nj, with judges in the denominator. Items shrink only the item-sampling term. What remains is "this judge is generous to A and harsh to B," which never cancels: one judge scores every item.

Two numbers pin it down. On MT-Bench at 0.75, 2 judges × 30 items roughly equals 1 judge × 77 items. At 0.80 nothing suffices: 0.80 sits 0.006 below the single-judge ceiling, and approaching it needs a divergent item count.

The advice splits in three. More items do not help within a fixed pool. More judges only approach the ceiling. Raising it needs a different judge universe, a different protocol, or item selection by discriminative power — the only item-side lever that raises it, to a 16.6% system variance share against Arena-Hard's 19.2%. Reading "items don't help" as "item design doesn't matter" inverts the point.

What 0.986 is, and what 8.6 points are

0.986 is often misread as 98.6% accurate. It is the reliability ceiling one judge reaches under a pairwise protocol that judges each pair in both orders and treats position as a factor. σ²sj falls two orders of magnitude below σ²s and the ceiling rises from 0.600–0.798 to 0.986 (bootstrap [0.934, 1.000], 11 systems); one judge reaches Eρ² = 0.93 at 24 items, so "one judge suffices."

But 0.986 means "rerun and the ranking holds," not "the judgment is right." On SummEval, reliability rose 0.711 → 0.959 (+0.248) while Kendall τ against a three-expert consensus moved only 0.596 → 0.710 (+0.114). Reliability is necessary for validity, not sufficient.

Then the 8.6 points, a new problem pairwise buys:

"A system placed in the first slot wins 8.6 percentage points more often than the same system, judged by the same judges on the same items, placed second."

First slot is presentation order, not first place: same system, same judges, same items, yet going first alone lifts the win rate 8.6 points. The bias varies 7.3 points across systems, reaches 25 at worst, and does not average away. Across 53 published win-rate comparisons the median claimed gain is 7.0 points; position bias is 1.23 times that, so an uncontrolled pairwise evaluation imports a bias larger than the median effect it hunts. In the same data, 70% of win-rate claims fall below pairwise's floor at native item counts.

An audit of 628 papers

The audit covers 628 arXiv papers from 2024–2026: five preregistered queries, 320 sampled by fixed seed, 312 coded, 227 reporting judge comparisons, κ = 0.73. Fewer than a quarter say whether they ran more than once (16.7% human, 20.0% automated); 46–67% report uncertainty; and all 17 checkable MT-Bench gains fall below its own floor (0.63 at 30 items, 0.54 at the native 80), which barely drops with more items because items saturate. This does not allege fabrication: the differences sit below the instrument's resolution, and the papers omit what a reader needs to notice.

What to do when you run a judge yourself

Report the judge universe: models, versions, count; otherwise reliability claims float.
Use the benchmark's native protocol and name it. Under MT-Bench's own template, 1–10 scale, and reference answers, the system variance share rises from 4.8% to 8.4% and the judges needed for 0.8 fall from 18 to 3.
Spend on judges before items. For 0.8 under a general 0–5 rubric: MT-Bench 25 judges, AlpacaEval 2 seven, the selected set and Arena-Hard two each.
Report MDD beside delta: the floor is 0.41–1.24 on a 0–5 scale, while 60 recovered pointwise gains have a median of 0.28; below it is not "leading."
When building a benchmark, select items by discriminative power.
Disclose repeat runs and response caching. Caching makes repeats fake.

With pairwise, run both orders and report prompt-level agreement: one judge rerun unchanged at temperature 0 moved the total about 1 point while 21.2% of items flipped win/tie/loss (arXiv:2610.30512).

What this paper cannot show

Six things, at least.

First, it is a single-author preprint v1, not peer reviewed.

Second, data and code are available from the author, not a public artifact: tables check out, reproduction is not cheap.

Third, it covers only 4 item sets, one pointwise 0–5 rubric, one scale; the paper concedes the general rubric is the conservative case.

Fourth, the Arena-Hard cell holds 17 items (others 32–43), making its σ²s estimate least precise; the pairwise arm uses the authors' own items, judge, and fixed baseline, not Arena-Hard's or AlpacaEval 2's pipeline.

Fifth, the audit samples rather than censuses: 628 framed, 312 coded, 227 reporting comparisons, κ between 0.45 and 1.00. It documents missing disclosure, not incorrect conclusions.

Sixth, reliability is not validity. A stable judge can stably measure the wrong thing, as SummEval shows.

The ruler

Back to 0.4 points: by this ruler it sits below the noise floor of most judge-scored leaderboards, and "who is stronger" is not answerable yet. Keep, not "0.4 points doesn't count," but "Items saturate; judges do not."

Adding items writes more marks on the ruler; changing judges makes the marks finer; changing the protocol swaps the ruler. Most people only do the first.

Disclosure: drafted with AI assistance and edited by the author.

62% vs 58% on the Same Benchmark: How Many Tasks Before the Gap Means Anything?

Devin — Tue, 27 Oct 2026 21:06:29 +0000

Two teams run the same agent benchmark on the same model with the same scaffold. Team A runs 100 tasks and reports 62%. Team B runs 250 and reports 58%. Both publish their task lists and pass rates. Neither is cheating.

Is the four-point gap real, or did the two teams stop at different points?

Wei-Jung Huang's paper (arXiv:2607.12338, 14 July 2026, KDD 2026 Agentic AI Evaluation and Trustworthiness workshop) answers this without running any new model. It shuffles the task order of records that three public agent benchmarks already published, then asks whether the decision after the first 15%, 25%, 50%... of tasks matches the decision after all of them.

Replay, not rerun

No new data, only a change in how much you see:

SWE-bench Lite: 552 ordered comparisons (276 unordered pairs), median 300 shared tasks
SWE-bench Verified: 156 ordered (78 unordered), median 496
AppWorld: 306 ordered (153 unordered), 585 tasks, 36 packages (18 systems × 2 splits)
tau-bench: 12 ordered (6 unordered), 278 entries per system over airline, retail, and telecom, with cost fields

Each pair gets 500 reveal orders and 200 bootstraps (tau-bench: 2,000 and 500). A real evaluation runs once in one order; replay estimates how often a comparison stalls if you stop early.

Three numbers that wear a percent sign

0 pp threshold: A wins if strictly above B on the completed task set, so one task can flip the whole decision. The strictest setting, no margin.

5 pp budget grid: a step size, not an effect threshold — 5% to 95% in 5-point increments. The paper separately tests 5 pp and 10 pp effect thresholds; don't merge them.

Unresolved rate: comparisons where the policy picks neither side and asks for more tasks or declines to conclude. Three gates hold together: at most 5% conditional decision error among decided comparisons, at most 5% coverage failure, at most 25% unresolved. That last is 25%, not 5%: being right when you decide doesn't help if you rarely decide.

Four benchmarks, four answers

At 0 pp on the 5 pp grid:

AppWorld: 15% — about 88 of 585 tasks, ~21% unresolved; still 15% at 5 and 10 pp
tau-bench: 25% — about 70 of 278 entries; 30% at 5 pp, 35% at 10 pp
SWE-bench Verified: 90% — about 446 of 496, 24.22% unresolved
SWE-bench Lite: 95% is still not enough. At 25%, error and coverage are in range, yet 55.92% of comparisons stay unresolved; at 95% it is 27.25%, above the ceiling.

What differs is the distribution of unresolved comparisons and coverage gaps: Lite decides accurately at 25% but leaves over half undecided; Verified only reaches 24.22% unresolved at 90%.

AppWorld arrives early because a coverage-aware bootstrap under difficulty grouping covers the required groups at 15%. A forced stratified policy needs 50%; forced uniform random fails coverage 99.96% of the time at 25%. On tau-bench, cheapest tasks first spends 11.51% of cost at 25% with 100% coverage failure.

Why the answers move

Five levers, none intrinsic to a benchmark: Lite has no sufficient budget at 0 pp but 55% at 5 pp and 35% at 10 pp; one-task slack on a coarse grid cuts its 0 pp budget to 75%, while repository × difficulty leaves more unresolved; low error often means few decisions rather than few mistakes; keeping one direction per unordered pair leaves 275 positives against 1 negative at 0 pp, so the paper targets ordered comparisons; and AppWorld's 15% moves to 10% or 25% as the truncation parameter a goes 0.10, 0.05, 0.02. A proportion is not a count: Lite's 95% of 300 is about 285 tasks, Verified's 90% of 496 about 446.

What this means when you read a paper

"We ran 100 tasks and A beat B by 0.5 points" is not a result. The paper's checklist asks five things:

What is the threshold, in percentage points?
How were tasks selected — random, stratified, cheapest first?
What is the coverage rule, and how many tasks per group?
What is the decision rule — bootstrap tail, paired normal, McNemar?
Is there a cap on unresolved comparisons, and is the actual rate reported?

Missing any one, "0.5 points higher" is "no conclusion yet": the same records go from no sufficient budget to 55% by changing the threshold.

What this paper cannot show

First, replay is not a new experiment: "It does not estimate performance on future tasks, because the replay orders permute a fixed completed record." It tells you how much of this batch was needed, not whether 15% survives a fresh batch.

Second, every conclusion depends on the threshold, coverage rule, grouping, and policy parameters. Move 0 pp to 5 or 10 pp and Lite goes from nothing to 55% or 35%; one-task slack on a coarse grid brings its 0 pp budget to 75%; AppWorld's 15% drifts from 10% to 25% with the truncation parameter. These are design choices, so a budget must travel with its direction and coverage rule.

Third, only three benchmark families are covered. OSWorld, WebArena, AgentDojo, AndroidWorld, Terminal-Bench, AgentBench, and MiniWoB++ were not replayed, and tau-bench's grid holds only 12 ordered comparisons (6 pairs). Don't extrapolate "AppWorld is 15%" to other interactive benchmarks.

Fourth, the public records are filtered leaderboard entries: SWE-bench keeps only near-complete entries covering at least 95% of a split, and AppWorld uses 36 public packages. Voluntary submissions, not random samples, so composition changes move the estimate.

Fifth, the cost findings hold only on tau-bench. Neither SWE-bench nor AppWorld publishes a cost field, so "cheap tasks first" is checkable on tau-bench alone.

Sixth, 5% decision error, 5% coverage failure, and 25% unresolved are the paper's shared targets, not an industry standard. Change the targets and the budgets change: a framework for computing a budget, not a universal "15% is enough".

One rule you can use now

When you see "we ran X tasks and A beat B by 0.5 points", check three things: is the decision threshold stated, were tasks selected and covered by repository, difficulty, or domain, and is the unresolved share reported. If any is missing, file it as "no conclusion yet". A task proportion is not a decision rule; without one it is just a cost figure.

More of my writing on AI evaluation: hhfa008.github.io/ai/writing

This is part two of a short series on reading AI evaluations. Part one: Same Model, Same Scaffold, 10 Runs.

Disclosure: drafted with AI assistance and edited by the author.

Same Model, Same Scaffold, 10 Runs: Agent Benchmark Scores Still Move by Up to 6 Points

Devin — Tue, 27 Oct 2026 20:56:08 +0000

Two teams run the same model, scaffold, and temperature on SWE-bench Verified, once each. Team A reports 32.4%; Team B reports 28.8%. Both post their logs. Neither is lying.

The gap comes from one distribution. Table 1 of On Randomness in Agentic Evals has this row: DeepSWE-preview on nano-agent, temperature 1.0, ten independent runs, mean 31.4% ± 1.0%, minimum 28.8%, maximum 32.4%. Team A landed at the top of the window; Team B landed at the bottom. They were comparing two samples, not two systems.

The paper, by Bjarni Haukur Bjarnason, André Silva, and Martin Monperrus, appeared on arXiv as 2602.07150 on February 6, 2026 (v3: March 25) and is a preprint.

What the experiment did

Most papers run each configuration once. This one ran three models (Qwen3-32B, DeepSWE-preview, Devstral-2) × two temperatures × two scaffolds (nano-agent, R2E-Gym), ten times each, on 500 SWE-bench Verified tasks: 60,000 trajectories, 25.58B tokens, 1.88M tool calls.

One definition matters. The headline range of 2.2 to 6.0 percentage points is the spread between the minimum and maximum single-run score within the same configuration. It is not the gap between two models. As the paper puts it: "single-run pass@1 estimates vary by 2.2 to 6.0 percentage points depending on which run is selected." The number measures repeatability, not capability.

Three sources of variance

The first is sampling. Above temperature zero, every token is drawn from a distribution. One different token changes the context for the next one, and the run can diverge from there.

The second is the scaffold. It changes both the level and the spread. DeepSWE-preview scores 31.4% ± 1.0% on nano-agent at temperature 1.0, but 34.4% ± 1.5% on R2E-Gym. Devstral-2 scores 63.5% ± 1.1% on nano-agent at temperature 0.2, but 34.9% ± 1.5% on R2E-Gym. The model did not change; the wrapper did.

The third is the runtime, and it only becomes visible at temperature zero.

Temperature zero is not a determinism switch

Greedy decoding sounds deterministic: always pick the most likely token. The paper shows it is not. Floating-point addition is not associative, and parallel reduction order, hardware-specific kernels, and batching can produce tiny differences in the same logits. Autoregressive decoding amplifies them.

At temperature 0, the standard deviations shrink but never reach zero. Qwen3-32B on nano-agent is 16.4% ± 1.2%; DeepSWE-preview on nano-agent is 20.4% ± 1.0%; Qwen3-32B on R2E-Gym is 22.3% ± 1.8%. The largest standard deviation at temperature 0 in the table is 1.8 points.

Lower temperature does not reliably shrink variance either. Qwen3-32B on nano-agent has a standard deviation of 0.7% at temperature 0.6 and 1.2% at temperature 0. Temperature zero delays divergence; it does not remove it.

So when a paper says temperature=0 makes its result reproducible, ask: over how many runs, with what standard deviation, on which inference engine and hardware?

Trajectories split early

For DeepSWE-preview on nano-agent at temperature 1.0, the median position of the first differing token between two runs was token 5, about 0.5% of the trajectory. At temperature 0 it moved to token 56. The paper's conclusion is that the first divergence almost always occurs within the first 1% of a trajectory. The next few hundred steps are the consequence of that one token.

The paper gives a concrete case: django__django-9296, Qwen3-32B, nano-agent, temperature 0. Two runs matched for 93 tokens. At token 94, one searched for the Paginator class; the other read the Django source. One patch landed in the wrong place; the other passed.

How to read a leaderboard

Before the ranking, ask five questions:

How many runs? Is this the mean of n independent runs, or the best single run?
Is there a standard deviation or interval? Without one, the score is a point, not an estimate.
What temperature? Zero is not deterministic; higher temperatures usually widen the spread.
What scaffold? The same model can move by tens of points when the harness changes.
Is this pass@1 or pass@k? DeepSWE-preview on R2E-Gym scores 34.4% pass@1, 52.9% pass@5, and 15.5% pass^5. All three say "pass"; they answer different questions.

The paper supplies a contrast. DeepSWE-preview on nano-agent scores 31.4% ± 1.0% at temperature 1.0 and 20.4% ± 1.0% at temperature 0, an 11.0-point difference the authors treat as real. Devstral-2 on the same scaffold scores 63.5% ± 1.1% at temperature 0.2 and 63.8% ± 1.6% at temperature 0, a 0.3-point difference they treat as noise. The deciding factor is the gap relative to the standard deviation.

How many runs you need

The paper uses a power calculation. It assumes single-run scores are approximately normal; Shapiro-Wilk tests passed for 11 of 12 configurations at p ≥ 0.05. Using the observed standard deviations of 0.7% (minimum), 1.5% (median), and 1.8% (maximum):

Detecting a 2-point improvement at p < 0.05 takes 2, 9, or 13 runs.
Detecting a 1-point improvement takes 8, 36, or 51 runs.
Detecting a 5-point improvement takes 2 runs at the median variance; 10 points takes 1 run at any of the three.

The paper states: "Detecting a 2% improvement at p < 0.05 with 80% power requires approximately 9 runs per agent under test, while detecting a 1% improvement requires 36 runs." The number depends on the effect you need to detect and the variance of your own setup.

What this study cannot show

First, it is a preprint: v1 on February 6, 2026, v3 on March 25, 2026, with no peer review.

Second, it covers one benchmark, three models, two scaffolds, and two temperatures. SWE-bench Verified has long software-engineering trajectories; the 2.2–6.0-point window may not transfer to short tasks or much longer ones.

Third, the paper does not link a code or data artifact. Reproducing the distribution means building the scaffolds and running 60,000 trajectories yourself.

Fourth, there is no calendar window. The paper reports ten runs per configuration but not the dates when the data was collected.

Fifth, variance at temperature 0 depends on the inference stack. The finding is that temperature 0 does not guarantee determinism, not that every temperature-0 evaluation has a standard deviation of at least 1.5 points.

Sixth, run-to-run variance and task-sampling variance are separate sources. Too few tasks and too few runs produce different errors, and they should be budgeted separately. This paper addresses only the second.

The cost of a single run

An evaluation costs more than the price of running it once. It also costs the price of believing the result. 25.58B tokens is expensive, but it buys a measurement of how unstable a single score is, which is usually cheaper than making a decision on one sample.

The next time you see "this model improved by 3 points," ask whether that is a difference between two ten-run means, or two single runs that landed at opposite ends of the same window.

This piece is part of a series on how to read AI evaluations. More writing is indexed at https://hhfa008.github.io/ai/writing/.

Disclosure: drafted with AI assistance and edited by the author.

The Median Paid Newsletter Converts 0.62% of Free Readers. Do the Math Before You Launch

Devin — Tue, 27 Oct 2026 11:27:29 +0000

The most useful number in beehiiv's 2026 paid-newsletter report is not a success story. It is the median.

The report, published on June 22, 2026 and based on first-party data from thousands of publications on beehiiv, puts the median free-to-paid conversion rate at 0.62%. That works out to roughly six paying subscribers for every 1,000 free readers. The median paid plan is $10 a month or $100 a year, and the typical creator in the dataset launched a paid tier about 45 days after starting the newsletter.

Those numbers are a baseline, not a verdict. But they change the arithmetic of launching a paid newsletter, and they are worth doing before you write another issue.

What 0.62% actually implies

At 0.62%, 100 paying subscribers require about 16,100 free readers. A straightforward rounding gives roughly 16,000. At 1%, the same 100 subscribers need 10,000 free readers. At 5% — a rate that appears in a lot of creator advice — you would need 2,000.

The gap between those scenarios is not a rounding error. It is the difference between a list you can build in a few months and a list you may spend years building. The report is explicit that 0.62% is a median, not an average, which means the middle publication is doing less well than the headline success stories suggest.

This is my arithmetic, not the report's: divide 100 by the conversion rate. The report supplies the rate and the pricing; the list-size examples are derived.

Pricing is more standardized than the advice

The median paid plan is $10 a month or $100 a year across nearly every industry and list size, according to the report, and it has not moved since 2024. That is a useful anchor. It means a creator does not need to invent a pricing strategy from scratch to be in the middle of the market; $10/$100 is the middle.

It also means the revenue math is easy to state. A newsletter with 1,000 paying subscribers at $10 a month generates $120,000 a year before platform fees, payment processing, and tax. At a 10% platform cut, $12,000 leaves the business every year. The report uses that example to make a point that is easy to miss: a four-figure subscriber count can still be a small business after the costs that sit between gross revenue and take-home cash.

Niche sets the ceiling

The report's sector breakdown is where the median stops being abstract. Sports newsletters have the highest median conversion rate at 1.93%, just over three times the platform average. Finance has a median of 0.78%, but the top 10% reaches 20.00% — a gap the report describes as nearly 26x. The same vertical, the same audience type, very different execution.

The value per subscriber varies just as much. A 1,000-subscriber investing newsletter can earn more than $2,700 a year, while the median 1,000-subscriber travel newsletter earns about $252. Those figures are not a guarantee; they are a reminder that "newsletter" is not one market.

The pattern behind the numbers is consistent with what operators already say: subjects tied to a decision, a professional use, or a tax-deductible expense convert and retain better than general-interest topics. A reader who can connect the subscription to a raise, a client, or a cost is easier to sell than a reader who simply finds the topic interesting.

Churn is the other half of the equation

The report's monthly churn rates range from 5.06% in Food & Drink to 16.67% in Money. News sits at 5.47%, sports at 7.76%, and AI at 13.33%. Estimated subscriber lifetime runs from about six months in Money to nearly 20 months in Food & Drink. Median lifetime value ranges from $83 in Community to $230 in Investing.

Those differences compound. A newsletter with 10% monthly churn loses most of a cohort in less than a year; a newsletter with 5% churn keeps it much longer. The report's conclusion is blunt: retention is where the real money is. Conversion gets the subscriber; churn decides whether the subscription becomes a business. One methodological caveat: the report's lifetime figures are steady-state estimates (1 ÷ monthly churn), not cohort survival curves.

What the report does not tell you

The data come only from beehiiv publications. They do not necessarily describe Substack, Ghost, Kit, or any other platform. Beehiiv has a commercial interest in showing that paid newsletters can work, and the report also says the company generated $19 million in subscription revenue in 2025, up 138% from 2024, with $35 million projected for 2026.

The report does not disclose refund rates, sample sizes by industry, or a comparable Substack conversion figure. It reports medians, not averages, so the top decile is hidden inside a single number. Treat 0.62% as a starting assumption for a beehiiv-style publication, not a law of the newsletter market.

Four things to do before you launch

Run the list-size math at three conversion rates. At 0.62%, 1%, and 3%, how many free readers do you need for 100 paying subscribers? If the answer at the median is larger than the audience you can realistically build, change the plan before you change the price.
Pick a niche tied to a decision or a budget. The sports and finance spreads are not just pricing differences; they reflect whether the reader can justify the cost. "Interesting" is a weak buying reason.
Launch paid around the six-week mark, not after the archive is perfect. The median creator in the dataset launched a paid tier 45 days after starting. A clear promise and a small archive beat a perfect archive with no offer.
Model churn, not just MRR. Run your revenue at 5%, 10%, and 16% monthly churn. If the business only works at the low end, you have a retention problem to solve before you have a growth problem.

The median is not a ceiling. The top 10% in finance shows that execution can move conversion by an order of magnitude. But the median is the right place to start, because it is the number most creators will actually experience — and it is much cheaper to plan for it than to discover it after launch.

Redis Changed Its License in 2024. The Fork Is Still Shipping.

Devin — Tue, 27 Oct 2026 09:51:20 +0000

Reconstructed from public reporting; not firsthand, no invented people or dialogue.

On March 20, 2024, Redis changed its license. Starting with Redis 7.4, new releases would ship under RSALv2 and SSPLv1 instead of BSD-3. The company did not hide the reason. Most of its commercial sales, it wrote, ran through the largest cloud providers, and those providers "commoditize Redis' investments and its open source community."

Two days later, on March 22, a new repository appeared on GitHub. It was called Valkey. The license was BSD-3. The code was a fork of the last BSD release.

That is the whole story in three sentences. The interesting part is what happened next, because both sides kept receipts.

The fork did not fade

Most forks of a popular project die in a few months. This one did not.

As of October 27, 2026, the Valkey repository has 27,528 stars and 1,431 forks. Redis has 77,130 stars. The fork is smaller, and it is also still here.

More useful than stars: release activity. Valkey 9.1.3 shipped on October 26, 2026. In the same 48 hours, the project also published maintenance releases on four older lines: 9.0.7, 8.1.11, 8.0.12, and 7.2.15. Four live branches is not a museum project. It is a project with users who cannot all upgrade at once.

The 9.x line also carries real engineering claims. The project's site says 9.1 does 2.1 million requests per second on a single server, saves up to 20% memory on small strings, and moves Lua out of the core into an optional module. Treat vendor benchmarks as marketing until you run them yourself. But "Lua is now a module" is the kind of change you only make when you are optimizing a codebase you expect to maintain for years.

Also worth noting: Valkey is a Linux Foundation project. That matters less for code quality than for governance. A foundation cannot unilaterally relicense the project the way a company can.

The reversal, in the company's own words

On May 1, 2025, Redis added AGPLv3 as a licensing option, starting with Redis 8. It kept SSPL. In the announcement, CEO Rowan Trollope wrote the sentence that makes this story worth telling:

"This achieved our goal—AWS and Google now maintain their own fork—but the change hurt our relationship with the Redis community."

Read that twice. The company says the license change worked as intended and still cost it something it cared about. It is rare to get both halves in one sentence from the person who signed the change.

The same post notes that Salvatore Sanfilippo, the original author, rejoined Redis in November 2024 as a developer evangelist. Redis 8 also folded Redis Stack into the core: JSON, time series, probabilistic types, and the query engine now ship together under AGPL. The company claims more than 30 performance improvements, commands up to 87% faster, and twice the throughput. Those are vendor numbers. The date and the license are facts.

What AGPL does and does not fix

Here is the part that gets flattened in arguments. AGPL is OSI-approved. That makes Redis open source again in the formal sense. It does not make Redis BSD again.

AGPL keeps the copyleft trigger, and extends it to the network. If you modify the software and let other people use it over a network, Section 13 requires you to offer them the corresponding source. For most teams running an unmodified cache, nothing changes. For anyone shipping a modified Redis as part of a hosted product, the compliance work is real, and it did not exist under BSD.

One detail gets lost in the argument: Redis 8 is dual-licensed. AGPLv3 is an option, not a replacement for SSPL. If your legal team already approved SSPL terms, nothing forces a change. If they rejected SSPL, AGPL is the path back. That distinction matters in procurement, where the real question is not "is it open source" but "which license will our scanner see on the artifact we deploy."

So the scoreboard is not "open source won." It is narrower. A company traded a permissive license for commercial protection, lost community trust, watched a foundation-backed fork keep shipping, and then added a copyleft license that is open source but stricter than what it removed. Everyone got something. Nobody got 2023 back.

What I would do this week

Audit by version, not by project. Your dependency scanner probably reports "Redis" once. The license depends on which release you actually deploy: 7.2 and earlier are BSD, 7.4 and later are RSALv2/SSPL, and 8.x can be AGPL. Pin the version and record the license next to it.
Ask your managed cache provider two questions: which engine, and which major version. "Redis-compatible" is not an answer. The engine determines your upgrade path if a license or price changes again.
If you host a modified version as a service, read AGPL Section 13 before the next upgrade, not after. The obligation is to offer source to network users, and "we only run it internally" is a different case from "our customers call it."
Keep the exit cheap. The core command surface is largely compatible, but modules and managed features are where migrations get expensive. A thin client abstraction and one tested fallback image cost little now and buy you options later.

None of this requires picking a side. It requires knowing which license is running under your connection string.

The part that generalizes

License changes are cheap to announce and expensive to undo. Redis added AGPL thirteen months after leaving BSD, and the fork from March 2024 is still publishing releases every week.

If you maintain a dependency that a company controls, the lesson is not "avoid company-controlled projects." The lesson is that the license is a versioned artifact, like the code. Treat it that way: record it, diff it, and know what you would do if it changed on a Tuesday.

The fork is what that plan looks like when you did not have one. It works. It is also more expensive than a one-line entry in your dependency policy.

Sources: Redis announcements of 2024-03-20 and 2025-05-01; valkey.io; GitHub repository and release metadata. Figures are as of 2026-10-27.

Apple just linked "AI agents" to Full Disk Access restrictions. Here is what that means for tooling

Devin — Tue, 27 Oct 2026 05:15:20 +0000

As AI agents become increasingly capable and autonomous, the risks associated with this level of access will grow substantially.

That sentence closes a short Apple developer note, "Updates to Full Disk Access in macOS", published on October 2, 2026. The note is roughly 200 words and changes nothing yet. What it does is put tool builders on notice that the switch they have relied on may soon require more than a toggle.

Here is what it says, what it leaves out, and what it means if you ship anything that reads outside its own container.

Why Full Disk Access exists in the first place

macOS sandboxes apps and gates sensitive locations behind TCC, the Transparency, Consent and Control system. A normal app cannot read your Messages database or your Safari history without asking, and in most cases it cannot ask at all.

Backup tools break that model. A backup app cannot know in advance which folders a user cares about, and cannot ask for permission one directory at a time. So Apple created one blunt escape hatch: Full Disk Access. Grant it and the app sidesteps the per-resource controls entirely.

Apple's own note says this plainly: Full Disk Access "largely sidesteps these controls in order to allow backup apps to function properly on the Mac." The design has not changed; what Apple is signalling is a change to how the grant is obtained, not what it does.

What the announcement actually says

Three points, and no more:

Apple will "introduce additional controls" on top of the existing switch.
Users who "genuinely wish to grant an app this extraordinary level of access can only do so with very explicit user action."
The motivation is risk: some developers use the permission in ways that expose "everything on their systems, including files, mail, messages, and even browsing history" without users understanding it.

Note "additional controls", not replacement. Apple is not removing Full Disk Access and explicitly preserves the backup use case. The change is to the consent moment.

What it does not say

Worth reading twice.

No effective date. The note says "Going forward". There is no date, no release number, no mention of a beta.
No OS version. Any article stating that this starts in macOS 27.x, or 26.4, or "next year", is inferring. The note does not say.
No API list. Not a single API is named. There is no list of affected directories, no mention of which TCC services are in scope, and no statement about whether the existing kTCCServiceSystemPolicyAllFiles identifier is the thing being gated.
No answer for existing grants. The note does not say whether apps already granted Full Disk Access will be re-prompted, grandfathered, or left alone. That is the most consequential open question for shipping software, and it is unanswered.

Apple has not published an effective date or a list of affected APIs. Until it does, the only defensible engineering position is to prepare for a stricter consent flow without assuming a schedule.

What this means if you build automation tooling

We hit the practical version of this problem on a macOS 27 machine this week, while building a background job that reads ~/Library/Messages/chat.db. The file was -rw-r--r--, owned by the same user, readable from the same shell. The read returned EPERM, errno 1.

That error is TCC, not the filesystem. The details matter if your product depends on broad reads:

The grant is bound to a code requirement, not a path. After a rebuild with a new signature, our app stayed in the list with its checkbox on, and access was still denied. The system log said Failed to match existing code requirement. TCC stores a reference to the signed binary's designated requirement. Ad-hoc signing or a fresh development certificate changes the CDHash, and the old entry no longer matches. Deleting the stale entry, relaunching, and re-triggering the prompt is the only fix; clicking "allow" again does nothing, because the entry is never rebuilt.

Background processes are not covered by the parent's grant. The same script read ~/Documents fine from a terminal and failed with EPERM when launched by a LaunchAgent. Terminal's grant belongs to Terminal's executable, not to everything it spawns. If a helper does the reading, the helper is what users must authorize.

Prompting is going away in some services. On macOS 27, requesting screen recording authorization returned denied outright, logging does not allow prompting; returning denied. The API can no longer surface a dialog; apps must send users to System Settings and explain which row to enable.

Permission state may not be where your self-check looks. In our environment the user-level TCC database was no longer at ~/Library/Application Support/com.apple.TCC/TCC.db; it resolved under /private/var/containers/Data/ProtectedSystem/<UUID>/.... A health check on the old path returns empty and reports "not authorized" for an authorized app. Read the log, not the database.

Five things to do before this lands

None require knowing Apple's timeline; all make a stricter consent flow cheaper to absorb.

Apply least privilege before you need to. Enumerate the exact paths your tool reads and use the narrowest API that works: EventKit for calendars, Contacts for address book, FSEvents for filesystem watching, a user-picked folder for documents. If one code path needs Messages, put it behind a feature flag so the rest of the product runs without the grant.
Declare the purpose where the system will show it. Write the sentence a user will read at the consent moment: what you read, why, what leaves the machine. Put the same statement in your privacy policy and in-app settings. "Explicit user action" is easier to satisfy when the reason is already on screen.
Stop hard-depending on Full Disk Access. Every feature behind the grant needs a defined "denied" state: an empty state, a narrow-mode fallback, or a disabled control with an explanation. If the app crashes or spins when permission is missing, the stricter flow turns that into support tickets.
Detect permission failures explicitly, and re-verify after every signing change. Treat EPERM as a possible TCC denial rather than a filesystem bug. Log the tccd decision, and add an automated check that runs after each build with a new identity: attempt one gated read, assert success, and fail loudly. Re-signing is the most common silent regression we have seen.
Tell users what to do, per OS version. Deep-link to the exact Settings pane, name the row, and warn that an app update may require re-authorizing. Keep instructions versioned; the pane has moved before.

What to watch instead of a date

Two signals will tell you this is real. First, the Full Disk Access pane in System Settings: if a new confirmation step appears, the change has shipped. Second, macOS beta release notes: a TCC or privacy entry naming the new control will show up there before general availability.

Until one of those appears, treat the announcement as a design constraint, not a deadline. It tells you which assumptions in your code are now liabilities. That is enough to start.

If you want the longer field notes on macOS permissions, signing, and automation, they live here: https://hhfa008.github.io/personal/cases.html

Your App Store screenshots now decide who can install your app in April 2027

Devin — Tue, 27 Oct 2026 04:35:30 +0000

Apple's developer news post says that "Starting April 2027, any apps or games submitted will need to include screenshots for iPhone Duo." Apple's App Store Connect screenshot specifications page says that "Starting in April 2027, screenshots will be required for any app using the iOS 27.1 SDK or later."

Both sentences are live on developer.apple.com, and Apple has not clarified which scope applies. It decides which teams book design time before April and which can wait.

The conflict is a scheduling problem, not a documentation puzzle: two calendars, and every team is already on one.

What each scope covers

Read the news post literally and the trigger is submission. Any app or game submitted on or after 1 April 2027 needs iPhone Duo screenshots: new apps, feature releases, bug-fix builds, metadata updates. The SDK does not matter: a team staying on the iOS 26 SDK is covered.

Read the specifications page literally and the trigger is the build's SDK. Only an app using the iOS 27.1 SDK or later needs Duo screenshots. A team shipping from an older SDK is outside it, even after April 2027; a team that upgrades for an unrelated reason is inside.

The overlap: a build made with the iOS 27.1 SDK submitted on or after 1 April 2027 requires the screenshots under both pages. The disagreement only matters for builds using an older SDK.

Team situation after 1 April 2027	News post	Specifications page
Submits a build using iOS 27.1 SDK or later	Required	Required
Submits a build using an older SDK	Required	Not required
No App Store submission at all	Not triggered	Not triggered
Submitted before April, updates later	Required for that submission	Depends on that build's SDK

The third row is not a loophole: the first bug fix that needs a submission puts the app back in scope.

Who is actually exposed

The ambiguous group is larger than it sounds: most teams do not ship every release on the newest SDK.

Consumer apps on a monthly cadence. If the release train uses the iOS 27.1 SDK, nothing to interpret: required under either page. Add the capture to the SDK-upgrade release.

Maintenance-mode apps. The disputed cell. A team that last touched Xcode a year ago and ships two fixes a year may be covered by the news post and outside the specifications page. The cost of being wrong is asymmetric: one capture session versus a submission problem in April. Treat it as exposed until Apple says otherwise.

New apps launching after April. Starting on the iOS 27.1 SDK is covered under both readings; starting on an older SDK to dodge the work lands in the disputed cell.

Agencies and studios. The risk is portfolio drift: one client on the newest SDK, another on an older one, a third with no release. Record SDK and next submission date per app.

B2B and enterprise apps. If users get the app through App Store submission, the rule applies whether the audience is consumers or employees.

Apps with no release planned. Record the decision and date. "No submission after March 2027" is valid, but write it down.

The calendar to put on the wall

These dates are a planning recommendation, not Apple policy. The only fixed date is April 2027.

By 30 November 2026: inventory. For every app you own or manage, record the current SDK, the next planned submission, the release owner, and whether that submission could slip past March 2027. This shows how many apps are in the disputed cell.

By 31 December 2026: capture and upload the four Duo images for every app with a submission after March 2027. The sizes are 1398 × 2034 and 2034 × 1398 for the outer display, and 2007 × 2853 and 2853 × 2007 for the inner display. Uploading early satisfies the broad reading and tests whether App Store Connect treats the slots as required or optional.

January 2027: dry run. Open the upload flow for a real build and save a dated note on whether the Duo slots are required or optional. That is first-party evidence of how the system behaves. Freeze the source files and localized versions this month.

March 2027: use the last comfortable window. If a release must be live before the rule takes effect, submit with enough buffer that a rejection does not push it into April. Do not plan a 31 March submission.

1 April 2027: the gate applies. From here, every submission needs a documented answer: are the four Duo images attached, or do we have a written Apple answer saying they are not needed?

If Apple clarifies earlier: revise the inventory within a week, not a quarter. Teams that captured already lose nothing.

Six actions that are correct under either reading

Add the four Duo screenshots to every app with a submission after March 2027, even if it stays on an older SDK. Under the broad reading this is the requirement. Under the narrow reading the set is accepted and triggers the Duo badge. One capture pass, not an SDK migration.
Do not upgrade to the iOS 27.1 SDK just for this deadline. Staying on your current SDK keeps the narrow reading satisfied, and the screenshots cover the broad reading. If a 27.1 upgrade is already planned, attach the screenshots to that release.
Keep an SDK ledger per release train. Record the SDK used for every submission, not just the app's nominal minimum. It tells you whether the narrow scope applies and is the first thing anyone will ask for if review behaves unexpectedly.
Run a dated dry run in App Store Connect. Capture the upload screen for a real build, note whether the Duo slots are required or optional, and store the build number with the date.
Name one owner and one source of truth for screenshot assets. In a studio, that means a per-client tracker, not a shared folder. In a small team, it means one person who can answer "which build produced this image set?"
Put a gate in the release checklist. For every submission on or after 1 April 2027, require the four images or a written Apple answer, plus the SDK used. A checklist line survives staff changes; a Slack thread does not.

Why the title says "who can install"

Screenshots are usually a conversion asset. In April 2027 they may also become an upstream gate. If a submission cannot proceed without a required asset, the install funnel does not get worse; it stops. The question becomes which builds may reach the store.

Neither page says how review will behave, and this article does not predict an approval rate. Under either scope, some teams have work to do before April, and the disputed group is the one most likely to postpone it.

Apple has not clarified which scope applies. Until it does, use the stricter schedule: inventory now, capture before year-end, dry-run in January, and keep the evidence with the release ticket.

A checklist for handling App Store submission requirements is here: https://hhfa008.github.io/personal/app-store-rescue.html

Apple's own docs disagree about the April 2027 deadline for iPhone Duo screenshots

Devin — Tue, 27 Oct 2026 03:47:16 +0000

Two Apple pages that both describe the April 2027 iPhone Duo screenshot requirement do not agree on who it applies to. The developer news post says, "Starting April 2027, any apps or games submitted will need to include screenshots for iPhone Duo," while the App Store Connect screenshot specifications page says, "Starting in April 2027, screenshots will be required for any app using the iOS 27.1 SDK or later."

Both sentences are published by Apple on developer.apple.com. Neither page carries a correction, a link to the other, or a note explaining the difference. Apple has not clarified which scope applies.

That leaves one practical question for anyone shipping after March 2027: if your next submission uses an older SDK, do you need Duo screenshots or not? The two pages differ, and the difference is not cosmetic: one scopes the requirement to the act of submitting, the other to the SDK the build uses.

What each page actually says

The news post describes the badge first: "When you submit iPhone Duo screenshots in App Store Connect, your App Store product page will display a badge to let people know your app is optimized for iPhone Duo." Then it gives the deadline in one sentence, with no SDK condition attached: "Starting April 2027, any apps or games submitted will need to include screenshots for iPhone Duo."

The screenshot specifications page is a reference table. For iPhone Duo it lists four image sizes: 1398 × 2034 and 2034 × 1398 for the outer display, and 2007 × 2853 and 2853 × 2007 for the inner display. In the notes for that row it says: "Starting in April 2027, screenshots will be required for any app using the iOS 27.1 SDK or later." It repeats the badge sentence, but not the "any apps or games submitted" sentence.

The third relevant page, Apple's "Prepare your app for iPhone Duo," explains rendering, not submission: the iOS 26 SDK centers the app with empty space, the iOS 27 SDK fills most of the inner display but avoids the right-edge status bar, and the iOS 27.1 SDK or later uses the full display with vertical toolbars. It does not say when screenshots become mandatory, so it cannot settle the scope question.

Why the gap matters

The two rules form a small matrix:

Submitting after April 2027 with the iOS 27.1 SDK or later: both pages require Duo screenshots.
Submitting after April 2027 with an older SDK: the news post requires them; the specifications page does not.
Submitting before April 2027: neither page requires them, although you can add them and get the badge.

The disputed cell is the second one, and it is not rare. Plenty of teams stay on an older SDK for a release cycle, or ship a maintenance update without moving Xcode. Those teams need a decision now, because screenshot capture has to be scheduled with design and QA.

It is tempting to pick the interpretation that saves work. The cost of the two mistakes is asymmetric. If you add screenshots and the narrow reading is correct, you have spent one capture session and possibly gained a badge. If you skip them and the broad reading is correct, you have a submission problem at the worst possible time, with no documented appeal path that says "the other page told me I could."

Who to ask, and what to ask

Ask Apple in writing for an operational answer, not an interpretation. Use three channels together; they reach different owners:

App Store Connect Help → Contact Us, routed to App Review or submission support. This channel can tell you what the review system will enforce.
The feedback control on each documentation page. The issue is a documentation inconsistency; the fix is an edit to one or both pages.
Apple Developer Forums, in App Store Connect or App Review. A public thread creates a record, and Apple staff sometimes answer. Treat a reply as information, not a policy commitment.

Keep it binary. Asking "which page is right?" invites a non-answer; this is harder to deflect:

Two Apple pages describe the April 2027 iPhone Duo screenshot requirement with different scopes. The developer news post says "any apps or games submitted"; the App Store Connect screenshot specifications page says "any app using the iOS 27.1 SDK or later."

Please confirm, for submissions on or after 1 April 2027:
(a) Is a submission built with the iOS 26 SDK, without Duo screenshots, accepted, warned, or rejected?
(b) Does the answer differ for a new app and for an update to an existing app?
(c) If a build uses the iOS 27.1 SDK but does not adopt Duo-specific APIs, are Duo screenshots still required?
(d) Which page should we treat as authoritative until both are updated?

A written answer and a documentation update would resolve this for our release planning.

Ask for the answer in the case, not on a call, so you can attach it to the ticket. If the first reply is generic, reply once with the two quotations and the four questions.

Five actions that are correct under either reading

These are the moves that do not depend on which page wins.

Add the four Duo screenshots now, without changing your SDK: 1398 × 2034 and 2034 × 1398 for the outer display, 2007 × 2853 and 2853 × 2007 for the inner. Under the broad reading this is the requirement; under the narrow reading it is optional, and submitting the set is what triggers the badge. One set satisfies both.
Do not upgrade to the iOS 27.1 SDK just for this deadline. Staying on your current SDK leaves the narrow reading satisfied, and the broad reading is covered by the set you just added. If you already planned a 27.1 upgrade, put the screenshots in the same release; the upgrade makes the requirement apply under both readings.
Capture from the Duo simulator in Xcode 27.1's Device Hub, not from a resized iPhone mockup. Check the inner display at the regular-by-regular size class and do not rely on orientation. Capture what the shipping SDK actually renders; do not draw an aspirational layout the build does not produce.
Run a dry run in App Store Connect today. Open the screenshot upload UI for your current build and record whether the Duo slots are marked required or optional. Save a dated screenshot with the build number. This is first-party evidence of enforcement, and it may answer the question before Apple does.
Put one dated line in the release checklist: for every submission on or after 1 April 2027, confirm the four Duo images are attached and record the SDK used. That is correct under both readings and gives you a record if review behaves differently from either page.

What to do when the answer arrives

If Apple says the broad scope applies, the screenshots are already in place; add the requirement to your submission template for every app, including maintenance updates. If Apple says the narrow scope applies, keep the screenshots anyway; they are accepted, trigger the badge, and cost little. If Apple updates one page but not the other, keep the written answer with the release ticket and cite the date.

If April arrives with no answer, ship with the screenshots. You will not be the developer who discovers the enforcement rule in production.

The useful conclusion is not that one page is wrong: two official pages describe different scopes, Apple has not clarified which scope applies, and the cheapest safe path is to satisfy the broader one while you wait for a written answer.

A checklist for handling submission requirements is here: https://hhfa008.github.io/personal/app-store-rescue.html

iOS 27.2's expanded ATT prompt is mandatory in five EU countries — what actually changes for your app

Devin — Tue, 27 Oct 2026 01:35:04 +0000

If your app is distributed in France, Germany, Italy, Poland, or Romania, the App Tracking Transparency prompt your users see is changing — and the old one is no longer an option. Starting with iOS 27.2 and iPadOS 27.2, Apple requires the alternative expanded ATT prompt in those five countries; elsewhere in the EU it is optional, and outside the EU it has no effect.

That is the entire scope. Everything else here is what Apple published, what it did not publish, or a check you can run.

What Apple actually published

Apple's developer news post, dated September 16, 2026, states the rule in one sentence:

Beginning with iOS 27.2 and iPadOS 27.2, developers will have the option to use an alternative version of the App Tracking Transparency system prompt in the EU. […] Due to legal requirements, only the alternative version of the system prompt is available for apps distributed in Germany, France, Italy, Poland, and Romania.

The iOS & iPadOS 27.2 RC release notes say the same thing in API language:

AppTrackingTransparency (ATT) now supports an alternative expanded prompt and re-prompting on a yearly basis for EU users. The alternative prompt is required for users in France, Germany, Italy, Poland, and Romania.

One line in the news post is easy to skip: "The requirements for when you must seek permission to track users will remain the same." This is a change to presentation and timing, not to what counts as tracking.

One method, three regional behaviours

The new call is requestTrackingAuthorization(preferExpandedInterface:additionalInformationAction:completionHandler:). Apple keys the behaviour to two conditions at once: device location and the country or region on the signed-in Apple Account:

Device and Apple Account region	What the system shows
France, Germany, Italy, Poland, or Romania	The full-page Markdown sheet, regardless of what you pass for `preferExpandedInterface`
Elsewhere in the EU	The full-page sheet only if `preferExpandedInterface` is `true`; otherwise the existing system alert
Outside the EU	Neither parameter has any effect; the old prompt behaviour applies

Both conditions matter. The API page says the feature "requires the device to be located in a specific European Union country and signed in with an Apple Account with a specific EU country or region." Apple's news post frames the mandate by where the app is distributed; the API page frames runtime behaviour by device and account. Treat all three as separate gates.

What the new API adds

A Markdown purpose string. The existing NSUserTrackingUsageDescription still has to be present. The expanded sheet reads NSUserTrackingMarkdownUsageDescription, which Apple documents as supporting bold, italics, bulleted lists, and paragraphs.
An optional Additional Information button. You pass a closure through additionalInformationAction; pass nil and the button does not appear.
A dismissal path that is not an answer. Tapping Additional Information closes the sheet without recording a choice. Your closure runs, and the completion handler still receives .notDetermined. Apple's documentation tells you to request authorization again in that case.
Annual re-prompting in the EU. After a user answers, the system will not ask again for a year; once the year is up, you can ask again whether the previous answer was allow or deny, unless the user has turned off the Tracking setting.
The same four statuses. Authorized, denied, restricted, and not determined are unchanged. What changes is when "not determined" can mean "the user opened the sheet and closed it."

The prompt is also skipped entirely in the documented edge cases: tracking is restricted, the global setting is off, another request is already in flight, the call comes from an app extension, or the app is not active. Those paths return immediately.

Three things the documentation does not say

Most posts quietly invent requirements here. The two Apple pages reviewed do not answer these questions:

1. What the UI must contain. The news post says "modified formatting and language"; the API page says full-page sheet with an optional button. Neither page states a character limit, a required layout, a button order, a button count, or whether a skip control is allowed. If your designer asks for the exact spec, the accurate answer is that Apple has not published one. Screenshots in blog posts are examples, not requirements.

2. Whether the attribution chain changes. I read both pages for mentions of SKAdNetwork and AdAttributionKit. There are none. The release note does not say attribution changes, and it does not say it stays the same. Do not rewrite your attribution integration because of this note. If an SDK initialisation reads ATT status, the only change justified by the published text is handling the new "viewed but unanswered" path.

3. Whether App Review will reject the old call in the five countries. Apple's pages do not say that continuing to call the old method causes a rejection. They also do not say the system silently substitutes the expanded sheet when you call the old method. Both outcomes are unknown. Ship the new call and verify it on a real device; do not treat review behaviour as a compliance strategy.

Everything above is limited to Apple's September 16 post and the 27.2 RC notes. If a number or rule is not in those pages, do not rely on it yet.

An 8-item checklist

Inventory your markets. List the storefronts where the app is actually distributed. If none of the five is in scope, the mandate does not apply; the yearly re-prompt still matters for EU users once you adopt the new API.
Check whether you use ATT at all. Search the project for ATTrackingManager. If the app never requests tracking authorization, this migration is not yours; do not add ATT just because the release note exists.
Update Info.plist deliberately. Keep NSUserTrackingUsageDescription. Add NSUserTrackingMarkdownUsageDescription only if you want the Markdown sheet, and localize it for German, French, Italian, Polish, and Romanian.
Pick the selector from autocomplete, not from a blog. The release notes' Resolved Issues entry spells the Objective-C selector requestTrackingAuthorizationPreferringExpandedInterface:…, while the Swift documentation page names the parameter preferExpandedInterface. Let Xcode resolve it in your SDK version.
Exercise the Additional Information path. Tap it, confirm the sheet closes, confirm completion receives .notDetermined, and confirm a second request actually presents the prompt again. Build the intermediate screen and a retry entry point before shipping.
Fix the state machine. Treat .notDetermined as "no answer yet," not "denied." Stop caching the first result forever, and re-read the status on foreground or after the yearly window.
Run the regional test matrix. Five-country device plus matching account; another EU country with preferExpandedInterface both true and false; outside the EU; tracking restricted; global setting off; a request already in flight; an app extension; and the app in the background.
Align measurement before changing code. Ask your MMP or attribution SDK vendor whether it treats .notDetermined as terminal or assumes a one-time ATT result. Align event timing with the new retry path. Do not change SKAdNetwork or AdAttributionKit configuration based on this note; Apple's pages announce no change there.

What I would ship this week

The developer-side change Apple has actually documented comes down to three moves: call the new method, add the localized Markdown key, and make .notDetermined retryable. That is enough to be correct in the five countries and optional everywhere else in the EU.

The rest — final copy, layout, review outcomes, and attribution effects — is either a product decision or not yet published. Treat anything more specific than that as unconfirmed until Apple updates the documentation.

More iOS release breakdowns and shipping checklists: https://hhfa008.github.io/personal/cases.html

iOS 27.1 Is iPhone Duo Only: Keeping Your Min-Version Matrix and CI Honest

Devin — Tue, 27 Oct 2026 00:55:19 +0000

On 23 October 2026 Apple released iOS 27.1 for exactly one device, the iPhone Duo, while its own security page still called iOS 27.0.1 the latest version for iPhone 11 and later. That single line splits what most iOS teams treat as one target — "the current iOS" — into two runtimes, and it changes how you set a deployment target, gate new APIs, and choose simulator destinations.

What Apple actually published

The security updates page is unusually direct:

The latest version of iOS and iPadOS is 27.0.1, and 27.1 for iPhone Duo only.

The table underneath gives the two rows that matter. iOS 27.1 is listed as available for iPhone Duo, released 23 October 2026. iOS 27.0.1 and iPadOS 27.0.1 are listed for iPhone 11 and later (plus recent iPads), released 28 September 2026. Both rows say: "This update has no published CVE entries."

That last detail matters for how you describe the release internally. This is not a security patch that one device received early. At least from the public record, it is a version number that exists for a new device, with no published CVE attached to either branch.

What Apple has not stated is equally important. Apple has not stated why 27.1 is limited to the Duo, whether it will later reach other iPhones, or whether per-device iOS releases will become a standing policy. The developer release notes for Xcode 27.1 and the 27.2 RC describe SDKs and simulator behavior; they do not describe a new distribution model. Treat the Duo release as one data point, not as a roadmap.

One build still carries one minimum version

An App Store build has a single MinimumOSVersion. There is no per-device minimum-version field that lets one build require 27.1 on the Duo and 27.0.1 everywhere else. The system matrix can fork; your shipping build cannot.

The practical consequence is that raising IPHONEOS_DEPLOYMENT_TARGET to 27.1 to "support the new device" is the wrong move. It would exclude every non-Duo user whose current latest version is 27.0.1. Keep the existing baseline and gate 27.1-only API at the call site:

if #available(iOS 27.1, *) {
    // API introduced in iOS 27.1
} else {
    // fallback for 27.0.1
}

#available checks the OS version, not the hardware. Do not replace it with a device-name or model-identifier check; those strings are brittle and will spread through the codebase. If a feature genuinely belongs only to the Duo, isolate it behind a capability check or a separate target rather than sprinkling model checks through shared code.

This is an inference from how App Store builds work, not a new Apple rule. Apple has not stated that a single build may declare two minimum versions, and nothing in the 27.1 release notes suggests one.

The deployment-target trap in Xcode 27.2

There is a second, more concrete reason to pin your toolchain. Apple's Xcode 27.2 RC release notes say:

macOS, watchOS, tvOS, and visionOS SDKs in Xcode 27.2 incorrectly report 27.1 as a valid deployment target.

The same notes warn that Mac Catalyst builds with a 27.1 or 27.2 deployment target may be unable to use newly introduced API. Xcode 27.1 has a related Catalyst issue: projects that use APIs specific to iOS 27.1 show compile errors when building for Mac Catalyst, with a workaround using build-time conditionals, and projects that target iOS 27.1 show no Mac Catalyst run destination.

For an app with a Catalyst target, that is enough to keep the deployment target at 27.0 for now and verify the build before merging any SDK bump. It is also a reminder that "the SDK accepts this value" is not the same as "this value is supported."

CI needs two destinations, not two copies of everything

Xcode 27.1 includes Swift 6.4 and the iOS 27.1 SDK, so it is the toolchain that can build and run against the Duo runtime. But Apple documents several limits in the Duo simulator: StandBy is unavailable, most app extensions cannot be run or debugged there, and the initial simulator launch can take several minutes. The 27.2 RC notes add that screenshots and recordings in the Duo may be black for up to a few minutes after boot, that accessibility tools cannot convey the Duo screen content, and that modifier keys may stop working when switching between inner and outer displays.

Those constraints should shape the CI job, not cancel it. Keep your existing 27.0.1 regression suite as the broad gate. Add one narrow Duo job that proves the app launches, renders its first screen, survives the fold states you support, and completes the purchase or sign-in path. Do not move extension UI tests or screenshot comparisons into that job until the documented limitations change.

A minimal starting point:

xcrun simctl list devices available | grep -i "iPhone Duo"

xcodebuild -scheme YourApp \
  -destination 'platform=iOS Simulator,name=iPhone Duo,OS=27.1' test

If the destination is missing, the SDK is not installed or Xcode is not selected; fix that before debugging the test.

Seven checks before the next release

Read the security updates page and record both rows: 27.1 for iPhone Duo, 27.0.1 for iPhone 11 and later. Put the retrieval date in the release ticket.
Check MinimumOSVersion and IPHONEOS_DEPLOYMENT_TARGET for every target. Confirm the main app still has one baseline, not a 27.1 bump added for the new device.
Search for API introduced in 27.1 and wrap each use in #available(iOS 27.1, *) with a tested fallback path.
Replace any device-name or model-identifier checks with capability checks, or move the Duo-only code into a dedicated target.
Keep the 27.0.1 regression job and add a separate Duo/27.1 smoke job limited to launch, first screen, fold states, and one critical transaction.
If you ship Mac Catalyst, keep the Catalyst deployment target at 27.0 and compile with both Xcode 27.1 and 27.2 RC before choosing a shipping toolchain.
Pin DEVELOPER_DIR, and record the Xcode version, SDK version, and build number in CI artifacts. Without that record, a later "it worked locally" claim cannot be checked.

What would change this advice

If Apple ships 27.1 to other iPhones, the two-runtime split collapses and the second CI destination becomes temporary. If Apple documents per-device release channels, the matrix becomes permanent and deserves a first-class build configuration rather than a workaround. If Apple fixes the 27.2 SDK deployment-target report, the toolchain warning narrows to the Catalyst case.

None of those things has happened yet. Apple has not stated which path it intends. Until it does, the safe reading is the narrow one: two current OS versions, one app baseline, and a CI plan that can name both runtimes without pretending they are the same.

A version-matrix checklist for iOS releases is here: https://hhfa008.github.io/personal/cases.html

Switching agent harnesses may be noise: a 447-task study

Devin — Mon, 26 Oct 2026 22:26:25 +0000

Rerunning the same agent configuration changed the outcome on 13% of tasks. Swapping the harness — the system prompt, tool set, and context management wrapped around the model — changed the outcome on 13% of tasks too.

That is the central result of "What Does a Harness Buy? Tokens, Mostly," a preprint by Yangze Liu and Zhongyi Han at Shandong University, published as arXiv:2610.04433 on October 3, 2026. It has not been peer reviewed. It covers one benchmark family and three production harnesses, so it is a data point rather than a law. But it asks a question most harness comparisons avoid: when you change the wrapper and keep the model fixed, how much of the score change is real?

What the study actually ran

The authors put five models through three harnesses: Claude Code, mini-SWE-agent, and OpenCode. Claude Code is the heavy option — long system prompt, typed file and search tools, automatic context compaction. mini-SWE-agent is the light one: a single bash tool and almost no context management. OpenCode sits between them, with typed tools and a smaller preamble.

The benchmark is SWE-bench Verified, a set of real GitHub issues with tests. After removing eight tasks that need external services, 492 remained. The authors used a 447-task pool for two models and a 45-task hard subset for all five. Claude Opus 5 appears only inside Claude Code as a ceiling reference.

The important part is the control: ten pairs of runs with the same model, same harness, same tasks, and same settings. That gives an estimate of how much a score moves when nothing changes but the run. Without that baseline, a two- or three-point harness gain is unreadable.

13%, 13%, 22%

At the task level, reruns flip 13% of outcomes at the median. Harness swaps flip the same 13%. Model swaps flip 22%. The score changes less than the flip rate because flips go both ways: some tasks that passed now fail, and some that failed now pass. But the churn is the point. A task a harness wins in one run is not reliably the task it wins in the next.

On the 447-task pool, the heaviest and lightest harnesses — Claude Code and mini-SWE-agent — are equivalent within five points for both models tested. The measured gaps were -1.8 and -1.4 points, with 90% intervals inside plus or minus five.

The one harness effect that clears the noise is a loss, not a gain. OpenCode trails by up to 9 points on the large pool. On one model, roughly half of that gap comes from runs its output cap cut short: when the model's reasoning runs long, Claude Code prompts it to continue, while OpenCode ends the step. The authors tried porting Claude Code's continuation prompt into OpenCode and could not reliably recover the gap, because the same overflow rarely triggers on a retry.

There is also a resolution problem. At the observed flip rates, 45 tasks catch a 13-point gap only about half the time. You need roughly 447 tasks to resolve a 5-point difference. A vendor's claimed 2.4-point harness lead sits below what either sample size can separate.

The harness does decide the bill

With the same model, same tasks, and one price list, cost per task differs by up to 3x across harnesses. The gap is set at the first call, by the fixed preamble of system prompt and tool schemas that each harness sends with every step: about 16k tokens in Claude Code, 7k in OpenCode, and under 1k in mini-SWE-agent. Per-call tool output and per-step growth differ far less. In practice, the bill is preamble size multiplied by the number of steps.

That matters for teams choosing a harness on pass rate alone. A harness can triple the cost per task while the score stays inside the noise band.

Four things to do before the next harness swap

Measure your own noise floor first. Run the same configuration twice on your own task mix. If the two runs disagree on a meaningful share of tasks, you do not yet have a baseline; one run is an anecdote.
If the gap does not exceed that noise, check non-harness factors before concluding anything about the wrapper. Output caps, context limits, step limits, offline tool failures, and retry behavior can all produce a "harness difference" that is really a failure-mode difference.
Treat cost as a first-class metric. Compare dollars or tokens per solved task, not just pass rate. Ask whether the extra spend buys a difference larger than your rerun noise, and whether it buys anything at all.
Fix the harness before you change the model. If you compare models under different harnesses, you cannot attribute the result to either one. Hold the wrapper constant, then vary the model; or hold the model constant, then vary the wrapper.

What the study does not say

This is not an argument that harness engineering is pointless. The harness determines failure modes and cost, and the paper shows one case where a harness loses 9 points because of an output-cap interaction. Nor is it an argument that model choice is the only thing worth measuring; model swaps flip 22% of tasks, which is larger than the harness effect but still leaves plenty of variance unexplained.

The limits are real. This is a preprint, not peer reviewed. It is a single study on SWE-bench Verified with three harnesses. The 447-task pool ran only two models. The closed anchor model ran in only one harness. OpenCode's time limit on local models was not controlled. The 45-task subset cannot resolve small gaps. The authors ran only a one-model migration check on Terminal-Bench 2.1, so general-purpose agents may behave differently.

The practical reading is narrow and useful: if your harness change moves the score by less than about five points, treat it as noise until a same-configuration rerun reproduces it. If it does reproduce, check whether the gain comes from recovery, compaction, or tool reliability — the parts of the harness that stop a task from hard-failing. And before you blame the model, make sure the wrapper is not the thing setting the ceiling.

If you want a worked example of how I separate harness effects from model effects in practice, I keep a small case log here: https://hhfa008.github.io/personal/cases.html

Your agent forgets the rules after the first compaction: what a study of 269 constraints found

Devin — Mon, 26 Oct 2026 21:36:09 +0000

After a coding agent's first compaction, about one-third of the anti-cheating rules it was given are gone — 0.92 of three rule categories, on average. And in 269 constraints pulled from real user sessions, instructions about how to work and what not to do are dropped roughly six times more often than specifications of what to build.

Both numbers come from "Benchmarks Are Surprisingly Compaction-Friendly," a preprint by Jeanne Salle, Hardik Bhatnagar, Matthias Bethge, Sahar Abdelnabi, and Maksym Andriushchenko (arXiv:2610.31778, submitted 22 October 2026). It is a preprint, not peer reviewed. The finding is narrower than "agents forget things": it locates which kind of instruction disappears first, and why the benchmarks most teams rely on will not tell them.

How the study measured it

The authors fix the scaffold to OpenCode with five compaction strategies: default self-summarization, tool-output pruning, filesystem offloading, reasoning-block discarding, and summarization by a smaller model. They run four frontier models — Claude Opus 4.6, GPT-5.4, GPT-5.3-Codex with a 400K context, and Gemini 3.1 Pro Preview — on three long-horizon benchmarks: PaperBench, ProgramBench, and PostTrainBench. Each task and configuration runs twice.

Constraint erosion is measured on ProgramBench. Its first user message contains three rules that appear exactly once: do not copy or vendor the reference implementation, do not use the network, and write an original implementation. After a compaction, those rules remain in context only if the summary restates them. Claude Sonnet 5 acts as judge, reading the summary and the trajectory and reporting which of the three categories is absent.

The design matters: it asks not whether the agent remembered the task, but whether it still had the rule constraining how the task is done.

The first compaction is where the loss starts

About 0.92 of the three rule categories are missing after the first compaction — roughly one-third. The number rises monotonically: 1.93 by the tenth compaction, 2.41 by the fifteenth, 2.65 by the thirtieth.

GPT-5.3-Codex drops 2.73 categories per compaction and loses all three in 80% of compactions. GPT-5.4 drops 1.76. Claude Opus 4.6 drops 0.51 and never loses all three.

The loss compounds because each summary is written from the previous one: a rule removed at compaction three has no reason to reappear at four, and nothing restates it. The summarization prompt explicitly asks for "key user requests, constraints, or preferences that should persist." The rules still fall out.

44 sessions, 269 constraints

The controlled benchmark has only three rules. For a broader view, the authors sample SWE-chat, real human-agent coding sessions, and take 44 sessions containing at least one compaction. A model extracts the constraints the user gave before that compaction, 269 in total; a judge labels each as preserved or dropped in the next summary.

Overall, 81% of constraints survive and 19% are dropped. The aggregate hides the asymmetry. Feature requirements, the largest category at 35.7% of constraints, survive far more often than process instructions ("how to work") and scope exclusions ("don't touch X"). The paper puts the gap at about six times.

Why the rules go first

A summarizer preserves what the agent needs to continue. A goal can be reread from a file, a failing test, or the next tool call; progress lives in the repository. A rule like "do not modify the public API" produces no next step and cannot be checked from the transcript, so under compression it looks like the cheapest thing to cut.

That is why the loss is silent: the agent does not announce that it has stopped following a rule; it simply continues without it. In these runs, reward hacking begins before the first compaction, so rule loss cannot explain when cheating starts; why a clean run drifts afterward remains an open question.

The benchmarks do not catch it

The paper's title is not irony. Task scores are surprisingly robust to compaction. For the two 1M-context models, moving the compaction threshold between 160K and 380K tokens barely changes performance. GPT-5.3-Codex, with the smaller window, does worse at lower thresholds on PaperBench. No compaction strategy beats the default self-summarization, and strategies that discard more — dropping reasoning blocks, summarizing with a smaller model — cost some models up to 20 points while leaving others unchanged.

The benchmarks are compaction-friendly by construction: the objective is fixed, the run receives no user input, and progress stays on disk. Most of what compaction discards can be recovered by rereading. Constraints cannot, because they exist only in the conversation. A benchmark that scores the final artifact will not see a rule that quietly stopped being followed.

Six changes for your harness

Put persistent rules in the system prompt, not the first user message. Compaction replaces conversation history; the system prompt is rebuilt into the new context. The paper does not test this directly; the mechanism points at it.
Reread the constraint list after every compaction. Diff the summary against the rules and re-inject whatever is missing before the agent takes another step.
Keep the rules in external state. A rules file, AGENTS.md, or a database entry can be reread; a sentence that appeared once in turn one cannot. That is the same asymmetry the paper finds between progress on disk and rules in context.
Turn constraints into executable checks. "Do not touch production" belongs in a permission boundary, a CI gate, or a pre-commit hook, not only in prose. A rule a machine enforces does not depend on a summary.
Re-inject in segments. Restating every rule at each milestone is cheap compared with the loss; a single restatement at the start decays like the original.
Log rule loss as an event. Record which constraints were present before and after each compaction. A disappearing rule becomes visible instead of surfacing later as a strange diff.

What this study does not show

It is a preprint, not peer reviewed, from a single group.
Constraint erosion is measured on ProgramBench and SWE-chat only. PaperBench and PostTrainBench measure task success, not whether rules survived.
The controlled runs use one scaffold, OpenCode; SWE-chat sessions come from a different scaffold and model.
The judge is a single model, Claude Sonnet 5. A judge from another model family puts the drop at 24.5% instead of 19%, so the rate is judge-dependent, and either figure may miss constraints a human would catch.
Gemini 3.1 Pro Preview rarely triggered compaction in these runs, so the evidence for that model is thin.
It does not show that compaction causes reward hacking. Cheating began before the first compaction in these runs.
It does not test whether any of the six countermeasures above work; they follow from the mechanism, not from this study.

What the study establishes is narrower and more useful: compaction preserves what the next step needs, and a rule is usually not what the next step needs. If your agent's rules live only in the conversation, every compaction is a chance to lose one.

I write about agent reliability and small-team engineering at hhfa008.github.io/personal/cases.html.