DEV Community: Ivan Zdravkov

How much does an adapter forget?

Ivan Zdravkov — Sat, 31 Oct 2026 10:59:58 +0000

Less than expected, and the honest story is why that answer took three extra checks to trust. Across a 3x3 grid of LoRA rank and training steps on Qwen2.5-3B-Instruct, general-task retention never once dropped below the unadapted baseline.

Key takeaways

The registered kill-gate for H1 (forgetting grows with adapter rank and training steps) did not fire: retention rose 9.1 percentage points between the smallest and largest configuration, the opposite of the hypothesis.
Every one of nine grid cells showed general-task retention at or above the unmodified baseline, a pattern suspicious enough to investigate before trusting it.
Two checks ruled out ordinary bugs: the raw generated text showed no parsing artifact, and reloading the identical unmodified model twice produced zero differences in its answers.
The real gap was a missing seed control in the original grid. Retraining the two extreme configurations under new seeds showed the smallest configuration's exact 100% retention was a coincidence, while the largest configuration's gain replicated in three out of three independent runs.
A configuration with meaningful domain accuracy gain and high general-task retention existed in 8 of the 9 tested combinations.

What did the experiment measure, and what did it show?

A LoRA adapter fine-tuned on a narrow domain is supposed to specialize a model without erasing what it already knew. I registered two hypotheses to test that. H1: forgetting on an unrelated general task grows with adapter rank and training steps. H2: a configuration exists that gains real domain accuracy while keeping general-task retention above 95%.

The domain task was Overruling, a binary legal classification: does a sentence overrule a prior case holding, yes or no. The general task was 120 fixed multiple-choice questions sampled across 57 MMLU subjects, evaluated identically before and after every fine-tune. The grid covered LoRA rank 4, 16 and 64 crossed with 50, 200 and 500 training steps on Qwen2.5-3B-Instruct, nine cells, one training run per cell.

KG2 fired comfortably: 8 of 9 cells reached over 10 percentage points of domain accuracy gain while keeping general-task retention above 95%. KG1 did not fire at all. Retention moved from 100.0% at the smallest configuration to 109.1% at the largest, a rise, not the registered 5-point drop that would have confirmed forgetting scales with rank and steps.

What ruled the pattern out as a bug?

A 9-point rise sounds like good news, but the full table was stranger than one rise: all nine cells landed at or above the 100% baseline, never once below it. Under ordinary sampling noise on a 120-question eval, roughly half the cells should have landed under the baseline by chance alone. A number that only ever moves one direction is a reason to check the method before writing the conclusion.

The first suspect was parsing: if the unmodified model answered more verbosely and the scoring regex failed to extract a letter from it, that alone could manufacture a fake accuracy gap. Comparing the raw generated text from the baseline model against the highest-gain configuration on the same 120 questions ruled this out: both answered in a clean one-letter format almost every time, 1 unparseable reply out of 120 for the baseline and 0 out of 120 for the adapted model.

The second suspect was the 4-bit quantized loading itself. Every cell reloaded the base model fresh from the same checkpoint, and quantization kernels can in principle introduce small numerical differences between loads. Loading the unmodified model twice, independently, and comparing its raw answers on the same 120 questions showed zero differences. Loading and quantization were fully deterministic here, and could not explain the pattern either.

What did reseeding the two extreme configurations show?

The original grid trained each of the nine cells exactly once, with no fixed seed for the LoRA adapter's random initialization. That is the real gap, not a bug but a missing control. I retrained the smallest configuration (rank 4, 50 steps) and the largest (rank 64, 500 steps) under two new explicit seeds each.

The smallest configuration's retention landed on both sides of the baseline across the new seeds, 98.5% and 104.5%, consistent with a near-zero true effect and explaining why the original single run happened to land on exactly 100.0% by coincidence. The largest configuration stayed above baseline in both reseeded runs, 110.6% and 106.1%, for three out of three independent trainings above 100% in total. That pattern replicates. It is not the artifact of one lucky run, and it is the opposite of what H1 predicted. The two cells chosen were the extremes of the grid on purpose: if the real effect were forgetting that scales with rank and steps, the gap between them was the place most likely to show it clearly, and it still did not.

What does this mean for fine-tuning a narrow domain?

Within this tested range, LoRA fine-tuning on Overruling did not cost general capability, and the strongest configuration mildly, reproducibly improved it instead. I do not have a confirmed mechanism for why. A plausible but untested explanation is that training toward a short, committed answer format transfers a general "commit to an answer" behavior that happens to help multiple-choice scoring too, independent of any actual knowledge gained. A control run, fine-tuning on irrelevant or randomly labeled data to isolate whether the effect comes from this domain specifically or from LoRA training in general, was not part of this registered run. It is the obvious next step, not a claim this one makes.

The practical lesson is less about LoRA and more about measurement discipline: a result that looks too clean, with every cell moving the same direction, earned three separate checks before it earned a sentence in this article. Two of those checks found nothing. The third found a real design gap and, underneath it, a real effect.

FAQ

Does this mean LoRA fine-tuning never causes forgetting?
No. It means forgetting was not observed in this specific range: one 3B model, LoRA rank up to 64, up to 500 steps, one narrow binary-classification domain. A longer generative domain task, a different model family, or a larger rank could behave differently.

Why only 120 general-task questions?
It kept nine training-and-evaluation cycles plus the follow-up reseeding runs inside a reasonable time on a single consumer GPU. The tradeoff is coarseness: 120 questions across 57 MMLU subjects cannot detect forgetting that is small and spread thinly across many subjects.

What is the best explanation for the improvement?
No confirmed one. The leading untested hypothesis is a general answer-format effect from fine-tuning toward short, committed responses, rather than genuine new general knowledge.

Will the full grid be rerun with multiple seeds?
Not committed yet. The two-cell reseed check here was a targeted fix for the specific anomaly this run produced, not a full redo of all nine cells.

References

Yifeng Xiong, Xiaohui Xie. OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning, 2026. arxiv.org/abs/2510.13003
Seungju Back et al. Understanding LoRA as Knowledge Memory: An Empirical Analysis, 2026. arxiv.org/abs/2603.01097

Who gets to set your kill-gate?

Ivan Zdravkov — Sat, 31 Oct 2026 10:59:52 +0000

Context. A reader asked a plain question under one of my articles: what does a kill-gate even look like for a social robot, when "did the conversation work" does not give you a clean number the way a benchmark does. The question came from a reader building Bite, a continual-learning project for social robots. Answering it honestly surfaced a different question underneath, one that applies well beyond robots. Anything built between pure research and a product that will eventually ship raises the same problem: who is allowed to decide the threshold the research has to pass?

Decision. The threshold has to come from the research question itself, never from whatever a later product, client, or roadmap decides to call success. A kill-gate exists to tell me honestly whether an idea works, not to confirm a number someone already needed. The moment an external target gets to define "good enough," the measurement stops being a test and becomes a formality the work is expected to pass.

In practice this means I write the threshold before I know whether a product, an article, or a client deliverable depends on the result. I write it from the mechanism I am trying to understand: how much does fine-tuning actually cost in forgetting, where does a masking layer silently miss, how much does quantisation degrade a model's output. None of those questions care what a roadmap wants the answer to be, and the threshold should not either.

Alternatives considered. The obvious shortcut is to let the application define success first and work backward: decide what counts as "fixed," "retained enough," or "accurate enough" based on what the product needs, then run the experiment to confirm it. I rejected this, and not on principle alone. My EWC-versus-L2 comparison is the concrete case. The registered retention gate technically fired, both methods looked like they reached the same bar. If I had stopped at that single pass or fail number, the way a product requirement would, the article would have reported a tie. The full picture, the cost each method paid on the new task to reach that same retention, told a different and more useful story. A threshold borrowed from an application would have hidden exactly the part that mattered.

The same risk shows up anywhere a pass or fail number is the only thing reported. A silent miss, like a recognizer never active for one language, produces zero errors and a clean-looking pass rate right up until someone checks per entity type instead of on average. The kill-gate has to be shaped by the question, not by the report a stakeholder wants to see.

Consequences. What this buys me is a measurement I can trust when it disagrees with what I hoped to find. The canary experiment that caught a masking recognizer silently missing an entire language only caught it because the gate was built around detection fidelity, not around "does the pipeline run without errors." A product-shaped threshold would have called that pipeline healthy.

What it costs is control over the timeline. Sometimes the honest answer is that a method does not clear the bar I would have liked, and the product, the article, or the client conversation has to adapt its claims to the result, not the other way around. That is slower than writing the requirement first and designing a test that is guaranteed to satisfy it. It is also the only version of the measurement worth publishing.

A canary for silent PII misses

Ivan Zdravkov — Sat, 31 Oct 2026 10:59:24 +0000

Testing masking recall one PII type at a time, instead of as one average, found two real detection gaps a single number would have hidden, including a credit card recognizer that was never active for Bulgarian text at all.

Key takeaways

Planting around 40 synthetic examples per PII type and measuring recall separately showed a 30 to 100 percentage point gap between the best and worst covered type, a spread no single aggregate recall number reveals.
The same canary caught an injected regression immediately: disabling one pattern recognizer dropped recall for its type by 75 points while every other type's recall stayed exactly unchanged.
The clearest finding was not in either registered hypothesis: a credit card recognizer was wired up for English only, so a card number mentioned in a Bulgarian sentence passed through completely undetected, with no error anywhere.
A second, narrower gap hit English too: card numbers with a length outside the recognizer's expected range were missed regardless of language.
Every example is synthetic, generated from a frozen, seeded script, not drawn from any annotated dataset or real registry.

Why measure recall per type instead of as one number?

A reader comment on an earlier note, the masking layer matters more than the model, made a point I had not stated clearly: a bug in restoring a masked value gets caught fast, because the output looks visibly wrong. A detector that quietly fails to flag a phone number in the first place looks like nothing at all. The reader's suggestion was direct: stop reporting one recall number and start continuously running synthetic, "planted" PII through the pipeline, checking recall per type.

That is a fair correction. An aggregate recall of, say, 0.85 can describe a system that is uniformly decent, or one that is excellent at emails and blind to credit cards. The two situations call for different fixes, and the average cannot tell them apart. This note is the experiment that comment asked for.

What does a planted-PII canary actually look like?

The method is small on purpose. A seeded script generates synthetic values per entity type, given name, surname, email, phone, card number, date, city, street, and a checksum-valid but entirely fake Bulgarian personal number, then drops each one into a neutral carrier sentence such as "Please contact {name} about the renewal." The exact character span is recorded at generation time, so no manual labelling happens afterward.

Nothing here comes from an annotated dataset or a real registry; every value is either library-generated or built from a public formula, the same discipline as the recall measurements in why I pre-register a kill-gate for every experiment. The set is run through the same ensemble of standard open components measured there, Presidio orchestration, an XLM-RoBERTa NER model, and pattern recognizers for formatted identifiers, unchanged, except for one deliberate change described below.

What did the per-type recall actually show?

Across 240 English and 280 Bulgarian examples, recall by type ranged from 1.000 (email, both languages) down to 0.700 (credit card, English) and 0.000 (credit card, Bulgarian). The spread between the best and worst type was 30 points in English and 100 points in Bulgarian, well past the 20-point bar registered before the run. A system report built from one recall figure would have shown something in the comfortable middle of that range and called it a day.

Why did Bulgarian credit card detection fail completely?

Not a partial miss, a complete one, so I checked the recognizer registry directly rather than trusting the number. The credit card recognizer was registered only under the English language configuration. Several country-specific identity recognizers in the same stack are explicitly duplicated across every supported language; this one was not. A card number sitting in a Bulgarian sentence never reaches a recognizer that could have caught it, no exception thrown, no failing test, because nothing had tried that exact combination before. That is precisely the shape of failure the original comment described: invisible until someone specifically looks for it.

Does a canary like this actually catch a regression?

To find out, I reran the same planted set with the phone recognizer deliberately switched off, simulating an accidental regression during a future change. Phone recall dropped 75 points in both languages. Every other type's recall moved by exactly zero. The canary isolates the break instead of producing generic noise, which is the property that makes it worth running again, not just once for an article.

What's next?

The plan, the generator script, the raw numbers, and the evaluation code are published alongside the earlier experiments at github.com/divanski/kill-gate. "Continuously" above still means a reproducible, seeded script rather than a job on a schedule; turning it into one is the next step, now that the one-off version has already found something worth watching for.

FAQ

Is this the same masking stack as earlier articles?

Yes, the same ensemble of standard open components measured in the kill-gates article, unchanged except for deliberately removing one recognizer to test whether the canary would notice.

Why only English and Bulgarian?

Those are the two languages already covered by the earlier recall measurements on this stack. The method applies to any supported language; this run simply did not extend further.

Is 40 examples per type enough to trust these numbers?

Enough to catch a complete gap like 0% recall with confidence. Not enough to pin down a precise recall figure for a type that is merely mediocre rather than absent; a larger run is the natural next step.

Could other languages have the same kind of gap?

Possibly, and this experiment did not check. It tested the two languages already in scope, found one clear miss, and stopped there rather than claiming a broader audit it did not do.

What triggered this experiment?

A comment from a reader on the masking layer article, arguing that visible restoration bugs get more attention than silent detection misses. The argument was right, and this note is the direct answer to it.

Your model is not the product. The pipeline around it is.

Ivan Zdravkov — Wed, 28 Oct 2026 06:59:15 +0000

Swap the model and almost nothing changes. Swap the masking layer, the permission model, or the monitoring, and the system stops running, starts leaking, or quietly drifts wrong. The part everyone demos is the part that matters least.

Key takeaways

Model choice sits behind an abstraction in a well-built system, which means it is the cheapest part to change and the least differentiating.
The expensive 90% is plumbing: masking before any prompt, permissions that travel with the person, audit trails, kill-gated evaluation, drift detection, none of which shows up in a demo.
Research on production ML lifecycles increasingly measures workflow completion, artifact traceability and drift recovery, not just predictive accuracy, because that is where systems actually fail.
A team that can name its out-of-scope cases and its monitoring signal is pipeline-first; a team that can only name its model is not, yet.
None of this is an argument against caring about models. It is an argument about where the engineering hours actually need to go.

What is "the pipeline" here, concretely?

Everything between a request arriving and an answer leaving, minus the forward pass itself. Ingestion and normalisation of whatever format the input actually shows up in. A masking boundary that decides what a model is ever allowed to see, the subject of why the masking layer matters more than the model. An authorisation layer that scopes what an assistant can reach, described in what an internal MCP server actually solves and permissions that travel with the person. Logging, evaluation and the discipline of registering a kill-gate before trusting a number, covered in why I pre-register a kill-gate for every experiment. None of that is the model. All of it is what makes the model usable inside a real company.

The model sits in the middle of this as one component with a clean interface: text in, text out. Everything upstream and downstream of that interface is where the actual engineering risk lives, because it is where identity, authorisation, sensitive data and operational failure modes are handled.

Why does swapping the model change so little?

Because a well-built pipeline puts the model behind an abstraction on purpose. One of the four principles I hold myself to on this site is no vendor lock-in: open models behind an interface, swappable without rewriting anything around them. If swapping the model breaks the system, the system was never really decoupled from it, and that coupling is a design defect in the pipeline, not a fact about models.

This is not just a personal preference. A 2026 framework for LLM-orchestrated data pipelines measures success across workflow completion, artifact traceability, deployment readiness, reproducibility and drift recovery, deliberately alongside predictive performance rather than instead of it. Lifecycle reliability, not raw model quality, is the axis the field is now building evaluation around, because that is the axis production systems actually fail on. A model a few points better on a leaderboard rarely rescues a pipeline with no masking, no permission model and no drift detection; a mediocre model behind a disciplined pipeline usually survives long past the leaderboard's relevance.

Where does the 90% actually go?

Concretely, in the systems I have built and measured this year: designing the masking boundary took longer than building the AI features it protects, because free text detection has real failure rates that have to be measured, not assumed, as the entity-level numbers in detecting PII before the model sees it show directly. The permission model, deciding that an assistant inherits its user's rights rather than owning any of its own, is infrastructure work with no model in it at all. Kill-gate discipline, writing a threshold down before a single experiment runs, is process, not code, and it is the difference between a result and an impression. None of these three things appears in a five-minute demo. All three decide whether the demo's system survives a normal Tuesday.

The pattern generalises past my own work. Structured evaluation of LLM agents keeps finding that failures cluster in tool invocation, planning under constraints and long-horizon coordination, categories that live entirely in the pipeline around a model, not in the model's raw capability. Scaffolding does not reliably fix this either; the same research finds that additional orchestration does not consistently improve reliability, which means the fix is disciplined engineering around the model, not more layers stacked on top of it.

FAQ

Does this mean model choice does not matter at all?

No. It means model choice is the reversible, comparatively cheap decision, while masking, permissions and monitoring are expensive to retrofit and dangerous to skip. Spend engineering hours where mistakes are costly and hard to undo; treat the model as the part you can change your mind about later.

What is the fastest way to tell if a team is pipeline-first or model-first?

Ask what happens on malformed input and who gets paged when output quality drops silently. A pipeline-first team answers with a specific mechanism. A model-first team answers with the name of the model, which is not an answer to either question.

Isn't "the pipeline" just a rebrand of MLOps?

Overlapping, not identical. MLOps usually centres training and deployment tooling; the pipeline argument here includes things MLOps tooling rarely owns, such as the masking boundary and the permission model, both security decisions as much as operational ones.

How do you get budget for work that produces no visible feature?

Frame it by what it prevents, not what it adds: a masking failure is a data breach, a missing permission model is unlimited blast radius from one prompt injection, a missing kill-gate is a metric nobody can trust. Those are costs a stakeholder already fears; the pipeline work is the line item that keeps them from materialising.

References

Aueaphum Aueawatthanaphisut, Badri Raj Lamichhane. Trustworthy Self-Composable Big-Data-as-a-Service: An LLM-Orchestrated Multi-Agent Framework for Automated Data Engineering, AutoML, MLOps Deployment, and Drift-Aware Lifecycle Optimization, 2026. arxiv.org/abs/2606.17915
Wael Albayaydh, Rui Zhao, Ivan Flechais. Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents, 2026. arxiv.org/abs/2607.05775

What the EU AI Act actually requires of a masking layer

Ivan Zdravkov — Wed, 28 Oct 2026 06:58:39 +0000

The Act never uses the word masking. It does not need to. Three articles, on data governance, record-keeping and human oversight, each describe a property a masking layer either has or does not, and retrofitting any of the three after a system is running costs far more than building toward them from the start.

Key takeaways

Article 10 permits processing special category personal data only for bias detection, and only under conditions that already sound like a masking layer: strict security measures, no transmission to third parties, deletion once the purpose is served.
Article 12 requires automatic logging over the lifetime of a high-risk system, with no specified retention period, which means the logging capability has to exist before anyone decides how long to keep it.
Article 14 requires a human overseer to be able to interpret, override and stop a system, which fails silently if nothing in the pipeline explains what data reached a decision.
Recent research on regulatory compliance argues that audit-time, binary compliance is structurally the wrong model for a running system; the requirement is a continuous signal, not a certificate filed once.
None of this depends on whether a given system is formally classified as high-risk. The three properties are worth building toward regardless, because they are expensive to add later and cheap to design in now.

What does the Act actually say about data?

Article 10 governs training, validation and testing data for high-risk systems, and most of it reads like ordinary data-quality discipline: datasets have to be relevant, sufficiently representative and, to the best extent possible, free of errors. The part that matters for a masking layer is narrower and stricter. The Act allows processing of special category personal data, health, ethnicity, and similar sensitive fields, but only for the specific purpose of detecting and correcting bias, and only when a list of conditions holds: no alternative data would serve the purpose, technical limits restrict reuse, strict security measures apply, the data is never transmitted to third parties, and it is deleted once the bias has been corrected.

Read as an engineering requirement rather than a legal one, that is a masking and lifecycle policy with a narrow, purpose-bound exception carved into it. A system that cannot say which fields are sensitive, cannot restrict a specific processing purpose to a specific dataset, and has no deletion trigger tied to that purpose ending, has no way to satisfy this even if the underlying use case is legitimate. The masking boundary I described in why the masking layer matters more than the model is the component that makes the distinction between sensitive and non-sensitive fields exist at all; Article 10 is one concrete reason that distinction has to be enforced, not merely documented.

What has to be logged, and why does that matter for masking?

Article 12 requires that high-risk systems technically allow automatic recording of events over their lifetime, so that the system's functioning is traceable. Notably, it does not specify a retention period. That omission is easy to misread as leniency; in practice it means the logging capability has to exist unconditionally, because nobody can decide how long to keep records that were never captured in the first place.

A masking layer sitting at the boundary between internal systems and a model is naturally the place this logging belongs. It already sees every field that leaves the boundary, so it is the cheapest point to record what left, in what form, and under what placeholder mapping. Building this after the fact means reconstructing months of decisions from application logs that were never designed to answer a compliance question. Building it into the boundary from day one means the audit trail is a side effect of normal operation, which is the only version of record-keeping that survives contact with an actual audit.

What does human oversight mean for an automated pipeline?

Article 14 requires that a human assigned to oversee a high-risk system can understand its capacities and limitations, remain aware of the tendency to over-rely on its output, correctly interpret what it produced, and override, disregard or stop it. None of that is possible if the system presents a final answer with no visible path to the data behind it.

This is where masking and oversight pull in the same direction rather than against each other. A masking layer that replaces sensitive values with stable placeholders, the design I described in the same earlier article, gives a human reviewer something concrete to check: which fields were substituted, whether the substitution was correct, and whether the model's answer still makes sense once the real values are restored. An oversight requirement bolted onto a system with no visibility into what the model actually saw reduces to a person clicking approve on a black box, which satisfies the letter of the article and defeats its purpose.

Why does audit-time compliance fail for a running system?

A 2026 paper on regulatory compliance for deployed LLM systems argues that treating conformity as a binary, audit-time verdict is structurally mismatched to a requirement built around ongoing oversight and the detection of behavioural drift. The authors propose governance from metrics: compliance as a continuous signal derived from runtime observability, not a static assessment filed once and forgotten. Their own validation, a panel of small on-premise models scoring compliance criteria, found agreement rates between 51.5% and 69.1% depending on the model, with no single model dominating across every criterion, which is itself evidence that compliance judgement is not a solved, one-shot problem even with dedicated tooling.

The relevance here is architectural rather than a specific product recommendation: whatever machinery ends up checking compliance, it needs a continuous stream of real signal to check, not a document written before launch and never revisited. A masking layer with logging built in produces exactly that stream as a by-product of running. A compliance binder produced once for an audit does not, and updating it by hand every time the system changes is precisely the kind of expensive retrofit the rest of this article has been describing.

FAQ

Does this mean my system is legally classified as high-risk under the Act?

Not necessarily, and that classification depends on specifics this article does not cover, listed in the Act's Annex III. The three properties discussed here, purpose-bound handling of sensitive data, unconditional logging and genuine human oversight, are worth building regardless of classification, because the cost of retrofitting them after a system is running is far higher than the cost of designing them in.

Is a masking layer enough to satisfy Article 10 on its own?

No. Article 10 is about the training, validation and testing data pipeline as a whole, including bias examination that a masking layer does not perform. What a masking layer provides is the enforcement mechanism for the purpose-bound, security-conditioned handling that the special category data exception requires; the bias assessment itself is separate work.

Do I need a dedicated compliance tool like the one in the cited paper?

Not necessarily this specific one; it is research software, not a product recommendation. The useful idea to take from it is the framing: compliance as a continuous signal your system produces, not a document written once. Whether that signal comes from a purpose-built tool or from disciplined logging at your masking boundary is a separate decision.

Is this legal advice?

No. I am reading the Act's text as an engineer deciding what to build, not as a lawyer assessing legal exposure. Treat the article numbers as a starting point for a conversation with counsel, not as a substitute for one.

References

Regulation (EU) 2024/1689 (Artificial Intelligence Act), Official Journal version of 13 June 2024. Articles 10, 12 and 14. Article 10, Article 12, Article 14.
Jehanne Dussert. Who judges the judges? Governance from metrics: a runtime framework for continuous LLM compliance monitoring, 2026. arxiv.org/abs/2605.24737

What EWC buys you that flat L2 does not

Ivan Zdravkov — Wed, 28 Oct 2026 06:52:48 +0000

A flat L2 penalty can match EWC's exact Task A retention number on Permuted MNIST: 97.63% against 97.63%. It does this by collapsing Task B accuracy to 18%, against EWC's 97%. The same headline number hides two very different costs.

Key takeaways

A registered gate asked whether flat, uniform L2 regularisation matches EWC's Fisher-weighted penalty at comparable cost. The cheapest tested setting landed within 0.86 points of EWC's Task A accuracy, inside the registered 3-point bar - the gate fired.
No tested setting met the registered "fair budget" of costing at most 1 point of Task B accuracy. The cheapest one still cost 2.70 points, over six times what EWC paid.
At a stronger setting, flat L2 hits EWC's exact Task A number, 97.63%, while Task B accuracy collapses to 18.42% against EWC's 97.43%.
Fisher weighting is not required to match EWC's retention number in this setup. It is required to match it without wrecking the model's ability to learn the new task.

What did the registered gate actually test?

An earlier experiment found that Elastic Weight Consolidation (EWC, Kirkpatrick et al., 2016) recovers 98.0% of the accuracy a small network loses to catastrophic forgetting on Permuted MNIST. EWC works by penalising drift away from Task-A-optimal parameters, weighted by a diagonal Fisher information estimate: parameters that mattered more to Task A get pulled back harder. That estimate costs something to compute, 1,000 samples and a forward-backward pass through each. The open question this experiment registered: does the Fisher weighting itself earn that cost, or would any quadratic pull-back toward the old parameters, uniform and uninformed, do just as well?

The registered kill-gate compared a flat L2 penalty, same shape as EWC's but with every parameter weighted equally, against EWC's actual result. To keep the comparison honest, the plan fixed a rule before running anything: search a range of L2 strengths, and report the one whose cost to Task B accuracy sits closest to, but under, 1 percentage point, matching EWC's own 0.41-point cost. If Task A accuracy at that point comes within 3 points of EWC's, the claim that Fisher weighting matters would be rejected.

Where did the gate fire, and what did that hide?

The registered gate fired, literally. Across six tested strengths (0.01 to 1000, log scale), the cheapest one, lambda=0.01, reached 96.77% Task A accuracy, 0.86 points under EWC's 97.63%, comfortably inside the 3-point bar. By the letter of the pre-registered rule, the hypothesis that Fisher weighting provides a measurable advantage is rejected.

But the fair-budget rule that gate was supposed to respect was never actually satisfied. That same lambda=0.01 setting cost 2.70 points of Task B accuracy (97.84% down to 95.14%), not the 1 point the plan called "fair." Every cheaper option would have cost less on Task B but also recovered less of Task A; every stronger option recovered more of Task A at a steeper Task B cost. No point in the sweep lived inside the budget the plan defined as comparable to EWC. fair_pick_in_budget: false sits in the raw metrics file precisely so this gets reported rather than smoothed over.

What does the lambda=10 result reveal?

One point in the sweep makes the trade-off impossible to miss. At lambda=10, flat L2 reaches 97.63% Task A accuracy, identical to EWC to two decimal places. Read as a single number, EWC and flat L2 look interchangeable. Task B accuracy at that same setting is 18.42%, against EWC's 97.43%. The network did not learn to protect what mattered on Task A while staying free to absorb Task B. It nearly stopped learning Task B at all, and freezing a network is a cheap way to protect old knowledge from any source.

This is the shape a naive comparison misses: pick one number, retention on the old task, and two very different mechanisms can look tied. Add the number nobody asked for and the picture reverses completely. It also explains why the sweep never found a point inside the fair budget: Task A accuracy rises smoothly toward EWC's figure as lambda climbs, but Task B accuracy falls off a cliff well before Task A catches up. There is no gear between "too weak to matter" and "strong enough to freeze the network," at least at this granularity of search.

So does Fisher weighting matter?

Both, depending on which question is being asked. If the question is "can a simpler method hit EWC's retention number," this experiment says yes, at least in this mild Permuted MNIST setting, with enough uniform pressure. If the question is "can a simpler method hit that number while leaving the network able to learn the next task," the answer across every tested setting is no. Fisher weighting is not a hard requirement for the headline number. It is the reason EWC can post that number while still being useful for anything after it.

The distinction matters beyond this one benchmark. Any report that states only "method X preserved Y% of old-task accuracy" is stating half a result. The other half, what the method spent to get there, decides whether it is a usable continual-learning method or a network that has quietly stopped learning. A retention number without its corresponding new-task cost is not a false claim, but it is an incomplete one, and this experiment is a case where the missing half changes the conclusion entirely.

FAQ

Does this mean EWC is unnecessary, and flat L2 works just as well?

Not for the actual goal of continual learning. Flat L2 can copy EWC's Task A number, but only by paying far more of the model's capacity to learn new tasks, an option that defeats the point of learning continually in the first place. The two methods are not interchangeable once both accuracy numbers are read together.

Why sweep several L2 strengths instead of tuning one directly to match EWC?

Tuning a single value to match a known target after seeing the result is exactly the hindsight pre-registration exists to prevent. The sweep and the fair-budget rule were both fixed before any run, so the reported pick is the honest output of that rule, not a value chosen because it looked good afterward.

What prompted this specific comparison?

A correspondent, following up on the earlier EWC experiment, asked directly whether a flat L2 baseline had been tested. It had not. Answering from intuition would have been easy and probably wrong in an interesting way, given how the lambda=10 result turned out, so the honest option was to register a plan and run it rather than guess.

What would make this comparison more convincing?

A finer sweep near the two extremes that bracket the trade-off, more task pairs than one, and ideally a network trained from scratch by the same author, since this Permuted MNIST setup is a mild benchmark where forgetting itself was measured, not catastrophic, in the original experiment.

References

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, Raia Hadsell. Overcoming catastrophic forgetting in neural networks, 2016. arxiv.org/abs/1612.00796

What an internal MCP server actually solves

Ivan Zdravkov — Wed, 28 Oct 2026 06:52:09 +0000

An internal MCP server does not make a model smarter. It solves a governance problem: without one, an assistant either guesses from its training data or receives internal data through copy-paste, and neither path has authentication, permissions or an audit trail. The server is where those three things finally get a place to live.

Key takeaways

Before any tool access, an assistant answers from model memory, and employees fill the gap by pasting internal data into chat windows, which is an unmanaged leak channel.
The Model Context Protocol standardises how assistants discover and call tools, turning N assistants times M systems into N plus M.
Typed schemas matter more than clever prompts: the server validates every call against a contract and rejects what does not fit, instead of hoping the model behaves.
A survey of 247 papers on agent security finds prompt injection and tool-mediated control-flow hijacking dominate, and that current defences compose poorly, so enforcement belongs server-side.
Per-person authentication and role-scoped tool lists turn an assistant into something that inherits its user's rights instead of having rights of its own.

What breaks before there is an MCP server?

Two failure modes appear in every company that gives employees AI assistants without giving those assistants data access. The first is confident guessing: the model answers questions about internal state from its training data, which contains nothing about the company, so the answers are fluent and wrong. The second is worse: people close the gap themselves by pasting customer records, contracts and internal reports into whatever chat window they use. Copy-paste is an integration layer with no authentication, no scoping and no log, and it appears wherever an official path is missing.

Building official paths one at a time does not scale either. Each assistant product has its own plugin format, each internal system has its own API, and every combination needs custom wiring. The integration count grows as the product of the two, and every integration is another place where credentials and permissions are handled slightly differently. What is missing is not capability but a standard boundary, one place where access is granted, scoped and recorded.

What does the protocol actually standardise?

MCP, the Model Context Protocol, standardises the conversation between an AI client and a tool server: how the client discovers what tools exist, what parameters each tool takes, and how calls and results travel. A server written once works with any client that speaks the protocol, which collapses the integration problem from N times M to N plus M. The protocol carries typed tool definitions, so every tool declares a schema for its inputs, and the server validates calls against that schema before anything executes.

How far this stretches is best shown by a hard case. A 2026 paper describes controlling scientific instrumentation, including electron microscopy workflows, through MCP: a reference server exposing 30 typed tools, with a schema-bound surface that validates requests against physical bounds before any command reaches an instrument, precisely because natural-language interfaces "can emit physically unreasonable arguments". If schema-bound MCP control is trusted to keep an agent from sending a physically impossible command to a microscope, the same pattern covers reading order data or querying a contract database. Business systems are the easy case.

Why do typed schemas matter more than clever prompts?

Prompt instructions are requests; schemas are contracts. A system prompt that says "only query orders from your own region" depends on the model honouring it under every phrasing a user can produce. A tool whose schema requires a region parameter that the server fills in from the authenticated user's profile does not depend on the model at all. The difference decides what an attacker or a careless user can reach.

The instrumentation paper cited above is built on that observation: arguments are validated against declared bounds before dispatch, and invalid calls fail at the boundary instead of reaching the instrument. The business translation is direct. Models produce plausible parameters, and plausible is not the same as valid: a date range spanning ten years, an identifier from the wrong system, a status value that sounds right and does not exist. Server-side validation catches all of it in one place, and the assistant gets a structured error it can correct from. My rule for the internal server is that nothing the model sends is trusted: every call is checked against the schema, and everything the server-side context already knows, identity, role, scope, is filled in by the server rather than requested from the model.

Where does security actually live?

Research on agent security keeps arriving at the same place. A 2026 survey synthesising 247 papers finds that prompt injection and tool-mediated control-flow hijacking still dominate the threat landscape, and its verdict on defences is sobering: current mechanisms are useful building blocks but remain weakly compositional. Stacking model-level mitigations does not add up to a guarantee. The authors argue for explicit trust boundaries, principled privilege control and provenance-aware state management, which are properties of infrastructure, not of prompts.

An internal MCP server is where those properties become enforceable. Each person authenticates with their own key; the design leaves no shared service account behind which everyone could hide. The tool list an assistant sees is derived from its user's role: a tool that is not listed cannot be called, which is a stronger statement than a tool that is listed but forbidden. Every call is logged with who, what and when, which turns incident response from archaeology into a query. And the same boundary is the right place for the masking layer I described in why the masking layer matters more than the model: tool results pass through masking before they reach an external model, so grounding the assistant does not become a new leak.

The permission model behind this is the subject of its own article: permissions that travel with the person.

What does this look like in practice?

In the systems I work on, an internal MCP server is currently in testing. It sits in front of the company's data, and the design is that colleagues connect their own assistants to it: each person authenticates with a personal key, their role decides which tools their assistant can list and call, and answers come from the systems instead of from the model's imagination. Read-only tools come first, because a wrong read costs a correction while a wrong write costs an incident.

Not everything is solved. Injection through retrieved content remains the open flank: a tool result can carry text that tries to steer the model, and the survey's finding that defences compose weakly matches what I see in testing. The mitigations in the design are scope and privilege, not cleverness: an assistant reads what its user could read, writes almost nothing, and the interesting attack surface stays as small as I can keep it. I do not yet have a monitoring signal that would reliably tell me an injection attempt happened rather than a user asking something odd. That is measurement work of the kind I described in the demo is the easy part, and it is next on the list.

FAQ

Is an MCP server just another API wrapper?

Structurally yes, practically no. The difference is discovery and client support: any MCP-speaking assistant can list the tools, read their schemas and call them without custom integration work per client. A REST API needs bespoke wiring for every assistant product; an MCP server needs none, which is what makes the N plus M economics work.

Do we need MCP if our systems already have good APIs?

Yes, if you want assistants to use those systems. Existing APIs were designed for developers, not for language models: no discovery, error formats models cannot correct from, and authentication schemes assistants cannot complete. The MCP server is the adapter layer that presents your systems in a form a model can use safely.

Is it safe to let an assistant write to internal systems?

Treat writes as a separate, later decision. Read-only tools deliver most of the value at a fraction of the risk, and role-scoped reads limit even that. When writes become necessary, put them behind narrow schemas, server-filled identity fields and human approval steps, and log everything. A wrong read costs a correction; a wrong write costs an incident.

Does an internal MCP server need a masking layer?

Yes, whenever the assistant behind it talks to an external model. Tool results are exactly where sensitive data enters the conversation, so they pass through masking at the server boundary before leaving the infrastructure. Grounding an assistant in real data and protecting that data are the same project, solved at the same choke point.

References

Roberto dos Reis, Vinayak P. Dravid. Schema-Bound LLM Control of Scientific Instrumentation through Model Context Protocol Skills, 2026. arxiv.org/abs/2607.17012
Yuchen Ling, Shengcheng Yu, Zhenyu Chen, Chunrong Fang. Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation, 2026. arxiv.org/abs/2606.10749

The real trade-off between an API model and a local one

Ivan Zdravkov — Wed, 28 Oct 2026 06:46:18 +0000

An API model costs nothing until the bill arrives, and a local model costs everything up front: hardware, quantisation loss, engineering time, before it answers a single question. Which one is cheaper depends on volume and constraints, not on which sounds more modern.

Key takeaways

The break-even point between API and local hosting depends on request volume and GPU utilisation, not a fixed rule of thumb. A card that sits idle most of the day never pays for itself, however cheap its per-token math looks on a spreadsheet.
Real numbers from a registered quantisation experiment on an RTX 5070 Ti (16GB): Q4_K_M quantisation delivered 2.22x the throughput of fp16 with 0.9835 average embedding similarity, a genuine trade, not a free lunch.
API providers get batching economies of scale that a single-GPU local deployment cannot replicate at low request volume, so a naive per-token price comparison is not comparing like with like.
Privacy and data residency requirements can override the entire cost calculation. Some data cannot leave the building at any price, and no API pricing tier changes that.

What does an API call actually cost?

The advertised per-token price is the smallest part of the real cost. It buys elastic capacity: no idle hardware, no upgrade cycle, and access to whatever batching efficiency the provider has built into their serving stack. Yao and Zhou (2026) study exactly this kind of efficiency, showing that batched serving performance depends heavily on how fairly a scheduler shares GPU resources across requests of different sizes, an engineering problem a provider solves once across millions of customers and a single company solves, if at all, alone. That difference in scale is why per-token API pricing can undercut a naive local cost estimate even when the underlying hardware is comparable: the provider's utilisation is close to constant, while a single company's local GPU sits idle outside business hours or peak load.

The real cost that shows up later is variance. A quiet month costs little. A viral feature, a marketing push, or a seasonal spike costs whatever the new volume multiplies out to, with no ceiling unless one is deliberately engineered in. That unpredictability is the actual price of "nothing until the bill arrives": not the rate itself, but the absence of a cap on the total.

What does a local model actually cost?

Hardware first, and it is paid whether or not the card is used. A consumer GPU capable of running a useful open model, an RTX 5070 Ti with 16GB of VRAM in this case, is a fixed cost that starts amortising from day one, independent of how many requests actually arrive. A registered experiment on exactly this card measured what changes when a model is quantised to fit that budget: Q4_K_M quantisation reached 2.22x the throughput of fp16 inference, comfortably past a pre-registered 1.3x bar, while average embedding similarity between quantised and full-precision outputs held at 0.9835, against a 0.90 bar. Knoop and Holtmann (2026) describe the same trade-off across a range of consumer Blackwell cards: real cost-effectiveness for SMEs, bounded by exactly how much precision a business is willing to trade for how much throughput.

The trade is real, not free. Quantisation loss is small in this measurement but not zero, and 16GB sets a hard ceiling on model size regardless of how well quantisation works. A model that needs more context or more parameters than that budget allows is not a local option at any quantisation level, only a different, more expensive hardware conversation.

Where does privacy override the price question?

Some data cannot leave the premises under any pricing model, and that constraint decides the architecture before cost enters the conversation at all. Healthcare records, certain financial data, and anything under a contractual data-residency clause fall into this category regardless of how favourable an API's per-token rate looks. In these cases the honest framing is not "which is cheaper" but "which options are even legally or contractually available," and a local deployment, whatever its hardware ceiling, is sometimes the only one left standing once that filter is applied.

This is also the case where a masking layer changes the calculation, not the venue. Stripping identifying data before it reaches any model, local or API, can turn an otherwise-forbidden API call into a legitimate option, because what leaves the building is no longer the sensitive part. That reframing is worth doing before assuming a privacy requirement forces local hosting by default.

So which one should you pick?

There is no single answer, only a smaller set of questions worth answering in order. What is the realistic request volume, and how steady is it across the day? Does the data need to leave the building at all, with or without a masking layer in front of it? Does the task fit inside what a quantised model on the hardware actually available can deliver, measured, not assumed? A low, steady volume with no privacy constraint usually favours the API. A high, steady volume with spare GPU capacity to amortise usually favours local, once the quantisation trade-off has been measured rather than guessed. Anything in between, most real cases, tends to end up as both: API for the unpredictable long tail, local for the predictable core that justifies the hardware.

A concrete shape this takes in practice: an internal document-tagging pipeline that runs continuously against a steady inbound queue is exactly the profile that justifies the RTX 5070 Ti and the quantisation work above, because the GPU is busy most of the day and the throughput measured on it directly offsets its cost. A marketing-copy generator that fires only around occasional campaign launches, in bursts nobody can predict a month ahead, is the opposite profile: the hardware would sit idle between launches, and the API's elastic capacity is worth its per-token premium precisely because nobody is paying for idle time. The same organisation can reasonably run both at once, on the same hardware budget, once the two workloads are told apart rather than treated as one decision.

FAQ

Does buying a GPU always pay for itself eventually?

No. It pays for itself only if utilisation is high enough, often meaning steady daily volume, not occasional bursts. A card that runs one hour a day is amortising over a tiny fraction of its available capacity, and the API alternative for that same hour may already be cheaper than the electricity.

Does quantisation quality loss actually matter in production?

It depends on the task. The 0.9835 embedding similarity measured on one card and one model is a real number, not a universal one, and a task more sensitive to small output differences than embedding similarity captures could show a larger practical gap. Measure the specific task, do not assume the number transfers.

Is running some models locally and others through an API a realistic setup?

Yes, and it is common in practice once volume is uneven across use cases. A steady, high-volume internal tool justifies local hardware; an occasional, unpredictable one rarely does. Splitting by workload rather than picking one venue for everything usually reflects the actual cost structure better.

What is the simplest way to decide without building a full cost model?

Estimate daily request volume and multiply by the API's per-token price for a rough monthly bill, then compare that to a GPU's amortised monthly cost at realistic utilisation, not full-time. If the API estimate is comfortably lower, that is usually enough to decide without further modelling; if the two are close, only then does a fuller analysis, including the privacy and volatility questions above, earn its time.

References

Jonathan Knoop, Hendrik Holtmann. Private LLM Inference on Consumer Blackwell GPUs: A Practical Guide for Cost-Effective Local Deployment in SMEs, 2026. arxiv.org/abs/2601.09527
Dayi Yao, Zijie Zhou. Efficiency and Cost Alignment in Batched LLM Serving via Resource-Fair Scheduling, 2026. arxiv.org/abs/2608.02244

The demo is the easy part

Ivan Zdravkov — Wed, 28 Oct 2026 06:45:41 +0000

A working demo is the cheapest part of an AI system and the least informative. It proves the happy path exists. Everything that makes the system usable every day, on inputs nobody curated, still remains to be built. I have watched that remainder quietly swallow projects that looked finished on the projector.

Key takeaways

A demo proves feasibility on inputs chosen by the person presenting it; it says nothing about the input distribution of a normal working day.
A synthesis of 27 benchmark, taxonomy and audit papers on LLM agents finds performance degrades nonlinearly with task complexity, and component success does not guarantee end-to-end completion.
Additional scaffolding does not consistently improve reliability; narrowing the scope works better than adding machinery.
The work after the demo is a lifecycle: monitoring, drift detection and feedback loops, which current MLOps research treats as first-class components.
Judge any demo by asking what happens on malformed input and who notices when quality drops.

What does a demo actually prove?

A demo is an existence proof. It shows that for some input, under some conditions, with a person steering, the system produces the intended result. All three qualifiers matter. The inputs were selected, consciously or not, by someone who knows where the potholes are. The person driving retries silently when something goes wrong. The audience sees a selection, not a sample.

None of this makes demos dishonest. A demo answers a real question: is this feasible, and roughly what would it look like. That is worth knowing, and cheap to find out. The mistake is reading the demo as evidence about reliability, when reliability was never on trial. The demo was built to win a meeting, and everything in it serves that goal.

My rule is to treat the demo as the start of the estimate, not the end. When a demo works, the honest statement is that the cheap part is done and the expensive part is now better specified.

What changes when the system runs every day?

Production removes all three qualifiers at once. Inputs arrive uncurated, nobody steers, and errors compound across steps instead of being quietly retried. The gap this opens is documented systematically. A 2026 synthesis of 27 benchmark, taxonomy and audit papers, spanning 19 distinct benchmarks, assembled a taxonomy of LLM agent failures: tool invocation and parameter errors, planning and constraint-satisfaction failures, long-horizon degradation as context accumulates, multi-agent coordination failures, safety failures under adversarial input, and measurement validity problems.

Two findings in that synthesis match what I see in practice. First, performance degrades nonlinearly with task complexity: a system that handles each step at ninety percent does not handle a ten-step chain at anything close to that. Second, additional scaffolding does not consistently improve reliability. Wrapping a fragile agent in more orchestration mostly relocates the failure. The same review notes where genuine progress holds: narrow domains, single-turn tool use, short-horizon tasks. Constrained scope is not a compromise; it is the condition under which these systems currently work.

Why do benchmark scores not settle the argument?

Benchmark results are demos at scale, and they inherit the same selection problem. The tasks are fixed, the distribution is known, and the score summarises performance on that distribution rather than yours. The synthesis cited above puts it bluntly: reported benchmark gains often obscure recurring failure modes that keep being documented across otherwise unrelated evaluations.

A leaderboard position answers the question the leaderboard asks. Whether a system survives your Tuesday afternoon traffic is a different question, and no public number answers it. The only score that predicts production behaviour is one computed on your own data, which is why the first piece of engineering after any demo should be measurement: log every model call, label a sample of real traffic, and watch the number move over time. I made the same argument about masking layers in why the masking layer matters more than the model.

The incentive problem has no methodological fix. Demos and leaderboards exist to persuade. Production metrics exist to disappoint you early, and a team that has only ever built the persuasive kind of evaluation has never been disappointed at the right time.

What does it take to close the gap?

The distance between demo and daily operation is filled with unglamorous components: monitoring on real traffic, drift detection when the input distribution shifts, feedback loops for correcting the system, access control, data masking, audit trails, and a plan for the day your model version is retired. None of these appear in a demo, and each takes longer than the demo did.

Research infrastructure is catching up to this reality. A recent framework for LLM-orchestrated data pipelines dedicates specialised agents to ingestion, monitoring and drift detection, includes human-in-the-loop checkpoints, and reports drift recovery among its lifecycle-level reliability improvements. Whatever one thinks of the architecture, the allocation of effort is telling: the post-deployment half of the lifecycle now gets as much machinery as training, because systems fail there, not on the projector.

The practical conclusion is budgetary. If the demo took two weeks, the production system is a matter of months, and most of those months go into components the demo audience will never see. A team that plans for this is slower to celebrate and far more likely to still be running a year later. This has not made me popular in demo meetings.

FAQ

How can I tell whether an AI demo will survive production?

Ask three questions: what happens on malformed input, who notices when output quality drops, and what is explicitly out of scope. A team that answers concretely has engineered for the boring days. A team that answers with another demo has not, and the gap will surface after the contract is signed.

Why do agent demos look so much better than agent products?

Agent demos are short-horizon and steered, precisely the conditions where the research literature finds agents reliable. Documented failure modes, degradation over long contexts, planning errors and compounding tool mistakes need time and complexity to appear, and a five-minute demo offers neither.

Should teams stop building demos?

No. A demo is the cheapest way to test feasibility, negotiate scope and expose hidden assumptions with stakeholders. The failure mode is treating it as evidence of production readiness. Present a demo as an existence proof with a price tag attached, and it does honest work.

What should be built first once the demo convinces everyone?

Measurement. Log every model call, hand-label a few hundred real examples, and compute a quality number you can watch weekly. Every later decision, about scope, retraining or cancellation, depends on that number existing. Register the thresholds before the numbers arrive; the practice is described in why I pre-register a kill-gate for every experiment. Features can wait; the ability to notice degradation cannot.

References

Wael Albayaydh, Rui Zhao, Ivan Flechais. Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents, 2026. arxiv.org/abs/2607.05775
Aueaphum Aueawatthanaphisut, Badri Raj Lamichhane. Trustworthy Self-Composable Big-Data-as-a-Service: An LLM-Orchestrated Multi-Agent Framework for Automated Data Engineering, AutoML, MLOps Deployment, and Drift-Aware Lifecycle Optimization, 2026. arxiv.org/abs/2606.17915

Permissions that travel with the person

Ivan Zdravkov — Wed, 28 Oct 2026 06:39:48 +0000

An AI assistant should own no permissions. It borrows the rights of the person using it, for the duration of the request, and nothing more. The shortcut that replaces it, one shared service account for everyone, eventually hands a language model more access than any single employee has, which is exactly backwards.

Key takeaways

A shared service account accumulates the union of every user's needs, so the assistant behind it ends up with more access than any individual person.
Prompt injection is a matter of when, not if: ARGUS, one of the strongest published defences, cuts attack success from 28.8% to 3.8% on its benchmark, which is impressive and still not zero.
Because no defence reaches zero, the permission model decides what a successful attack is worth, and per-person scope makes that price small.
Tool lists derived from the user's role make out-of-scope actions invisible rather than forbidden: a tool that is not listed cannot be called.
Identity, role and scope belong in fields the server fills in; nothing the model writes should decide who is asking.

Why is a shared service account the default failure?

Convenience produces the same architecture in every company: one service account for the assistant, wired once, working for everyone. The account needs to serve the support team, the finance team and management, so its permissions grow into the union of all their needs. The result is a single identity with more access than any employee in the building, standing behind a component that takes instructions in natural language from anyone who can type at it.

Every request then looks the same to the downstream systems. The log says the assistant read a contract; it cannot say for whom, because the identity that reached the database was the assistant's own. Access reviews stop working too: nobody can answer which people would lose reach if a permission were removed from the shared account, so permissions are only ever added.

None of this involves an attacker yet. It is just entropy: shared identities absorb scope, and scope, once absorbed, has no owner who could give it back. The failure is structural, and it is in place before the first malicious prompt arrives.

What does it mean for permissions to travel with the person?

Inversion is the whole idea: the assistant is not a user, it is a proxy. When a person asks their assistant a question, the request that reaches internal systems carries that person's identity, their role and their scope, and the systems answer exactly what they would answer the person directly. The assistant adds convenience, not reach.

Mechanically that means per-person credentials instead of one shared key, so every downstream call is attributable to a human. It means scope is evaluated at request time against the person's current rights, not against a snapshot: a permission revoked at noon is gone from the assistant at noon. And it means offboarding needs no special case, because revoking the person's access revokes their assistant's access in the same motion; there is no separate assistant account to remember and forget.

The attribution alone changes behaviour. A log line that reads "the assistant queried salaries" starts an investigation with no suspect. A log line that reads "the assistant queried salaries for this authenticated person" is an answer, and everyone knows their questions arrive under their own name.

How do roles decide what an assistant can even see?

Filtering the tool list is the cleanest enforcement point, and it is the reason I put role logic in the MCP server I described in what an internal MCP server actually solves. When an assistant connects, the server derives the visible tool list from the authenticated person's role. A tool outside that role is not rejected when called; it is never listed, so the model does not know it exists. Invisible is a stronger property than forbidden: a forbidden action can be requested in a hundred phrasings, an invisible one cannot be requested at all.

The parameters that decide scope get the same treatment. Nothing the model writes should determine who is asking: identity fields are filled server-side from the authenticated session, and a schema whose region or owner parameter is server-filled cannot be talked into someone else's data. The model chooses what to ask for; the server decides, from information the model never touches, whom it is asking as. Prompts remain what they are, requests, and the contract lives where contracts belong, in the code that validates every call.

What do permissions buy when an injection succeeds?

Assume the attack works, because sometimes it will. The research on this is converging: a 2026 survey of 247 papers on agent security finds prompt injection and tool-mediated control-flow hijacking still dominate the field, and judges current defences useful but weakly compositional. The most encouraging result I have seen this year is ARGUS, a defence against context-aware injection, where a malicious instruction hides inside the very evidence the agent must read to decide what to do. On the AgentLure benchmark, ARGUS cuts attack success from 28.8% to 3.8% while keeping 87.5% of clean-task utility.

A drop by a factor of seven is real progress, and 3.8% is still not zero. At production volume, a few percent means a matter of time. So the practical question is not whether an injection will ever land but what it can reach when it does, and permissions are that answer. Under a shared account, a landed injection speaks with the union of everyone's rights. Under per-person scope, it speaks with the rights of one person, on whose behalf every action was logged, in a tool list that hides everything outside their role. Detection tells you how often things go wrong; the permission ceiling decides how expensive wrong ever gets.

What does this look like in practice?

In the internal MCP server I am building, currently in testing, permissions travel exactly this way by design: each colleague authenticates with a personal key, the tool list is derived from their role, identity fields are server-filled, and read-only tools come before anything that writes. None of that required inventing new security machinery; it required refusing the shortcut of a shared account on day one, when the shortcut was tempting and the user count was small.

Open questions remain, stated honestly. Delegation is unsolved in the design: when a manager's assistant prepares something using an analyst's data access, whose permissions apply and how is that recorded. Long-running background tasks are a second gap, because rights valid when a task starts may be revoked before it finishes, and re-checking at every step has a cost I have not measured yet. Those are the next design decisions, and I would rather publish them as questions than pretend the model is complete.

FAQ

Why not one service account with careful prompt instructions?

Prompt instructions are requests to a model, not constraints on it, and the survey cited above finds injection attacks remain the dominant, unsolved threat. A shared account puts the union of everyone's permissions behind those breakable instructions. Per-person scope moves the constraint into infrastructure, where phrasing cannot negotiate with it.

What happens to the assistant when someone leaves the company?

Nothing special, which is the point. Offboarding revokes the person's credentials, and their assistant loses every capability in the same motion, because it never had rights of its own. Under a shared-account design the leaver's reach survives inside the service account, invisibly, until someone remembers to audit it.

Do defences like ARGUS make strict permissions unnecessary?

No, they stack. Detection and provenance decide how often an attack lands, and the residual few percent that ARGUS still leaves shows detection alone is not enough; the permission model decides what a landed attack is worth. Run both, and size the ceiling as if detection will occasionally fail, because it will.

How granular should assistant roles be?

Start from the roles your organisation already has, and map each to a tool list; resist inventing a per-person permission matrix on day one. Coarse roles with server-filled identity fields already bound the damage to one person's scope. Refine where the audit log shows real need, not where the org chart suggests theoretical elegance.

References

Shihao Weng, Yang Feng, Jinrui Zhang, Xiaofei Xie, Jiongchi Yu, Jia Liu. ARGUS: Defending LLM Agents Against Context-Aware Prompt Injection, 2026. arxiv.org/abs/2605.03378
Yuchen Ling, Shengcheng Yu, Zhenyu Chen, Chunrong Fang. Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation, 2026. arxiv.org/abs/2606.10749

The masking layer matters more than the model

Ivan Zdravkov — Wed, 28 Oct 2026 06:39:12 +0000

Once sensitive data reaches a model provider, no later decision can bring it back. The layer that detects and masks that data before any model call deserves more design time than model selection. In the AI systems I build, designing the masking layer was among the hardest parts of the work, harder than choosing the models behind it.

Key takeaways

A masking layer detects and replaces sensitive data before any text reaches a language model, local or external.
Masking belongs at the boundary of the pipeline, as one service in front of every model call, not code repeated inside each feature.
Detection does not need a large model. GLiNER2-PII covers 42 PII entity types with 0.3 billion parameters.
Detection is never complete: the strongest configuration compared in the PIIBench paper reaches an F1 of 0.6455 across 82 entity types, so design for leakage.
A few hundred labelled examples from your own traffic tell you more than any vendor benchmark.

Why does the masking layer matter more than the model?

Model selection is a reversible decision. A pipeline built behind an abstraction can swap one provider for another or move to a local model without touching business logic. Sending a customer's name, address and order history to the wrong place is not reversible. The data has left the building, it may be logged, cached or used for training, and no later architecture change undoes that.

The asymmetry is why I treat masking as the core of any AI system that touches operational data. The model decides how good the answers are. The masking layer decides whether the system may run inside a real company at all. GDPR obligations follow the data, not the model, so the question a data protection officer will ask is not which model I chose but what exactly left the infrastructure.

The other reason is practical. Models improve every quarter with no effort on my side, because vendors and open-source teams keep releasing better ones. The masking layer improves only when I work on it. Nobody will ship me a better understanding of which fields in my own systems are sensitive or which partners put phone numbers in comment fields.

What does a masking layer actually do?

A masking layer performs three steps: it detects spans of sensitive data in the text about to be sent to a model, replaces them with placeholders, and restores the original values in the response before a human sees it.

Detection covers two different problems. Structured fields are easy: the schema already says which column holds a name or a bank account, so masking is a lookup, not a guess. Free text is the hard part. A support ticket or an invoice line can contain a phone number, an identification number or an address in any format and any language, and finding those spans is a named entity recognition task, not a string match.

Replacement, in the design I would recommend, uses deterministic placeholders within a request, so the same person resolves to the same token everywhere. A model that sees CUSTOMER_1 ordering and CUSTOMER_1 complaining can reason about the connection; one that sees two unrelated tokens cannot.

A consistent token is also a pseudo-identifier. With enough surrounding context, a stable CUSTOMER_1 can be linked back to a person, so the mapping should be scoped as narrowly as the task allows and long-lived tokens should be treated as data that still needs protection.

Restoration means the mapping between placeholders and real values never leaves the infrastructure: the model only ever sees tokens, and unmasking happens on the way back, under the same access rules as the rest of the system.

Where does the masking layer sit in the pipeline?

One service in front of every model call is the placement that works. Every feature that talks to a model, whether it classifies tickets, checks invoices or answers a colleague's question, routes its request through the same masking gateway, and nothing else in the codebase holds an API key to a model provider.

Centralising the layer buys three things. First, one place to test: a single suite and one labelled dataset cover every feature at once. Second, one place to improve: when a new leak pattern shows up, the fix lands in one service instead of five prompt builders. Third, one audit log: the gateway records exactly what left the infrastructure and in what form, the record a GDPR conversation needs.

The alternative, masking logic living inside each feature, decays fast. Different features drift to different rules, a new feature ships in a hurry without any masking, and no one can answer what the system as a whole sends out.

Can a small model handle the detection?

Detection does not require a large generative model. GLiNER2-PII, published in May 2026, is a 0.3 billion parameter model extracting 42 types of personally identifiable information across languages, trained on a constraint-generated synthetic corpus of 4,910 annotated texts rather than real people's text. On the SPY benchmark it reports higher span-level F1 than the systems it was compared against, the OpenAI Privacy Filter and three GLiNER-based detectors. A model of that size runs on modest hardware inside your own network, where a masking component should run.

A second result points at where the effort should go. On PIIBench, a corrected multi-source dataset with 82 entity types, direct token-classification fine-tuning of DeBERTa reached an F1 of 0.6455 on a test set of 100,002 records, beating a source-conditioned hierarchical model and a curriculum-based variant from the same author. The conclusion of that paper matches my experience: diverse task-specific training data moves detection quality more than architectural sophistication does.

Using a large hosted model to detect PII defeats the purpose. The detector would receive the raw sensitive text, precisely the exposure the masking layer exists to prevent. The detector has to be the one component you can run entirely inside your own boundary. My own later measurement of GLiNER2-PII on public data confirmed the strengths and found the weak spot, surnames; the numbers are in why I pre-register a kill-gate for every experiment.

What slips through, and how do you measure it?

An F1 of 0.6455 reported in that paper across 82 entity types means a substantial share of entity mentions are missed or mislabelled. Narrower, well-represented types score much higher, but the honest reading is that free-text detection leaks, and the design has to survive that.

Leakage is not only a missed entity. Three unmasked details that look harmless on their own, a town, a delivery window and an unusual surname, identify a person more reliably than any single field does.

Benchmark scores are measured on someone else's data. Mine has Bulgarian names in Cyrillic and in several Latin transliterations, national identifiers whose format collides with other numbers, and addresses buried in free-text comments. No public benchmark contains that mix, so no public score predicts my leakage. The same gap between benchmark scores and production behaviour shows up everywhere in AI systems, not only in PII detection: see the demo is the easy part.

The measurement is small and boring. Take a few hundred real examples from your own traffic, hand-label the sensitive spans, and compute recall per entity type. Recall matters more than F1 here: a false positive masks too much and costs a little answer quality, a false negative sends personal data to a provider. The errors are not symmetric, so tune thresholds toward over-masking.

For categories where a miss is unacceptable, the fallback is structural, not statistical: route those requests only to a locally hosted model, or keep those fields out of model input entirely. I do not have a method that reaches zero leakage, and I distrust claims of one.

What does this look like in practice?

In the systems I work on, several features share one masking layer: text-heavy request handling, document reconciliation, and an internal service that lets colleagues' assistants query company data. Each of them sends text to a language model, and none of them is allowed to do so directly.

Designing that layer was among the hardest parts of the whole effort. I consider the time well spent, because it is the reason the features may run in production at all. The building blocks are ordinary: schema-driven masking for structured fields, a named entity recognition model for free text, one gateway in front of every model call, and tracing on each request. The mapping between placeholders and real values stays inside the infrastructure, and what any given assistant may see is decided by the role of the person using it, not by the model.

Not everything worked on the first attempt. The structured-field side was straightforward, because the schemas were known. The free-text side was not. The spans I trust least are those where a harmless-looking string carries identity: organisation names shaped like personal names, addresses written into comment fields, and numbers whose meaning depends entirely on context. The per-entity recall measurement described above now exists, and its first run turned that impression into numbers.

FAQ

Do I still need a masking layer if the model runs on my own hardware?

Yes. A local model removes the external provider, but prompts still end up in logs, traces and evaluation datasets. Masking limits what spreads into those secondary stores. It also keeps the pipeline portable if part of the workload later moves to an API model.

Is regex enough for PII detection?

For structured fields and strictly formatted identifiers, yes: regex is fast and auditable. For free text it is not enough: names, addresses and mixed-language content do not follow patterns. The practical setup is layered: schema-based masking first, regex for formatted identifiers, an NER model for everything else.

Does masking hurt the quality of the model's answers?

Little, provided placeholders are deterministic: the model reasons about CUSTOMER_1 and ORDER_17 as consistent entities, and real values return during unmasking. Quality drops when the masked value carried meaning the task needed, such as a city name in a routing question, so masking rules must be scoped per task.

What is the difference between masking and anonymisation?

Anonymisation is irreversible: once applied, nobody can recover the original values, which suits published datasets. Masking is reversible inside a trusted boundary: the mapping to real values is kept, and authorised users see restored output. Assistants that act on real operations need masking: an answer about ORDER_17 is useless if nothing can say which order that is.

References

Urchade Zaratiana, Ash Lewis, George Hurn-Maloney. GLiNER2-PII: A Multilingual Model for Personally Identifiable Information Extraction, 2026. arxiv.org/abs/2605.09973
Pritesh Jha. Fine-Tuning Over Architectural Complexity: Broad-Coverage PII Detection on PIIBench with DeBERTa, 2026. arxiv.org/abs/2605.25816

LoRA on a 16 GB card: what fits and what does not

Ivan Zdravkov — Wed, 28 Oct 2026 06:33:20 +0000

A 7B model is the honest ceiling for LoRA fine-tuning on a 16 GB consumer card, not because anything bigger throws an error, but because a 14B model silently spills into system memory and runs 50 times slower without ever reporting a failure.

Key takeaways

7B is the real ceiling on an RTX 5070 Ti at LoRA rank 16, sequence length 512, batch size 1: 11,897.9 MiB peak, well inside the card's 16,303 MiB.
14B completed without a raised exception in both tested conditions, but peak VRAM read 22,189.8 MiB (136% of the card) and 17,470.9 MiB (107%) - both over the physical limit, through Windows' silent system-memory fallback.
The cost of that fallback is severe: 21.7 seconds per training step at 14B against 0.4 seconds at 7B, roughly 51 times slower, with no error message anywhere.
Gradient checkpointing is close to free at 7B and below - same or less time, meaningfully less memory - but does not rescue 14B: it leaves the model still over the limit and makes the step 59% slower, not faster.
The registered hypothesis that checkpointing moves the ceiling up a tier is rejected; the wording of the first kill-gate also needed correcting, because "did not throw an out-of-memory error" turned out to be the wrong proxy for "fits".

Why doesn't "no error" mean "it fits"?

I pre-registered two kill-gates on an RTX 5070 Ti, 16,303 MiB of physical VRAM: whether a 14B Qwen2.5-Instruct model, loaded in 4-bit with a LoRA adapter, completes one real training step without a raised CUDA out-of-memory exception, and whether gradient checkpointing moves that ceiling up by at least one tested size, following the same discipline described in why I pre-register a kill-gate for every experiment. Neither 14B run threw an exception. By the literal wording of the first gate, it fired - 14B "fits". But torch.cuda.max_memory_allocated() reported 22,189.8 MiB without checkpointing, 5,887 MiB above the card's physical limit, and the step took 21.7 seconds instead of the sub-second range every smaller model showed. Windows ships a driver-level system-memory fallback policy that lets CUDA allocations succeed past a card's physical limit by quietly paging into ordinary RAM instead of raising an error, and nothing in the console said so. The gate measured the wrong thing: "did not throw an exception" is not the same claim as "fits in 16 GB", and only comparing the allocator figure against the card's actual VRAM, plus noticing the 50x time cliff, exposed the gap.

What did the clean numbers actually look like?

Three of the four sizes behaved exactly as expected. Without checkpointing, 1.5B peaked at 4,000.7 MiB, 3B at 6,749.0 MiB, 7B at 11,897.9 MiB - all comfortably under 16,303 MiB, all completing a forward-backward-optimizer step in under half a second. 7B used 73% of the card's VRAM at rank 16, sequence length 512, batch size 1, with real room to spare. Full numbers for all eight configurations sit in the frozen plan and results in the public experiment repository. Enabling gradient checkpointing lowered every one of those three figures - 2,855.5 MiB, 4,490.8 MiB and 9,709.9 MiB respectively - at essentially no time cost: 7B ran in the same 0.425 seconds either way, 3B cost 25% more time for a third less memory. The 1.5B timing is excluded from this comparison because it ran first in the script and likely carries a one-time CUDA initialisation cost. The honest ceiling for this exact setup, Qwen2.5-Instruct, 4-bit base, LoRA rank 16, one sequence of 512 tokens, batch size 1, is 7B. Push any of those variables up and the real number moves down, not up.

Does gradient checkpointing rescue the 14B case?

No. With checkpointing enabled, 14B still reported 17,470.9 MiB, 1,168 MiB over the card's physical limit - smaller than the 5,887 MiB overflow without checkpointing, but still spilling into system memory, not fitting inside it. The step time went the wrong direction: 34.5 seconds, 59% slower than the 21.7 seconds without checkpointing. Checkpointing buys memory by deleting intermediate activations during the forward pass and recomputing them during backward, and that recompute cost is normally hidden by the time saved from avoiding memory pressure, which is exactly why it helped 7B and below for free. Once the card is already paging to system RAM, the saved memory only shrinks an already-severe penalty instead of removing it, and the extra computation adds straight on top. The registered hypothesis, that checkpointing moves the ceiling up by at least one tested tier, is rejected. On this card, for this model family, checkpointing is a real, close-to-free lever below the ceiling, not a way past it.

FAQ

Does this mean a 16 GB card can never run a 14B model?

Not for training, no, not with this exact setup. A smaller rank, a shorter sequence, a smaller LoRA target set, or a more aggressive quantisation scheme could plausibly bring a 14B model under the limit; none of those were tested here. For inference rather than training, the memory math is different again and was not measured in this experiment.

Why trust torch's reported VRAM figure at all, if it can be wrong?

It is not wrong about what it allocates - torch.cuda.max_memory_allocated() reports real tensor memory the CUDA allocator granted, honestly. What it does not tell you is whether that memory was physically on the card or transparently paged to system RAM by the driver. Comparing the figure against the card's known physical limit, rather than trusting the absence of an exception, is what catches the difference.

Why only test rank 16 and batch size 1?

Those are close to the smallest realistic defaults most people reach for first. Testing a single point cleanly, rather than sweeping many configurations in one pass, keeps the registered claim specific and checkable. A larger rank or batch size sweep is a legitimate follow-up, not something this experiment's kill-gates covered.

Where do the numbers come from?

Every figure in this piece is read directly from results.json in the registered experiment, not estimated or rounded from memory. The frozen plan, the full experiment code, and the raw per-run output all sit in the public repository linked above, so the eight numbers in the table can be checked against the source instead of taken on trust.

References

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, 2021. arxiv.org/abs/2106.09685