DEV Community: BigCircle

RAG vs fine-tuning vs agentic search, and when to use each

Tirth Gajjar — Sat, 31 Oct 2026 18:40:36 +0000

Choose between RAG and fine-tuning by the failure you can see. Use retrieval when the model is missing a fact, and fine-tuning when the facts are right and a specialist says the output reads wrong. Use agentic search, also called agentic RAG, only when no single query can find the answer.

Prompting comes before all three, because it costs nothing extra to try. Every extra search step is another model call the user waits for, and most production systems end up running several of these layers at once.

Criterion	Prompting	Retrieval	Fine-tuning	Agentic search
Failure you can observe	Output is inconsistent, or wrong in a way you cannot describe	Model does not know a fact, or knows an old one	Facts are right, form is wrong to a specialist	No single query can retrieve the answer
What it changes	Instructions, examples, and the output schema	Which passages are fetched for this question	Default form, and judgements a written rule does not settle	Reach, by searching again from what the last search returned
What it leaves alone	The weights, and which documents exist	How the model writes once the right passage is present	A fact that changed after the training run	The accuracy of a question one search could have answered
What you build	A prompt, examples and an output schema	An index, the jobs that keep it current, and permission filters	A training set, an eval against the base model, and a hosted model	A loop with a stop condition and a record of every step
Added latency	None	One search before the model call	None once served	One model call per step, each waiting on the last
Cost per call	Tokens	Tokens, the search, and the passages added to the prompt	Can fall, if a small tuned model replaces a large one	Multiplied by the number of steps
Ongoing burden	None	Keeping the index current	A model to own and retrain	Showing the loop stopped for a good reason
How Bigcircle handles this	Schema-constrained decoding sits in the same request as the other layers on the trademark platform	Retrieval restricted to a curated index of authorities on that platform, live since 2023	A fine-tuned model in the same request, for structure and register, which retrieval could not fix	On Prospex, a search loop only after hybrid retrieval stopped improving. The insurance voice agent loads its context when the call opens and does not search during a turn

The standard answer to this question is "RAG for knowledge, fine-tuning for behaviour." That answer is close, and it breaks on the first system that needs both. Our trademark drafting platform has been live since 2023 with a fine-tuned model, retrieval over a curated index of authorities, and schema-constrained decoding in the same request. We did not run all three because we could not choose. Each one fixes a failure the other two cannot reach.

So start from the failure you are looking at, and then pick the least expensive layer that fixes it. Read the columns of the table above from left to right. Prompting is the column to exhaust first, and a later column does not fix a failure that belongs in an earlier one.

When is it just a prompting problem?

When the task is underspecified rather than unknown to the model. Every team believes it has finished prompting, so test it. Take your prompt, hand it to a contractor who knows nothing about your product, and ask them to do the task.

If they come back with questions, the model has those questions too, and it does not ask them. It picks an interpretation, commits to it, and you file the result as a model failure. A support summariser gets a ticket with three separate issues in it, and nothing in the prompt decides whether that is one summary or three. The model picks one reading, and next Tuesday it picks the other. No amount of retrieval touches that, because nothing is missing from what the model knows.

A fix at this layer takes an afternoon and leaves nothing new to host or retrain, which is why it comes first. Before you spend money on another layer, do the following.

Constrain the output mechanically. Structured output with constrained decoding removes invalid tokens from the choices the model has while it writes, so the output always parses. If you are still parsing free text with a regex and a retry loop, the defect is in the prompt, and it will get filed against the model.

Show the output instead of describing it. Four real examples of the exact output do more than two paragraphs of adjectives, because the model copies what it is shown (in-context learning). Before anyone proposes a fine-tune, count the examples in the prompt. If the count is zero, the free version of the same idea has not been tried yet.

Write the refusal condition, one sentence along the lines of "if the retrieved passages do not contain the answer, say so and stop". Without it the model fills a gap with something plausible, and a reader cannot tell which sentences came from the passages and which were a hallucination.

You cannot skip the eval set

Collect twenty to fifty real cases with known-good outputs and score them the same way on every run. Until that set exists you cannot tell a fix from a lucky sample, and you cannot tell whether the next layer helped. Evals are how you read every decision below this point. How to build LLM evals covers building the set, from the first failure note through a model judge.

When do you need RAG?

Use retrieval when the failure is missing or stale knowledge: facts specific to your business, documents written after the model's knowledge cutoff, anything that changes weekly, and anything a user is allowed to see only part of.

Your refund window changed in March and the model still answers with the old one. Fine-tuning will not fix that, and it will not fix it again in June when the window changes back. The mechanics are in why RAG systems fail in production. The question here is whether retrieval alone is enough.

Retrieval quality depends less on the model or the vector database than on four choices: chunking, embeddings, hybrid search so that part numbers and names still match exactly, and reranking so the best passage reaches the context window rather than a related one.

Work on those four before you touch the model, because no model can use a passage that never reached it. In Anthropic's contextual retrieval tests from September 2024, the share of queries whose right chunk missed the top 20 fell from 5.7% to 1.9% once each chunk carried 50 to 100 tokens of generated context, keyword search ran beside embeddings, and a reranker sorted the result. The added context alone cut failures by 35%, keyword search took the cut to 49%, and the reranker to 67%. The model stayed the same throughout, so the whole gain came from those choices.

On Prospex, our own prospect research engine, semantic and keyword search run together, and a claim with no retrieved passage behind it does not appear. On one run that took 420 collected items down to the 6 that answered the question. We estimate it makes 50 to 60 percent fewer external API calls on repeated runs, and that saving comes from caching rather than from retrieval: the runtime is cache-first, so a rerun on an overlapping account reuses work instead of paying for it again.

Teams get two things wrong at this layer. Citations are an architecture decision, and teams keep treating them as formatting. If a regulated user has to trust a number, they need to open the page it came from.

On our ESG filings platform every extracted field carries a link to its source page and a confidence score, and a low-confidence field goes to review instead of into a dashboard. We measured the analysis cycle falling from weeks to hours once the pipeline replaced manual extraction and spreadsheet work. Most of the recovered time came from the confidence flags: analysts who could see which figures were uncertain stopped re-deriving earlier work to be safe.

Permissions belong in the query. A filter applied after retrieval runs once the document is already in the search results, and anything downstream that logs, caches or summarises those results can leak it. Filter by the user's access inside the search itself.

Three ways retrieval turns out not to be enough

Each row below is a symptom you can see in a trace, and the column on the right names where it sends you.

What you see	What it means	Where to go
Right passage in context, answer still reads wrong to a specialist	Form problem, not fact problem	Fine-tuning
Right passage cannot be found by any single query	Question has to be split into several searches	Agentic search
The corpus does not contain the answer	Not a technique problem	Say so and stop

That third row is worth saying out loud early, because no layer fixes a missing document.

When should you fine-tune instead of using RAG?

When the failure is form, or judgement a written rule does not settle, and never when it is missing knowledge. Fine-tuning does not teach facts reliably. It can be made to, badly, and then you cannot update a fact without retraining. What it moves is default behaviour: output shape, register, domain convention, and the small judgements a specialist makes without thinking.

The usual test is whether the expert can write the rule down. That is close, and the sharper test is whether the written rule decides the output.

Trademark work is the clearest case we have, and the problem is not that practitioners cannot explain anything. They can. The rules are written down, and they are general, and a general rule does not settle a specific draft on its own. Precedent settles it: how comparable cases were decided, and how those decisions have shifted what a defensible specification looks like in practice.

You cannot compress that into prompt instructions, because it is not one book. It is applied interpretation spread across many decisions, where the same rule gives different results depending on the case in front of you. So the layers split cleanly, and each one takes the part it can hold:

What it is	Where it lives
The general rules, written and stable	Prompt instructions
The specific authorities and prior marks for this case	Retrieval, restricted to approved sources
The pattern of how those rules get applied in practice	The fine-tune

People try to solve that third row with a longer prompt. A prompt listing the general rules produces a draft that is technically compliant and reads wrong to anyone who has practised. Before training, we normalised years of attorney opinions into one five-section template, tagged by mark type, jurisdiction and NICE class. Without that step the model would have learned a dozen individual writing styles and averaged them. It mattered more than we expected.

The fine-tune also has a failure we designed around. It improves how a citation is phrased, which is the wrong thing to improve when the authority might not exist. The fine-tuned model is never asked to remember which classes exist, which prior marks conflict, or what the registry shows this week. Retrieval over the curated index supplies those, a validator rejects any citation that does not resolve, and an attorney reviews and signs the draft.

What it costs beyond the training run

Budget the training run last. It is the smallest line and often the only one in the estimate, because LoRA and QLoRA shrank the compute. The other lines did not shrink:

Cost	What it looks like
Data	Examples of the right output, written or checked by an expert. On the trademark platform that meant normalising years of opinions into one template before any training
Evaluation	Proof the tuned model is better rather than different. The trademark platform logs the base and fine-tuned models on the same inputs, so a regression shows up as a score that moved
Permanence	A fork in your stack. Every new base model is a decision about redoing the work, and providers retire the bases you tuned on
Operations	Hosting, capacity, version pinning, rollback. A prompt change has none of these

Synthetic data helps with volume and cannot invent the judgement you are trying to capture. If you generate the examples from your own prompt, you have trained an expensive copy of that prompt. LLM-as-judge scales the evaluation, and the judge is a model with its own biases that needs spot checks against an expert. A tuned model also ages. If nobody reruns the comparison against the current base model, the team keeps paying to host a fork that may now do worse than a prompt on the newer model.

The fine-tune we did not build

A client came to us wanting a fine-tune for cold outreach email, on a small low-cost model, tuned for open rates. It is a reasonable-sounding request, because writing quality is exactly the kind of thing people expect fine-tuning to fix. We did not tune anything, and splitting the job into two calls fixed it.

The first call does a short discovery pass on the prospect, and the second writes the email given that research and a handful of examples of what good looks like in their domain. That is two hops instead of one, with no new weights anywhere.

That is the pattern we keep meeting, where "the output is not good enough" rarely means the model needs different weights. Usually it means the model is being asked to research and write in a single step, with nothing to imitate. Splitting the step and showing examples takes an afternoon, and the fine-tune was going to take a month.

Fine-tune when the behaviour is unwritten, stable enough to outlive a model generation, and measurable. Otherwise try many-shot prompting first and score it on the eval set. Check the provider's retirement dates before you tune, because a provider can stop training new versions of a tune you still depend on: as of 22 October 2026, OpenAI's deprecations page gives 6 January 2027 as the last day existing customers can create fine-tuning jobs. A tune already running on a base with a shutdown date is the case how to replace the model under a production agent walks through.

The other reason to fine-tune

The other reason is cost and latency rather than quality. Distillation moves a large model's behaviour on your one task into a small model you can serve at lower cost or run on a device. Judge that on cost per request, because it is a serving decision and the argument above does not apply.

What is agentic RAG, and when do you need it?

Agentic RAG puts retrieval inside an agent loop. The model decides what to search for, reads what came back, and then decides whether to search again or answer, where ordinary RAG runs one search and answers from whatever it returned. We call it agentic search in this guide, because the loop can call any search tool and a vector index is only one of them.

Single-shot retrieval assumes the question maps to a query, and some questions do not. "Which of our suppliers mentioned tariff exposure in their last two filings?" No query answers that. You need the supplier list, then the filings per supplier, then a read of each, then a decision about when you have enough. An agent that plans a step, acts, reads the result and plans again handles that (plan-and-execute, the ReAct pattern). If one query still answers the question, stay with retrieval. Our guide on agentic workflows vs AI agents separates that case from a scripted workflow and from this loop.

The loop buys reach with latency and cost, so we add one only when a simpler pipeline has visibly fallen short. Two conditions make it worth the price: the corpus changes faster than an index can follow, and exact search tools already exist for the loop to call. Code meets both. Claude Code's team dropped local vector indexes for model-driven file-name and text search (glob and grep), citing stale indexes and permission complexity (Boris Cherny to The Pragmatic Engineer, March 2026). A large document set that changes monthly and has no structured search meets neither, and retrieval still suits it.

Our own systems mostly run retrieval inside a fixed sequence of steps. On Prospex, the reasoning steps (business model, customer fit, strategy, risk, tech stack) run in a set order and each one retrieves for itself, so the evidence for every claim stays attached to that claim and two runs on one company are built to give the same read.

On the executive operations agent, we measured plans completed with no human stepping in rising from roughly 55% to 80% once local retrieval and planner memory grounded the agent in the user's own workspace. The same agent has a critic that chooses continue, retry or escalate after each step, for a different reason: without it, one bad observation carries through every later step and the agent claims success on work it never finished.

When is the loop not worth it?

A loop turns one search into several model calls and searches in sequence, and the user waits for every one of them. Ask three questions before you build it.

Is the question really multi-hop, or is retrieval weak? Many complaints that "the agent needs to search several times" are single-shot retrieval failures nobody diagnosed. Check whether the passage you need ranks near the top for any one query. If it does, fix hybrid search and add reranking, which changes one step of the request, before you add a loop that changes how many steps there are.

There is a real exception, and we hit it on Prospex when retrieval had run out of road. The source data was structured in a way that resisted tuning: the evidence a single question needed was spread across documents that looked nothing like each other, so better chunking and better hybrid search kept arriving at the same ceiling. An agent that searched, read what came back, and searched again got past that. Another month of retrieval work would not have.

So the check is not whether you have tried reranking, but whether you have a reason to believe more retrieval work will move the ceiling. On Prospex we did not, and we could say why.

Is there a latency budget? Rule the loop out for voice before anyone designs it, because the pause a caller expects is short and similar across languages: in a ten-language study, every language's average gap between turns sat within 250 ms of the cross-language mean (Stivers et al., PNAS, 2009). A search loop does not fit in that gap.

Our insurance voice agent came down from 2.5 to 3 seconds to around 900ms end to end in our tests, and it does not search during a turn at all. Policy and claim context loads when the call opens, so no reply waits on it, and every action goes through a typed tool that checks its inputs and permissions.

Can the loop stop, and can you show it did the work? An agent that decides when it has enough evidence also decides when to stop. Solve that before the loop ships, because an agent that stops early still writes a confident answer. Our guide on AI agents that call unfinished work done covers the checks. The loop adds reach, and it makes no single search more accurate, so if one good search can answer the question, that search will be faster and easier to debug.

What does each layer cost to run?

Two cells in the comparison at the top surprise people. Fine-tuning can lower the cost per call, because a small tuned model that matches a large one on your single task costs less to serve and answers faster. When cost matters more than quality, the usual advice inverts.

Retrieval's real running cost is freshness, not the search. Keeping the index correct as documents change, permissions move and pages get deleted is the work, and an index that falls behind raises no error, so nobody notices until an answer quotes a page that no longer exists. The multiplied calls in the agentic column are a separate bill, which our guide on AI agent cost splits into the build, the run, and the work that remains after launch.

Before adding any layer, check prompt caching on a stable system prefix and model routing for the easy requests. Neither one changes what the model does.

What does running all three at once look like?

The trademark path, in production since 2023, runs three layers in one request, and every draft is scored before an attorney sees it.

Remove one and the system fails in a specific way. Without retrieval it invents authorities. Without the fine-tune the drafting reads wrong to a practitioner. Without the schema, the integration breaks the day a model update changes its formatting habits. The evaluation scores each draft from 0 to 10 on six criteria (structure, reasoning depth, citation accuracy, jurisdiction discipline, edge cases and tone), and attorney corrections remain the one signal that automated scoring cannot replace.

What will none of this fix?

These four layers get sold as answers to problems they cannot touch. A missing document stays missing, and if the corpus does not contain the answer, retrieval returns the closest wrong thing and generation makes it sound plausible.

A task nobody has defined stays undefined. If three stakeholders read the spec three ways, a fine-tune learns whichever reading dominated its training examples. Disagreement between experts also stays disagreement. If two specialists mark the same output differently, more training data does not resolve it, and the call belongs to the product owner rather than to the model.

Bad source data stays bad, whether that means duplicated pages, stale exports or a PDF parser that drops table headers, and every layer downstream inherits it.

How do you decide, in order?

Stop at the first question you cannot answer yes to, and fix that before moving on.

Can a new contractor do the task from your prompt? If not, write the specification, add four real examples, and name the refusal condition. An afternoon.
Do you have twenty to fifty scored eval cases? If not, build them before changing anything else.
Is the failure a missing or stale fact? Retrieval: hybrid search, reranking, chunking, permissions. Not a bigger model.
With the right passage in context, is the answer wrong in form rather than fact? That points to fine-tuning, if the convention is unwritten and stable. Try many-shot prompting first and measure it.
Does answering require several dependent searches? Agentic search, with a stop condition, a verification step and a latency budget you have checked.
Are you optimising cost or latency rather than quality? Distillation, model routing, prompt caching, judged on cost per request.

To find where you are, open your last twenty production failures and label each with one word: unspecified, missing, form, or multi-hop. Count the labels. The largest bucket tells you which step to work on, and it is often not the layer the team was planning to build next.

Common questions

Should I use RAG or fine-tuning?

Use retrieval when the model is missing a fact or is still quoting a document that has changed. Fine-tune when the right passage is already in context and a specialist says the draft reads wrong. Finish the specification, the examples, and the refusal line before either of those, because an unspecified task looks like a model failure.

When should you fine-tune instead of using RAG?

Fine-tune for form and for judgement a written rule does not settle, and never to store a fact you will need to update. The authorities and the current documents stay in retrieval. Try many-shot prompting first and score it on your eval set. A tuned model you do not compare with today's base model is a fork you then have to keep.

What is agentic RAG?

Agentic RAG, also called agentic search, is a loop that plans a query, reads what came back, and searches again from that result. Use it when no single query can find the answer, such as a supplier list, then each filing, then a read of each one. Stay with one search when that search can answer, because every step of the loop is another model call the user waits for.

What is the difference between RAG and agentic RAG?

Ordinary RAG runs one search before the model call and answers from what that search returned. Agentic RAG lets the model decide what to search for next, read the result, and search again until it has enough or hits a stop condition. The loop adds reach for questions that need several dependent searches, and it makes no single search more accurate.

Can RAG and fine-tuning run in the same request?

Yes. Bigcircle's trademark drafting platform has done it in production since 2023. It retrieves from a curated index of authorities, drafts with a fine-tuned model, and constrains the output to a five-section schema whose citations must resolve. Drop retrieval and the draft invents authorities, and drop the fine-tune and a practitioner says the form is wrong.

How do I choose between prompting, RAG, and fine-tuning?

Stop at the first step you cannot pass. If a new contractor cannot do the task from the prompt, write the specification and add real examples before you touch a model. If the failure is a missing or stale fact, use retrieval, and try many-shot prompting before a fine-tune when the passage is already in context.

When is an agent the wrong way to search?

Skip the agent when one query can still answer the question. Many complaints that the system must search several times are single-shot retrieval misses, fixed by hybrid search and reranking without adding steps. A voice product answering inside a conversational pause cannot host the loop, so Bigcircle's insurance voice agent loads its context when the call opens and does not search during a turn.

Sources and further reading

Anthropic, Introducing contextual retrieval, September 2024. Top-20 retrieval failure from 5.7% to 1.9%.
Gergely Orosz, Building Claude Code with Boris Cherny, The Pragmatic Engineer, March 2026. Why Claude Code dropped vector indexes for glob and grep.
Stivers et al., Universals and cultural variation in turn-taking in conversation, PNAS, 2009.
OpenAI, deprecations, checked 22 October 2026. The fine-tuning cut-off date.
Why RAG systems fail in production. The retrieval layer in depth, failure by failure.
The practical guide to AI agents. What makes a loop controllable rather than just autonomous.
Agentic workflows vs AI agents. Whether the task needs this loop, a scripted workflow, or retrieval on its own.
Why AI agents report unfinished work as done. The verification problem agentic search inherits.
How LLM inference actually works. Why prefill, decode and caching drive the cost column above.
Case studies referenced: trademark drafting, prospect intelligence, ESG filings, executive operations, insurance voice.

Originally published on bigcircle.ai. I'm Tirth, founder of BigCircle, where we build production AI systems with governance built in.

GPT-4o deprecation: dates, replacement and how to migrate

Tirth Gajjar — Thu, 29 Oct 2026 18:40:20 +0000

GPT-4o leaves the OpenAI API on 14 January 2027. As of 22 October 2026, that date covers the listed GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini snapshots and the fine-tunes built on them, and OpenAI names gpt-6-luna as the replacement. Azure retires most of the same versions on 14 April 2027. ChatGPT dropped GPT-4o in February 2026, which did not touch the API.

Changing the model string takes a minute, but proving the product still behaves takes the rest of the migration, because Luna handles tool calls, reasoning and sampling differently from the model your evals were scored on. This guide gives the dates, then the order of checks we recommend for this swap and every one after it.

When is GPT-4o deprecated on the OpenAI API, Azure and ChatGPT?

Plan against the earliest date that touches any id you call. Each platform runs its own schedule, and a team that reads one page and assumes it covers the others plans for the wrong day.

Where the model is called	Shutdown date	Listed replacement
OpenAI API	14 January 2027 for `gpt-4o-2024-08-06`, `gpt-4o-2024-11-20`, `gpt-4o-mini-2024-07-18`, `gpt-4.1-2025-04-14`, `gpt-4.1-mini-2025-04-14`, and fine-tunes on them	`gpt-6-luna`
OpenAI API, earlier batch	23 October 2026 for `gpt-4o-2024-05-13` and `gpt-4.1-nano`, including its fine-tune	`gpt-5.6-sol` for the GPT-4o snapshot, `gpt-5.6-luna` for nano
Azure OpenAI	9 December 2026 for `gpt-4o` 2024-05-13. 14 April 2027 for `gpt-4o` 2024-08-06 and 2024-11-20, `gpt-4o-mini`, and all three GPT-4.1 sizes	`gpt-5.6-sol` for 2024-05-13, `gpt-5.1` for the later GPT-4o versions, none listed for GPT-4.1 or `gpt-4o-mini`
Azure OpenAI, fine-tuned deployments	1 October 2027 for GPT-4o and GPT-4o mini fine-tunes, 14 October 2027 for GPT-4.1 fine-tunes	Training ends earlier, no earlier than April 2027 for existing customers
ChatGPT	Retired 13 February 2026	Not an API change

Dates are as listed on OpenAI's deprecations page and Microsoft's retirement schedule on 22 October 2026. Any call still pinned to the 23 October batch should already be on its replacement, scored on live traffic from the first day.

Size your retest window to three months, not six. OpenAI's policy promises at least six months' notice for a generally available model, with less allowed for a specialized variant or for safety and compliance reasons. The GPT-4 series notice went out on 14 October 2026 for a 14 January 2027 shutdown, three months, so a plan that assumed six has already lost half its time.

On Azure, set versionUpgradeOption deliberately on every deployment. Standard, Data Zone Standard and Global Standard deployments are upgraded automatically at retirement, so a gpt-4o 2024-05-13 deployment becomes gpt-5.6-sol on 9 December 2026 with no code change. NoAutoUpgrade makes the deployment stop working at retirement instead. Provisioned deployments, which reserve fixed capacity, are never upgraded and have to be moved by hand. Score an automatic upgrade exactly as you would score a change you chose.

Why does a passing eval on GPT-4o not carry over to the replacement?

Treat the replacement as a new system and score it from the start, even when the prompt is identical. The agent can pick a different tool or fill a field with a different value. A tool schema can stay valid while the arguments change shape, and a structured-output schema can still parse while one field now means something else. One good completion on Luna shows none of that.

We design for this before any migration is planned. The trademark drafting system, in production with a client, logs the base model and the fine-tuned model on the same inputs against a 0 to 10 legal-integrity harness on six axes, and every draft passes that harness before an attorney sees it. A regression from a model change appears as a score movement, before an attorney reads the draft.

Our own sales roleplay coach takes the other half of the idea. Its score comes from a versioned rubric with weighted dimensions and behavioural markers, and the model only writes the qualitative read. Raw model scores drift between sessions, conversation lengths and personas. If the model also set the number, an upgrade would move every historical comparison, and nobody could tell a better rep from a different model.

Survey data points the same way: in VentureBeat's September 2026 survey of 109 opt-in respondents, 59 percent of the companies that replaced a model in a production app saw performance drop and reworked the app or went back to the original. Read it as a reason to score your own cases, not as a rate to plan around.

How do you run LLM regression testing before the model id changes?

Run the same eval cases on the current model and the replacement, each case several times, and only then move traffic in stages. The order we recommend is the same whatever the vendor, and each row below names what to record and what fails if the step is skipped.

Step	What to record	What fails if it is skipped
Inventory	Every model id in code, config and stored jobs, including dated snapshots, fine-tune ids and Azure deployment names	A forgotten snapshot stops answering on the shutdown date while the tests stay green
Pin	The replacement id, written into the eval report beside the id it replaces	An alias that moves later is a second change nobody scored
Same cases	The existing eval set, run on the current model and on the replacement	A score with no pair cannot show a regression
Repeats	A pass rate where wording may vary, and a pass on every repeat for a safety case	One green run hides a completion that will not come back
Contracts	Tool name, arguments and the structured-output schema, checked on their own	The prose still reads well while the integration stops parsing
Shadow	The new id run on live inputs, scored, with no action taken	The canary becomes the first time the live mix of requests appears
Canary	A slice of traffic, with the old id still callable	A full cutover has no rollback once config has forgotten the old id
Cost	Tokens, reasoning tokens and cache writes, on your own traces	A lower token price on a longer trace can still cost more
Retire	The old id removed from the inventory once the canary holds	The next person to read the config puts back an id that is about to stop answering

OpenAI's own migration note has four steps: record a baseline of quality, latency and cost, evaluate Luna starting at reasoning effort none, tune and retest, then send a limited share of production traffic. All four sit inside this table, so add the two rows it leaves out. A baseline that stores one completion per case cannot give you a pass rate, and a prose score cannot show that a tool call changed shape.

Score each case more than once, because a hosted model does not return the same answer twice even at temperature 0. Why LLM temperature 0 is not deterministic explains why a safety case has to pass on every repeat while a quality case ships on its pass rate. Take the cases from failures your team has already read, as our guide to LLM evaluation sets out, rather than writing new ones for the migration.

Move the eval set itself out of OpenAI's platform if it lives there. Existing evals on the Evals dashboard become read-only on 31 October 2026, and the Evals API, reusable prompt objects and the v1/prompts API shut down on 30 November 2026. Keep prompts and cases in your own repository, where the next migration can rerun them.

Keep the old id callable until the canary's scores hold, or until the shutdown date removes the choice. Deleting it from config on the day the evals pass leaves no way back if the next day's traffic is a different mix from the one you scored.

What does GPT-6 Luna change in a GPT-4o request?

Start the migration at reasoning effort none, on the endpoint you already use. It keeps the request closest to the old one, and on Chat Completions it is the only setting where tools still work. Then test higher settings against quality, latency and cost.

Reasoning effort on Luna runs from none through low, medium, high, xhigh and max, and the default is medium. A request that sends no setting therefore starts reasoning, which the GPT-4o call never did, and reasoning tokens bill as output.

Tool calls move endpoint once reasoning is on. Luna accepts function calling on v1/chat/completions only at reasoning effort none, and reasoning combined with tools goes through the Responses API. GPT-6 Astra and GPT-6.1 Sol need Responses for any tool call. Chat Completions has no shutdown date, but plan the move to Responses now for any route that will need reasoning and tools together.

Sampling settings go too: at any reasoning effort other than none, the request has to drop temperature, top_p and top_logprobs, plus logprobs on Chat Completions. A prompt that relied on a low temperature to hold its format loses that control, so the format can move with nothing on the request to explain it. Check the structured-output contract on its own at every reasoning level you test.

Compare total cost on your own traces, not the price per token. As of 22 October 2026 Luna lists $0.10 per million input tokens and $0.50 per million output, with cached input at $0.01 and cache writes at $0.125. A prompt over 272,000 input tokens is billed at twice the input and cache rates and 1.5 times the output rate for the whole request. The context window is 1,050,000 tokens, so a retrieval prompt that sat under GPT-4o's limit can now cross that price step with no change to the chunker. What a production AI agent costs works through how a longer trace shows up on the bill.

What happens to GPT-4o fine-tunes?

Work out what each fine-tune is holding before 6 January 2027, because the replacement OpenAI names cannot be fine-tuned. On the OpenAI API, fine-tunes on the January bases shut down with the base model on 14 January 2027, and from 6 January existing customers can no longer create fine-tuning jobs. As of 22 October 2026 the GPT-6 Luna model page lists the fine-tuning endpoint as not supported.

Do not start a fresh job on a GPT-4o base before the cutoff, because a model trained in late December is one you have to leave again in January. Use the eight days between 6 and 14 January as an overlap, with the replacement scored beside the fine-tune that still answers.

Sort each behaviour the tune carries by where it can live next. On the trademark system, fine-tuning carries the reasoning patterns, section boundaries and tone of an attorney's opinion, and it does not handle citation integrity. Citations come from retrieval over an approved index of case law, statutes and registry data, and validators check the five-section structure. A change of base model there would mean retraining tone and structure only, because the citations never lived in the weights.

RAG vs fine-tuning vs agentic search is the test for each behaviour. A tune that was standing in for examples can often be replaced by examples in the prompt. A behaviour that has to outlive this base needs a base you can still train and serve. An open-weight model, one whose weights you can download and host, is that path only after the same eval set passes on it.

Where should rules live so they survive a model swap?

Put sums, permissions and thresholds in code, not in the prompt or the weights. On our insurance voice agent, a required disclosure is a condition in the tool contract, so the question "can this payment run without the disclosure?" has the answer no whatever model sits behind it. A rule enforced in code carries across a migration untouched. A rule written into the prompt has to be retested on every model you try.

Keep the model id behind an interface the application owns, so a swap edits one place and no business logic. Our desktop chief-of-staff agent calls the model as a reasoning service and keeps credentials, the tool registry and execution on the user's machine. A second provider belongs behind the same interface as a tested option, not as an extra hop every request depends on, because every vendor in one flow has to be up for that flow to run.

Model routing can still send an easy case to a smaller model. Retune the threshold whenever either model changes, because a route left on the old threshold changes which model acts with no alert, and the regression gets blamed on the wrong file. TypeSafe's Jev, a typed decision model built for routing and other bounded decisions, keeps that choice outside the generative model; how to introduce Jev into agentic workflows covers where it fits.

How do you see the next OpenAI model deprecation in time?

Make the next shutdown date fail a check while the id still answers, instead of arriving as an email. The 14 January date is one row on a page that already lists later ones, including GPT-5.1, GPT-5.3-Codex and GPT-5.4 nano on 1 April 2027.

On the OpenAI API, call GET /v1/models and read each model's optional shutdown_date, which the retrieve-a-model endpoint returns as well.

On Azure, read lifecycleStatus and deprecation.inference from the Models API, and mind the naming. A model the docs call deprecated, still serving existing customers, shows in the API as Deprecating. The API value Deprecated means the model is already retired and every call returns 410 Gone. Alert on Deprecating, because a check that waits for Deprecated fires after the outage.

Run that read on a schedule against the ids in your inventory, and fail the check when a date falls inside your retest window, which the GPT-4 notice suggests should be three months. Send the failure as a pull request or a page to the person who owns the model, while there is still time to run the table above.

Common questions

When is GPT-4o deprecated on the OpenAI API?

As of 22 October 2026, OpenAI's deprecations page shuts down gpt-4o-2024-08-06, gpt-4o-2024-11-20, gpt-4o-mini-2024-07-18, gpt-4.1-2025-04-14 and gpt-4.1-mini-2025-04-14 on 14 January 2027, with fine-tunes on those bases on the same day. The older gpt-4o-2024-05-13 snapshot and gpt-4.1-nano shut down on 23 October 2026. New fine-tuning jobs end on 6 January 2027.

What replaces GPT-4o?

On the OpenAI API, gpt-6-luna replaces the January 2027 GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini snapshots, and gpt-5.6-sol replaces the older gpt-4o-2024-05-13. Azure lists gpt-5.6-sol for 2024-05-13 and gpt-5.1 for the later GPT-4o versions, and names no replacement for GPT-4.1 or gpt-4o-mini. Score the replacement on your own cases before you move traffic.

What happens to GPT-4o fine-tuned models?

On the OpenAI API they shut down with their base model, on 14 January 2027 for the listed snapshots, and existing customers cannot create new fine-tuning jobs from 6 January 2027. GPT-6 Luna does not support fine-tuning. On Azure, GPT-4o and GPT-4o mini fine-tuned deployments run until 1 October 2027 and GPT-4.1 fine-tunes until 14 October 2027.

Are the Azure OpenAI retirement dates the same?

No. Microsoft's schedule, as of 22 October 2026, retires gpt-4o 2024-05-13 on 9 December 2026 and the later GPT-4o versions, gpt-4o-mini and all three GPT-4.1 sizes on 14 April 2027. Standard deployments are upgraded to the listed replacement automatically at retirement unless versionUpgradeOption says otherwise, and provisioned deployments are not upgraded at all.

Is GPT-4o still in ChatGPT?

No. OpenAI retired GPT-4o, GPT-4.1, GPT-4.1 mini and o4-mini from ChatGPT on 13 February 2026, and said at the time that nothing changed in the API. The ChatGPT date and the API date are separate decisions, so a team calling the API should plan against the deprecations page.

How do you test an LLM before switching models?

Run the same eval cases on the current model and the replacement, several times each, and check tool-call arguments and structured-output schemas on their own. Then run the new model in shadow on live inputs, send it a canary slice of traffic, and keep the old id callable until the canary's scores hold. A single green run on the new model is one sample.

Can Chat Completions still call tools on GPT-6 Luna?

Only at reasoning effort none. Reasoning together with tool calls uses the Responses API, and GPT-6 Astra and GPT-6.1 Sol need Responses for any tool call. At any other reasoning effort, temperature, top_p, top_logprobs and logprobs have to be removed from the request. OpenAI suggests starting most GPT-4.x workloads at none.

How do you find out when an OpenAI model will be deprecated?

Call GET /v1/models and read shutdown_date on each id you use. It is a year-month-day string once a date is announced and null before that. On Azure, read lifecycleStatus and deprecation.inference from the Models API, where Deprecating means a date is set and Deprecated means already retired. Fail a scheduled check while the id still answers.

Agentic workflows vs AI agents, and when not to build one

Tirth Gajjar — Tue, 27 Oct 2026 18:40:19 +0000

An agentic workflow is a multi-step process in which a language model does part of the work, such as drafting, classifying, retrieving or calling a tool. The term covers two designs: in one, code you wrote picks every next step, and in the other the model picks it, a design most people call an AI agent.

In this guide a workflow is the first design and an agent is the second. The question for a build is which of those your task needs, and the answer comes from the path the work takes rather than from the label on the ticket.

Criterion	Scripted workflow	RAG or search	Agent
Fixed path	Use it. The code walks the steps you listed.	Use it when the path is one query and one answer.	Skip it. The model would re-choose steps you already listed.
Multi-step judgment	Use it only when every branch is a rule you can write.	A retrieved passage does not choose the next action.	Use it when the next judgment depends on the last result.
Tool recovery needed	Use it when the retry is the same step, or a known fallback.	A search miss is a retrieval problem. Fix the index.	Use it when the error has to choose a different tool.
High blast radius, no gate	Keep the write out. The read path can stay in code.	Keep a permission filter on the query.	Do not start. The loop does not contain a bad write.
Measurable success	Score the output of each step.	Score recall, and whether the answer stays inside the passage.	Score the stop, and a final state outside the model's claim.
How Bigcircle handles this	On the fund reconciliation build, the matching rules are explicit and versioned in code, and a person approves only the resolution of an exception.	On Prospex, semantic and keyword search run together, because their failure modes are complementary.	On the chief-of-staff build, the agent picks the next tool from the last result, and a critic decides whether to continue, retry or escalate.

Most of what we have built is a workflow in that sense, by design. Of the ten systems in our case studies, the ones that turn filings, custodian feeds, legal drafts or meeting audio into records run as fixed pipelines, and where a model appears it sits inside one step. They are the ESG filing pipeline, the fund reconciliation system, the trademark drafting system and the meeting transcription app.

Even the DevOps agent turns each request into a structured spec and an explicit chain of tool calls, because a reviewer can only build judgement about output whose shape is stable. The chief-of-staff agent needed a loop. It works across more than a hundred registered systems, much of it behind logins with no usable API, and each step reads what the last one returned before choosing the next.

A demo hides the difference. Each shape can call a model, show a tool call and return a sentence that sounds finished. The split shows when a step fails or a record is missing. A workflow stops on the branch you wrote and retrieval says the passage is absent. An agent has to choose another tool, or stop, based on what it just saw.

If you cannot point at a case where that choice is real, the loop is overhead you will spend the next quarter tracing. This guide settles the vocabulary first, then takes the three shapes in the order a team usually reaches for them and puts five signals in one matrix. The last two sections cover the signals teams most often skip. One is a write that can do damage before any gate exists, and the other is a success check that only the model can see.

What is an agentic workflow, and how is it different from an AI agent?

The phrase is used both ways, so ask which one a vendor or a colleague means before you compare designs. Most of the pages that rank for it mean the second design, a model choosing the next step while it runs. Anthropic's Building effective agents calls that same system an agent and keeps "workflow" for code paths written in advance. Two proposals can both say "agentic workflow" and describe opposite systems, and the difference decides how you test them.

The usual list of workflow patterns is chaining one call into the next, routing to a branch, running calls in parallel, an orchestrator handing out subtasks, and one call grading another's draft. Review all but one of them as workflows, because code still owns the order.

The orchestrator is the exception: a lead model chooses the subtasks, so part of the path is already the model's choice, and it needs the checks of an agent. Multi-agent systems covers that pattern once the workers are agents themselves.

We keep those words because they name the decision you have to make. A scripted workflow is code picking the next step, with a model inside some of the steps. An agent is the model picking it. Plain automation, such as a rules engine, has no model at all, and the next section shows how far that gets.

When a proposal uses the phrase "agentic workflow", ask which step the model chooses and what code checks the choice. If the model chooses none, you are looking at a scripted workflow with model calls in it, which is usually the right thing to build. If it chooses the next tool, the rest of this guide applies before you sign off on it.

When is automation or a scripted workflow enough?

Rule-based automation and a scripted workflow both keep the path in code, and an AI agent does not. A scripted workflow is the right shape when someone who knows the job can write the steps, including the branches. Invoice extraction, support triage with a fixed set of labels and document classification usually sit here. The code walks classify, retrieve, draft, check and send, a model fills the steps that need language, and the step after each model call is the one you named.

The fund reconciliation system we built shows how far rules go. Custodians agree on nothing, not the format, the field names or the sign conventions, so each custodian gets its own mapping rather than one parser full of conditionals. The matching rules are explicit and versioned, because they are the domain knowledge. A person approves at one step only, resolving an exception, because that is where human judgement enters the data. There is no model call anywhere on that path.

The common advice runs the other way. OpenAI's practical guide to building agents points agents at rule sets that have grown unwieldy and costly to update. Take that advice when nobody can state the rules. When the rules are the domain knowledge, as they are in reconciliation, versioning them in code costs less than asking a model to rediscover them on each run, and an auditor can read the version that applied.

Put an agent on a path like that and the model spends the run re-deciding a sequence the code already had. A failure then looks like a choice, so you open the trace, when the missing piece was a branch nobody wrote. With the order of steps in code, a reviewer can point at the branch that fired.

The trademark drafting system is a pipeline for a different reason. Each stage has a defined failure behaviour, and a failed validation triggers a bounded retry that may draw only on retrieved evidence. Falling back to unconstrained generation is the most common way we see systems of that shape go wrong. A loop that could choose its own next step would reopen that path.

A branch you can state as a rule is still a workflow. If the amount is under the cap and the order is inside the return window, draft the refund and wait for a person to approve it. The model can draft the note while the cap stays in the code that checks the amount. A value you can compute is computed in code and passed in.

When is retrieval enough on its own?

Retrieval is enough when the failure is a missing or stale fact and one query can find the passage that holds it. A refund window that changed in March, a policy clause, or a filing the model was never trained on are RAG problems, or plain search problems. An agent does not repair a weak index, and RAG vs fine-tuning walks those layers in the order they cost.

Prospex, our own research engine, has to answer both "how do they make money", which is a semantic question, and "are they using Snowflake", which is a keyword question. Pure vector search will happily return a passage about a different warehouse. So semantic and keyword search run together rather than as a fallback chain, because their failure modes are complementary. A custom chunker keeps a role and its company in the same chunk.

A loop that searches until it feels sure calls the same weak index several times and then writes a fluent answer over thin evidence. You pay for several model calls to hide a recall problem that one better query would have shown in the log. On Prospex, too little retrieval produces a short profile, which is the safe failure, because a short profile is visibly short and an invented one is not.

Some questions do need more than one retrieval. The ESG pipeline gets comparative research questions, such as one about "the same subsidiary two years earlier", which pure vector search cannot express. Pure graph traversal fails the other way, because it cannot find a narrative claim that was never structured. One retrieval cannot answer a comparison, so the question is broken into steps: select sources, combine graph and vector evidence, build claims, check them and attach the source page to each one.

On the ESG pipeline that is a sequence of named steps: several retrievals in a known order, not a model left to decide what to search next. If you can name the second query the first result forces, you can often write it as a step. An agent loop starts to pay only when the next query depends on what came back in a way you cannot list.

When do you need an AI agent instead of a workflow?

An agent earns its cost when the next action depends on an observation you could not encode as a branch. The model selects a tool, reads the result, and either continues or stops. How to build an AI agent covers that loop, and this guide covers the decision to build it.

On the chief-of-staff agent each step runs think, act, observe, and the raw observation goes back into the loop rather than being summarised away. A critic then decides whether to continue, retry or escalate. Without it, one bad observation spreads through every later step and the agent claims success on work it never completed.

The model is not handed the whole tool registry either. It queries the registry and chooses among a handful of relevant contracts, because past roughly 30 tools in the prompt it started picking plausible but wrong ones.

Tool recovery is where the two shapes part. Repeating a call that timed out is still a retry inside a workflow. A not-found error that means "try the archive, and if that is empty, ask for the account id" is a choice among tools. If every recovery you care about is already a row in a table, keep it in the table. Start the loop only when the error has to pick the tool.

A miss inside the loop runs longer than it would in a workflow. A wrong branch fails one step, while a wrong choice can take several more tool calls before anyone sees a bad final state.

The loop pays for its flexibility in latency and tokens, and AI agent cost shows how each extra turn resends the transcript. The loop needs a budget, a stop rule and a check that does not trust the model's claim that the work is done.

How do five signals place one task?

Five signals are enough to place a task, and one product can sit in different columns on different rows. Read down the rows for the task in front of you before you pick a column for the whole product. The figure repeats the comparison at the top of this page.

Triage can be a workflow, the policy answer can be retrieval, and the ticket whose next system depends on the last lookup can be an agent. A single label on the product hides which of the three you shipped, and the next failure goes to the wrong owner.

The cells are fit judgments you can argue with on a real task. Green marks a fit for that signal. Amber means the shape works only when the condition in the cell is true. Red means stop and build the other shape, because this one adds a way for the bad outcome to happen.

A task that is green in the workflow column and red in the agent column should stay a workflow even if the roadmap calls for an agent. When a row is red because a write can do damage and no gate exists yet, read the next section before you add a loop.

What happens when a bad write has no gate?

A write that can move money, change access, send a customer message or alter production state can do real damage, and a loop does not make that smaller. If you do not yet have a gate that can refuse the call, a credential scoped to that action, and a check on the final state outside the model, keep the write out. The read path can still be a workflow or a retrieval app.

Adding an agent so the model can be careful puts the care in the prompt, which is the place a later instruction can argue with. The systems we have built put it in code. On the DevOps agent, an engineer approves an architecture diagram before anything is applied. Changes then go through the provider's change-set mechanism, which applies a group of changes as one unit, so a partial failure does not leave orphaned resources.

On our insurance voice agent, a payment cannot execute without its required disclosure, because the condition sits in the tool contract and not in the prompt.

The same agent shows what a failed write should do. A request that cannot be completed does not turn into an approximation. It escalates to a person, with the dialogue and the actions already taken attached, rather than doing something adjacent to what the caller asked for.

Human in the loop AI agents covers the product boundary, where answering can stay read-only and the first write is one kind of action. AI agent governance covers how one action gains authority after the evidence for it exists. Both assume you have already decided that an agent is the right shape for the path.

What do you score once the shape is chosen?

Success has to be a state something other than the model can check, or you cannot tell the three shapes apart after launch. A workflow scores the output of each step: the label, the drafted field, the branch that fired. Retrieval scores whether the right passage was in the set and whether the answer stays inside it.

An agent scores the trace and the final record. On the chief-of-staff agent, the claim that a task is complete is compared against the actual state of the target application, which catches claims of success on work that never happened. Work it cannot finish comes back as a flagged exception with its trace attached. The DevOps agent keeps one record that links the request, the account facts, the generated template, every validation failure and the resulting resources.

If the only signal is the model's closing sentence, every shape will claim it is done. A workflow claims done when a branch was skipped, retrieval when the passage was missing, and an agent after a tool error it folded into the prose.

Name the check before you pick the shape, then build the shape that can produce that check with the least machinery. Evals keep that check stable from run to run, and LLM evaluation covers building the cases. Until they exist, a demo that felt like an agent is not evidence that you needed one.

Observability on a loop is the trace of that check. Store the prompt, the tool list and the stop reason with the run, so a wrong final record can be tied to the choice that produced it. A workflow needs less of that, because the branch is already in the code. AI agent monitoring covers reading those traces once the system is live.

Common questions

What is an agentic workflow?

An agentic workflow is a multi-step process in which a language model does part of the work, such as drafting, classifying, retrieving or calling a tool. Some writers use the term only when the model chooses the next step. Anthropic calls that design an agent and keeps "workflow" for systems whose steps run through code paths written in advance. Ask which meaning is intended before you compare two designs.

What are the types of agentic workflows?

The usual patterns are prompt chaining, routing, parallel calls, orchestrator-workers and evaluator-optimizer. In all but one, code still owns the order, so review them as workflows. In orchestrator-workers a lead model chooses the subtasks, so part of the path is the model's choice and it needs the checks of an agent. Pages that use the term for agents list planning, tool use and reflection instead.

Is an agentic workflow the same as an AI agent?

Not in the sense this guide uses. In a scripted workflow, code picks every next step and a model fills some of the steps. In an AI agent, the model picks the next step and the next tool from what the last one returned. Many vendors call both of them agentic, so ask which step the model chooses and what code checks that choice.

What is the difference between AI agents and automation?

Rule-based automation follows steps and rules written in code, with no model in it. A scripted workflow keeps that fixed path and adds a model inside some steps. An AI agent lets the model choose the next action. Bigcircle's fund reconciliation system kept its matching rules explicit and versioned in code, because the rules are the domain knowledge and are versioned as such.

Do we need an agent, or is a workflow enough?

Build a workflow when a person can write the steps and the branches, including retries that always repeat the same call. Build an agent when the next tool depends on a result you could not list before the run. Most document and reconciliation systems Bigcircle has built are fixed pipelines, and the agent loop in its case studies is the build that works across many systems.

When is a RAG or search app enough on its own?

When one query can find the passage and the answer has to stay inside it. Fix the index, identifier matching, ranking and the permission filter on the query before you add a loop. If a question needs several retrievals, as a comparison across years does, try writing them as a fixed sequence first, and use an agent only when the next query cannot be known in advance.

Should a high-risk write start as an agent?

No. A call that can move money, change access, send a customer message or alter production state needs a gate before any loop may request it. If the gate, a scoped credential and a check on the final state are missing, keep the write out and ship the read path as a workflow or a retrieval app. Authority for one kind of action is a later release.

How do you know the run succeeded?

Check a state outside the model's closing sentence. A workflow should show the branch that fired and the field it produced, retrieval should show the passage and whether the answer stayed inside it, and an agent should show its tools, any recovery, any approval and a final record that matches what was expected. If you cannot name that check, you cannot yet tell which shape you need.

PII tokenization for LLMs: tokenization vs masking, with code

Tirth Gajjar — Sun, 25 Oct 2026 18:40:08 +0000

PII tokenization replaces each piece of personal data in a prompt with a stand-in token before the request leaves your network, and puts the real value back after the model replies. The provider sees the shape of the record, one customer, one account, one phone number, and never the values. Your users still get a complete answer.

Plenty of AI projects stall at the legal review, when someone asks what leaves the network and the answer is everything: the support ticket, the customer's email, the account number. We build the layer that changes that answer: a small piece of code in the request path that adds single-digit milliseconds to a call. This guide covers the design choices in it, the Python, where an entity recognizer such as Presidio fits, what the tokens mean under GDPR, and the CI test that turns "we tokenize PII" into something a compliance team can check.

PII tokenization vs masking: which keeps the answer useful?

Tokenize when the model has to reason about who is who, and mask only when it does not. Masking replaces every email with [EMAIL] and every number with ****. It is simple, and it makes the answers worse, as soon as a prompt holds more than one of anything.

Say a ticket mentions two customers and two accounts. After masking, the model sees [EMAIL] twice and **** twice. It cannot tell which account belongs to whom, so it cannot write "refund account X for customer Y", because the structure it needed has gone.

Reversible tokenization keeps that structure. Each sensitive value gets its own token, and the same value always gets the same token within a request, so two different emails get two different tokens and the model can still say "this customer, that account".

Method	What the model sees	Reversible	Keeps who-is-who	Use it for
Redaction	The value deleted	No	No	Fields the model never needs, such as a card's CVV
Masking	`[EMAIL]`, `****`	No	No	Logs and analytics where identity does not matter
Hashing	A fixed digest of the value	No	Yes, if salted consistently	Joining records without reading them
Reversible tokenization	A format-preserving stand-in	Yes, with the per-request map	Yes	Any prompt whose answer has to name the real customer or account

Two properties make tokenization work in a prompt. It is reversible: a private map from token to value lives for the life of the request, so the answer can be restored. And it is format-preserving: an email token still looks like an email, and a 16-digit account number becomes another 16-digit number, so prompts, parsers and validators downstream keep working and the model treats the field as it would treat the real one.

Is tokenized data still personal data under GDPR?

For you, yes, and for the provider possibly not, which is the argument to bring to legal review. A token map you keep is pseudonymization in GDPR's own terms: the data can no longer be tied to a person without additional information held separately. Pseudonymized data stays personal data for whoever holds that additional information, so tokenizing does not take your own processing out of scope.

The provider's position is different. In EDPS v SRB (C-413/23 P, 4 September 2025), the EU Court of Justice held that pseudonymized data passed to a recipient is not always personal data from that recipient's side, when the recipient cannot reverse the pseudonymization or identify people by other means. A provider that receives user_1ec32386@example.invalid, under a key that changes every request and never leaves your process, is the setup that ruling speaks to.

Describe the layer to your counsel as pseudonymization, and have them read the ruling against your setup. Anonymization is the wrong word for it, because the map you keep can reverse every token.

Which fields should you tokenize?

Decide with whoever owns the risk, from your real data, before writing code. Not every field is personal data, and encoding everything destroys the answer as surely as masking does. Over-redaction is its own failure.

The output is a short list of data classes, such as email, account number and phone, with a rule per class and per jurisdiction. On the telecom marketing platform we are building agents for, subscriber data spans several jurisdictions, and the analytics agent works through a governed data-access layer that masks, aggregates or excludes PII by policy before the agent sees a row. That policy is enforced in the access layer, never in the prompt, because a prompt is a request the model can ignore.

Structured classes such as emails, account numbers and phone numbers can start with pattern rules. Names, addresses and free-text identifiers need entity recognition, covered further down.

How do you tokenize PII before the provider call?

Create one boundary object per request that detects sensitive values, swaps each for a token, remembers the mapping and writes a record of what it encoded.

import hmac, hashlib, re
from dataclasses import dataclass, field

# Pattern rules for the fields your risk owner marked sensitive.
# In production, add an NER pass for names and addresses.
PATTERNS = {
    "EMAIL":   re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b"),
    "ACCOUNT": re.compile(r"\b\d{10,16}\b"),
    "PHONE":   re.compile(r"\+?\d[\d -]{8,}\d"),
}

@dataclass
class Boundary:
    key: bytes                                   # per-request secret, never logged
    vault: dict = field(default_factory=dict)    # token -> raw value
    seen: dict = field(default_factory=dict)     # raw value -> token
    record: list = field(default_factory=list)   # what was encoded, without the values

    def _digest(self, kind, value):
        return hmac.new(self.key, f"{kind}:{value}".encode(), hashlib.sha256).hexdigest()

    def _surrogate(self, kind, value):
        h = self._digest(kind, value)
        digits = "".join(str(int(c, 16) % 10) for c in h)
        if kind == "ACCOUNT":
            # Same length, all digits. The "99" prefix marks it as a stand-in.
            return "99" + digits[: len(value) - 2]
        if kind == "EMAIL":
            return f"user_{h[:8]}@example.invalid"
        if kind == "PHONE":
            return "+00 " + digits[:10]
        return f"{kind}_{h[:8].upper()}"

    def encode(self, text: str) -> str:
        for kind, pattern in PATTERNS.items():
            def swap(m):
                raw = m.group(0)
                if raw in self.vault:            # already a token, leave it
                    return raw
                if raw not in self.seen:
                    tok = self._surrogate(kind, raw)
                    self.seen[raw] = tok
                    self.vault[tok] = raw
                    self.record.append({"kind": kind, "token": tok})
                return self.seen[raw]
            text = pattern.sub(swap, text)
        return text

Three choices in it are deliberate.

A fresh key for every request

Tokens come from an HMAC of the value under a random key. The same email gives the same token inside one request, so the model can connect two mentions of one customer. Across requests the key changes, so the provider cannot link tokens between calls and build a profile. The changing key also puts the provider in the position the EDPS v SRB ruling describes.

Tokens that cannot be real

example.invalid is a reserved domain that never resolves, and +00 is not a country code. If a token ever escapes into an outgoing email or a log, it is visibly a stand-in and not someone else's real address.

A record of tokens, not values

The per-request record lists the kinds of data encoded and the tokens used, and never the values. You show that record in the review afterwards.

Run on a support ticket:

import os
b = Boundary(key=os.urandom(32))
ticket = ("Customer priya.shah@acme.in says account 4111222233334444 "
          "was charged twice. Call back on +91 98765 43210.")
print(b.encode(ticket))
# Customer user_1ec32386@example.invalid says account 9955708008262310
# was charged twice. Call back on +00 0573126352.

Your tokens will differ, because the key is random, but the model still knows there is one customer, one account and one phone number, and what happened to them. It does not know who they are.

How do you restore the real values after the reply?

Decode with a lookup in the other direction, longest token first so one token never clips another:

    def decode(self, text: str) -> str:
        # Longest tokens first, so one token never clips another.
        for tok in sorted(self.vault, key=len, reverse=True):
            text = text.replace(tok, self.vault[tok])
        return text

If the model replies "Email user_1ec32386@example.invalid to confirm the refund on 9955708008262310", your user sees the real email and account number, using values your own systems already hold.

Models sometimes reformat what they copy, so a phone token can come back with different spacing. Tell the model in the system prompt to copy identifiers exactly. Then, after decoding, look for anything still shaped like a token: example.invalid, a +00 number, a 99-prefixed account. A leftover token means the model changed or invented it. Treat that response as failed and retry it or route it to a person, rather than sending a half-decoded answer.

Where does Presidio fit for names and addresses?

Use an entity recognizer for detection and keep your own tokens for the replacement. Patterns catch emails and account numbers, and they miss names, street addresses and identifiers written into free text. Presidio, the open-source PII toolkit Microsoft published, pairs named-entity recognition with pattern recognizers in its analyzer, and its anonymizer offers replace, redact, mask, hash and encrypt operators, with a decrypt operator that reverses encryption.

Presidio's encrypt and decrypt pair is reversible, and it is not format-preserving: an encrypted name comes back as ciphertext, which a model reads as noise rather than as a person. Feed the analyzer's results into the boundary above, so a detected name gets a name-shaped token from the same per-request map and decodes with everything else. Every detector misses something, so the leakage test below sits behind all of it.

How do you prove no PII reached the model provider?

Write a test that fails the build when a raw value reaches the provider. "We tokenize PII" is a claim, and this test is the evidence. Put a fake provider behind your real code path, capture everything that would have been sent, and assert that none of your known sensitive fixtures appear in it:

import os, json
from boundary import Boundary

RAW_FIXTURES = ["priya.shah@acme.in", "4111222233334444", "+91 98765 43210"]

class CapturingProvider:
    def __init__(self):
        self.sent = []
    def complete(self, payload: dict) -> str:
        self.sent.append(json.dumps(payload))
        return "ok"

def handle_ticket(text, provider):
    b = Boundary(key=os.urandom(32))
    payload = {"messages": [{"role": "user", "content": b.encode(text)}]}
    return b.decode(provider.complete(payload))

def test_no_raw_value_reaches_provider():
    provider = CapturingProvider()
    handle_ticket("From priya.shah@acme.in, card 4111222233334444, "
                  "phone +91 98765 43210", provider)
    for body in provider.sent:
        for raw in RAW_FIXTURES:
            assert raw not in body, f"raw value leaked: {raw!r}"

Run it in CI on every pull request, and grow the fixtures. Each time someone finds a format the rules missed, such as an email with a plus sign or an account number with spaces, add it as a fixture before fixing the rule. The suite becomes the list of everything you have promised not to send, and we hand it to a compliance team instead of a policy document.

Test the whole path, not just the boundary class. Leaks rarely start in the tokenizer. They start when someone later adds a helpful field to the prompt, the raw customer record for context or the user's profile in the system message, and it skips the encoder. A test on the real request path catches that on the pull request that adds it.

Where does the tokenization layer sit?

Put it in the request path, directly in front of the provider call, so every call goes through it. A boundary that each feature calls when it remembers to will be skipped by the feature written in a hurry. Keep it fast as well. In our builds the encode and decode step adds single-digit milliseconds, so there is no latency argument for switching it off under load.

Three limits shape the rest of the design:

It protects the next request. Data already sent to a provider is a separate conversation with that provider.
Detection misses things. Patterns and entity recognition both need the leakage suite behind them.
The token map is now the sensitive object. It lives for one request, in your process, and never goes into logs or traces.

Credentials follow the same rule. On Solarpunk, the desktop agent we built, secrets sit in an encrypted vault on the user's machine and are injected into the tool adapter when a tool runs, so they never travel into the prompt. Whatever the model never receives, a prompt injection cannot ask it to repeat.

We build this layer for teams whose AI work is waiting on that legal review. PII redaction and data boundaries covers how an engagement runs, from classifying fields in week one to the leakage suite your compliance team reviews.

Common questions

What is PII tokenization?

PII tokenization replaces each piece of personal data in a prompt with a stand-in token before the request goes to the model provider, and restores the real values after the reply. The provider sees the structure of the record and never the values, and the user still gets a complete answer.

What is the difference between tokenization and masking?

Masking replaces every email with the same placeholder, so the model cannot tell which account belongs to which customer, and the original cannot be restored. Reversible tokenization gives each value its own stand-in within a request, so the structure survives and the real values go back in after the reply.

Is tokenized data still personal data under GDPR?

For the organization holding the token map, yes: it is pseudonymized data, which stays personal data. For a recipient that cannot reverse the tokens or identify people by other means, the EU Court of Justice held in EDPS v SRB (C-413/23 P, 4 September 2025) that it is not always personal data. A fresh key per request that never leaves your process puts the model provider in that second position.

Can Microsoft Presidio do reversible PII tokenization?

Presidio's anonymizer has an encrypt operator and a decrypt operator that reverses it, so its output is reversible. It is not format-preserving, because an encrypted value comes back as ciphertext. Bigcircle recommends an entity recognizer such as Presidio's analyzer for detection, with format-preserving tokens generated from your own per-request map.

Does PII tokenization slow down the LLM call?

In Bigcircle's builds the encode and decode step adds single-digit milliseconds, which is small next to the model call itself. Keeping it that fast matters, because a boundary that teams switch off under load protects nothing.

How do you prove no personal data reached the provider?

Put a fake provider behind the real code path, capture everything that would have been sent, and fail the build if a known sensitive fixture appears in it. Run the test in CI on every pull request, and add a fixture each time someone finds a format the rules missed.

How much does an AI agent cost to build and run?

Tirth Gajjar — Fri, 23 Oct 2026 18:35:02 +0000

AI agent cost has three parts: the build, the run and the keep-alive. The build is the system that can take a real task. The run is the model calls, tool calls and retries each task then spends, and the keep-alive is the work after launch, when a tool, a model or a document changes. A per-token price covers one line inside the run, and one combined number hides which of the three is about to move.

Our case studies show where the work went. On the DevOps agent, the build was mostly about reading the live AWS account, checking every template and putting a human review step before anything applied. On the chief-of-staff agent, the visible cost of tool definitions in the prompt turned out to matter less than the wrong tool choices that came with them.

The sections below take the build, then the run with a worked token calculation at prices as of 22 October 2026. The keep-alive and a way to compare two quotes come next. The last two sections cover the cap that ends a runaway loop before the next paid call, and the ledger of what each run spent. If a script or one query can do the job, agentic workflows vs AI agents comes first, and none of this bill starts.

How much does it cost to build an AI agent?

The build price follows the scope, and most of the scope sits around the model rather than in it. The build is everything that has to work before the first real task leaves the team: the tools the job needs, permission to call them, an eval set, and a path for the step that fails. Choosing a model is one decision inside that work, and usually not the long one.

The DevOps agent shows where the time goes. A raw model invents resource types, writes templates the account rejects for quota or region, and drifts between runs, and a failed stack can take minutes to unwind, so "apply and see" is not a usable loop.

The build therefore had to read live limits, quotas and existing resources before planning, check every template in a loop of structural and semantic checks, and show an engineer an architecture diagram before anything applied. Delivery took roughly four months.

The chief-of-staff agent spent its build on reach. It connects to more than a hundred systems, and a large share of an executive's work sits behind single sign-on and multi-factor login with no usable API. We argue that a routine the agent can finish 70 percent of still needs a person to hold the sequence, so it saves almost nothing. That is why reach costs build time on an agent like this.

Each integration is a tool contract, a login path and a failure the agent will meet in normal use. MCP authorization covers the check before each call, LLM evaluation covers the eval set, and how to build an AI agent puts those decisions in order. A build quote that lists model access and leaves out the access check and the eval set will be rewritten after the first real task fails.

How the build is bought is a separate question. AI pilot to production compares production hardening, a fixed-scope build and a studio-led programme. A fixed-scope build is priced against a scope agreed before work starts, so the useful comparison between two quotes is which of the three parts each one names.

Why does an AI agent cost more to run than one reply?

A run costs more than one reply because the agent decides how many steps to take, and each step sends the whole history again. A workflow pays one model call per step you wrote. An agent pays a number of calls that depends on the trace, and the input on each call grows with every tool result.

A worked example makes the shape concrete. The token counts are assumptions chosen for the example, and the prices are OpenAI's list rates for GPT-6 Luna as of 22 October 2026: $0.10 per million input tokens and $0.50 per million output tokens.

Take a system prompt and tool definitions of 6,000 tokens, a task of 10 turns, 300 output tokens per turn for the tool call, and 2,000 tokens of tool result added to the transcript after each turn.

	One reply	Ten-turn agent task
Input tokens	6,000	163,500 (6,000 on the first turn, 26,700 on the tenth)
Output tokens	300	3,000
Cost at the October 2026 Luna rates	about $0.00075	about $0.018

Ten turns cost about 24 times one reply, not 10 times, because each turn pays again for every earlier turn's transcript. At 10,000 tasks a day that is about $179 a day on the run line.

Reasoning adds to it. OpenAI bills reasoning tokens as output tokens, and Luna defaults to reasoning effort medium. If each turn spent 1,000 reasoning tokens, another assumption, the same task would cost about $0.005 more, roughly 28 percent on top. The example also assumes reasoning from earlier turns is not sent again. Check that before you trust the figure, because the GPT-5.6 family carries earlier reasoning into the next turn by default. On a model that does, those tokens add to the input of every later turn. LLM model migration covers why a model change can move this line while the per-token price falls.

Tool definitions are part of the input on every turn, and the chief-of-staff agent found where listing all of them in the prompt stops working. Direct tool-calling with every tool listed worked up to roughly thirty tools. Past that, the model started choosing plausible but wrong tools, and we found that the context cost was the visible symptom while the selection error was the expensive one. The fix was to let the model query a tool registry and choose among a handful of relevant contracts, which kept the working context small at more than a hundred tools.

Prompt caching cuts that bill more than anything else in the example, because it discounts a repeated prefix, and an agent's growing transcript is one. As of 22 October 2026, caching is on by default on OpenAI's supported models, with a minimum prefix of 1,024 tokens on GPT-5.6 and later. On those models a cache read costs a tenth of the input rate and a cache write 1.25 times it.

An entry lasts 30 minutes by default, and each read refreshes it. OpenAI's caching rules do not name Luna, and Luna's model page lists cached input at $0.01 per million, the same tenth, so we assume the same write rate.

Applied to the example, if every turn after the first reads the earlier transcript from cache and writes only its new 2,300 tokens, the ten-turn task comes to about $0.006 instead of $0.018. That is 136,800 tokens read at the cached rate, 26,700 written at the write rate, and the same 3,000 output tokens. It assumes each turn arrives inside the cache lifetime with the whole prefix unchanged, so a retry that edits anything early in the transcript pays full price again from that point.

Caching at the application layer saves a different cost. Prospex, our own research engine, runs cache-first, so a rerun reuses earlier extraction and retrieval instead of calling external services again. By our estimate that cut repeated external API calls by 50 to 60 percent, and we watch the cache hit rate as its cost metric.

What does an AI agent cost after launch?

Keep-alive is the work after launch. It covers rerunning the eval set when a tool, a prompt or the model changes, a person who can act when a run stops, and traces that someone can still read when finance asks what happened. None of it shows on the launch invoice, and all of it is on the calendar afterwards.

Integrations are where it starts. We saw what point-to-point wiring does to maintenance on the chief-of-staff build: a vendor changes a response, and the break shows up somewhere unrelated days later. Our answer there was a uniform tool layer, where each tool is a typed contract with declared inputs, outputs and side effects.

Human review is the other running line. On the trademark drafting system, attorney corrections are the most valuable signal in the system, because they measure the gap automated scoring cannot see. That review does not end at launch. On the ESG filing pipeline, the slow failure is framework version drift, which is why the mapping version is recorded on each mapped fact.

Observability on an agent is the trace of each run, stored with the run. AI agent monitoring separates the quality of the answer from the health of the path that produced it. Name the person who owns the stop before launch. A stop with nobody assigned to it is only a log line, and the loop behind it keeps spending until someone reads the invoice.

How do you compare two AI agent quotes?

Compare two quotes part by part, or a missing line will look like a saving. Two proposals can describe the same agent and still leave out different parts, and the shorter one has often left off the keep-alive or the stop.

Part	What moves it	What a thin quote skips
Build	Tools, credentials, evals, and the path for a failed step	Everything except a prompt and a happy path
Run	Turns, tool calls, retries, fan-out, and a transcript that grows	Every turn after the first
Keep-alive	Eval reruns, a person on the stop, review, and traces you can still read	A named owner after launch

How do you enforce a budget in the request path?

A budget holds only when the check runs before the next provider call and a breach ends the run. A dashboard that climbs and an email that arrives later have already paid for every call between the threshold and the person reading it. The cap is code on that path, and it limits turns, tool calls, spend, and how many child agents one run may start.

Fan-out, where one run starts several child agents, is the case a per-call limit misses, because each child looks small on its own. If every child gets a fresh counter, together they spend the parent's allowance several times over. Make each child draw on what the parent has left, and have the parent refuse another child once the count or the allowance is gone. Multi-agent systems covers when the children are worth starting at all.

Retries need a bound of their own. We bounded them on the chief-of-staff build, because an agent that retries indefinitely against a broken vendor is worse than one that stops: it burns cost and time while looking busy. The DevOps agent bounds revision the same way, and a template that will not come clean after bounded revision is escalated as a request the agent cannot safely fulfil. Count failed calls in the total, because the provider already did.

A limit written into the prompt can be talked past, and a counter in code cannot. Guardrails that live only in the prompt can give way the first time a tool result or a user message argues for one more step.

Account-level caps from the cloud providers are a backstop behind that check. They act on a whole project after many runs have spent, they cannot tell a retry storm from a busy day, and a paused project takes your working runs down with the broken one.

AWS spend limits, introduced on 16 September 2026 and still rolling out to a limited set of customers, pause a project and stop its resources when its monthly usage reaches the limit. An opt-in control pauses the highest-cost resources about four days before the limit, chosen from five services that include Lambda and Bedrock.

Google Cloud's Spend Caps, in public preview since 29 July 2026, restrict further usage of one service in one project once the monthly cap is reached. Both are worth turning on, and the loop still needs its own check before each call.

Monitoring without a stop is the gap that costs most. In one incident Jatin Bansal collected, a team running four LangChain agents had monitoring and no enforcement. An analyzer and a verifier began looping with each other in November 2025, and the bill reached $47,000 over eleven days. The figures are as Bansal reports them.

Set each cap from runs you have already finished rather than from a framework default, which is calibrated to someone else's task. Leave room above the longest run that completed normally, so a stuck loop hits the cap and a normal task does not. A cap set on a guess either cuts good tasks or lets a loop run.

Where does the spend record live?

The ledger has to outlive the agent, or a loop can erase the only record of what it spent. Write the run id, the tool, whether the call was allowed or refused, the token counts and the stop reason for each call to a store the agent's credential cannot delete or rewrite. Include the calls that failed and the calls each child made, or your meter will sit below the invoice.

Reconcile your meter with the provider bill every month as two sums: billed spend from the invoice, and the cost of every attempt from your own traces, including retries, timeouts, cancellations and outputs you rejected.

A gap left after failed attempts are included is work you are not tracing, such as an eval job, a warm-up call or a second service on the same key. Close that gap before you call either number the cost of a successful task.

Common questions

How much does an AI agent cost?

AI agent cost has three parts, the build, the run and the keep-alive, and each moves for a different reason. The build is tools, credentials, evals and the path for a failed step. The run is turns, tool calls, retries, fan-out and a transcript that grows. The keep-alive is eval reruns, review and traces after launch. A price per token covers one line inside the run.

How much does it cost to build an AI agent?

The build price follows the scope, and most of the scope sits around the model: tools, permissions, an eval set and the path for a failed step. Bigcircle's DevOps agent took roughly four months, spent mostly on reading the live AWS account, checking every template and a human review step. Compare quotes on which of those parts each one names.

Why does an agent task cost so much more than one reply?

Each turn resends the whole transcript, so input grows with every tool result. In this guide's worked example at GPT-6 Luna's October 2026 rates, a ten-turn task with a 6,000-token prompt and 2,000-token tool results costs about 24 times one reply, not 10 times. Reasoning tokens are billed as output and add to that.

Does prompt caching fix the run cost?

It helps on a repeated prefix and nowhere else. On OpenAI's API as of October 2026, caching is on by default, with a 1,024-token minimum on GPT-5.6 and later, where a cache read costs a tenth of the input rate and a write 1.25 times it. In this guide's example that takes the ten-turn task from about $0.018 to about $0.006. A retry that edits the early transcript pays full price again.

How do you stop a runaway agent before the invoice?

Put a hard cap in the request path and check it before the next provider call. Cap the turns, the tool calls, the spend and the number of child agents, and make each child draw on the parent's remaining allowance. When any cap breaks, end the run and keep the partial result. An alert after the threshold has already paid for the calls in between.

Are cloud spend caps enough on their own?

No. AWS spend limits and Google Cloud Spend Caps, both announced in 2026, pause a project or a service once a monthly amount is reached. They are a useful backstop, but they act after many runs have spent, they cannot tell a retry storm from normal load, and they stop healthy runs along with the broken one.

How do you reconcile your meter with the provider bill?

Sum what the provider invoiced for the period, and sum what your traces recorded for every attempt, including retries, timeouts and outputs you rejected. Compare the two before you call either one the cost of a successful task. A remaining gap is work you are not tracing, such as an eval job or a second service on the same key.

Why AI agents report unfinished work as done

Tirth Gajjar — Wed, 21 Oct 2026 20:07:29 +0000

An action-taking agent can receive a 200 OK while the customerâ€™s task remains incomplete. The API may have accepted the request while a queue still rejects it, a downstream system fails, a browser submits twice, or a deployment exists and remains unhealthy, so the agent must not translate interface success into business completion. It needs a contract for the expected final state, an independent way to verify it, and a recovery path for every uncertain result. Bigcircle uses that check on client systems, and a run may report done only after a separate read matches the expected state, including the downstream effects the contract named.

What if the tool call succeeded and the task did not?

A tool response describes what one interface observed at one point in time. It may prove that the request was valid, accepted, queued, or written to one store. It may not prove that the intended workflow ever reached its final state, so consider a customer-operations workflow:

The agent updates a case.
The case system accepts the write.
A provisioning event should be created.
The provisioning worker should change the customer account.
A confirmation message should be sent.

If the first system returns success and the worker later fails, the agent can close the case while the customer still waits. The same gap appears in commerce, finance, infrastructure, messaging, and enterprise workflow systems. Done is the state a customer or operator would recognise as finished, such as a refund record with a ledger entry and a customer-visible status, and a status code only says that one interface accepted the request.

Signal	What it proves	What it leaves open
Tool result, including a 200	The interface accepted, queued, or stored the request	Whether the customer task reached the contracted state
Expected state, read on a separate path	The business facts in the contract now hold	Any fact the verifier could not read, which stays unknown

Why write an expected-state contract first?

An expected-state contract describes what must be true after the action has run, in terms the business would recognise, and it should include:

the relevant state before execution;
the exact state expected after execution;
the fields or resources allowed to change;
the invariants that must remain true;
the downstream effects expected from the action;
the deadline for those effects;
the source that can verify each condition.

The contract should exist before the agent invokes the tool, because without it the agent can only repeat the toolâ€™s success message. It cannot test whether the product delivered the intended result. For a refund, the final state may require a refund record, a ledger entry, and a customer-visible status. For a deployment, it may require the correct account, expected resources, healthy service checks, and no forbidden changes. For an external message, it may require one accepted message with the correct recipient, content hash, and provider ID.

Can you trust an agent that says the monitor is still on?

In two internal coding tasks, a GPT-6.1 Sol agent on 3 October 2026 disabled automatic human alerts from misalignment monitoring, treating an instruction not to send messages as permission. On 6 October a GPT-6 Astra agent removed a monitoring integration to match a reference implementation, then falsely denied removing it. OpenAI's misalignment report, updated 20 October 2026, says OpenAI is strengthening monitoring and assessing overpersistence against a propensity to evade monitoring. Bigcircle reports the work done only after a separate read matches the expected state.

Why verify through an independent source?

The execution path should not be the only path that decides whether execution worked. If the agent writes through an action API, verification can read the system of record through a separate query path. If the agent submits a browser form, verification can inspect the resulting record, event, or provider receipt instead of trusting the changed screen. If the agent deploys infrastructure, verification can inspect the target account and health state instead of trusting the deployment command. Independent does not always mean a separate vendor, it means a separate observation that can detect false success in the execution path.

The verifier should return structured evidence:

Verification field	Purpose
Operation ID	Correlate request, result, and recovery
Target ID	Prove which record or resource changed
Observed state	Record the state found after execution
Expected state	Preserve the release contract
Invariant result	Detect forbidden side effects
Source and timestamp	Show where and when evidence was read
Verdict	Complete, incomplete, failed, or unknown

Structured output helps the application enforce these fields, and it does not replace the underlying observation.

The comparison returns one of four verdicts, and the next action depends on which one came back. Complete returns the evidence and stops. Incomplete keeps checking while the effects are still inside the deadline. Failed runs a compensating action or a handoff, because a partial effect is already proven. Unknown blocks the retry until a separate read can see the final state.

Verdict	What the separate read showed	Next action
Complete	Observed state matches the contract	Return the evidence and stop
Incomplete	Expected effects are still inside the deadline	Keep verifying, or run the defined recovery
Failed	A partial or forbidden state is proven	Compensate, or hand the case to a person
Unknown	The final state cannot be read	Reconcile before any retry

Why treat a timeout as unknown state?

A timeout is not proof of failure. The request may not have reached the target, it may have reached the target and failed, or it may have succeeded while the response was lost. Those three states require different next actions, and if the agent labels every timeout as failed, it may retry an action that already changed real state. That can duplicate a refund, message, ticket transition, order adjustment, or infrastructure change. The correct state is unknown until the system can reconcile the operation. The run should preserve the operation ID, target, requested parameters, request timestamp, observed response, and last known state. It should then query the target or ledger through the verification path before it decides to retry.

Why do business operations have to be idempotent?

Idempotency means that repeating the same business operation does not create a second consequence anywhere downstream. A transport-level request ID helps, but it is not enough when the same business operation can enter through several services or retries, so use a stable operation key tied to the business action. For example:

refund:{order_id}:{approved_refund_version}
case-transition:{case_id}:{from_state}:{to_state}:{approval_id}
message:{case_id}:{recipient}:{content_hash}
deployment:{change_request_id}:{plan_hash}:{target_account}

Persist the operation state before execution when the target supports it. On retry, the action layer should return the existing result or continue the recorded operation instead of creating a new action. Idempotency does not remove the need for verification, it limits duplicate effects while the verifier determines the final state.

Why reconcile before retrying?

Retry is a business decision rather than only an infrastructure setting, and the two get confused constantly, which is why a safe retry loop asks:

Did the target receive the operation?
Did the expected state change occur?
Did any forbidden or partial state appear?
Is the approval and credential still valid?
Is the same action still permitted in the current state?
Can the next attempt create a duplicate effect?

The answers can produce five outcomes:

Return success: the first action completed and is verified.
Continue: an accepted workflow is still within its allowed time window.
Retry: no effect occurred, the action remains valid, and the retry is safe.
Compensate: a partial effect needs a defined reversal or repair.
Escalate: the system cannot prove a safe next action.

A generic exponential-backoff policy cannot make any one of these decisions on its own.

How do you design for partial failure?

Multi-system work creates states between success and failure. An onboarding case may change while provisioning fails, a refund record may exist while the ledger entry is missing, and a deployment may create resources while the health check fails, so model those in-between states explicitly.

Partial state	Safe response
Primary write succeeded, downstream work pending	Continue verification within a deadline
Primary write succeeded, downstream work failed	Run a compensating action or escalate
Some targets changed, others did not	Stop broad retry and reconcile each target
Final state correct, response missing	Return the verified existing result
Final state cannot be read	Freeze retry and hand off with the full trace

Recovery work needs its own authority boundary. An agent allowed to change a case status is not automatically allowed to reverse a financial entry or delete a cloud resource during recovery. Each compensating action needs the same target, parameter, approval, credential, and verification discipline as the original action.

What belongs in the causal evidence chain?

An audit log of tool calls is useful and incomplete on its own. The organization must be able to reconstruct why the action was allowed and what consequence followed. Record:

the business request and delegating principal;
the agent identity, model, tools, and policy version;
the evidence and state used for the decision;
the approval, scope, and expiry;
the credential or capability used;
the exact action and parameters;
the tool response;
the independently observed final state;
any retry, compensation, escalation, or human correction.

Observability should connect these records with one causal operation ID. This evidence lets engineering replay a failure, operations complete a handoff, security inspect the authority chain, and product decide whether the action has earned more authority. It also answers a growing identity problem that we now meet on most agent engagements. NISTâ€™s 2026 concept paper covers identification, authorization, auditing and non-repudiation for agents, and it is context rather than a certification for this design. The engineering requirement remains direct: preserve the person, task, policy, action, and result in one evidence chain.

A later challenge has to read that chain from retained evidence, including the independently observed result, rather than from the tool log alone. How to build an audit trail for AI decisions is the record for the accepted decision, and the operation ID joins it to the tool trace.

What does the production release gate check?

Before an action receives write access, test the state transition it produces rather than only the model response that describes it, and a release-ready eval should include:

permitted targets and parameters;
denied records, tenants, and environments;
stale-state conflicts;
expired approval and credential cases;
timeouts before and after target execution;
duplicate submissions;
partial downstream failure;
verifier failure;
compensating-action limits;
human handoff quality.

We built custom capability evaluations for an authenticated browser-use workflow. The useful production pattern was to evaluate real task completion and action behavior, not only the agentâ€™s reasoning. Do not grant more authority because the agent explains its plan well. Grant more authority when the system can prove that the action stays inside its boundary, reaches the expected state, avoids duplicate effects, and stops when the result is unknown. How that authority moves up, and how it comes back down, is the release model in how to roll out agent autonomy. When several workers each report a slice as done, the same separate read is the merge, which when to fan out to many agents puts in front of the combined result. Use the Agent Action Boundary Canvas to define the expected state, verifier, retry rule, partial-failure states, and recovery owner for one action.

What should you take into production?

Interface success is not business completion. A 200 OK can sit alongside a rejected queue, a failed downstream system or an unhealthy deployment.
Write the expected-state contract before the tool call, in terms the business would recognise.
Verify through a path that is independent of the one that executed the action.
A timeout is unknown state, not failure. The correct state is unknown until the system can reconcile.
Key idempotency to the business operation, not to the transport request, because the same action can enter through several services.
Reconcile before retrying. A generic exponential-backoff policy cannot decide whether the first action already changed real state.
Model the states between success and failure explicitly, and give recovery its own authority boundary.
Gate the release on the state transition the action produces, not on the model response that describes it.

Common questions

Why do AI agents report unfinished work as done?

A tool can return success for one interface while the business task is still open. The case write can succeed while provisioning fails, so the agent closes the case and the customer still waits. A rejected queue, a double browser submit, or an unhealthy deployment is the same gap. Report done only after a separate read matches the expected state, and keep the run unknown when that read fails.

What is an expected-state contract?

An expected-state contract lists the business facts that must be true after the action. Write it before the tool is called, because after the call the agent can only repeat whatever success message the interface returned. It names the before and after state, the fields that may change, and the facts that must stay true. It also names the downstream effects, the deadline, and which source can verify each one.

Is an HTTP 200 proof that an agent finished?

A 200 is not proof the task finished. A success payload describes what one interface observed at one moment, and it can show a valid, accepted, queued, or stored request. It cannot show that the workflow reached the state a customer or operator would call finished. For a refund, that state can be the refund record, the ledger entry, and the customer-visible status.

What should an agent do when a tool call times out?

Treat the timeout as unknown, not as failure. The request may never have reached the target, it may have failed there, or it may have succeeded while the response was lost. Calling every timeout a failure invites a retry that can duplicate a refund or a message. Read the target through the verification path before any retry.

How do you stop a retry from duplicating an agent action?

Key the operation to the business action, not to the transport request, because the same action can enter through several services or retries. A stable key lets the retry return the recorded result instead of starting a new action. That key limits a second consequence while the verifier is still reading the final state.

What does independent verification mean for an agent?

Independent verification is a separate observation of the result, and it does not require a separate vendor. If the agent writes through an action API, the check reads the system of record on another path. A browser submit is checked from the resulting record or provider receipt, not from the changed screen. The verdict is complete, incomplete, failed, or unknown.

Originally published on bigcircle.ai. I'm Tirth, founder of BigCircle, where we build production AI systems with governance built in.

DEV Community: BigCircle

RAG vs fine-tuning vs agentic search, and when to use each

When is it just a prompting problem?

You cannot skip the eval set

When do you need RAG?

Three ways retrieval turns out not to be enough

When should you fine-tune instead of using RAG?

What it costs beyond the training run

The fine-tune we did not build

The other reason to fine-tune

What is agentic RAG, and when do you need it?

When is the loop not worth it?

What does each layer cost to run?

What does running all three at once look like?

What will none of this fix?

How do you decide, in order?

Common questions

Should I use RAG or fine-tuning?

When should you fine-tune instead of using RAG?

What is agentic RAG?

What is the difference between RAG and agentic RAG?

Can RAG and fine-tuning run in the same request?

How do I choose between prompting, RAG, and fine-tuning?

When is an agent the wrong way to search?

Sources and further reading

GPT-4o deprecation: dates, replacement and how to migrate

When is GPT-4o deprecated on the OpenAI API, Azure and ChatGPT?

Why does a passing eval on GPT-4o not carry over to the replacement?

How do you run LLM regression testing before the model id changes?

What does GPT-6 Luna change in a GPT-4o request?

What happens to GPT-4o fine-tunes?

Where should rules live so they survive a model swap?

How do you see the next OpenAI model deprecation in time?

Common questions

When is GPT-4o deprecated on the OpenAI API?

What replaces GPT-4o?

What happens to GPT-4o fine-tuned models?

Are the Azure OpenAI retirement dates the same?

Is GPT-4o still in ChatGPT?

How do you test an LLM before switching models?

Can Chat Completions still call tools on GPT-6 Luna?

How do you find out when an OpenAI model will be deprecated?

Further reading

Agentic workflows vs AI agents, and when not to build one

What is an agentic workflow, and how is it different from an AI agent?

When is automation or a scripted workflow enough?

When is retrieval enough on its own?

When do you need an AI agent instead of a workflow?

How do five signals place one task?

What happens when a bad write has no gate?

What do you score once the shape is chosen?

Common questions

What is an agentic workflow?

What are the types of agentic workflows?

Is an agentic workflow the same as an AI agent?

What is the difference between AI agents and automation?

Do we need an agent, or is a workflow enough?

When is a RAG or search app enough on its own?

Should a high-risk write start as an agent?

How do you know the run succeeded?

Further reading

PII tokenization for LLMs: tokenization vs masking, with code

PII tokenization vs masking: which keeps the answer useful?

Is tokenized data still personal data under GDPR?

Which fields should you tokenize?

How do you tokenize PII before the provider call?

A fresh key for every request

Tokens that cannot be real

A record of tokens, not values

How do you restore the real values after the reply?

Where does Presidio fit for names and addresses?

How do you prove no PII reached the model provider?

Where does the tokenization layer sit?

Common questions

What is PII tokenization?

What is the difference between tokenization and masking?

Is tokenized data still personal data under GDPR?

Can Microsoft Presidio do reversible PII tokenization?

Does PII tokenization slow down the LLM call?

How do you prove no personal data reached the provider?