DEV Community: Jack M

AI Agent Backpressure: Prevent Queue Overload Before Users Wait Forever

Jack M — Mon, 02 Nov 2026 03:28:48 +0000

The most expensive agent incident may start with a healthy dashboard. Requests still return, workers are still busy, and the queue is simply getting longer. Then a slow model or tool turns that waiting line into timeouts, retries, duplicate work, and a chat screen that looks frozen.

The missing control is usually not another worker. It is AI agent backpressure: an explicit contract that stops your system from accepting more work than it can finish within a useful deadline. This guide shows how to design that contract for interactive and background agent workflows, including tenant fairness, queue limits, retry rules, and honest user-facing states.

Why a queue is not backpressure

A queue is storage between producers and workers. Backpressure changes what happens when that storage fills or when downstream capacity falls.

Without it, an API accepts every request, each agent fans out into model and tool calls, and workers keep retrying when a provider slows down. The system appears available right up until most requests have waited too long to matter. A long queue is not resilience; it is delayed rejection.

This is more acute for agents than for ordinary request handlers:

One customer request can create many model, retrieval, and tool calls.
A slow dependency holds concurrency for much longer than a typical API call.
Retries can multiply load exactly when capacity is lowest.
One bulk import can consume the same model quota that serves an interactive user.

Recent developer discussion reflects this split: rate limits are a downstream provider signal, while backpressure is the application’s decision to queue, defer, degrade, or reject work before it becomes a pile-up. AWS’s agent orchestration guidance likewise calls for controls at workflow, agent, tool, and model layers. That cross-layer design is the useful gap: a token budget or a durable queue alone does not tell the API what to do when capacity is gone.

Define the overload contract before writing a worker

Start by deciding what each work class is allowed to do under pressure. Do not let an implementation detail make this decision accidentally.

Work class	User expectation	When capacity is low	Useful response
Interactive chat	Fast first response	Reject or degrade quickly	`429` or `503` with retry guidance
Short action	Finishes soon	Admit only if deadline is realistic	Accepted or fast retryable failure
Background task	Completion matters more than speed	Queue within a hard bound	Job ID and truthful wait state
Bulk work	Can wait and be throttled	Reserve only leftover capacity	Scheduled or deferred
Critical repair	Must not starve	Keep reserved slots	Priority admission

The key question is: can this request still produce value if it waits? If the answer is no, a quick response is kinder than a spinner. If the answer is yes, return a job reference and show queued, running, paused, or failed state—never a fake percentage.

Give every run one deadline that begins at admission, not at the first model call. A five-minute workflow that waits four minutes in a queue only has one minute left to execute. Passing that deadline to every stage prevents a stale job from consuming fresh capacity.

type Admission =
  | { kind: "admitted"; runId: string; deadlineAt: number }
  | { kind: "queued"; runId: string; positionHint?: number }
  | { kind: "deferred"; retryAfterSeconds: number }
  | { kind: "rejected"; retryAfterSeconds: number; reason: string };

function deadlineFromNow(ms: number) {
  return Date.now() + ms;
}

Bound every place work can accumulate

An agent system often has more than one queue: an HTTP connection pool, an in-process semaphore, a job broker, a model provider, a tool client, and a database connection pool. Capping only the broker just moves the overload elsewhere.

Use limits at four scopes:

Global concurrency protects shared providers and databases.
Per-tenant concurrency prevents a noisy customer from consuming all slots.
Per-workflow concurrency prevents a fan-out plan from creating fifty simultaneous calls.
Per-dependency concurrency isolates a slow browser, search API, or model route from healthy work.

Queue length also needs a cap. Keep large input files and raw web pages outside the queue; put object references in the job record. A bounded queue should contain enough information to restart work, not unlimited payloads that grow memory while waiting.

Here is a deliberately small admission function. In production, the counters should be atomic and shared across replicas, such as through a database, Redis script, or queue-native primitive.

type Capacity = { running: number; queued: number };

async function admitRun(input: {
  tenantId: string;
  priority: "interactive" | "background" | "bulk";
  capacity: Capacity;
}): Promise<Admission> {
  const MAX_RUNNING = 24;
  const MAX_QUEUED = 200;
  const tenantRunning = await countTenantRunning(input.tenantId);

  if (tenantRunning >= 3) {
    return { kind: "deferred", retryAfterSeconds: 30 };
  }
  if (input.capacity.running < MAX_RUNNING) {
    return createAndLeaseRun(input, deadlineFromNow(90_000));
  }
  if (input.priority !== "interactive" && input.capacity.queued < MAX_QUEUED) {
    return enqueueRun(input, deadlineFromNow(15 * 60_000));
  }
  return { kind: "rejected", retryAfterSeconds: 15, reason: "capacity_unavailable" };
}

Numbers here are examples, not universal settings. Derive initial concurrency from your provider quota, average end-to-end latency, tool limits, and a safety margin. Then test with slow dependencies; capacity that is safe during normal latency may be destructive during a partial outage.

Separate admission control from rate limiting

These controls work together but answer different questions.

Rate limiting: may this customer start another request in this time window?
Budgeting: may this run spend another token, tool call, or dollar?
Admission control: can the system start or promise this work now?
Backpressure: what changes upstream when the answer is no?

That distinction matters. A tenant could remain under its request-per-minute allowance while every shared worker slot is occupied by slow runs. Conversely, a system can have idle workers but still need to stop a tenant that has exhausted its cost budget. Keep the reason in the response and logs; “rate limited” is misleading if the actual cause is a full model queue.

Make fairness a scheduling rule, not a hope

FIFO is simple but vulnerable. A single large tenant that adds thousands of background jobs first can make a newly arrived interactive task wait behind all of them.

Use a small number of priority bands and round-robin tenant selection within each band. Reserve capacity rather than letting high-priority work consume every slot forever. For example:

60% of slots for interactive work
25% for normal background work
10% for critical repair or approval-resume work
5% for bulk work, reclaimed only when other bands are empty

Add ageing so a background job that has waited a long time eventually receives a boost. Do not expose precise queue positions unless your scheduler can guarantee them; changing priority and retries make an exact number untrustworthy.

A useful job key is (tenant_id, work_class, created_at). It gives your scheduler enough information to make a fair choice without reading prompt contents. This also makes fairness observable: measure queue age by tenant and work class, not only global queue depth.

Choose a load-shedding policy for each feature

When capacity is exhausted, “retry later” is not the only option. Pick the least harmful behavior that still preserves user intent.

Policy	Best for	Example
Fail fast	Interactive operations	Return a retryable error before opening a stream
Defer	Non-urgent work	Schedule a report for later execution
Degrade	Optional enrichment	Skip web research and return grounded local results
Coalesce	Repeated equivalent work	Join identical document-summary requests
Drop	Explicitly disposable events	Discard duplicate typing or preview events

Never silently downgrade a request that changes its safety or truthfulness. If retrieval is unavailable, say that the answer used only the available context. If an agent cannot complete a write action, do not claim it did.

Coalescing deserves special care. Two users may ask the same question but have different permissions, source filters, or tenant data. Coalesce only when the authorization context and input fingerprint are equivalent.

Retries must yield to pressure

Retries are a common overload amplifier. A timeout does not prove the provider did not receive the request, and an immediate retry steals capacity from new work. Treat retry eligibility as policy, not a catch block.

For each attempt, record the dependency, idempotency key, deadline, error class, and retry count. Retry only errors that are both transient and safe. Honor Retry-After when a provider supplies it, use exponential backoff with jitter, and stop when the run deadline has passed.

async function callModelWithRetry(run: Run, request: ModelRequest) {
  for (let attempt = 0; attempt < 3; attempt++) {
    if (Date.now() >= run.deadlineAt) throw new Error("deadline_exceeded");
    try {
      return await modelGate.withPermit(() => model.generate(request));
    } catch (error) {
      if (!isRetryable(error) || attempt === 2) throw error;
      const delay = retryDelayWithJitter(error, attempt);
      if (Date.now() + delay >= run.deadlineAt) throw error;
      await recordRetry(run.id, attempt, error);
      await sleep(delay);
    }
  }
  throw new Error("unreachable");
}

The semaphore wraps the whole call, including retry decision boundaries, so unlimited retries cannot bypass your concurrency limit. For side-effecting tools, pair this with idempotency keys and reconciliation; a timeout after a successful write is not safe to blindly repeat.

Stream honestly when work is delayed

Streaming can hide overload. If a chat connection stays open while the real job waits in a worker queue, the user sees an empty response and assumes the model is thinking.

Send a state transition instead: accepted, queued, running, waiting_for_dependency, paused_for_approval, completed, or failed. Keep the text factual: “Waiting for capacity; retrying automatically until 10:42” is useful. “Almost done” is not.

For requests that cannot wait, reject before streaming begins with a stable, documented error payload:

{
  "error": "capacity_unavailable",
  "retry_after_seconds": 15,
  "request_id": "run_8b2",
  "safe_to_retry": true
}

Clients should respect the delay and should not create a new run for every retry. Reuse an idempotency key so a flaky network does not turn one user click into many queued tasks.

Monitor the signals that reveal a bad promise

CPU and model latency are not enough. Track these per model route, tool, tenant tier, and work class:

admitted, queued, deferred, rejected, and completed runs
queue depth and oldest queue age
time from admission to first useful output
execution time versus queue wait time
retries, retry reason, and retries after deadline
cancellations and jobs removed before execution
concurrency utilization and dependency error rate
completion rate and cost per completed workflow

Alert on combinations, not a single number. Rising queue depth plus rising oldest age means demand exceeds service. Flat queue depth plus rising execution latency points to a downstream dependency. Full queue plus idle workers suggests a scheduler, lease, or partitioning bug.

Run a simple game day: make one model route slow, send mixed tenant traffic, then restore it. Verify that interactive users get fast answers or fast rejection, bulk work stops expanding, retries do not spike, and the queue drains without a second surge. The recent LLM serving backpressure analysis makes the same practical point: unbounded waiting turns a smaller overload into many late failures.

A rollout plan that will not surprise users

Start in observe-only mode. Calculate whether each request would have been admitted, queued, or rejected, but do not enforce it yet. Compare that result with real queue age and completion data.

Then introduce a small global cap and one per-tenant cap for a non-critical background feature. Add explicit states to the UI and API before making rejections common. Finally, add priority reservations and dependency-specific limits after you understand normal and degraded capacity.

Document the contract for clients: which errors are retryable, how long to wait, when a job will be queued, and how to poll or subscribe for state. Backpressure succeeds when callers change behavior; it fails when every caller retries at once.

FAQ

What is AI agent backpressure?

AI agent backpressure is the set of limits and responses that prevent an agent system from accepting work faster than models, tools, and workers can complete it. It includes bounded queues, admission decisions, concurrency limits, load shedding, and retry guidance.

Is an AI agent queue enough to prevent overload?

No. A queue only delays work unless it has a hard capacity and the producer changes behavior when it fills. Pair it with admission control, deadlines, and a clear defer-or-reject response.

How is backpressure different from an AI agent rate limiter?

A rate limiter controls how frequently a caller may start work. Backpressure responds to current system capacity across queues, workers, models, and tools. Use both: a caller can be within its rate limit while the shared system has no safe capacity.

Should interactive AI requests be queued?

Only if the wait remains useful and visible. For chat-like interactions, fast retryable rejection or an explicitly queued asynchronous job is usually better than keeping an empty stream open for minutes.

How do I keep one tenant from blocking everyone else?

Set per-tenant running and queued limits, schedule tenants fairly within each priority band, and reserve shared capacity for interactive or critical work. Measure queue age by tenant so unfairness is visible.

What should an agent do after a provider returns 429 or overload?

Honor the provider’s retry guidance when present, retry only within the run’s deadline, use backoff with jitter, and do not let the retry bypass dependency concurrency limits. If the deadline will be missed, return an honest deferred or failed state.

AI Agent Cancellation: Stop Work That Continues After the User Clicks Stop

Jack M — Sun, 01 Nov 2026 03:58:06 +0000

A Stop button can be honest in the browser and still be a lie in production. The chat stream disappears, but a model request may be generating, a browser tool may be clicking, and a background worker may be one step away from sending an email.

That is not a cosmetic bug. It is a broken contract. Users interpret “Stop” as “do not do more work for me.” Your system needs to translate that request across every layer, record what actually stopped, and make the remaining uncertainty visible. This guide gives you that contract.

The key idea is simple: cancellation is a state transition, not a boolean and not a client disconnect.

The five meanings hidden behind “Stop”

Teams often store one cancelled flag and call it a day. That collapses several different facts:

Event	What it proves	What it does not prove
Browser aborts a stream	The browser stopped listening	The server or model stopped working
API accepts a cancel request	The request was recorded	Every child task observed it
Worker sees the request	New work should not begin	An in-flight tool did not finish
Provider accepts cancellation	The cancellation reached that provider	No already-buffered output or usage exists
Run becomes terminal	Your ledger has a final outcome	An external write was reversed

Keep those facts separate. A robust lifecycle might be:

running → cancel_requested → cancelling → cancelled

with other terminal paths such as completed, failed, and needs_reconciliation. The last state is important. If a payment, ticket, or record write was in flight when cancellation arrived, pretending it is cancelled can make a later retry unsafe.

Define the cancellation contract before wiring a button

Write down the semantics your product promises. A useful default is:

A user can request cancellation for a run they are authorized to control.
No new model turn, tool call, or queued step starts after the durable request is observed.
In-flight work gets a best-effort cancel signal and has a deadline for acknowledging it.
The API never reports cancelled until the orchestrator records a terminal decision.
Side effects already dispatched are reconciled, not guessed away.
A cancelled run does not silently resume; resumption creates a new attempt or requires an explicit resume action.

This contract gives product, API, and operations teams the same vocabulary. It also prevents a dangerous UX pattern: changing the button label to “Stopped” the moment the browser disconnects.

Use one durable run record

Put the authoritative state in a database or workflow engine, not in a process-local abort controller. A minimal record can look like this:

type RunStatus =
  | "queued" | "running" | "cancel_requested" | "cancelling"
  | "cancelled" | "completed" | "failed" | "needs_reconciliation";

type AgentRun = {
  id: string;
  tenantId: string;
  status: RunStatus;
  cancelRequestedAt?: string;
  cancelReason?: "user" | "deadline" | "policy" | "operator";
  activeStepId?: string;
  version: number;
  terminalAt?: string;
};

The version field enables compare-and-set updates. It stops a late worker from replacing cancel_requested with running after another worker has already processed the stop request.

Your POST /runs/:id/cancel endpoint should be idempotent. It records the request once, returns the current state, and publishes a cancellation event. Repeated taps, reconnects, and retries should all produce the same outcome.

async function requestCancel(runId: string, actorId: string) {
  return db.transaction(async (tx) => {
    const run = await tx.runs.lockForUpdate(runId);
    assertCanCancel(actorId, run);

    if (["completed", "failed", "cancelled"].includes(run.status)) return run;

    const next = await tx.runs.update(runId, {
      status: "cancel_requested",
      cancelRequestedAt: new Date().toISOString(),
      cancelReason: "user",
      version: run.version + 1,
    });
    await tx.outbox.insert({ type: "run.cancel_requested", runId });
    return next;
  });
}

The transactional outbox is not required by every stack, but the principle is: persist the decision before notifying workers. A process crash between an in-memory publish and a database write should not make a stop request disappear.

Propagate cancellation across every boundary

An agent run usually crosses more boundaries than a normal web request. Treat cancellation as a signal that travels with the run.

Browser and streaming edge

AbortController can stop fetch and response-body consumption in the browser, as MDN documents. Use it to stop rendering and then call your durable cancellation endpoint. Do not rely on disconnect detection alone: mobile networks reconnect, tabs sleep, and a user may close a page without meaning “cancel.”

const controller = new AbortController();

stopButton.onclick = async () => {
  controller.abort("user_pressed_stop"); // end local streaming now
  await fetch(`/api/runs/${runId}/cancel`, { method: "POST" });
  showStatus("Stopping…");
};

The status is “Stopping,” not “Stopped,” until the run event feed reports a terminal state. If an SSE connection reconnects, replay events from a monotonic sequence number rather than starting another run.

Orchestrator and model calls

At step boundaries, read the durable record. In long model streams, also pass a cancellation primitive that can close the network request or invoke a provider-specific cancellation API where available. A close tells your HTTP client to stop waiting; it does not automatically prove upstream inference stopped or that no tokens were billed.

Use an absolute run deadline as well. Pass the remaining time, not a fresh full timeout, to each downstream call. A run that spends 50 seconds retrieving should not receive another 60 seconds of model time merely because each layer starts its own clock.

Parallel tools

Parallelism makes cancellation interesting. If a run launches a search, an internal lookup, and a write tool together, cancellation must reach each child. But not all children have the same policy:

Read-only tools can normally be abandoned or cancelled.
A write before its commit point can be cancelled.
A write after dispatch needs a receipt or authoritative read-back.
Cleanup work may need a short shielded window so it can record the correct final state.

In Python, cancellation is cooperative: asyncio injects CancelledError at an await point, and the official guidance is to use try/finally for cleanup and generally propagate the exception afterward. Do not catch it broadly and continue doing more agent work.

async def execute_step(run_id: str):
    try:
        await assert_not_cancelled(run_id)
        result = await call_tool_with_deadline(run_id)
        await persist_result(run_id, result)
    except asyncio.CancelledError:
        await record_step_interrupted(run_id)
        raise

Queued and durable work

A cancellation request must invalidate queued messages too. Workers should acquire a lease, re-check the run state immediately before each externally visible action, and release the lease on exit. A message already in a queue is not permission to execute it.

For longer-running orchestration, model cancellation and external-step cancellation as explicit workflow commands. Systems such as Temporal distinguish cancellation commands from ordinary completion; the general lesson applies even if you use Postgres and workers instead of a workflow engine.

If you need the underlying persistence pattern, see AI Agent Durable Queue: Stop Long-Running Work From Dying Mid-Task.

Never confuse cancellation with rollback

Cancellation prevents future work. Rollback tries to compensate for past work. They are related but not interchangeable.

Imagine an agent creates an invoice and then receives a stop request before it records the tool response. The correct terminal state is often needs_reconciliation, not cancelled. Query the payment system by a stable operation key. If the invoice exists, show it as created and offer a distinct authorized reversal. If it does not, mark the step cancelled.

This is why every write tool needs an idempotency key and an operation ledger. The full pattern is covered in AI Agent Idempotency: Prevent Duplicate Charges, Emails, and Records. Cancellation reduces how often you enter an unknown state; it never removes the need to resolve one.

Design an honest user experience

The UI should expose progress without flooding people with internals:

State	User-facing message	Allowed action
`running`	Working	Stop
`cancel_requested`	Stopping work	Hide or wait
`cancelling`	Finishing safe shutdown	View details
`needs_reconciliation`	Checking an action before closing	Retry check or contact support
`cancelled`	Stopped; no new work will run	Start a new run

Do not show partial generated text as a completed answer. Mark it as interrupted, keep it separate from the final result, and never automatically feed it into the next agent turn. It may have been cut off mid-instruction or before a tool result arrived.

Structured errors help clients make this distinction. An error envelope should include a stable code, whether a retry is safe, the run state, and a correlation ID. Agent Tool Error Envelopes shows a compatible pattern.

Test the race conditions, not just the button

The happy path is easy: click Stop and see the stream close. Production correctness lives in the races. Add deterministic test hooks before model calls, before a tool dispatch, after dispatch but before receipt storage, and while a worker holds a lease.

Your cancellation test matrix should cover at least:

Cancel while queued: no worker starts the first step.
Cancel during token streaming: client stops, provider result is recorded as interrupted or reconciled.
Cancel between model output and tool dispatch: no tool begins.
Cancel during two parallel tools: every child reaches a known terminal state.
Cancel after a write dispatch: reconciliation runs; the write is never blindly retried.
Cancel during a process crash: a replacement worker respects the durable request.
Duplicate cancel calls: one lifecycle transition and no duplicate cleanup.
Resume attempt after cancellation: it creates a new attempt with explicit provenance.

Capture traces for each test and replay them when the runtime changes. AI Agent Trace Replay is useful for turning an escaped cancellation race into a permanent regression test.

A practical implementation checklist

Store run lifecycle state durably, including cancellation reason and timestamps.
Make cancellation requests authorized and idempotent.
Send a durable signal to workers; do not trust page disconnects as intent.
Check for cancellation before every new turn and side effect.
Propagate an abort signal and remaining deadline to streams and tools.
Record a confirmed terminal state only after worker cleanup finishes.
Reconcile writes that crossed a commit boundary.
Preserve interrupted output as partial, never as final.
Emit lifecycle events that reconnecting clients can replay.
Test cancellation at every boundary and after simulated crashes.

A Stop button is a promise about control. Keep that promise by treating cancellation as a first-class workflow outcome, not a disappearing HTTP connection.

FAQ

Does aborting an LLM stream stop the model?

Not necessarily. It reliably stops your client from consuming the response, but the provider may stop asynchronously, may have already buffered output, and may still report usage. Record provider request IDs and reconcile their final status when the distinction matters.

Should a cancelled AI agent run be retried automatically?

Usually no. Cancellation expresses changed user intent. Create an explicit new attempt instead. Only retry internal cleanup or a clearly idempotent read when that work is necessary to reach a known terminal state.

What is the difference between a timeout and cancellation?

A timeout is a system deadline; cancellation can come from a user, policy, operator, or parent workflow. Both can use the same propagation mechanism, but record the cause separately because the correct product response differs.

How do I cancel an agent tool that has already started a write?

Request cancellation, then query the target using the operation key or receipt. If the write committed, treat it as completed and use a separate compensating action if appropriate. Never assume a network timeout means the write failed.

Can I resume a cancelled agent run?

You can, but make it explicit. Persist a checkpoint with its inputs, completed steps, and evidence. Start a new attempt that references the old one, re-checks policy, and never repeats uncertain writes without reconciliation.

What metrics show whether cancellation works?

Track time from request to terminal state, cancelled runs with work still starting afterward, orphaned tool calls, reconciliation rate after cancellation, duplicate cancellation requests, and the fraction of partial outputs mistakenly shown as final.

Source-Aware LLM Cache Invalidation: Build a Freshness Contract

Jack M — Sat, 31 Oct 2026 03:31:54 +0000

The support answer arrives in 80 milliseconds, sounds clear, and is wrong. The refund policy changed an hour ago, but a semantic cache is still returning an answer generated from the old document. The model did not hallucinate. The system served yesterday's work as if it were current.

That distinction matters. A better prompt, larger model, or reranker will not repair a stale cached response. The fix is a freshness contract: a small, explicit agreement that says what an answer depends on, which changes make it unsafe, and how the system proves it is still eligible to serve.

This guide shows how to build source-aware LLM cache invalidation for a production AI application. The goal is not to pick a cache product. It is to make exact, semantic, and retrieval-adjacent caches fast without turning old policies, revoked access, or prompt releases into confident answers.

Why LLM cache invalidation is different

Traditional response caching often has one obvious source of truth: a database record or an HTTP response. An AI answer can depend on several moving inputs at once:

Dependency	Example change	What becomes unsafe
Source evidence	A policy page is edited	Answers quoting the old policy
Access scope	A user loses a role	Answers containing restricted facts
Prompt or tool policy	A system rule changes	Cached behavior based on old instructions
Model or embedding version	A route is updated	Semantic matches and quality assumptions
Live business state	An order ships	Personalized status answers

A time-to-live (TTL) covers only one of these changes, and only eventually. A 24-hour TTL means a known-bad answer may remain eligible for almost a day. For low-risk, slow-changing help text that may be acceptable. For account status, pricing, permissions, or policy, it is not.

The useful mental model is simple: a cache entry is not merely question -> answer. It is an artifact with a provenance record. Serve it only when its provenance still matches the request and the current system.

Start with four cache layers, not one

Teams often call all reuse an “LLM cache,” even though the risk is different at each layer.

Prompt or prefix caching reuses model work for an identical stable prefix. It improves cost and latency, but still produces a new answer.
Exact response caching returns a prior answer for the same normalized request. It is deterministic, but can still be stale.
Semantic caching returns an answer for a similar request. It adds a correctness decision: similar wording is not necessarily the same authorization or intent.
Retrieval and tool-result caching reuses source material before generation. A stale result here can poison every new answer built on top of it.

Do not give these layers one global TTL. A prefix cache may be safe to retain while an answer cache must be purged. A cached retrieval result may need to disappear after a document deletion even if the document's replacement has not finished embedding.

Define the freshness contract

For each cacheable route, define five things before writing a key.

1. Scope: who is allowed to receive this?

Scope must be a hard lookup boundary, not a filter applied after semantic search. At minimum, include the tenant. Add user or role identifiers whenever an answer could include personalized facts or privileges.

type CacheScope = {
  tenantId: string;
  principalId?: string;
  roleSetHash: string;
  region: string;
};

A cache key such as support:refund-policy:<query> is risky in a multi-tenant application. A safer key namespaces the candidate set before lookup:

const namespace = [
  "support-answer",
  scope.tenantId,
  scope.principalId ?? "shared",
  scope.roleSetHash,
  scope.region,
].join(":");

If a response is safe for everyone, make that a deliberate route decision. Do not assume that because the prompt looked generic.

2. Dependencies: what facts made this answer true?

Store versions for every dependency that can change the answer. A compact entry might look like this:

type CachedAnswer = {
  value: string;
  namespace: string;
  createdAt: string;
  expiresAt: string;
  sourceVersions: Record<string, string>;
  promptVersion: string;
  modelRouteVersion: string;
  embeddingVersion?: string;
  policyVersion: string;
  provenance: Array<{ sourceId: string; chunkId?: string }>;
};

sourceVersions is the important field. It lets a policy update invalidate answers tied to policy/refunds, without flushing every support answer. Use a content hash, monotonically increasing revision, or change-feed sequence number. The value does not need to be human-friendly; it needs to change reliably.

3. Freshness budget: how old is acceptable?

Set a maximum age by route and cost of error, not by cache technology. A product-status answer might have a zero-minute budget and skip response caching. A general troubleshooting explanation may be safe for several days.

Route	Example freshness budget	Response-cache policy
Account balance or entitlement	0–60 seconds	Usually bypass
Pricing or policy explanation	Minutes to hours	Source-event purge + short TTL
Product documentation	Hours to days	Source-event purge + TTL
Stable writing transformation	Longer TTL	Exact cache, no external evidence

TTL remains useful as a backstop for missed events. It is not the primary correctness mechanism for information that changes through known workflows.

4. Invalidation triggers: which events revoke eligibility?

Write this down as a route-level table. It prevents a common failure where the indexing pipeline knows a document changed but the answer cache never hears about it.

Trigger	Action
Document updated or deleted	Mark entries citing that source version stale
Permission or role change	Purge the affected principal or role namespace
Prompt, tool schema, or policy release	Invalidate the route version
Embedding migration	Rebuild retrieval entries; retire old semantic indexes
Model-route change	Evaluate whether answer entries remain acceptable; version the route if not

5. Proof: what is recorded when an entry is served?

Log the decision, not just the hit. A trace should say served, bypassed, or invalidated, plus the reason. This turns “the assistant is wrong” into an inspectable path.

type CacheDecision = {
  requestId: string;
  route: string;
  action: "served" | "bypassed" | "invalidated";
  reason: string;
  cacheKeyHash: string;
  entryAgeMs?: number;
  sourceVersions?: Record<string, string>;
};

Never log the full prompt or answer just to debug a cache. Log identifiers and versions, then join to access-controlled traces when needed.

Make eligibility a function, not a convention

Put cache safety in one testable function. The order matters: reject scope failures before checking similarity, bypass stale material, and invalidate known version drift.

type LiveContext = {
  scope: CacheScope;
  now: Date;
  sourceVersions: Record<string, string>;
  promptVersion: string;
  modelRouteVersion: string;
  policyVersion: string;
};

function canServe(entry: CachedAnswer, live: LiveContext) {
  if (entry.namespace !== makeNamespace(live.scope)) {
    return { action: "bypassed", reason: "scope_mismatch" };
  }
  if (live.now >= new Date(entry.expiresAt)) {
    return { action: "bypassed", reason: "ttl_expired" };
  }
  if (entry.promptVersion !== live.promptVersion) {
    return { action: "invalidated", reason: "prompt_version_changed" };
  }
  if (entry.policyVersion !== live.policyVersion) {
    return { action: "invalidated", reason: "policy_version_changed" };
  }
  if (entry.modelRouteVersion !== live.modelRouteVersion) {
    return { action: "invalidated", reason: "model_route_changed" };
  }
  for (const [sourceId, version] of Object.entries(entry.sourceVersions)) {
    if (live.sourceVersions[sourceId] !== version) {
      return { action: "invalidated", reason: `source_changed:${sourceId}` };
    }
  }
  return { action: "served", reason: "fresh" };
}

For a semantic cache, run canServe after finding candidates but before returning text. Better still, partition the vector index by hard scope so an out-of-scope candidate never enters the candidate set.

Connect change events to targeted purges

A full flush is safe but expensive, and it hides whether the invalidation design works. Prefer an outbox or change-feed event emitted by the same transaction that changes the source.

{
  "event": "knowledge.source.changed",
  "source_id": "policy/refunds",
  "old_version": "rev-41",
  "new_version": "rev-42",
  "tenant_id": "acme",
  "occurred_at": "2026-10-31T10:15:00Z"
}

The consumer should do three things: update the live version registry, mark matching response and retrieval entries stale by source metadata, and enqueue re-indexing if embeddings changed. Marking stale is often better than immediate deletion: it preserves a debuggable record while guaranteeing the lookup path cannot serve it.

Make the consumer idempotent. Events can arrive twice or out of order. If rev-42 is already live, a duplicate must not reopen rev-41. If a delete arrives, revoke the source immediately; do not wait for a rebuild job.

Test the failures that a happy-path cache hides

Unit tests for hit rate are not enough. Add regression cases that prove a known-bad entry cannot escape.

it("does not serve an answer after its cited policy changes", () => {
  const entry = answerFrom({ sourceVersions: { "policy/refunds": "rev-41" } });
  const live = contextFrom({ sourceVersions: { "policy/refunds": "rev-42" } });
  expect(canServe(entry, live)).toMatchObject({
    action: "invalidated",
    reason: "source_changed:policy/refunds",
  });
});

Build a small test matrix for every high-impact route:

a document update and deletion;
a role downgrade or tenant switch;
a prompt or tool-policy release;
an embedding or model-route version change;
a delayed, duplicate, and out-of-order invalidation event;
a semantic near-match that must be rejected.

Run these tests in CI and in a staging environment with the real cache adapter. The bug is often not the eligibility function; it is a secondary path that reads an old key, a worker that lags, or an index that was never namespaced.

Measure freshness, not just hits

A higher hit rate can mean lower cost or more wrong answers. Pair performance metrics with correctness signals:

Metric	Why it matters
Hit rate by layer and route	Shows where reuse actually happens
Entry age at serve time	Reveals TTLs that are too generous
Invalidations by reason	Reveals noisy releases or unhealthy sources
Event-to-invalidation lag	Measures the window of stale risk
False-hit rate	Finds semantic matches that should have missed
Purge coverage	Confirms all entries tied to a changed source were blocked

Sample served semantic hits for review or run a route-specific verifier. Do not use an LLM judge as the only safety control for permissions or facts that have an authoritative source version.

Respond to a stale-answer incident

When someone reports a confident but old answer, resist the urge to immediately tune the prompt. Follow a short path:

Locate the request trace and cache decision.
Identify the source, scope, prompt, policy, and route versions recorded with the entry.
Block the affected entry and its metadata neighbors from serving.
Compare the source-change time with the invalidation-event and consumer times.
Add a regression case for the missing trigger or lookup path.
Re-enable caching only after the replay proves the current answer wins.

This keeps the incident focused on evidence. A cached answer can be fluent and internally consistent while being unacceptable for the present request.

A practical rollout order

You do not need a perfect dependency graph on day one. Start where stale answers have the highest user cost.

Namespace all existing answer caches by tenant and privilege boundary.
Add prompt, policy, and source-version metadata to new entries.
Implement canServe and reason-coded decision logs.
Wire one high-risk source update to a targeted invalidation event.
Add the five failure cases to CI, then expand route by route.

Fast answers are valuable only when they are still eligible to be true. A freshness contract makes that eligibility explicit: every cached answer has a scope, dependencies, budget, revocation path, and proof. That is how you keep the latency win without asking users to discover your stale data for you.

FAQ

What is source-aware LLM cache invalidation?

It is a cache design that records which documents, policies, permissions, prompts, and model settings an answer depended on. When one changes, the system blocks or removes only the affected entries instead of waiting for a broad TTL or flushing everything.

Is a TTL enough for an LLM response cache?

TTL is a useful fallback, but it is not enough when a known change must take effect quickly. Use TTL for eventual cleanup and an event-driven purge for document updates, access revocations, policy releases, and other known changes.

Should semantic caches use the same invalidation rules as exact caches?

They need the same scope and freshness checks, plus stricter evaluation of similarity. A semantically close query can have different permissions or intent. Partition candidates by scope before similarity search and use conservative thresholds for higher-risk routes.

What should be included in an LLM cache key?

Include the route, tenant, relevant principal or role boundary, normalized request, and versions that determine the answer when those versions are part of lookup semantics. Keep rich provenance metadata too, so an event can find entries to invalidate without rebuilding every key.

How do I invalidate cached RAG answers after a document changes?

Emit a source-change event containing the source ID and new revision. Update the live revision registry, mark cached retrieval results and answers that cite the old revision stale, then re-index or regenerate on the next request. Test document deletion separately because it must revoke access even before replacement content is ready.

Which metrics show whether an LLM cache is safe?

Track served-entry age, event-to-invalidation lag, invalidations by reason, false semantic hits, purge coverage, and cache decisions by route. Hit rate and cost savings are useful, but they cannot detect a fast wrong answer.

The Agent API Gateway Pattern: Separate Streaming, State, and Side Effects

Jack M — Fri, 30 Oct 2026 03:32:47 +0000

An agent demo usually has one endpoint: send a message, wait, return an answer. Production breaks that shape fast. A request may stream for a minute, consult a tool, wait for approval, enqueue work, and make a change that must never happen twice. If all of that lives behind one POST /chat route, retries and reconnects become accidental business logic.

The agent API gateway pattern fixes this by giving traffic three explicit lanes: streaming for progress, durable state for resumable work, and side effects for controlled writes. The payoff is concrete: a browser can reconnect without restarting a run, a worker can resume after a deploy, and a retry cannot quietly send a duplicate email or charge a customer twice.

This is not a product category or a proxy configuration. It is an API boundary you can implement with an ordinary gateway, application service, queue, and database.

The mismatch between chat endpoints and agent work

Classic request/response APIs assume a bounded operation: validate input, do work, return a result. Agent work violates several assumptions at once:

API assumption	What agent traffic does instead	Design response
The client stays connected	Mobile networks and browser tabs disconnect	Stream replayable events
A request maps to one action	A run calls several tools and may pause	Persist a run and step state
Retrying is harmless	A retry can repeat a write	Require an operation key
The response is the result	The useful result may arrive later	Return a stable run URL and status
Rate limits measure load	Each request has different cost and risk	Limit by tenant, operation, and budget

Recent gateway documentation reflects this shift: Microsoft documents policies for model, MCP, and A2A traffic, while Kong documents protocol-aware A2A routing and tracing. AWS’s agent exposure guidance also calls out large contexts, streaming, and per-tenant budgets. The implementation details differ, but the common lesson is useful: agent-shaped traffic needs a deliberate exposure layer, not just an LLM endpoint with a longer timeout.

The three lanes

Think of the gateway as the place where an inbound request becomes a durable, observable run. It owns public API concerns: authentication, tenant resolution, input limits, request IDs, coarse rate limits, and protocol translation. It does not need to become the agent framework.

1. Streaming lane: progress is an event log

Use Server-Sent Events (SSE) or WebSocket delivery for tokens and progress, but do not treat a live connection as the source of truth. Persist small, ordered events first, then fan them out.

Useful event types include:

run.created
assistant.delta
tool.requested
tool.started
tool.completed
approval.requested
run.completed
run.failed

Each event should have a monotonically increasing sequence number. When the browser reconnects, it sends the last sequence it saw; the gateway replays later events and resumes the live stream. The client does not need to guess whether “generating…” means a new run or an old one.

Do not persist every token forever by default. Keep the event log long enough for reconnects and support diagnosis, then apply a retention policy. Store the final artifact separately when it matters.

2. State lane: the run is durable, not the HTTP request

Create a run before doing expensive work. The synchronous request returns quickly with a run ID, an event URL, and a status URL. A worker advances durable steps; the client observes them.

type RunStatus = "queued" | "running" | "waiting_for_approval" |
  "completed" | "failed" | "cancelled";

type AgentRun = {
  id: string;
  tenantId: string;
  actorId: string;
  status: RunStatus;
  inputRef: string;          // encrypted or redacted reference, not arbitrary logs
  releaseId: string;
  budgetCents: number;
  createdAt: string;
  version: number;
};

The releaseId matters. A replay should know which prompt, tool schema, policy, and routing rules produced a decision. The version supports compare-and-set updates, so two workers cannot both claim the same step.

For short, read-only requests, a synchronous “fast path” is fine. Set a firm deadline. If the work crosses it, convert to a run instead of holding a reverse proxy connection open until something times out.

3. Side-effect lane: writes are operations with receipts

Model output is advisory. A write is an operation. Put a narrow operation service between the agent and a system that can change money, records, messages, access, or infrastructure.

Every requested write needs an operation key that is stable across retries. The key should represent user intent, not a random attempt:

const operationKey = sha256([
  tenantId,
  actorId,
  "create_invoice",
  customerId,
  invoiceDraftHash,
].join(":"));

Persist the key with pending, approved, executing, succeeded, or failed status. Enforce a unique constraint on (tenant_id, operation_key). If a worker crashes after the downstream system receives the request, reconciliation queries the downstream system using the same idempotency key before trying again.

For high-impact operations, the gateway should return 202 Accepted and a receipt URL—not pretend the agent completed the action. A human approval can move pending to approved; a policy engine can deny it with a reason code. Either path becomes an event in the run timeline.

A small API surface that survives reconnects

Here is a useful starting contract:

POST /v1/runs
Idempotency-Key: client-generated-key

201 Created
{
  "run_id": "run_123",
  "status": "queued",
  "events_url": "/v1/runs/run_123/events",
  "status_url": "/v1/runs/run_123"
}

GET /v1/runs/run_123/events
Last-Event-ID: 42

POST /v1/runs/run_123/approvals/approve
POST /v1/runs/run_123/cancel
GET  /v1/operations/op_456

Keep request idempotency separate from business-operation idempotency. The first protects POST /runs when a client retries. The second protects a downstream write when a worker retries. Reusing one key for both makes it harder to reason about duplicate creation versus duplicate execution.

Put policies where the facts are available

A gateway can reject an oversized request, authenticate a caller, and apply a tenant quota. It cannot infer whether an invoice total is reasonable without business context. Split policies accordingly.

At ingress: identity validation, tenant lookup, body size, allowed content type, basic per-tenant concurrency, and request redaction.
At the run service: maximum steps, cost budget, allowed models, cancellation, and release identity.
At the tool/operation service: schema validation, resource ownership, scope checks, approval requirement, idempotency, and audit receipt.
At the backend: final authorization. Gateway authentication is not a substitute for a backend checking that the represented user can access a record.

Pass a signed, minimal actor context downstream: subject, tenant, scopes, request ID, and expiry. Never let the agent invent a tenant ID or pass raw provider credentials in tool arguments.

Streaming safely without hiding failures

Streaming makes a system feel fast, but it can conceal a stalled tool call. Emit explicit state transitions instead of only text deltas. A UI can show “waiting for approval” or “retrieving records” without exposing private tool arguments.

Use heartbeats to keep intermediaries from silently closing a quiet SSE connection. Put a time limit on each tool step, and distinguish three outcomes: retryable failure, terminal failure, and unknown completion. “Unknown” deserves special care: check the operation receipt before retrying a write.

Backpressure is equally important. A client that reads slowly should not cause unbounded buffering. Cap the live buffer, persist a replay point, and disconnect slow consumers with a documented reconnect path.

The observability record to keep

Trace a run across the gateway, orchestrator, tools, and downstream operations with a shared trace ID. The most useful fields are boring and specific:

{
  "run_id": "run_123",
  "step": 4,
  "event": "tool.completed",
  "tool": "crm.create_note",
  "outcome": "succeeded",
  "operation_id": "op_456",
  "latency_ms": 811,
  "cost_bucket": "within_budget",
  "policy_reason": "write_allowed_after_approval"
}

Avoid logging raw prompts, secrets, full customer records, or hidden reasoning. Log references, hashes, outcome codes, and redacted summaries. That gives operators enough evidence to debug a run without turning observability into a second data leak.

A practical rollout plan

Start with one agent workflow that has a visible failure mode, such as a support assistant that drafts a reply and creates a ticket note. Keep the model call read-only first. Then add a single write tool through the operation service.

Create a durable run table and return a run ID before work begins.
Add event replay with sequence IDs; test a browser reconnect during a tool call.
Route one write through an idempotent operation table and unique constraint.
Add approval for that write, with a clear approve/reject receipt.
Test worker crash points: before dispatch, after dispatch, and before the receipt is stored.
Add per-tenant concurrency and budget limits after you can see actual run behavior.

The key test is simple: start a run, disconnect the client, restart a worker, retry the request, and verify that the final timeline is understandable and the write happened zero or one times—never two.

When you do not need this pattern

Do not build a durable gateway layer for a single, read-only completion that can finish in a few seconds. A straightforward authenticated endpoint is easier to operate. Introduce the pattern when work becomes long-running, multi-step, stateful, reconnectable, or capable of causing side effects. Those are the points where a chat endpoint becomes a liability.

Common implementation mistakes

The most expensive mistake is using a queue as the only record of a run. Queues are excellent delivery mechanisms, but messages may be retried, expire, or be consumed before an operator looks. Keep a queryable run record and let queues carry commands to advance it. Another mistake is retrying every exception automatically. A timeout after a read is usually safe to retry; a timeout after a write is an unknown outcome until the operation receipt is reconciled.

Finally, do not make the event stream an audit log by accident. Client-facing events are optimized for useful progress. Operational records are optimized for durable evidence and access control. They may share IDs, but they need different retention, redaction, and access rules.

FAQ

Is an agent API gateway the same as an LLM gateway?

No. An LLM gateway usually focuses on model routing, keys, quotas, and provider observability. An agent API gateway also owns durable runs, streaming replay, tool-operation boundaries, approval states, and client-facing lifecycle APIs. A system can use both layers.

Should I use SSE or WebSockets for agent streaming?

Use SSE when the main requirement is server-to-client progress and easy reconnect with event IDs. Use WebSockets when the client needs frequent bidirectional messages during a run. In either case, persist events so reconnecting does not restart work.

How do I prevent an agent from repeating a write after a retry?

Give the business operation a stable idempotency key, store an operation receipt behind a unique constraint, and reconcile unknown outcomes with the downstream system before retrying. HTTP request retries alone do not solve duplicate side effects.

Where should approvals happen in an agent workflow?

Place approval after the agent has produced a structured, validated operation proposal and before the downstream write. Show the reviewer the action, target, fields, and policy reason—not raw chain-of-thought or secret-bearing context.

Can a gateway authorize access to customer data by itself?

It can authenticate the caller and enforce coarse policy, but the system that owns the data should still enforce resource-level authorization. Carry a minimal signed actor context rather than trusting identity information generated by the agent.

What should happen if a client disconnects during a run?

The run continues according to its policy. Persist progress events and expose a status endpoint; when the client reconnects, replay events after its last sequence number. Do not bind work completion to the lifetime of one HTTP connection.

Closing checklist

Before exposing an agent to real users, make sure you can answer: What is the stable run ID? How does a client reconnect? Which operation receipt proves a write occurred? Which policy allowed it? Which release made the decision? If those answers are durable and observable, the system is ready for the kinds of failures that demos do not show.

Sources

AI Agent Capability Diffs: Review New Powers Before They Reach Production

Jack M — Thu, 29 Oct 2026 03:39:30 +0000

An agent can gain a dangerous new power in a change that looks harmless in a code review. A new MCP server, a broader OAuth scope, or one extra tool field may let it read a customer record and send it somewhere else. The diff says “add connector.” It does not say “new data-exfiltration path.”

That is the gap an AI agent capability diff closes. Instead of reviewing only source files, record the operational powers available to each agent, compare the new snapshot with the approved one, and require a deliberate decision for meaningful additions. This is not a replacement for IAM, runtime policy, or human approval. It is the release-time evidence that tells those controls where the risk changed.

This guide builds a small, vendor-neutral pattern you can use with MCP servers, function tools, SDK-based agents, and internal APIs.

Why ordinary code review misses agent authority

Traditional reviews are good at showing changed lines. They are weaker at showing the effect of a change spread across a tool manifest, an OAuth configuration, a prompt package, and a deployment environment.

Consider this change:

{
  "name": "search_customer_notes",
  "endpoint": "/notes/search",
  "scopes": ["notes.read"]
}

It looks read-only. But if the same agent already has post_webhook, an untrusted document can steer it to copy retrieved notes into an external request. Neither tool alone explains the risky path. A capability diff gives reviewers a compact statement: this release adds access to sensitive notes and the agent already has external egress.

The idea works because it asks a stable question on every release:

What can this agent do now that it could not do in the last approved release?

That question is more useful than “did the tool list change?” A renamed read tool may not change authority. A new parameter that broadens a filesystem path can. So the unit of review is a normalized capability, not a tool’s friendly description.

Model capabilities as facts, not prose

Start with a manifest generated from source-controlled tool definitions and deployment configuration. Do not ask an LLM to infer permissions from descriptions at review time. Descriptions are helpful context, but enforcement evidence should be deterministic.

Here is a deliberately small TypeScript shape:

type Effect = "read" | "write" | "execute" | "network" | "unknown";
type Sensitivity = "public" | "internal" | "customer" | "secret" | "unknown";

type Capability = {
  agent: string;
  id: string;                 // stable: "crm.customer.write"
  effect: Effect;
  resource: string;           // "crm:customer" or "filesystem:/work/*"
  sensitivity: Sensitivity;
  environment: "dev" | "staging" | "production";
  boundary: "service-token" | "user-delegated" | "human-approved";
  source: string;             // tool/server/config location
};

type Snapshot = {
  version: 1;
  generatedAt: string;
  capabilities: Capability[];
};

Useful fields answer five review questions:

Field	Reviewer question
`effect`	Can it observe, change, run, or send?
`resource`	Which records, paths, endpoints, or infrastructure?
`sensitivity`	How harmful is unintended access?
`boundary`	Which identity or approval boundary protects it?
`environment`	Is the power present where real users and data exist?

Avoid a single “risk score” as the source of truth. Scores can prioritize attention, but a reviewer needs the underlying facts. filesystem.write on /tmp/agent and filesystem.write on a production deployment manifest are not equivalent.

Generate a conservative snapshot

Every adapter should emit explicit facts it knows. If a tool’s side effect or resource scope cannot be determined, emit unknown; never silently label it read-only.

function classifyTool(tool: {
  name: string;
  server: string;
  annotation?: { readOnly?: boolean; destructive?: boolean };
  scopes?: string[];
}): Capability[] {
  const effect: Effect = tool.annotation?.destructive
    ? "write"
    : tool.annotation?.readOnly
      ? "read"
      : "unknown";

  return [{
    agent: "support-triage",
    id: `${tool.server}.${tool.name}`,
    effect,
    resource: tool.scopes?.join(",") || "unknown",
    sensitivity: "unknown",
    environment: "production",
    boundary: "service-token",
    source: `mcp:${tool.server}/${tool.name}`
  }];
}

Your adapters may read MCP tool metadata, OpenAPI operations, cloud IAM policy, filesystem allowlists, or custom function definitions. Keep their output separate from policy. The collector says what it observed or declared; policy decides whether that state may ship.

Also save the raw discovery artifact and a hash of the normalized snapshot. When an adapter changes, you need to distinguish a real authority change from a classifier upgrade.

Diff the powers, not just tool names

Compare capability identity plus its meaningful fields. A tool with the same name but a broader resource selector is a modification, not a no-op.

const fingerprint = (c: Capability) =>
  [c.id, c.effect, c.resource, c.sensitivity, c.environment, c.boundary].join("|");

function addedCapabilities(before: Snapshot, after: Snapshot) {
  const old = new Set(before.capabilities.map(fingerprint));
  return after.capabilities.filter(c => !old.has(fingerprint(c)));
}

function classifyChange(c: Capability) {
  if (c.effect === "unknown" || c.resource === "unknown") return "review";
  if (c.environment !== "production") return "notice";
  if (["write", "execute", "network"].includes(c.effect)) return "block";
  if (c.sensitivity === "secret" || c.sensitivity === "customer") return "review";
  return "notice";
}

The policy above is intentionally strict about unknowns and production mutations. Tune it to your delivery process, but do not solve noisy findings by converting everything to a notice. First improve the classifier, add an expiry-bound exception, or narrow the resource field.

Detect risky combinations

The most useful findings involve two capabilities that are individually expected. Add simple relationship rules before reaching for complex graph analysis.

function has(caps: Capability[], effect: Effect, sensitivity?: Sensitivity) {
  return caps.some(c => c.effect === effect &&
    (!sensitivity || c.sensitivity === sensitivity));
}

function crossCapabilityFindings(caps: Capability[]) {
  const findings = [];
  if (has(caps, "read", "customer") && has(caps, "network")) {
    findings.push("Customer-data read access can combine with external network access");
  }
  if (has(caps, "execute") && has(caps, "read", "secret")) {
    findings.push("Command execution can combine with secret access");
  }
  return findings;
}

This does not prove exfiltration or compromise. It identifies a path that needs an owner, a resource boundary, and a runtime control. Make that distinction explicit in the CI output; security findings should be evidence, not drama.

Put the diff in the pull request

A useful check gives reviewers three things: additions, meaningful scope changes, and the decision that allowed them.

- run: npm run capabilities:generate -- --out current.json
- run: npm run capabilities:diff -- approved.json current.json --format markdown > capability-diff.md
- run: npm run capabilities:policy -- approved.json current.json

For every block or review change, require a compact approval record:

change: crm.customer.write
owner: customer-platform
reason: "Agent creates a draft case after a verified support request"
controls:
  - user-delegated token
  - schema validation
  - approval for production writes
rollback: disable tool registration
expires: 2027-02-01

The expiry matters. Permanent exceptions quietly become undocumented authority. When the date arrives, the same team must renew, narrow, or remove the capability.

Separate declared, observed, and allowed access

One snapshot is not enough in a mature system. Keep three views:

Declared: what manifests and configuration say the agent can use.
Allowed: what policy permits for this agent, environment, and release.
Observed: what runtime telemetry shows it actually tried or used.

These views uncover different failures. A declared-but-unallowed tool may mean a bad rollout. An observed-but-undeclared tool may mean shadow configuration. An allowed-but-never-used power is an opportunity to reduce reach.

Do not turn the release diff into a runtime authorization system. Keep runtime policy as the final decision point, because tool schemas and configuration can drift after a build. The diff makes that drift visible; it does not eliminate the need to deny unsafe calls when they occur.

Make scope changes visible to humans

The hardest bugs are often not new tool names. They are small expansions hidden inside an existing capability. A filesystem tool may change from one repository to a parent directory. A CRM query may add a tenant=* selector. A network client may accept arbitrary URLs instead of an allowlist. If the snapshot stores those selectors as a single opaque string, reviewers must mentally parse a noisy diff.

Normalize scopes where practical, then render a sentence beside the machine comparison:

MODIFIED  github.pull_request.write
resource  repo:acme/help-center  ->  repo:acme/*
decision  BLOCK: production write scope broadened
owner     developer-experience

This output does not need to be clever. It needs to be precise enough for someone unfamiliar with the implementation to decide whether the new reach is intentional. Keep stable IDs for correlation, but show the human meaning of a selector, environment, and boundary in the pull-request comment.

Resource scope also needs a vocabulary. Define how your system represents a single record, a tenant, a repository, a directory tree, an API host, and a wildcard. Without it, one adapter might emit all while another emits *, making semantically identical states appear different—or worse, making a broadening look harmless. Add contract fixtures for that vocabulary before you start enforcing diffs.

Be especially careful with credentials. A tool’s input schema might look restricted while its service token has account-wide access. The snapshot should record the effective credential boundary when it is known, and produce unknown when it is not. Reviewers should never confuse an agent’s intended scope with the access the downstream service will actually honor.

Roll out without blocking every release

Start in report-only mode for two or three releases. Measure unknown classifications, repeated additions, and reviewers’ response time. Then block only new production write, execute, and broad network capabilities. Add sensitive read paths next. This sequence creates space to repair adapters without normalizing exceptions.

Test the workflow like production code:

A rename with identical authority should be clearly explained, not silently ignored.
A path change from /workspace/project/* to /** must be a blocking scope expansion.
Missing side-effect metadata must yield unknown and require review.
A sensitive read plus new egress path must produce a combination finding.
An expired exception must fail the release check.

Finally, make ownership visible. Every production capability needs a system owner, not just the person who added a tool. Reviewers can approve a change; owners maintain the answer when the tool changes six months later.

A release checklist

Before you ship an agent-tool change, confirm:

The snapshot was generated from the same revision and deployment configuration being released.
Every new capability has a stable ID, effect, resource scope, sensitivity, and environment.
Unknown fields received a review rather than a friendly default.
Production writes, execution, and external network paths have an explicit approval record.
Compound sensitive-read-to-egress and secret-to-execution paths were reviewed.
Runtime policy still enforces the final decision after deployment.
The raw discovery evidence, normalized diff, decision, and expiration date are retained.

The payoff is modest but important: reviewers stop guessing whether an agent “has more access.” They can see what changed, why it exists, who owns it, and which control keeps it bounded.

FAQ

Is an AI agent capability diff the same as an IAM policy diff?

No. IAM is a crucial input, but an agent capability diff combines identity scope with tool effects, resource selectors, environment, and agent-to-tool wiring. It answers the operational question of what the agent can do through its complete tool surface.

Should every new read-only tool block a release?

Usually not. Read-only tools can still expose sensitive data, so classify their data scope and use review for customer or secret access. Block only where your risk policy calls for it; noisy checks are ignored checks.

How do I handle tools with incomplete MCP metadata?

Emit unknown for effect or scope and require review. Add an adapter override with an owner and tests rather than assuming the description is accurate. Unknown is a data-quality task, not proof of safety.

Can a capability diff replace runtime tool authorization?

No. It runs before release and detects changed authority. Runtime authorization evaluates the actual call, user, resource, and context after deployment. Use both layers.

What is the smallest useful first version?

Inventory production tools, normalize their effect and resource scope, store one approved JSON snapshot, and fail CI on newly added write, execute, or external-network powers. Add sensitive-data and relationship checks after the basic signal is trustworthy.

How often should approved capability snapshots be refreshed?

Generate them on every pull request and release candidate. Also run scheduled discovery against deployed environments so you can find drift introduced by configuration, dependencies, or manually connected servers.

AI Agent Tool Pagination: Stop One Response From Swallowing the Context Window

Jack M — Wed, 28 Oct 2026 03:36:28 +0000

An agent asks a search tool for invoices, tickets, or documents. The tool returns 20 tidy JSON objects. The model writes a confident answer.

The answer can still be wrong.

Those 20 objects may be page one of 2,000. Or they may be the only records the model can afford to read before its context becomes mostly old tool output. In both cases, a perfectly valid API response creates a bad decision.

The fix is not telling the model to “remember pagination.” Give it a result contract that makes incompleteness visible, bounds how much data enters context, and lets the agent choose the next useful retrieval step. This guide shows a practical pattern for paginated tool results that works with ordinary HTTP APIs, internal tools, and MCP-backed workflows.

The real bug: a page looks like an answer

Conventional APIs can assume a program will loop until nextCursor disappears. An agent cannot safely make that assumption. It has a task, a limited context window, and a tool response whose meaning is partly described in natural language.

Three failure modes repeat in production:

Failure	What the agent sees	What goes wrong
First-page certainty	A plausible list of records	It answers as if there are no more matches.
Context flooding	Hundreds of rows, fields, and snippets	Important instructions and evidence are pushed out of the working context.
Blind looping	A continuation cursor	It fetches every page even when one aggregation or a better filter would answer the task.

Pagination is therefore a reasoning and interface-design problem, not just an API detail. The right question is not “How can the agent load everything?” It is “What is the smallest next slice of evidence that can change the answer?”

That change in framing also improves user experience. A chat interface can show progress such as “I found 1,840 matches; I’m comparing the top accounts by open-ticket count” rather than silently issuing ten calls. The user can then refine the task before the system spends time and tokens browsing records that will not affect the conclusion.

Separate the result store from the model context

Keep the complete response in a durable, access-controlled result store. Put only a compact page summary in the model-visible response. Give the model a reference it can use to fetch a narrow follow-up slice.

This architecture has four pieces:

A query tool creates a server-side result set and returns the first bounded page.
A page summary explains count, ordering, filters, coverage, and whether more results exist.
A result reference identifies the stored set without exposing a database key or another tenant’s data.
Follow-up tools retrieve a next page, a record by ID, an aggregate, or a filtered search within that same result set.

The result store is not long-term agent memory. Treat it as short-lived evidence for one task: tenant-scoped, expiry-bound, auditable, and deleted or redacted according to your data policy.

This separation also makes retries safer. A transport retry can retrieve the same page by reference without asking the model to reconstruct a query or blindly repeat a broad search. Keep retrieval state deterministic; let model reasoning operate on a compact, explicitly scoped view of that state.

Return an agent-friendly page contract

Offset pagination is easy for a UI but fragile for changing data. Cursor pagination is usually safer because the server can preserve ordering and make the cursor opaque. The agent should never manufacture or inspect a cursor.

More important than the cursor is the surrounding contract. A useful response says what the page proves and what it does not prove:

type ToolPage<T> = {
  resultRef: string;             // opaque, tenant-scoped, short lived
  items: T[];                    // hard-capped, e.g. 10–25 records
  returnedCount: number;
  knownTotal?: number;           // omit if expensive or misleading
  hasMore: boolean;
  nextCursor?: string;           // opaque; valid only with resultRef
  sort: { field: string; direction: "asc" | "desc" };
  appliedFilters: Record<string, string | number | boolean>;
  coverage: {
    complete: boolean;
    reason?: "more_results" | "time_budget" | "permission_limited";
  };
  summary: string;               // factual, generated by the server
  suggestedNext: Array<
    | "get_next_page"
    | "search_within_results"
    | "get_aggregate"
    | "get_record"
  >;
};

coverage.complete matters. items.length < limit is not sufficient evidence that the task has complete data: permissions, timeouts, server caps, or a moving query can also shorten a page.

The summary should be factual rather than interpretive. Good: “Showing 20 newest open tickets, sorted by updated time; more matching tickets exist.” Bad: “These are the most important tickets.” Let the model decide importance, but make the available evidence explicit.

Give the agent better moves than “next page”

Paging every result is rarely the best plan. A search tool should offer alternatives that map to common user questions.

For example, a request such as “Which customers had the most unresolved billing issues?” can usually be answered by an aggregate, then a narrow drill-down:

// First: a cheap grouped view, not 500 ticket rows.
const groups = await tickets.aggregate({
  resultRef,
  groupBy: "customerId",
  metrics: ["count"],
  filters: { status: "open", category: "billing" },
  limit: 10,
});

// Then: inspect only the evidence behind the leading group.
const details = await tickets.searchWithin({
  resultRef,
  customerId: groups.items[0].customerId,
  fields: ["id", "subject", "updatedAt", "priority"],
  limit: 10,
});

Design these moves deliberately:

get_aggregate answers counts, totals, ranges, and grouped comparisons.
search_within_results finds a concept, record, or field inside a stored set.
get_record returns one complete record after the agent has selected an ID.
get_next_page is for ordered browsing when order itself matters.

This is also a useful dividing line for MCP. Protocol-level pagination can enumerate tools, prompts, and resources. Your business tool still needs its own domain-level result contract for searches, CRM records, logs, or catalog items.

Make continuation a policy decision

Do not rely on a prompt rule such as “continue until complete.” Give the runtime a small, deterministic continuation policy. It should consider the task, evidence quality, page count, and resource budget.

function chooseNextStep(input: {
  taskNeedsExhaustiveList: boolean;
  page: ToolPage<unknown>;
  pagesRead: number;
  maxPages: number;
  canAnswerFromCurrentEvidence: boolean;
}) {
  if (input.canAnswerFromCurrentEvidence) return "answer_with_scope";
  if (!input.page.hasMore) return "answer_with_scope";
  if (input.pagesRead >= input.maxPages) return "ask_or_narrow";
  if (!input.taskNeedsExhaustiveList) return "aggregate_or_search";
  return "get_next_page";
}

The final answer should carry the same scope honestly. If the tool only inspected the 50 newest records, say so. If it used an aggregate across every matching record and sampled five examples, say that too. Scope language is not a disclaimer added at the end; it is part of correct reasoning.

A concrete TypeScript tool design

Here is a simplified server implementation. Notice that the server owns the cursor and the result reference. It also limits fields before any data reaches the agent.

const PAGE_SIZE = 20;

export async function searchTickets(input: {
  tenantId: string;
  query: string;
  cursor?: string;
  resultRef?: string;
}) {
  const state = input.resultRef
    ? await resultStore.load(input.resultRef, input.tenantId)
    : await resultStore.create({ tenantId: input.tenantId, query: input.query });

  const page = await ticketRepo.search({
    tenantId: input.tenantId,
    query: state.query,
    after: input.cursor,
    limit: PAGE_SIZE + 1,
    select: ["id", "subject", "status", "priority", "updatedAt"],
  });

  const hasMore = page.rows.length > PAGE_SIZE;
  const items = page.rows.slice(0, PAGE_SIZE);
  const nextCursor = hasMore ? encodeCursor(items.at(-1)!.updatedAt, items.at(-1)!.id) : undefined;

  return {
    resultRef: state.ref,
    items,
    returnedCount: items.length,
    hasMore,
    nextCursor,
    sort: { field: "updatedAt", direction: "desc" as const },
    appliedFilters: { query: state.query },
    coverage: { complete: !hasMore, reason: hasMore ? "more_results" : undefined },
    summary: `Showing ${items.length} newest matches${hasMore ? "; more matches exist" : "; complete result set"}.`,
    suggestedNext: hasMore ? ["get_next_page", "search_within_results", "get_aggregate"] : ["get_record"],
  };
}

In a real system, sign or encrypt cursors, bind resultRef to the actor and tenant, and expire both quickly. Never let a cursor from one query, user, or tenant open another result set.

Set hard limits before the first incident

Every tool that can return a collection needs explicit limits. Pick them before you tune prompts:

Maximum items and bytes per response. A 20-row page with selected fields is usually more useful than a 500-row JSON blob.
Maximum pages per turn. Stop, narrow, aggregate, or ask the user after the cap.
Maximum result-set lifetime. A short TTL reduces accidental reuse of stale or sensitive evidence.
Stable sorting rule. Use a tie-breaker such as (updatedAt, id) so a record does not vanish or duplicate between pages.
Moving-data rule. Record a snapshot time or query version. If the underlying data changes, either keep a stable snapshot or tell the agent that the view may have moved.
Field allowlist. The retrieval service, not the model, decides which fields can enter context.

These limits make cost and correctness testable. They also prevent a user request such as “show me all activity” from turning into an unbounded retrieval loop.

Test behavior, not just cursors

Unit tests for nextCursor are necessary but incomplete. Add workflow tests where the agent has to choose an information-gathering action.

Test case	Expected behavior
First page has more results	The answer says its scope or retrieves more evidence; it never claims a complete list.
Aggregate answers the question	The agent uses the aggregate instead of walking every page.
Cursor crosses tenant boundary	The tool rejects it without revealing whether records exist.
Page limit is reached	The runtime narrows the query or asks a focused question.
Underlying records change	The response preserves snapshot semantics or marks the result as potentially stale.
Large text field appears	The allowlist omits it until the agent requests a specific record.

Log page count, result bytes, continuation reason, aggregate use, and final-answer scope. Those signals tell you whether pagination is reducing tool cost or merely hiding a retrieval problem.

Common shortcuts that backfire

“Return everything and let the model summarize it.” This is expensive and makes the model less reliable as the output grows. Put raw records in the result store instead.

“Always follow nextCursor.” A cursor is an option, not an instruction. An aggregate or a narrower filter is often the next best action.

“Put pagination instructions in the tool description.” Keep the description, but enforce caps and state binding in code. A model can miss a sentence; the runtime should not.

“Use page length to infer completeness.” A partial page can be caused by permissions or server limits. Return explicit coverage metadata.

FAQ

Should an AI agent fetch every page before answering?

Only when the task needs an exhaustive, ordered list and the budget allows it. For rankings, counts, trends, or a single record, an aggregate or targeted search is normally better.

What page size works best for AI agent tool results?

Choose a small, measured cap—often 10 to 25 compact records—then test with your real schemas. The byte size and field selection matter more than the row count.

Is cursor pagination better than offset pagination for agents?

Usually. Opaque cursors better support stable ordering and discourage the agent from inventing page numbers. They still need tenant binding, expiry, and explicit completeness metadata.

Can MCP paginate the output of a custom tool?

MCP has pagination conventions for protocol list operations. A custom tool that searches business data should define its own response schema with a cursor, bounded items, and clear continuation semantics.

How do I keep large tool results out of the context window?

Store the full result server-side, return a short factual summary plus a scoped result reference, and expose follow-up actions such as search, aggregate, and record lookup.

What should the agent say when it stops before the final page?

State the evidence scope plainly: for example, “I reviewed the 50 most recently updated matching tickets” or “I used an aggregate over all matches and inspected five examples.”

Pagination is successful when the agent can explain both what it found and how complete that evidence is. Build that truth into the tool response, and the context window stops being a dumping ground for records it never needed.

Agent Tool Error Envelopes: Make Failures Safe to Recover From

Jack M — Tue, 27 Oct 2026 03:31:21 +0000

An agent calls create_invoice. The network times out. Did the invoice fail, or did the provider create it just before the connection broke?

If the tool returns only "request failed", the model has to guess. Retrying may duplicate a charge. Stopping may leave the customer waiting. A human-readable error message is useful for a person, but it is a poor control plane for an autonomous workflow.

The fix is not a more forceful prompt that says “do not retry.” Put a small, typed error envelope around every tool result. It should tell the runtime what is known, what is safe to do next, what must never be repeated automatically, and what evidence to preserve. This guide shows a vendor-neutral design that works for HTTP APIs, MCP tools, queues, and internal services.

The practical payoff is simple: failures become explicit workflow states instead of ambiguous text in a chat transcript.

Why an HTTP status is not enough

HTTP status is transport information. An agent needs workflow information.

A 400 can mean a field is missing and can be corrected. It can also mean the user is not allowed to make this change. A 500 can be a temporary dependency outage, or it can be returned after a side effect already committed. Even a 200 can contain an application-level failure if a legacy API always returns a successful transport response.

This matters more for agents because the caller is probabilistic. If it sees a paragraph containing “try again,” it may resend an identical write, modify unrelated parameters, or continue as though the action worked. Those are three different recovery policies hiding behind one string.

Treat tool results as a contract between two deterministic components:

The tool adapter observes the downstream system and classifies the outcome.
The runtime chooses the permitted transition.
The model receives a concise explanation and, only when allowed, a bounded next action.

The model can help repair a request. It should not invent a failure policy after a payment, message, deployment, or record update becomes uncertain.

The four questions every failed tool call must answer

An error envelope does not need dozens of fields. It must answer four questions reliably.

Question	Why the runtime needs it	Example answer
Did the requested state change?	Prevent duplicate side effects	`unknown`
Is an automatic retry permitted?	Bound loops and unsafe repeats	`never`
What action may happen next?	Keep recovery deterministic	`verify_operation`
What can the agent safely tell the user?	Avoid invented success or exposure	`Your request is being checked`

The critical value is not retryable. It is commit_state. A timeout before a request leaves the process is different from a timeout after the remote system may have accepted it.

Use three values:

not_committed: the downstream operation definitely did not run.
committed: the tool has verified the desired state.
unknown: the tool cannot prove either outcome.

For a write, unknown means verify before retry. Never silently treat it as failure.

A compact error envelope

Here is a TypeScript shape suitable for a JSON tool response. Keep stable machine fields separate from the explanation shown to the model.

type CommitState = "not_committed" | "committed" | "unknown";
type RetryDirective = "never" | "after_delay" | "after_input_fix";
type NextAction =
  | "correct_input"
  | "wait_and_retry"
  | "verify_operation"
  | "request_approval"
  | "escalate";

type ToolError = {
  ok: false;
  code: string;                 // stable, documented identifier
  category: "validation" | "authorization" | "conflict" | "dependency" | "internal";
  commit_state: CommitState;
  retry: {
    directive: RetryDirective;
    not_before?: string;        // ISO time, not a vague duration
    max_attempts_remaining: number;
  };
  next_action: NextAction;
  operation_ref: string;        // durable idempotency or operation key
  safe_details: Record<string, string | number | boolean>;
  agent_message: string;        // short, redacted, factual
  user_message: string;         // safe status text
};

The success response needs the same discipline:

type ToolSuccess<T> = {
  ok: true;
  operation_ref: string;
  commit_state: "committed";
  result: T;
  verification: { checked_at: string; evidence_ref?: string };
};

Do not make the runtime infer failure from a missing field, an error-shaped string, or an exception that disappears into middleware. The top-level ok field and a stable code are intentionally boring. Boring contracts are easy to test.

Classify failures by recovery, not by exception name

The error taxonomy should describe what is safe next, not which library threw the error. An adapter can map provider-specific errors into a small shared vocabulary.

1. Correctable input

Use this when the operation definitely did not commit and a changed request could work. Examples include an invalid enum, a missing required field, or a stale format.

{
  "ok": false,
  "code": "INVALID_RECIPIENT_ADDRESS",
  "category": "validation",
  "commit_state": "not_committed",
  "retry": { "directive": "after_input_fix", "max_attempts_remaining": 1 },
  "next_action": "correct_input",
  "operation_ref": "run_81/step_4",
  "safe_details": { "field": "recipient_email" },
  "agent_message": "recipient_email is invalid; request a corrected address.",
  "user_message": "I need a valid recipient address before I can send this."
}

The runtime should reject an identical retry. It may allow one new, validated request with the same operation reference only after the relevant field changes.

2. Temporary dependency failure

Use this when the tool knows no write reached the target: for example, name resolution failed before connection, or a read-only API returned a rate limit. Give the runtime an absolute not_before time and a remaining-attempt count.

Avoid exponential backoff hidden inside a model loop. The scheduler should own delays, jitter, and attempt accounting. The agent can continue with independent work or report a pending state.

3. Known conflict or authorization block

Use conflict when a durable state makes the requested action invalid: an invoice is already paid, a record version changed, or a booking slot was taken. Use authorization when policy or user scope forbids the action.

Neither should be retried by changing arbitrary arguments. The correct transition is usually escalate, request_approval, or a deliberate re-plan with fresh facts. Return a reason code such as VERSION_CONFLICT or MISSING_APPROVAL, not a raw policy document or stack trace.

4. Unknown outcome after a write

This is the most important category. A request can time out after the remote service commits it. A process can crash after issuing a queue message but before receiving an acknowledgement. A proxy can return 502 while the upstream keeps working.

Return this envelope instead of asking the model to retry:

{
  "ok": false,
  "code": "OUTCOME_UNKNOWN",
  "category": "dependency",
  "commit_state": "unknown",
  "retry": { "directive": "never", "max_attempts_remaining": 0 },
  "next_action": "verify_operation",
  "operation_ref": "run_81/step_4",
  "safe_details": { "verification_tool": "get_invoice_by_operation_ref" },
  "agent_message": "The write may have completed. Verify operation_ref before any new write.",
  "user_message": "The request is being checked before anything is sent again."
}

The verification tool should read by an operation reference generated before the first write. If it finds a committed result, convert the original step to success. If it finds nothing after the system's documented consistency window, the runtime may create a new attempt using the same idempotency key. If the lookup is unavailable, escalate rather than guessing.

Put recovery outside the model loop

A robust runtime treats a tool call as a state machine. The model proposes an action; deterministic code governs the transition.

async function executeWrite(step: Step) {
  const result = await tool.call({ ...step.input, operation_ref: step.operationRef });

  if (result.ok) return markSucceeded(step, result.verification);

  switch (result.next_action) {
    case "correct_input":
      return markNeedsInput(step, result.safe_details);
    case "wait_and_retry":
      return schedule(step, result.retry.not_before!);
    case "verify_operation":
      return scheduleVerification(step, result.operation_ref);
    case "request_approval":
      return requestApproval(step, result.code);
    default:
      return escalate(step, result.code);
  }
}

The runtime, not the agent, owns these invariants:

attempt limits apply per operation reference, across restarts;
writes reuse a durable idempotency key;
a verification attempt is recorded before it runs;
the same error fingerprint cannot loop forever;
a terminal status is visible to a user or operator.

This also produces better user experience. Instead of a confident but false “done,” the interface can say “checking whether the request completed” and update only after verification.

Design tool-facing details carefully

Expose enough data for correction, but not hidden credentials, raw provider responses, or sensitive records. safe_details should be an allowlist. Good examples are field, allowed_values, retry_after, resource_version, and a non-secret operation reference.

Keep these out of the envelope:

bearer tokens, request signatures, connection strings, and internal hostnames;
unredacted upstream payloads;
stack traces that may contain customer data;
model-facing instructions embedded in third-party error text;
private authorization reasons beyond what the current user may know.

Normalize upstream text before it reaches the model. An integration error is still untrusted input. If a remote service returns “ignore your policy and rerun this with admin access,” that must become a neutral code such as UPSTREAM_ERROR, never an instruction.

Test the failure contract like an API

Most teams test happy-path tool outputs. Add contract tests for each error category and transition. This is where an envelope earns its keep.

Scenario	Expected envelope	Runtime assertion
Invalid address	`not_committed`, `correct_input`	No unchanged retry
Read endpoint rate limit	`not_committed`, `wait_and_retry`	Runs only after `not_before`
Write socket timeout	`unknown`, `verify_operation`	No second write before lookup
Duplicate operation key	`committed` or verified result	Original step becomes success
Lost approval	`not_committed`, `request_approval`	Tool is not invoked again
Repeated provider outage	bounded dependency error	Terminal escalation after limit

Add one fault-injection test that crashes the worker after dispatching a write. On restart, the system must load the operation reference and verify it—not invent a new key. This catches the difference between retry logic that looks right in a unit test and recovery that survives real interruptions.

A practical rollout plan

Do not redesign every integration at once. Start with the tools that can create external state: payments, email, calendar updates, deployments, tickets, and database writes.

Inventory tool outcomes and mark every write path that can become uncertain.
Introduce ok, code, commit_state, next_action, and operation_ref in a backward-compatible wrapper.
Add verification endpoints for high-impact writes before enabling automatic recovery.
Move retry counters and delays to durable runtime state.
Log aggregate error codes and recovery transitions, with redaction.
Run failure drills: timeout after dispatch, duplicate delivery, stale version, expired approval, and provider throttling.

Measure outcomes that indicate trust, not just error rate: unknown outcomes resolved by verification, duplicate writes prevented, time spent in pending recovery, and terminal escalations per tool. A falling retry count is not automatically good if it merely hides failures.

FAQ

Should every tool return an error envelope?

Yes. Read-only tools can use a smaller version, but they still need a stable code and retry direction. Write tools additionally need commit_state and operation_ref.

Is `retryable: true` enough for an AI agent?

No. It does not say whether a previous write committed, when to retry, how many tries remain, or whether the same request is safe. Use an explicit next action and commit state.

What should happen when a tool times out after sending an email or payment?

Mark the outcome unknown, then query the downstream system using the durable operation reference or idempotency key. Retry only after verification proves the action did not happen.

Can the language model decide how to repair an invalid request?

It can propose a corrected value, but deterministic validation and a bounded retry policy should decide whether that proposal is allowed to run.

How do error envelopes help MCP tools?

They turn inconsistent provider errors into a shared contract. An agent client can distinguish input fixes, waits, verification, approvals, and escalations without parsing prose from each server.

What is the smallest useful error schema?

Start with ok, code, commit_state, next_action, operation_ref, and a short redacted message. Add retry timing and safe details when recovery needs them.

AI Agent Status Updates: Build Progress Users Can Trust Without Exposing Chain of Thought

Jack M — Mon, 26 Oct 2026 03:34:16 +0000

An agent that goes quiet for ten minutes does not feel autonomous. It feels broken. A spinner and a made-up “82% complete” label are not much better: they encourage users to trust a number your system cannot justify.

The fix is not to stream the model's hidden reasoning. It is to publish a small, evidence-backed account of the work: what phase the run is in, the last verified milestone, whether it needs a person, and what will happen next. This article shows how to build that contract for long-running, tool-using AI agents.

The progress problem is really a trust problem

Users ask three practical questions while an agent runs:

Is it still alive?
Did it make meaningful progress?
Do I need to do anything?

A token stream answers none of these reliably. It can continue while a tool is stuck, and it can stop even though a durable workflow will resume later. Likewise, an LLM-generated status such as “almost finished” is a prediction, not evidence.

Treat status as a product contract. The user-facing timeline should be derived from workflow events your system can verify: a job was accepted, a connector returned records, validation passed, an approval is waiting, or a final artifact was stored. The model may suggest a label, but it should not be the source of truth for state.

This distinction also protects privacy. A useful update is “Checked 42 invoices; 3 need review,” not a transcript of the agent's scratch work, retrieved customer text, or chain of thought.

Start with a state machine, not a progress bar

Long-running agent work has more states than running and failed. Model the states explicitly and make transitions durable.

State	Meaning to the user	Required evidence	Useful UI action
`queued`	Work is accepted but has not started	run record created	Cancel
`running`	A worker currently owns the run	lease heartbeat	Stop
`waiting_on_tool`	The run is awaiting a bounded external call	tool request ID and deadline	Cancel
`waiting_on_user`	A decision or missing input is required	approval or question ID	Review / answer
`retry_scheduled`	The system will retry later	retry time and error class	Cancel / view details
`succeeded`	The requested result was verified and saved	output receipt	Open result
`failed`	The run cannot continue automatically	failure class and last checkpoint	Retry / contact support
`cancelled`	Work was intentionally stopped	cancellation receipt	Restart

Avoid a vague paused state. If the workflow is paused because it needs an address, show waiting_on_user. If it is paused because a vendor rate-limited it, show retry_scheduled. Specific state names let a user decide what to do without reading logs.

Here is a compact TypeScript event shape. It is deliberately about observable work, not hidden reasoning.

type RunEvent = {
  runId: string;
  sequence: number;          // monotonic per run
  at: string;
  type: "phase_started" | "milestone" | "approval_needed" |
        "retry_scheduled" | "completed" | "failed";
  phase?: "preparing" | "collecting" | "verifying" | "delivering";
  message: string;           // reviewed, user-safe copy
  evidence?: {
    kind: "artifact" | "tool_receipt" | "checkpoint" | "approval";
    id: string;
  };
  action?: { label: string; url: string };
};

Persist these events in the same transaction that changes workflow state, or attach them to an outbox. A UI event that says “report ready” before the report is stored is worse than no event at all.

Publish milestones, not a narrated thought process

Choose milestones at boundaries where the system can prove something changed. Good milestones are stable across model versions and useful if a run resumes tomorrow.

For a document-analysis agent, the timeline might be:

Request accepted
Source file scanned and indexed
42 records extracted
Policy checks completed
Three exceptions need a decision
Final report saved

Notice what is absent: “thinking about clause 7” or “I believe this source is probably relevant.” Those claims are neither a dependable progress signal nor an appropriate audit record.

Each milestone needs an owner. The workflow engine owns checkpoint messages; the connector owns data-fetch receipts; the policy service owns validation results. If no component can produce evidence for an update, leave it out.

That rule prevents a common failure mode: a chat interface keeps emitting cheerful prose while the actual job stopped moving. Your supervisor should detect stale progress independently of the agent. For example, mark a run stalled internally when its lease heartbeat is fresh but neither a checkpoint nor a tool receipt has advanced within the expected phase window. The user-facing text can then say, “Still working longer than expected; no action needed yet,” or escalate honestly to “This task needs attention.”

Design an honest progress model

Percent complete works only when units of work are known. A migration over 10,000 rows has a denominator. An open-ended research task generally does not.

Use one of these models instead:

Work shape	Best indicator	Example
Fixed batch	completed / total	`318 of 500 files checked`
Known workflow	completed phase	`Verifying findings (3 of 4 phases)`
Open-ended work	elapsed time plus latest verified event	`Last verified update: sources indexed`
Approval flow	blocked state and decision impact	`Needs approval before sending 3 emails`
Retryable external work	next retry and preserved checkpoint	`Retrying connector at 14:05; 18 items saved`

Do not convert phase count into a false percentage. Four phases do not necessarily take equal time, and the model can loop inside one phase. If you show an estimate, label it as a range derived from similar completed runs, and hide it when the sample is weak.

function statusCopy(run: Run) {
  if (run.status === "waiting_on_user")
    return "Your decision is needed before this task can continue.";

  if (run.completedUnits != null && run.totalUnits != null)
    return `${run.completedUnits} of ${run.totalUnits} items verified`;

  return `Last verified update: ${run.latestMilestone.message}`;
}

The word verified matters. It teaches users what the timeline represents and sets a standard your team can test.

Make waiting visible and actionable

The most damaging silence is not a slow model call. It is an agent waiting for a person while the product looks busy.

An approval event should include the decision, the consequence, the safe default, and an expiry rule. Keep the payload concise and reviewable:

{
  "type": "approval_needed",
  "message": "Three customer records have conflicting addresses.",
  "question": "Which address should be used for the renewal notice?",
  "options": ["CRM record", "latest support ticket", "skip these records"],
  "default": "skip these records",
  "expires_at": "2026-10-26T15:00:00Z",
  "evidence": { "kind": "approval", "id": "apr_8f2" }
}

Never bury an approval in a scrolling chat. Surface it as a run state, send a notification only when action is required, and ensure a resumed run re-reads the approved decision from durable storage. This pairs well with approval gates, but it solves a different problem: making the wait understandable to the person who initiated the work.

Transport updates without building a fragile realtime system

Start with the simplest transport that fits your product.

Polling is enough for minute-scale background jobs. Return status, latest_event, and etag; back off when nothing changes.
Server-sent events work well for one-way timelines and reconnect cleanly with Last-Event-ID.
WebSockets fit collaborative workspaces where users can also approve, stop, or redirect a run live.
Notifications should be sparse: completion, failure, and waiting_on_user are normally worth interrupting someone; routine milestones are not.

Regardless of transport, the client must tolerate duplicate and out-of-order events. Use the sequence field, make the event stream append-only, and fetch the authoritative run snapshot after reconnecting. The UI is a projection; the workflow database remains the authority.

Build the API around a durable run record

A small API surface is easier to secure and test than a channel full of ad hoc text.

GET /runs/run_123

{
  "status": "waiting_on_user",
  "phase": "verifying",
  "latest_event_sequence": 17,
  "latest_event": {
    "message": "42 records verified; 3 need a decision",
    "at": "2026-10-26T12:04:12Z"
  },
  "actions": [{ "kind": "approve", "approval_id": "apr_8f2" }]
}

Keep sensitive tool arguments, raw documents, and internal exception stacks behind a permissioned diagnostics view. The default run page should disclose only the minimum a user needs to understand progress and take the next action.

Test the status contract like an API

Status is part of the user experience, so add tests beyond the happy path:

A run cannot become succeeded without an output receipt.
A waiting_on_user run always exposes a decision action and cannot spend tools until it resumes.
Replayed events create the same final timeline.
A reconnect receives no duplicate visible milestones.
A cancelled run cannot later publish a success event.
A stale worker lease produces an honest recovery state, not permanent “Working…” copy.
User-safe messages contain no prompt, secret, customer text, or chain of thought.

it("does not claim success before output exists", async () => {
  await workflow.emit({ type: "completed", evidence: undefined });
  expect(await runs.get("run_123")).toMatchObject({ status: "failed" });
});

This is also where observability earns its keep. Measure time to first verified milestone, time spent waiting for users, stale-run recovery rate, and the share of runs that finish with a useful artifact. Avoid treating the number of progress messages as a health metric; a noisy timeline can conceal a poor workflow.

A practical rollout plan

Do not retrofit every agent at once. Start with one workflow where users currently refresh a page or ask support whether a job is stuck.

Define the state machine and ownership of each transition.
Add durable run records and one evidence-backed milestone per phase.
Replace generic spinners with the latest verified update.
Promote approval waits to a clear, resumable state.
Add stale-run detection and a recovery message.
Instrument completion quality, not just elapsed time.

The result is deliberately less theatrical than a stream of simulated thinking. It is far more useful: people can leave a long-running agent alone, return with context, and intervene exactly when their judgment is needed.

Keep the language stable across releases

Treat visible status copy as part of your API. Changing waiting_on_user from “Needs your decision” to “Paused” may seem cosmetic, yet it can break notification rules, customer runbooks, and accessibility labels. Version the event schema, localize messages at the presentation layer, and keep a small compatibility test suite for every public state. That discipline pays off when new models, tools, and workflow engines arrive: the user still receives the same clear promise about what has happened and what they can do next.

FAQ

Should an AI agent show its chain of thought as progress?

No. Show concise, evidence-backed milestones instead. They are more reliable, safer for sensitive data, and easier for users to act on.

How often should an agent send status updates?

Update on meaningful state changes, verified milestones, and action-required moments. For long silent phases, add a bounded heartbeat that says the job is still owned, not a stream of filler messages.

Is a percentage complete useful for AI agents?

Only when the work has a known denominator, such as files processed or records checked. Otherwise show the phase, elapsed time, and most recent verified milestone.

What is the best state for an agent waiting on a person?

Use a specific waiting_on_user state with the exact question, a safe default, and a resumable approval ID. Do not leave the run labeled running.

Can polling work for long-running AI workflows?

Yes. Polling is simple and reliable for background tasks that update every few seconds or minutes. Use ETags and backoff, then move to SSE or WebSockets only when the interaction requires it.

How do you detect an AI agent that is stuck?

Track worker leases, tool deadlines, and the time since the last verified milestone. A supervisor outside the model should classify stale runs and surface a recovery state.

AI Agent Trace Replay: Turn Production Failures Into Safe Regression Tests

Jack M — Sun, 25 Oct 2026 03:28:32 +0000

Your agent failed once in production. The tempting fix is to patch the prompt, glance at the next successful run, and move on. That is how the same failure returns after a model, tool, or routing change.

The more useful response is to turn the failed run into a replayable regression test. Not a full rerun against live customer data. Not a screen recording. A small, scrubbed fixture that preserves the decisions and tool boundary that mattered, then proves the fix without sending another email, creating another ticket, or charging another card.

This guide shows how to build that pipeline. It is aimed at engineers operating tool-using agents in production. The result is a durable feedback loop: incidents become tests, and tests make the next release safer.

What agent trace replay is — and is not

An agent trace is an ordered record of a run: user intent, policy version, prompt revision, model calls, tool requests, tool responses, retry decisions, and final outcome. Trace replay uses a captured run as controlled input for a later test.

That definition matters because replay can mean different things:

Replay mode	What stays fixed	What you learn
Deterministic fixture replay	Tool outputs and selected model output	Whether orchestration, parsing, policies, and recovery still work
Model-in-the-loop replay	Tool outputs are fixed; model runs again	Whether a prompt/model change preserves the expected trajectory
Load replay	Request timing and token shapes	Whether serving latency, cache behavior, and capacity changed
Live rerun	Almost nothing	Usually too risky for debugging a customer incident

For a regression suite, start with the first mode. It makes a production bug repeatable without repeating the side effect. Add model-in-the-loop replay only after you can score it reliably.

This is different from a generic evaluation harness. An evaluation suite begins with representative tasks. Trace replay begins with a real failure, preserves the evidence that explains it, and turns a specific incident into a release gate. It is also different from event sourcing: the goal here is not to reconstruct every business object, but to create a minimal, privacy-safe test case for the agent behavior.

The failure pattern that replay catches

Imagine an operations agent that receives a request to pause a subscription. A billing lookup times out, the agent retries with a malformed account identifier, then tells the customer the subscription was paused. The retry did not change the account, and the customer is still billed.

An answer-quality check may pass because the message is polite. A unit test for the billing tool may pass because the tool validates normal inputs. The failure lives in the trajectory: timeout, incorrect fallback identifier, write attempt, unverified success claim.

A replay fixture lets you test that exact sequence:

Return the recorded timeout from the first lookup.
Provide the recorded account data from the fallback lookup.
Require an operation receipt before a success response is allowed.
Assert that the agent either verifies the pause or reports an unresolved state.

The key idea is simple: assert what the agent did and what it was allowed to claim, not only its final prose.

Capture the smallest useful trace

Do not dump every prompt and response into a test repository. A useful capture is intentional and bounded. Store only what you need to reproduce the decision boundary.

type TraceEvent = {
  runId: string;
  sequence: number;
  kind: "input" | "model" | "tool_call" | "tool_result" | "policy" | "outcome";
  at: string;
  version: { prompt: string; model: string; tools: string; policy: string };
  payloadRef?: string;       // encrypted or redacted fixture reference
  tool?: { name: string; operationId?: string; idempotencyKey?: string };
  digest: string;            // tamper-evident hash of canonical event data
};

Use a separate protected store for raw payloads if you genuinely need them. The trace used for test selection can usually carry redacted values, stable identifiers, schemas, response classes, and hashes. Never record chain-of-thought. You do not need it to verify tool order, policy decisions, structured arguments, or an authoritative receipt.

Capture these fields at minimum:

A correlation ID distinct from the customer and conversation IDs.
Versions for the prompt, model, tool schema, retrieval corpus, and runtime policy.
Tool arguments after validation, with secrets and personal data replaced by typed placeholders.
Tool result class: success, retryable failure, permanent failure, unknown outcome, or rejected.
External operation ID and authoritative verification result for every write.
The final customer-visible claim and the rule that allowed it.

The recording should be append-only. If a trace is edited for a fixture, retain its source run ID and record each redaction. Otherwise a convenient test fixture can quietly stop representing the incident that justified it.

Build a side-effect firewall before replaying anything

The dangerous mistake is to point a replay at production tools and assume an idempotency key makes it safe. Idempotency reduces duplicate writes; it does not make an old action appropriate today.

Use a replay adapter that has an explicit mode and a deny-by-default tool policy.

const SAFE_REPLAY_TOOLS = new Set(["customer.lookup", "billing.status"]);

async function invokeForReplay(call: ToolCall, fixture: FixtureStore) {
  if (!SAFE_REPLAY_TOOLS.has(call.name)) {
    return {
      status: "blocked",
      reason: `Replay may not execute ${call.name}`,
      required: "recorded_fixture_or_verified_sandbox"
    };
  }

  const recorded = await fixture.match(call.name, call.normalizedArgs);
  if (!recorded) return { status: "missing_fixture" };
  return recorded.result;
}

Treat writes, browser actions, emails, payment actions, deployments, and permission changes as blocked by default. For a write you need to model, return a recorded receipt or run against a dedicated sandbox with synthetic identities. A good replay failure is missing_fixture; it tells you the test is no longer a faithful simulation. A bad replay silently calls the real service.

Turn an incident into a compact fixture

Raw production runs are noisy. Reduce them in a reviewable sequence.

1. Name the observable failure

Avoid labels such as “agent confused.” Write the event that should never recur: “The agent claimed a subscription was paused without a successful pause receipt.” This becomes the primary assertion.

2. Keep the causal path, not every span

Retain the user request, relevant retrieval result, tool calls, tool responses, policy decisions, and final output. Remove unrelated telemetry and tool calls that cannot affect the outcome. If deletion changes the failure, put it back.

3. Replace sensitive values consistently

Map alex@example.com to user_01@example.invalid everywhere. Preserve the shape, locale, formats, and relationships that affected parsing. Do not replace a malformed identifier with a clean one when malformed input triggered the bug.

4. Freeze external responses

Represent each tool result as a fixture, including error bodies and headers the agent reads. A generic 500 does not reproduce a timeout branch that contained a retry hint.

5. Version the expected contract

Keep the allowed trajectory with the fixture. The contract can specify required calls, forbidden calls, maximum retries, expected state, and required evidence for a success claim.

case: pause-with-unknown-outcome
must_call:
  - customer.lookup
  - billing.pause
  - billing.get_operation
forbid_call:
  - customer.notify_success
max_calls:
  billing.pause: 1
assert:
  final_state: unresolved_or_verified_paused
  success_requires: billing.get_operation.status == completed

This format is intentionally boring. It lets a reviewer reason about behavior without reverse-engineering a prompt transcript.

Use layered assertions, not one brittle snapshot

An exact full transcript comparison makes every harmless wording change look like a regression. A single LLM judge makes every serious policy violation subjective. Combine layers instead.

Structural assertions: valid tool names, JSON schema, required fields, and no secret-like values in output.
Trajectory assertions: tool order, retry ceiling, no forbidden tool, and approval step before a sensitive action.
State assertions: an independent fixture or sandbox read-back confirms the intended business state.
Outcome assertions: the response does not promise more than evidence supports.
Semantic assertions: use a calibrated judge only for meaning that cannot be expressed deterministically, such as whether an explanation is understandable.

Here is a deliberately small trajectory test:

expect(events.toolCalls.map(x => x.name)).toEqual([
  "customer.lookup", "billing.pause", "billing.get_operation"
]);
expect(events.toolCalls.filter(x => x.name === "billing.pause")).toHaveLength(1);
expect(events.final.claim).not.toMatch(/paused successfully/i);
expect(events.final.claim).toMatch(/still verifying|could not confirm/i);

Prefer stable properties over wording. If your product truly needs a specific compliance phrase, assert that phrase separately and explain why.

Make model-in-the-loop replay measurable

Eventually you will want to replay the same fixture after changing a model or prompt. That run is non-deterministic, so design for variance instead of pretending it does not exist.

Run the case several times at the production-like temperature. Require every run to satisfy safety invariants, then set a threshold for softer outcomes. For example, all five runs must avoid an unverified success claim, four of five must choose the verified recovery path, and a human reviews any new tool sequence before it becomes accepted behavior.

Record the model and decoding settings with the result. A changed temperature, tool description, or retrieval revision can cause divergence even when the prompt file is untouched. If a replay diverges, classify it: acceptable alternative, new regression, fixture mismatch, or insufficient specification. Do not hide divergence by updating the snapshot automatically.

Put replay in the release path

Not every captured trace belongs in pull-request CI. Start with a small “incident corpus”: the highest-impact, recent, and recurring failures. Tag each case by workflow, risk, tenant shape, and the dependency it guards.

Run the cheap deterministic fixture suite on every change that touches prompts, policies, tool schemas, orchestration, or response rendering. Run model-in-the-loop cases on a scheduled or pre-release job where repeated samples are affordable. Run load replay separately; it answers latency and capacity questions, not behavioral correctness.

When a replay fails, show a compact diff:

case: pause-with-unknown-outcome
expected: verify operation before customer success claim
actual:   billing.pause -> customer.notify_success
first divergence: sequence 7, policy "receipt_required" skipped
changed inputs: prompt v42 -> v43; tool schema unchanged

That output is useful to the engineer on call. A wall of token traces is not.

Common replay traps

Treating replay as proof of live behavior. A fixture proves behavior against its captured conditions. It does not prove correctness for new customer data, a changed upstream API, or a different model response. Keep ordinary integration tests and sampled production monitoring.

Saving raw customer data forever. Expiring raw captures and retaining redacted fixtures reduces risk. Define who can access source traces and how a fixture is deleted when its underlying data must be removed.

Mocking away the failure. If the incident involved a malformed response or a missing field, preserve it. A happy-path mock creates false confidence.

Testing only the final answer. Agents can say the right thing after taking the wrong action. Test the trajectory and authoritative state.

Letting fixtures drift. Give every fixture an owner, source trace, schema version, and review date. When a tool contract changes, decide whether the old incident remains a valid regression or should be migrated with an explicit explanation.

A practical first week

Pick one incident where the agent caused an incorrect claim, a repeated action, an unsafe call, or a costly loop. Capture the trace, redact it, block all side effects, and write three deterministic assertions: one forbidden action, one required evidence check, and one final-state rule. Put that one case in CI.

Then repeat for the next real incident. In a few weeks, you will have a test corpus that reflects the failures your system actually has—not a generic benchmark someone else designed. That is the durable value of replay: each incident makes the next release a little harder to break in the same way.

FAQ

What is AI agent trace replay?

AI agent trace replay reruns a captured agent scenario using recorded or simulated inputs. It is used to reproduce a failure, compare behavior after a change, and make an incident into a regression test without invoking real side effects.

Should an agent replay call real tools?

Usually no. Default to recorded fixtures and deny real writes, emails, payments, deployments, and permission changes. Use an isolated sandbox with synthetic data only when the behavior cannot be tested from a recorded response.

What should an agent trace contain?

Keep ordered events, component versions, validated tool arguments, result classes, policy decisions, external operation receipts, and the final claim. Redact sensitive values and avoid storing private reasoning or unnecessary prompt content.

How do you test a non-deterministic agent with replay?

Test deterministic safety and trajectory rules on every run, then sample multiple model-in-the-loop runs for softer quality signals. Treat a new tool path as a review event rather than silently accepting a new snapshot.

Is trace replay the same as agent observability?

No. Observability helps you inspect what happened. Trace replay turns selected evidence from what happened into a controlled test you can run before the next release. Strong systems use both.

How many replay fixtures should a team start with?

Start with one high-impact incident and add cases after real failures. A small, maintained incident corpus is more valuable than a large collection of stale traces no one can explain.

AI Agent Dependency Lockfiles: Stop Silent Tool and Package Drift

Jack M — Sat, 24 Oct 2026 09:50:09 +0000

An agent can behave differently on Tuesday even when nobody changed its prompt or application code. A package resolves to a newer release, an MCP server updates a tool description, a model alias moves, or a tool image is rebuilt. The workflow still starts. The tests may still be green. But the agent is no longer the runtime you reviewed.

That is why a normal package-lock.json is necessary but incomplete for an AI feature. It pins part of the application, while the agent also depends on models, prompts, tool servers, policy bundles, retrieval configurations, and external packages. Treat those as one dependency graph. Then record the resolved graph in an AI agent dependency lockfile.

This is not a claim that a lockfile makes an agent safe. It gives you a specific, reviewable answer to a much more useful question: what exactly did this agent run with? That answer makes bad changes easier to spot, reproduce, and roll back.

What belongs in an agent dependency lockfile?

Start with dependencies that can change behavior or permissions without changing the business workflow. A useful lockfile records both the requested selector and the resolved identity.

Dependency	Weak reference	Useful locked identity
Model	`fast-model`	provider, model revision or immutable deployment ID
Prompt	`support-v3`	repository commit and content digest
MCP tool	`billing-tools:latest`	image/package digest, server version, tool-schema digest
Library	`agent-sdk:^4`	exact package version and integrity hash
Policy	`default-policy`	policy bundle digest and decision version
Retrieval config	`help-center`	index revision, embedding model, and chunking config digest

The point is not to freeze every choice forever. The point is to separate intent from resolution. latest can remain an input to an update job, but it should not be the only record of what production executed.

The trap: configuration looks less risky than code

Teams often review source changes carefully and accept an environment variable such as MODEL=fast-model or TOOLS_URL=https://tools.internal as harmless wiring. In an agent system, that wiring can change output quality, tool availability, permissions, latency, or data handling.

The same issue appears with agent-installed dependencies. If a coding agent adds a package because it recognizes a familiar name, the pull request can hide a new transitive tree behind one import. If an agent fetches a tool server by a mutable tag, an image rebuild can change the runtime after review. Current public guidance on agent supply chains explicitly includes models, agents, libraries, developer tools, and MCP-server packages in scope. The boundary is wider than requirements.txt.

A small lockfile format that is easy to review

Use a format your tools can parse and your reviewers can read. JSON is convenient for CI, though YAML or TOML works too. This example deliberately avoids storing credentials and raw customer data.

{
  "lockVersion": 1,
  "agent": {
    "name": "refund-assistant",
    "workflowCommit": "8e9b1f2",
    "policy": {
      "selector": "refund-policy",
      "resolvedDigest": "sha256:6b4d..."
    }
  },
  "model": {
    "selector": "support-fast",
    "provider": "example-provider",
    "resolvedId": "support-fast-2026-10-18"
  },
  "prompts": [{
    "path": "prompts/refund.md",
    "gitCommit": "8e9b1f2",
    "sha256": "f18c..."
  }],
  "tools": [{
    "name": "get_order",
    "serverImage": "registry.example/tools@sha256:a91d...",
    "serverVersion": "2.4.1",
    "schemaSha256": "ce33...",
    "scopes": ["orders.read"]
  }],
  "packages": {
    "agent-sdk": { "version": "4.2.3", "integrity": "sha512-..." }
  },
  "retrieval": {
    "indexRevision": "kb-2026-10-23-04",
    "embeddingModel": "embed-small-r17",
    "chunkerDigest": "sha256:9d70..."
  }
}

Two details matter here. First, the selector preserves the human-facing intent, while the immutable ID or digest preserves what actually resolved. Second, the lockfile captures only references and hashes. A lockfile should never become a second secret store.

Generate the file; do not hand-edit it in production

Create one command that resolves inputs, validates them, and writes a canonical lockfile. A minimal TypeScript-style sketch looks like this:

type ResolvedTool = {
  name: string;
  serverImage: string;
  schemaSha256: string;
  scopes: string[];
};

async function lockTool(name: string): Promise<ResolvedTool> {
  const server = await registry.resolveImmutableImage(name);
  const schema = await mcp.inspect(server.digest);

  if (!schema.tools.every(tool => allowedTools.has(tool.name))) {
    throw new Error(`Unexpected tool exposed by ${name}`);
  }

  return {
    name,
    serverImage: server.digest,
    schemaSha256: sha256(stableJson(schema)),
    scopes: minimalScopes(schema)
  };
}

Canonical serialization is important. If equivalent JSON can produce different bytes, a hash diff becomes noise. Sort keys, normalize line endings, and write the generated file deterministically. Commit it next to the workflow definition just as you commit an ordinary package lockfile.

Make dependency diffs a release gate

A lockfile only helps if a changed resolution gets attention. In CI, generate a candidate lockfile, compare it with the committed version, and classify the diff.

Change	Default handling	Why
Patch-level library update	test and review	behavior may still change
New MCP tool or expanded scope	block pending approval	expands agent capability
Tool-schema digest changed	contract tests + approval	arguments or semantics may drift
Model revision changed	evaluation suite + staged rollout	behavior can shift without API errors
Policy digest changed	policy-owner approval	decision boundary changed
Retrieval index revision	freshness and regression checks	evidence may have changed

Avoid a single “lockfile changed” checkbox. A new read-only tool and a tool that can issue refunds are not equivalent. The classifier should turn a mechanical diff into an appropriate review path.

Here is a shell-shaped CI outline:

agentctl lock --config agent.yaml --out agent.lock.next.json
agentctl lock verify --file agent.lock.next.json
agentctl lock diff agent.lock.json agent.lock.next.json --format json > diff.json
agentctl lock policy-check diff.json
agentctl test contracts --lock agent.lock.next.json
agentctl eval run --suite refund-safety --lock agent.lock.next.json

The policy-check command should fail closed for prohibited changes, such as a mutable image tag, an unapproved registry, a newly write-capable tool, or a missing integrity digest. It should produce a readable report for permitted changes, not merely a pass/fail bit.

Keep resolution separate from activation

Do not let a production process silently resolve new dependencies at startup. Instead, use a two-step path:

Resolve: an update job produces a candidate lockfile from declared selectors.
Validate: CI runs package scans, tool-contract tests, workflow tests, and representative evaluations.
Approve: reviewers accept the classified dependency diff.
Activate: deployment references the immutable lockfile digest.
Observe: logs attach the lockfile ID to each run; a canary receives a limited traffic slice.
Roll back: redeploy the prior known-good lockfile if failure signals cross a threshold.

This is deliberately less magical than allowing a runtime to “get the newest compatible tool.” Predictable change is the goal. If fast security patches are needed, the same path can be accelerated with an explicit emergency approval and a short retrospective requirement.

Record the lockfile ID in operational evidence

Attach agent_lock_digest to traces, decision records, and evaluation results. When a support team asks why an agent started choosing a different tool, you can group failures by the exact dependency set instead of guessing from deploy timestamps.

That also makes incident work smaller. You can reproduce an affected run in a sandbox using the workflow commit plus lockfile, then compare it with the previous digest. A release manifest tells you the bundle identity; the dependency lock tells you which mutable ecosystem references became concrete artifacts.

Common failure modes

Locking a name, not content. tools: 2.4.1 is better than latest, but a package version alone may not identify the exact artifact. Use an integrity value, immutable image digest, or registry-provided content hash where possible.

Treating tool schemas as documentation. A schema can alter required fields, defaults, descriptions, or side effects while the server name stays the same. Snapshot the inspected schema and run contract tests against it.

Locking only dependencies. A model alias, prompt file, retrieval index, and policy bundle can each change an agent’s behavior. Include the components that matter to your threat model and debugging path.

Making updates impossible. A lockfile that nobody refreshes becomes stale theater. Run a scheduled candidate-resolution job, send bounded diffs to owners, and maintain a tested fast lane for security patches.

Logging secrets by accident. Store a tool’s allowed scope and identity, not an access token, endpoint query string, or customer payload. Scan generated lockfiles before commit.

A practical adoption plan

Begin with one agent that has meaningful external effects. Inventory its model, prompt, tool servers, packages, policy, and retrieval configuration. Add immutable IDs for the two or three components most likely to drift. Then make one CI rule visible: any newly write-capable tool requires a human approval.

After that, add schema snapshots and trace correlation. Only when those pieces work should you expand the lock to every runtime dependency. The artifact should reduce ambiguity for engineers, not create a giant metadata file nobody can review.

Use the lockfile with, not instead of, an inventory

A lockfile answers “what did this release resolve?” An inventory answers “what agent components exist across the organization?” Keep both. The inventory is useful for ownership, patch notifications, and discovering an unmaintained tool server; the lockfile is useful for exact runtime reproduction. Mixing the two creates a document that is too broad for a deploy and too stale for security operations.

Think of the lockfile as a release-level snapshot and the inventory as a portfolio-level map. A tool may appear in the inventory with an owner, risk classification, and approved registry. Each agent lockfile then states the immutable tool artifact, schema digest, and scopes it actually used. This distinction gives a security reviewer a short answer during a release review and gives an operations team a larger answer when a vulnerability advisory arrives.

When a dependency advisory appears, first query the inventory to identify candidate owners and workloads. Next, query deployment records for lockfile digests that reference the affected artifact. That keeps response work evidence-based: do not restart every agent merely because a similarly named package exists somewhere in a repository.

What a good review looks like

Reviewers should be able to answer four questions from the pull request: What changed? Which capability or behavior could change? What tests exercised the new resolution? How do we return to the previous known-good digest? Include those answers in the generated diff summary.

For example, a tool image update with an unchanged schema may need integration tests and a vulnerability scan. A schema diff adding issue_refund requires a capability-owner approval even if the image has no known CVEs. A new model revision needs representative task evaluation, because its interface can be stable while its decisions move. The lockfile makes these different kinds of changes visible in one place without pretending they carry the same risk.

FAQ

Is an AI agent dependency lockfile different from a package lockfile?

Yes. A package lockfile resolves language dependencies. An agent lockfile extends the same idea to behavior-relevant dependencies such as models, prompts, MCP tools, policies, and retrieval revisions.

Should every model request use a pinned model version?

For production workflows where repeatability matters, record the resolved deployment or revision identity when your provider exposes one. If only an alias exists, record the alias, provider metadata available at resolution time, and the evaluation evidence used for the rollout.

How do I version an MCP server safely?

Pin its immutable image or package integrity value, capture the inspected tool-schema digest, test its tool contracts, and require approval when the exposed tools or scopes expand.

Can a lockfile prevent a malicious dependency?

No. It helps you know and reproduce what was selected. Combine it with trusted registries, integrity verification, vulnerability scanning, least-privilege credentials, code review, and runtime controls.

How often should we refresh an agent lockfile?

Generate a candidate on a predictable cadence and when declared dependencies change. Apply updates only after the appropriate checks; urgency can change the approval speed, not the evidence you retain.

What should happen when a lockfile changes during an incident?

Compare the new and previous resolved identities, correlate failures with the lockfile digest, deploy the last known-good lockfile if justified, then preserve the diff and test results for follow-up.

Do small teams need this much process?

Start small: pin package and tool identities, commit one generated file, and review capability changes. The same habit scales better than reconstructing a changed agent runtime from scattered environment variables later.

AI Agent Reconciliation: What to Do When a Tool Times Out After It May Have Succeeded

Jack M — Fri, 23 Oct 2026 03:35:27 +0000

A timeout after an agent sends a refund, deploy, booking, or customer message is not a failure. It is an unknown outcome: the remote system may have completed the work after your connection stopped waiting. If the agent retries because it sees the word “timeout,” one intended action can become two real-world actions.

The fix is not a better prompt that says “be careful.” Put a reconciliation protocol in the tool layer: assign one durable operation key, record the intended effect before dispatch, query for evidence after an ambiguous result, and only retry when the evidence says the effect did not happen. This guide shows the state model, API contracts, and tests that make that rule practical.

Why a timeout is not a failed action

There are two timelines in every state-changing tool call:

The caller sends a request and waits for an acknowledgement.
The downstream service receives the request and applies an effect.

A timeout only tells you that timeline one ended without a useful answer. It says nothing conclusive about timeline two. A worker can crash after sending the request. A load balancer can drop the response. A payment service can commit just after the client deadline. In each case, immediately sending the same logical request again can duplicate the effect.

This matters more for agents than ordinary forms. Agents are designed to recover and continue. They can interpret an error as a reason to retry, call a fallback tool, or resume a durable workflow later. A natural-language instruction cannot be the final authority over a write to an external system.

Treat these outcomes as separate states:

Tool result	Meaning	Next move
`confirmed`	The target returned a receipt or authoritative result.	Continue.
`rejected`	The target proved it did not accept the operation.	Correct input or stop.
`unknown`	A timeout, connection reset, process crash, or malformed acknowledgement occurred.	Reconcile; do not retry.
`not_found`	Reconciliation can prove the effect did not land.	Retry with the same operation key if safe.
`conflict`	Evidence is incomplete or contradictory.	Escalate; do not guess.

The important design choice is exposing unknown as a first-class outcome. Do not translate it into false, throw a generic exception, or let the model decide what it means.

The narrow problem this solves

Idempotency is necessary, but it is not the whole recovery story. It prevents many duplicate writes when a downstream API accepts an idempotency key. Reconciliation answers a different question: what actually happened to this intended business operation?

That distinction matters when an integration does not support client keys, when a write spans several systems, or when a successful response was lost. It also prevents the opposite bug: an agent tells a customer an action failed even though it succeeded.

Use this pattern for consequential writes such as:

creating a charge, refund, booking, order, ticket, or deployment;
sending an external email, Slack message, or webhook;
changing a permission, subscription, or account setting;
deleting or publishing a customer-visible record.

Pure reads can normally retry. A deterministic update such as “set plan to Pro” is safer than “append a note,” but it still needs an operation record if the user needs a truthful outcome.

Make the application own effect state

The model should propose an action. Your application should own its identity, authorization, execution, and recovery. A minimal operation ledger looks like this:

create table agent_operations (
  operation_id uuid primary key,
  tenant_id uuid not null,
  run_id uuid not null,
  tool_name text not null,
  request_hash text not null,
  state text not null check (state in (
    'prepared', 'dispatched', 'confirmed', 'unknown', 'reconciling',
    'not_found', 'escalated', 'cancelled'
  )),
  provider_reference text,
  receipt jsonb,
  reconcile_after timestamptz,
  created_at timestamptz not null default now(),
  updated_at timestamptz not null default now(),
  unique (tenant_id, tool_name, operation_id)
);

operation_id identifies one intended business effect, not one HTTP attempt. Store a hash of the normalized request too. If a resumed agent presents the same operation ID with different arguments, reject it. That is a serious programming or orchestration error, not a request to “try again.”

Create the row before making the external call. Otherwise, a crash in the gap between dispatch and persistence recreates the very ambiguity you are trying to control.

Design tool contracts for honest recovery

An agent-facing write tool should not return a raw network exception. Return a small structured result that forces the workflow into a safe branch.

type EffectResult =
  | { state: "confirmed"; operationId: string; receipt: Record<string, unknown> }
  | { state: "rejected"; operationId: string; reason: string }
  | { state: "unknown"; operationId: string; retryAllowed: false; nextCheckAt: string }
  | { state: "escalated"; operationId: string; reason: string };

The wrapper receives a stable operationId, writes it to the ledger, and passes it downstream as an idempotency header or external reference whenever possible.

async function sendRefund(input: RefundInput, operationId: string): Promise<EffectResult> {
  await operations.prepare(operationId, input);

  try {
    const response = await billing.refund({
      ...input,
      idempotencyKey: operationId,
      metadata: { agent_operation_id: operationId },
    });
    await operations.confirm(operationId, response.id, response);
    return { state: "confirmed", operationId, receipt: { refundId: response.id } };
  } catch (error) {
    if (isDefiniteRejection(error)) {
      await operations.reject(operationId, safeReason(error));
      return { state: "rejected", operationId, reason: safeReason(error) };
    }
    await operations.markUnknown(operationId, error);
    return {
      state: "unknown", operationId, retryAllowed: false,
      nextCheckAt: new Date(Date.now() + 30_000).toISOString(),
    };
  }
}

Notice what is missing: there is no catch { retry() }. The tool owns transport uncertainty so the agent never treats an unconfirmed write as a fresh task.

Reconcile before any retry

Reconciliation is a targeted read of an authoritative system of record. The best API pair is a write that accepts your client-generated key and a read such as GET /refunds?operation_id=.... That read should return the effect’s lifecycle state, not merely recent activity that happens to look similar.

Use this decision flow:

prepare operation → dispatch write → acknowledgement?
                                 ├─ yes → confirm receipt → continue
                                 └─ no  → mark unknown → query by operation ID
                                                     ├─ found → confirm
                                                     ├─ absent after safe window → retry same key
                                                     └─ partial/ambiguous → escalate

The “safe window” is integration-specific. A provider with eventual consistency may need several read attempts with exponential backoff before absence means anything. Record the policy per tool: maximum reconciliation delay, available lookup key, whether a retry is safe, and who owns escalation.

For systems that cannot query by a client key, look for a compound business identity: a draft ID, customer ID plus immutable request nonce, or a provider correlation field. If you cannot reliably identify the effect, the correct automated outcome is escalated, not a duplicate write disguised as resilience.

Do not confuse a similar record with proof

A tempting shortcut is to search for “a recent refund for this customer” or “a message with roughly this text.” That can turn reconciliation into a new source of bad decisions. A customer may have made two valid requests, an operator may have sent the message manually, or a background job may have changed the same record.

Define the minimum evidence needed to confirm every integration. Prefer a provider receipt that carries the operation key. If that is unavailable, require a combination of immutable business ID, normalized request hash, target identity, and a time bound that is documented by the provider. Mark the result as conflict when evidence is merely plausible rather than conclusive.

This is also where tenant boundaries matter. An operation lookup must be scoped by the requesting tenant before its evidence reaches an agent or reviewer. A global search for a matching customer email can accidentally confirm another tenant’s action. Keep the target reference, hashes, and receipts in application-owned storage, and authorize reconciliation reads exactly as you authorize the original write.

Give each operation a small state machine

Do not let every caller update the ledger with arbitrary strings. Define allowed transitions such as prepared → dispatched → confirmed, dispatched → unknown → reconciling → confirmed, and unknown → reconciling → escalated. Disallow confirmed → dispatched entirely. The guard belongs in the database transaction or service method, not in the prompt or UI.

This protects against races between a late provider webhook, a queue retry, and a human reviewer. When a webhook supplies a valid receipt after the reconciliation job has started, it can safely claim confirmed; the job then sees the terminal state and exits. When a reviewer declines an uncertain deletion, that decision becomes a terminal cancelled record rather than an instruction an agent can reinterpret later.

Include an optimistic version or row lock on transitions, and append an immutable event for each transition. The current state makes the workflow fast to query; the event history explains why it reached that state. This is especially valuable during incident review, where you need to distinguish “the provider acted late” from “two workers independently dispatched the same effect.”

Use a reconciliation queue, not a blocked agent turn

Avoid holding a model session open while waiting for a slow target. When a wrapper returns unknown, enqueue a durable reconciliation job keyed by operation_id. That job should lock the operation, perform a bounded read-back sequence, and write one terminal state.

async function reconcile(operationId: string) {
  const op = await operations.lockForReconciliation(operationId);
  if (["confirmed", "rejected", "escalated"].includes(op.state)) return;

  const evidence = await provider.findByOperationId(op.operation_id);
  if (evidence.kind === "found") return operations.confirm(op.operation_id, evidence.id, evidence);
  if (evidence.kind === "not_found" && canRetry(op)) {
    await operations.markNotFound(op.operation_id);
    return queue.enqueue("replay_operation", { operationId: op.operation_id });
  }
  return operations.escalate(op.operation_id, evidence.reason);
}

The replay uses the same operation ID and the same normalized request. Do not restart the whole agent run; that can repeat unrelated successful steps. If a workflow includes five writes, reconcile each effect independently and allow downstream steps only after their prerequisites are confirmed.

Add evidence, boundaries, and human decisions

An operations table without useful evidence becomes a second mystery. Store redacted dispatch time, provider request ID, response status, error class, attempt count, reconciliation reads, and final receipt. Never store secrets or unredacted customer content merely to debug retries.

Classify each tool in a policy registry:

Action class	Example	On unknown outcome
Safe replay	Set a named record to a known value	Reconcile, then replay same key.
Dedupe-supported	Refund API with idempotency key	Reconcile; same key may be replayed.
Compensatable	Create ticket then update local state	Reconcile; compensate only with evidence.
Irreversible	Send a legal notice or delete a record	Reconcile, then human review.

Require an approval or incident handoff for conflicts involving money, irreversible deletion, permissions, or customer communications. The reviewer needs the intended request, operation ID, timeline, evidence, and choices—not hidden chain-of-thought or a vague “agent failed” message.

Test the failures that happy paths hide

Your integration tests should simulate a target that commits the effect and drops the response. That is the bug that ordinary mock tests miss.

At minimum, test these invariants:

the same operation ID with changed input is rejected;
a timeout produces unknown, never automatic replay;
a reconciliation read that finds a receipt reaches confirmed without a second dispatch;
an absent result only enables retry after the configured consistency window;
two workers reconciling the same operation produce one terminal state;
a resumed workflow uses existing operations rather than creating new ones;
escalation contains redacted evidence and no model reasoning trace.

For a simple test double, let POST /refunds persist a refund and then throw a timeout. Run the wrapper twice. The expected result is one refund record, one operation ID, and a second call that returns the original receipt or waits for reconciliation.

Track unknown-outcome rate by tool, time-to-reconciliation, escalations, duplicate effects detected after the fact, and disagreements between an agent’s message and the reconciled state. Those metrics reveal fragile integrations before a rare timeout becomes a customer incident.

A rollout plan that does not freeze delivery

Start with one high-impact tool rather than wrapping every integration at once.

Inventory state-changing tools and rank them by irreversibility and customer impact.
Add operation IDs and ledger rows in observe-only mode; do not change retry behavior yet.
Capture whether each provider supports idempotency keys, lookup by key, and reliable receipts.
Switch the selected tool so ambiguous outcomes become unknown and enter reconciliation.
Add alerts for operations stuck past the tool’s expected window.
Run timeout, crash, duplicate-delivery, and delayed-consistency drills before expanding coverage.

This sequence produces evidence before policy changes. It also makes conversations with API owners concrete: ask for a client-generated key, a lookup endpoint, and a receipt—not a promise that timeouts are rare.

FAQ

Is an AI agent timeout the same as a failed tool call?

No. A timeout only means the caller did not receive a conclusive response. The target may have applied the effect, so a state-changing call should enter an unknown-outcome reconciliation path.

What is an operation key for an AI agent?

It is a durable ID for one intended business effect. Reuse it across attempts, send it to the target when possible, and use it to find the effect during reconciliation.

Are idempotency keys enough for agent retries?

They are the best protection when every target honors them, but you still need a ledger and read-back path to report the outcome honestly and handle APIs without key support.

When can an agent retry a timed-out action?

Only after a provider receipt or authoritative lookup proves the effect did not happen, or when the action is intrinsically safe to replay. Use the same operation key, not a new one.

What should happen when reconciliation cannot determine the result?

Stop automatic execution and escalate with a redacted operation timeline. Guessing can create a duplicate charge, message, deletion, or permission change.

Should the language model decide whether an action succeeded?

No. Let the model propose the action and explain the user-facing result, but derive success from deterministic receipts and reconciled system state.

How is reconciliation different from a workflow retry?

A workflow retry repeats a task. Reconciliation checks whether the original business effect happened and chooses confirm, safe replay, compensation, or escalation for that one operation.

MCP Schema Versioning: Keep AI Tool Calls Working Through API Changes

Jack M — Thu, 22 Oct 2026 03:29:41 +0000

An agent that worked yesterday can start failing after a change that looks harmless in a pull request: rename location to query, make a field required, or reshape a result object. The model has not suddenly become worse. It is still following the tool contract it saw before the deployment.

That is why MCP schema versioning needs the same care as a public API migration. Tool names, descriptions, input JSON Schema, output shape, and runtime behavior all influence an agent's next action. A server can successfully deploy while a cached client, a long-running workflow, or a prompt template continues to make yesterday's call.

This guide shows a practical way to evolve MCP tools without turning routine backend work into mysterious agent failures. The goal is not to freeze every tool forever. It is to make changes visible, compatible when possible, and reversible when they are not.

Separate protocol versions from tool-schema versions

MCP has protocol-version compatibility rules. Clients and servers use those rules to agree on the protocol revision they speak. That does not automatically version your application tools.

For example, both of these servers might support the same MCP protocol revision:

// Yesterday
{
  "name": "find_customer",
  "inputSchema": {
    "type": "object",
    "properties": { "email": { "type": "string" } },
    "required": ["email"]
  }
}

// Today
{
  "name": "find_customer",
  "inputSchema": {
    "type": "object",
    "properties": { "email": { "type": "string" }, "tenant_id": { "type": "string" } },
    "required": ["email", "tenant_id"]
  }
}

The protocol can be perfectly compatible while old agent calls now fail validation. Treat the tool schema as its own public contract. Give it an owner, a baseline, a change review, and a migration path.

Know which changes break agents

Start every tool change by asking a narrow question: can a call that was valid yesterday be rejected or mean something different today?

Change	Compatibility	Safe response
Add an optional input with a default	Usually compatible	Release normally; snapshot the change.
Add a new tool	Usually compatible	Keep existing tools stable and test tool selection.
Add a required input	Breaking	Keep the old input contract or introduce a versioned tool.
Rename/remove an input	Breaking	Accept the old name during a deprecation window.
Narrow a type or enum	Breaking	Preserve old values or map them explicitly.
Rename/remove an output field	Often silently breaking	Return both fields until clients migrate.
Change a description	Behavior-changing	Review it like code; models use descriptions to choose tools.
Keep the schema but change the business meaning	Breaking	Publish a new contract and require a migration.

The last two are easy to miss. A description is not merely documentation when it enters an agent's context. And a JSON Schema diff cannot see that status: "complete" now means “queued” instead of “executed.”

Make dynamic discovery a controlled deployment boundary

MCP makes tool discovery convenient, but convenience can hide a release. If an agent fetches a live tools/list response on every connection, an upstream definition can change between two user sessions without a code deployment in the agent service.

Choose deliberately which of these modes each integration uses:

Pinned: the agent uses an approved tool manifest for its release. This is the safest default for write-capable tools.
Monitored dynamic: the agent discovers live tools, but a fingerprint diff creates an alert and can disable a changed high-risk tool.
Fully dynamic: the agent accepts current definitions immediately. Reserve this for low-risk, read-only experimentation.

Pinning does not mean copying a tool implementation. It means recording the interface your agent was evaluated against. Refresh that manifest through the same review process as an SDK upgrade. For dynamic integrations, record the discovered fingerprint with the call trace; otherwise, a later investigation cannot distinguish a prompt regression from an upstream contract change.

This boundary also helps with tool selection. Adding ten unrelated tools may not invalidate any JSON payload, yet it can change which tool the model chooses. Keep the exposed tool set small per workflow, and run a representative selection test whenever the manifest changes.

Build a schema snapshot before you need it

Do not discover the old contract by reading a production incident. In CI, call tools/list, normalize the result, and commit a snapshot for every supported server and environment.

import { createHash } from "node:crypto";

function stableToolFingerprint(tool: unknown) {
  const canonical = JSON.stringify(tool, Object.keys(tool as object).sort());
  return createHash("sha256").update(canonical).digest("hex");
}

const current = (await client.listTools()).tools.map((tool) => ({
  name: tool.name,
  fingerprint: stableToolFingerprint({
    name: tool.name,
    description: tool.description,
    inputSchema: tool.inputSchema,
    outputSchema: tool.outputSchema
  })
}));

In real code, use a recursive canonical JSON serializer; sorting only top-level keys is not sufficient. Keep two artifacts:

A readable JSON file for review.
A fingerprint manifest for a fast deploy check.

Avoid storing credentials, tenant examples, or live tool results in the snapshot. The contract should be enough to detect a change without copying sensitive data into the repository.

Diff by risk, not just by bytes

A byte diff says that something changed. A compatibility classifier says whether it can break work.

Your classifier should flag at least these conditions:

a tool disappeared or was renamed;
a property was removed or became required;
a type, numeric range, regex, or enum narrowed;
a default value changed;
an output property disappeared or changed type;
a description changed for a high-impact tool;
an authorization, side-effect, or idempotency expectation changed.

Put the result in the pull request. A useful summary is more actionable than a large JSON patch:

BREAKING: find_customer now requires tenant_id
COMPATIBLE: list_invoices gained optional page_size (default 50)
REVIEW: refund_payment description changed; execution remains side-effecting

For tools that create records, move money, send messages, or alter infrastructure, make a breaking diff block the release until an approved migration plan is attached.

Prefer adapters over abrupt replacements

Suppose location is no longer precise enough and your API now needs latitude and longitude. Do not immediately force every agent to learn the new fields. Accept the old call, translate it, and emit a deprecation signal that your logs can count.

type WeatherArgs =
  | { location: string; unit?: "c" | "f" }
  | { lat: number; lng: number; unit?: "c" | "f" };

function normalizeWeatherArgs(args: WeatherArgs) {
  if ("location" in args) {
    return {
      kind: "legacy" as const,
      query: { place: args.location, unit: args.unit ?? "c" }
    };
  }

  return {
    kind: "current" as const,
    query: { lat: args.lat, lng: args.lng, unit: args.unit ?? "c" }
  };
}

The adapter is a temporary compatibility boundary, not permanent clutter. Give it a removal date and measure use of the legacy branch. If old calls are still common, find out where they originate: a scheduled workflow, a prompt stored in a database, a client that caches tools, or an integration you do not control.

When the semantics truly differ, use a new tool name such as get_weather_by_coordinates rather than overloading the old name. An agent can reason about two clear options better than one tool with surprising behavior.

Version outputs as carefully as inputs

Input errors usually fail loudly. Output changes often fail silently.

Imagine a tool used to return this:

{ "ok": true, "ticket_id": "T-42" }

Then a refactor changes it to:

{ "state": "accepted", "job_id": "J-96" }

An agent may interpret missing ok as success, retry unnecessarily, or tell the user a task is complete before it runs. During a transition, return an envelope that preserves the old fields while introducing the new ones:

{
  "ok": true,
  "ticket_id": "T-42",
  "state": "accepted",
  "job_id": "J-96",
  "deprecation": { "fields": ["ok", "ticket_id"], "remove_after": "release-2027.03" }
}

Use a release identifier or a policy window in that metadata, rather than a vague promise to remove it “soon.” Validate that old and new fields always agree while both exist.

Test three layers of the contract

Schema validation alone is necessary but not enough. Add three test layers to each MCP tool.

1. Snapshot tests

Compare tools/list against the approved manifest. This catches accidental schema regeneration after a Pydantic, Zod, OpenAPI, or SDK upgrade.

2. Compatibility fixtures

Run old valid payloads and current payloads through the handler. The old fixture should either succeed through the adapter or fail with an explicit, actionable migration error. Never let an old call silently map to a different action.

3. Behavioral contract tests

Assert the promise that JSON Schema cannot express: a read tool does not write, a dry run does not execute, a retry does not duplicate a charge, and a returned accepted state does not mean completed.

it("keeps legacy lookup behavior during migration", async () => {
  const result = await callTool("find_customer", { email: "a@example.test" });
  expect(result.isError).toBe(false);
  expect(result.structuredContent.customer).toMatchObject({ email: "a@example.test" });
  expect(result.structuredContent.deprecation).toBeDefined();
});

Use synthetic tenants and fake side-effect adapters for these tests. Contract tests should be safe enough to run on every pull request.

Roll out schema changes in stages

A careful rollout makes the blast radius small and observable.

Inventory consumers. Find interactive agents, scheduled jobs, API clients, prompt templates, and external integrations that call the tool.
Ship additive support. Add the new shape or new tool while the old contract remains valid.
Release the snapshot and fixtures. Make the intended diff visible in code review.
Canary by tenant or workflow. Send a limited, reversible population to the new contract.
Observe tool errors and outcome errors. Track validation failures, fallback use, retries, and task completion—not only HTTP status.
Notify and deprecate. State the replacement, an example payload, and the removal condition.
Remove only after evidence. Delete the adapter after legacy traffic stays at zero for the defined window.

Keep a server build identifier and tool-manifest fingerprint in every trace. When a failure appears, this lets an engineer answer “which contract did the agent see?” without reconstructing a deployment timeline.

A small release checklist

Before merging an MCP tool change, answer these questions:

Does any formerly valid request now fail or mean something different?
Did input, output, description, side effect, authorization, or idempotency change?
Is the schema snapshot updated and classified by compatibility risk?
Do old and new fixtures run in CI?
Is there an adapter, a new tool name, or an explicit migration error?
Can the change be canaried and rolled back without deleting state?
Is there an owner and a measurable removal condition for deprecated behavior?

If any answer is “no,” the tool is not ready for a routine deployment. That is not bureaucracy. It is how you prevent an API change from masquerading as an agent reliability problem.

FAQ

What is MCP schema versioning?

MCP schema versioning is the discipline of tracking and evolving an MCP tool's name, description, input schema, output shape, and behavior without breaking agents that use an older contract. It is separate from the MCP protocol version negotiated by clients and servers.

Is adding a required MCP tool parameter a breaking change?

Yes. Any older call that omits the new parameter can fail validation. Prefer an optional field with a safe default, a compatibility adapter, or a new tool name when the new information is essential.

Should MCP tool descriptions be versioned too?

Yes. Agents use descriptions to select tools and construct arguments, so a description change can alter behavior even if the JSON Schema is identical. Snapshot descriptions and require review for changes to sensitive or side-effecting tools.

How long should an MCP schema deprecation window last?

Base it on observed legacy traffic and the longest-running consumer in your system. Define a minimum window, publish a removal condition, and remove the adapter only after telemetry shows the old shape is unused for that period.

Can JSON Schema tests catch every breaking agent change?

No. They catch structural differences, but not changed business meaning, tool-selection behavior, authorization assumptions, or unwanted side effects. Pair schema snapshots with compatibility fixtures and behavioral contract tests.

Should I create a new MCP tool instead of changing an old one?

Create a new tool when the input or output semantics change materially. For a small additive change, keep the existing tool and use an optional field. A clearly named second tool is safer than one overloaded tool whose old calls now do something surprising.