DEV Community: Modelflare

Why AI Agent Cache Hit Rates Collapse: GPT, Claude, and Auditable Gateways

Magnus — Thu, 29 Oct 2026 02:10:06 +0000

A source-backed guide to third-party agent cache failures, GPT and Claude prompt-caching mechanics, reproducible hit-rate measurement, and Modelflare's public compensation boundary.

AI agent caching is not a checkbox. It is a protocol, routing, and accounting contract. This article explains why an opaque third-party agent can show a very low cache hit rate even when the upstream model supports prompt caching, how GPT and Claude measure reuse differently, and what an auditable gateway should guarantee.

The decision in one sentence

Do not put a production, cache-sensitive agent behind a third-party agent or gateway that cannot expose native cache usage, preserve a stable prefix, keep model and route affinity, and reconcile the result against a durable ledger. “Supports caching” is a feature claim; a measured, eligible hit rate is an operating fact.

This is not a claim that every third-party agent fails. Public documentation cannot establish a vendor-wide percentage without a fixed corpus, a fixed model, a known observation window, and request-level usage evidence. The reliable conclusion is narrower: an opaque translation layer creates several independent ways to turn a valid provider cache into an almost-always-cold path. Treat that lack of evidence as a reason not to use the path for cache-dependent production work.

Define the denominator before discussing a percentage

The provider usually reports three different buckets. Let R be tokens read from a cache, W be tokens written to a cache, and U be input tokens processed without reuse. For cache-eligible prefixes, the comparable hit rate is:

eligible_hit_rate = R / (R + W)

For all input sent to the model, the reuse share is:

input_reuse_share = R / (R + W + U)

Those numbers answer different questions. A dashboard that divides by all input can look low when the agent sends many short, ineligible requests. A dashboard that divides only eligible traffic can look high while a large dynamic suffix still dominates spend. Report both, plus the eligibility rule and the time window.

The provider fields are the evidence, not a green badge in an agent UI:

Signal	OpenAI field	Claude field	What it proves
Reused prefix	`input_tokens_details.cached_tokens`	`cache_read_input_tokens`	Tokens actually served from a matching entry
New cache entry	`input_tokens_details.cache_write_tokens`	`cache_creation_input_tokens`	Tokens charged to create or extend an entry
Unreused input	`input_tokens - input_tokens_details.cached_tokens - input_tokens_details.cache_write_tokens` (derive carefully)	`input_tokens` after the breakpoint	Input outside the cached prefix; semantics differ
Correlation	request ID plus model and route	request ID plus model and route	Which provider attempt produced the usage

Worked example, using synthetic numbers rather than Modelflare production telemetry: R=720,000, W=80,000, and U=200,000 give an eligible hit rate of 720,000 / 800,000 = 90%, but an all-input reuse share of 720,000 / 1,000,000 = 72%. Publishing only one of those values hides the denominator.

input_tokens_details.cache_write_tokens=0 is not proof of “no cache.” A request can be a cache read and therefore create no new entry. Conversely, a missing usage field in a third-party response is not a zero; it is an observability failure that must be recorded as not auditable.

Why an agent gateway loses hits even when the model supports caching

The failure is usually in the path between the application and the provider. These are the high-frequency breakpoints:

Dynamic bytes arrive too early. Timestamps, request IDs, user names, experiment flags, or a changing “current date” in the system prompt alter the prefix before the reusable instructions.
Tools are reserialized. Adding, removing, reordering, or nondeterministically serializing a tool schema changes the exact prefix. A harmless-looking JSON key order can become a cache miss.
Fallback changes the cache identity. A load balancer that moves between model aliases, regions, organizations, or provider credentials does not share one universal cache entry.
The adapter drops native controls. Removing cache_control, prompt_cache_key, retention options, or usage details turns a provider feature into an invisible best effort.
The prompt is below the provider threshold. Short agent turns can be valid requests but still be ineligible for caching.
The TTL window is crossed. A five-minute Claude entry or a model-specific OpenAI retention window can expire during a human-in-the-loop pause.
Parallel warm-up races. Concurrent first requests may all arrive before the first response makes a cache entry available.
History is rewritten. Summarization, compaction, truncation, or a different serialization of prior messages changes the prefix instead of appending to it.

None of these failures requires a dishonest vendor. They are predictable consequences of a gateway that treats an agent request as free-form text instead of preserving a provider’s cache contract. The practical warning is stronger than a marketing comparison: if the agent cannot show which breakpoint failed, you cannot price or debug cache-dependent traffic.

GPT and Claude use related ideas, not interchangeable semantics

Both providers require an exact reusable prefix, but the control surface and usage accounting are different. The table below summarizes the official guides checked on 2026-08-25; model names, minimums, and retention policies can change, so recheck the linked pages before changing a production policy.

Dimension	OpenAI prompt caching	Claude prompt caching
Reusable unit	The full rendered context prefix, including instructions, tools, history, and multimodal parts	The ordered prefix through a `cache_control` breakpoint: tools, system, then messages
Minimum length	The current guide lists 1,024 visible tokens for GPT-5.6+ and generally 2,048 for older models	Model-specific minimums currently span roughly 512–4,096 tokens; shorter prompts run without caching
Controls	Implicit caching; explicit breakpoints and a stable `prompt_cache_key` on supported models	Top-level automatic caching or explicit block breakpoints; up to four breakpoints and a 20-block lookback
Retention	GPT-5.6+ supports a 30-minute TTL; earlier models expose retention modes with provider-defined typical windows	Five-minute default TTL, refreshed on use; optional one-hour TTL at a higher write price
Price shape	On the current GPT-5.6+ guide, writes are 1.25× base input and reads are 0.1×	Five-minute writes are 1.25× and one-hour writes are 2×; reads are 0.1×
Usage evidence	`input_tokens_details.cached_tokens` and, when applicable, `input_tokens_details.cache_write_tokens`	`cache_read_input_tokens`, `cache_creation_input_tokens`, and post-breakpoint `input_tokens`
Frequent invalidator	Model/tool/settings changes, machine overflow, or a prefix change before the breakpoint	Model/system/tools/message changes, a missing previous message, or a concurrent warm-up miss

OpenAI notes that cache entries live on individual machines and that a prompt_cache_key helps group and route requests but does not pin a machine or guarantee a hit. Claude documents exact matching, workspace-level isolation, and explicit diagnostics such as tools_changed and messages_changed. A gateway that hides those distinctions cannot honestly convert provider documentation into a single “cache rate” promise.

The price multipliers explain why this matters. If one write is followed by one read, the cached input portion costs about 1.25 + 0.10 = 1.35× the uncached input unit on the cited OpenAI model, rather than two uncached units. Ten fully reused requests cost about 1.25 + 9×0.10 = 2.15× instead of 10×. These are official illustrative calculations, not a Modelflare invoice or a promise for every model.

The adapter tax is measurable

Use a failure matrix instead of a screenshot. Keep the application prompt, model, credentials, and request rate fixed; change one variable at a time and retain the raw usage object.

Controlled change	Expected provider signal	What an opaque agent may hide	Release interpretation
Append only the user turn	`R` rises after the first `W`	Rewritten history or a new route	Prefix preservation works
Add a timestamp to the system prompt	`R` falls to zero or a new `W` appears	Which bytes changed	Dynamic prefix is a cache breaker
Reorder one tool property	`tools_changed` or a new write	Serializer behavior	Tool schema must be deterministic
Split traffic across aliases or machines	Lower and more variable `R`	Selected route and cache key	Affinity policy is missing
Sleep past the documented TTL	New write after expiry	Expiry time and retention mode	Measure human-pause workload separately
Fire two identical first requests in parallel	One or both may write	Warm-up race and attempt order	Do not infer capacity from a cold burst

The test must retain the request ID, stream mode, first event time, exact model, selected route, cache fields, and UTC timestamp. Redact prompts, keys, and customer content. If a gateway returns only a normalized “input tokens” total, label the run not auditable; do not turn missing dimensions into a fabricated zero or a favorable estimate.

A reproducible audit record

The following is a small, synthetic record format. It is deliberately provider-neutral at the envelope and preserves native fields inside usage. It is not a benchmark result.

{"request_id":"demo-001","timestamp":"2026-08-25T02:00:00Z","model":"MODEL_ID","route":"route-a","stream":true,"usage":{"input_tokens":1000000,"input_tokens_details":{"cached_tokens":0,"cache_write_tokens":1000000}}}
{"request_id":"demo-002","timestamp":"2026-08-25T02:00:03Z","model":"MODEL_ID","route":"route-a","stream":true,"usage":{"input_tokens":1000000,"input_tokens_details":{"cached_tokens":720000,"cache_write_tokens":80000}}}

Run at least five phases: a serial warm-up, an append-only conversation, a dynamic-system mutation, a tool-order mutation, and a TTL-gap replay. For Claude, map the native read/write fields into the same report without replacing them; for OpenAI, retain both cached and write fields when returned. Compare R/(R+W) and R/(R+W+U) per phase, route, model, and UTC day. A single blended number is not enough to decide whether an agent is safe for production.

What Modelflare guarantees—and what it does not

Modelflare’s public status page currently describes an OpenAI Cache Hit Rate Guarantee. For eligible requests that satisfy OpenAI’s caching requirements and form a valid cache, the daily rate is calculated by UTC calendar day. If the measured rate falls below the applicable tier, the shortfall is compensated and shown in Dashboard → Token Usage Analysis. The public tiers currently shown are 65%, 75%, and 85% for the published group levels.

The boundary is intentional:

The guarantee applies to eligible, valid OpenAI cache traffic, not to a short or constantly changing prompt that never qualifies.
It does not turn a changing model, tool schema, route, or TTL into a reusable prefix.
It is calculated per UTC calendar day and credited through the documented account surface; it is not an assertion that every individual request will hit.
The native usage evidence and eligibility calculation remain visible requirements. A compensation rule is useful precisely because the denominator is explicit.

That is the difference between an auditable service promise and a third-party agent slogan: the former names the eligible population, observation window, threshold, and credit destination.

Price advantage and operating legitimacy are separate proofs

The current public pricing page also displays a 0.015 first-top-up offer ratio for the applicable campaign/group, with the page’s stated minimum single top-up and eligibility terms. This is a price incentive; it must not be conflated with the cache guarantee. Check the live pricing page for the exact model scope, campaign status, and settlement rules before making a purchase decision.

Modelflare’s public trust and legal materials identify Havenbyte LLC as the operator and describe the operation as United States based. They list Stripe as the payment processor for payment handling, billing, invoices, refunds, and fraud controls. These are useful custody and accountability signals, not a substitute for reading the terms or testing the API. The article makes no claim about a particular state registration, certification, or provider affiliation beyond those public materials.

Recommendation for production agents

Choose a third-party agent only when it can pass this checklist:

It forwards the provider’s native cache controls and returns native read/write usage.
It keeps the stable instruction and tool prefix byte-identical, then appends dynamic state.
It exposes the selected model, route, organization/region boundary, TTL mode, and request ID.
It records serial and parallel warm-up behavior instead of averaging them away.
It defines eligible traffic, denominator, UTC window, threshold, and compensation in writing.
It lets you export request-level evidence without exposing secrets or customer content.

If any of those answers is “no,” do not use the path for a workload whose economics depend on cache reuse. Use a direct provider path for the experiment, or choose an auditable gateway such as Modelflare and verify the exact route with the same corpus. For adjacent controls, see reliable AI API routing, AI API cost tracking, pricing, and trust.

Sources and verification date

The following primary sources were checked on 2026-08-25. They describe provider contracts and Modelflare’s current public policy; they do not establish a universal hit rate for an unnamed third-party agent.

FAQ: Is a low third-party hit rate always fraud?

No. It may be a valid result of short prompts, a changing prefix, route fan-out, TTL expiry, or a missing usage field. The correct finding is “not reproducible” or “not auditable” until the controlled test shows the cause.

FAQ: Can I compare Claude and GPT with one number?

Only after normalizing the denominator and preserving each provider’s native fields. Compare the same corpus and workload phase, not a blended dashboard total.

FAQ: Does compensation remove the need to design a stable prefix?

No. Compensation limits the financial downside for eligible OpenAI cache traffic; it cannot make an ineligible request reusable or repair an agent that rewrites the prefix.

Originally published at modelflare.dev.

Disclosure: I'm part of the Modelflare team, which builds the AI API gateway mentioned in this article.

How to Evaluate an AI API Gateway: A Production Checklist

Magnus — Tue, 27 Oct 2026 02:05:27 +0000

A reproducible gateway evaluation covering protocol conformance, failure drills, latency, usage and cost reconciliation, security, operations, and exit risk.

Evaluate an AI API gateway by running your real protocol contract through it, forcing the failure modes you care about, and checking the request-level evidence it produces. A feature list or successful “hello world” request cannot prove streaming correctness, tool compatibility, fallback safety, cost accuracy, security boundaries, or a usable exit path.

The most reliable process uses two kinds of criteria: non-negotiable gates that cause a candidate to fail, and scored operational qualities that distinguish candidates after every gate passes.

Define the workload contract first

Do not begin with a vendor comparison table. Begin with one representative workload and write down its invariant contract:

exact endpoint: Responses, Chat Completions, embeddings, images, or another API;
exact model identifiers and whether aliases are permitted;
streaming and non-streaming modes in use;
Structured Outputs, Function Calling, hosted tools, reasoning, or other required fields;
typical and high-percentile input and output lengths;
concurrency, request rate, region, and user-facing deadline;
required usage, cache, cost, and request-correlation fields;
allowed fallback routes and whether model substitution is forbidden;
data-retention, access, residency, and deletion requirements;
application operations that can create side effects.

One gateway can pass for a text-only internal assistant and fail for a streaming coding agent. “OpenAI-compatible” is not a sufficient workload definition because compatibility can differ by endpoint, event type, tool, schema keyword, and provider route.

If the team is still deciding whether it needs a proxy or a model-aware control plane, start with LLM Proxy vs AI Gateway. This checklist assumes the gateway category is already justified and tests whether a specific implementation can own the required responsibilities.

Apply fast disqualifiers before a long trial

The first review should remove candidates that cannot satisfy a mandatory boundary. Ask for demonstrable behavior, not a roadmap statement.

Gate	Immediate failure condition	Evidence to request
Protocol	A required request field, output item, or stream event is dropped or rewritten incorrectly	Captured redacted request/response pair and parser result
Model identity	The gateway silently changes the requested model	Attempt record with requested and actual model
Streaming	It buffers the full response, loses cancellation, or corrupts Tool Argument fragments	Timestamped event sequence and cancellation trace
Authentication	Browser or workload clients receive provider credentials	Credential-flow diagram and a live key-rotation test
Tenant isolation	One project can use or inspect another project’s keys, usage, or logs	Authorization tests using real isolated accounts
Cost evidence	A final charge cannot be connected to model, route, price basis, and usage	One reconciled request ledger
Failure safety	A partial stream is transparently replayed or a cancellation starts another attempt	Forced partial-stream and cancel traces
Export and exit	Configuration and request contract cannot be recovered without rewriting the application	Export sample and provider-native rollback drill

A candidate that fails a mandatory gate should not receive a high aggregate score to compensate. Security isolation cannot be offset by a good dashboard, and an incorrect Tool Contract cannot be offset by a lower price.

Build a small protocol conformance corpus

Use deterministic, non-sensitive inputs and keep the expected wire behavior under version control. The corpus should call the real gateway endpoint; it should not mock the provider or reimplement the gateway’s conversion logic as the test oracle.

Case	Request	Required observation
Basic non-streaming text	Pinned model and fixed prompt	Correct status, model identity, text location, usage, request ID
Streaming text	Same prompt with streaming enabled	Ordered events, first effective output, final event, cancellation behavior
Structured output	Strict schema with required fields and `additionalProperties: false`	Valid output or an explicit unsupported error; no silent downgrade
Function Calling	One read-only function and one returned result	Function name, JSON arguments, Call ID correlation, final answer
No-tool path	Same tools declared but no tool required	Normal text without a fabricated Tool Call
Invalid field	Deliberately unsupported or malformed request	Stable client error; no provider fallback that hides the defect
Long input boundary	Input just below and above the approved limit	Documented acceptance or explicit rejection, not truncation
Usage detail	Request that exercises cached or reasoning usage when supported	Fields survive the route and reconcile with the billing record
Cancellation	Client cancels after connection and after first output	Upstream work stops and no new fallback attempt starts
Partial stream	Connection fails after effective output	One explicit partial failure; no invisible second answer

Run every required case against every route that may serve the workload. A primary route passing does not qualify its fallback. The Structured Outputs guide and Function Calling comparison provide field-level test cases for those two capabilities.

Record the gateway version, route configuration version, model ID, provider, region, timestamp, and sanitized result hash. Volatile model and provider behavior should be rechecked before rollout and after a material route change.

Test routing and failure behavior, not only success

Reliability claims matter only when the failure policy is visible. Force each condition before production:

primary route unavailable before headers;
provider rate limit with and without Retry-After;
upstream authentication or account failure;
slow headers and slow first effective output;
malformed provider response;
client cancellation while the upstream is pending;
connection loss after visible output begins;
all eligible routes exhausted.

For each case, capture the attempt order, selected route, status, timing, whether output had started, terminal reason, and any usage or cost. Confirm that the gateway preserves the requested model and protocol unless a separate model-substitution policy is explicitly enabled.

Measure attempt amplification across the SDK, application, gateway, and provider. One layer should own immediate same-contract fallback, while the application owns whether the entire user action can be repeated. The AI API Fallback Strategy provides a phase-aware failure matrix and retry-budget model.

Latency needs the same precision. Compare time to upstream headers, first SSE event, first effective output, first visible text, completion, and visible output speed at realistic concurrency. Do not accept one unexplained “latency” average. See AI API Latency Metrics for definitions and comparison controls.

Reconcile usage and cost from one request

Choose several completed requests and follow each one through the full accounting chain:

application request ID
  → gateway attempt sequence
  → selected model and route
  → provider or normalized usage
  → applicable price basis
  → final recorded charge

The evaluation should answer:

Are input, output, cached, reasoning, and tool-related units represented when applicable?
Which values come from the provider and which are estimated?
When is the model price selected, and is it frozen for the request?
Does a group, service tier, discount, or surcharge change the user charge?
Which failed attempts can create provider cost, and how are they recorded?
Can a final successful fallback hide earlier billable attempts?
Are currency conversion and rounding rules explicit?
Can finance reproduce a daily total from immutable request records?

Test a normal completion, a same-contract fallback, a cancelled request, and an upstream error. A dashboard total is not enough; the gateway must produce a defensible per-request record. AI API Cost Tracking separates provider usage, platform pricing, customer charge, and supplier cost.

Do not compare vendor savings unless the same model, workload, cache behavior, output length, failure rate, and provider price basis are held constant. A lower apparent cost can be caused by missing usage or silent model substitution.

Verify the security and data boundary

Draw the actual data flow from the client to the gateway and each provider. For every hop, identify who can read credentials, request content, response content, metadata, and administrative configuration.

At minimum, verify:

provider credentials are stored server-side, encrypted at rest, and never returned to ordinary clients;
application keys can be scoped by project or workload and revoked independently;
authorization is enforced server-side on every management and log endpoint;
logs avoid full API keys and have explicit prompt/response retention controls;
support access is attributable and bounded;
configuration changes have actor, time, before/after state, and rollback evidence;
exported traces redact secrets and personal or proprietary content;
deletion and retention behavior can be demonstrated, not only described;
region and subprocessor claims match the route actually used;
abuse limits run before expensive upstream work when possible.

Ask what happens during key rotation, an operator departure, a compromised application key, and a provider-key leak. Run a rotation and revocation exercise with real isolated test credentials. Do not copy a production secret into an evaluation environment.

The gateway cannot make unsafe application tools safe. Tool authorization, transactionality, approval, and idempotency remain application responsibilities even when model access is centralized. Use AI API Key Security and Cost Controls to separate credentials and workload limits.

Evaluate the operational control plane

The data plane may work while the control plane creates operational risk. Review how operators make and recover from changes:

Area	Questions to answer
Versioning	Is each route, price, policy, and key change versioned or attributable?
Validation	Can an invalid route or incompatible model be rejected before activation?
Rollout	Can a change target a small workload or percentage first?
Rollback	Can operators restore the last known-good configuration quickly?
Availability	What happens to existing and new requests if the control plane is unavailable?
Health	Is channel health based on current evidence, and can automatic disablement be inspected?
Incidents	Can one request be reconstructed without searching several unrelated systems?
Limits	Are rate and quota decisions atomic enough to remain correct under concurrency?
Change ownership	Are emergency edits separated from normal product configuration?

Run one configuration rollback and one unhealthy-route removal. Measure the operator steps and verify the resulting data-plane behavior. A screenshot of a rollback button is not the same as a completed rollback drill.

Test the exit path before signing

Gateway adoption can create dependencies on model aliases, custom headers, proprietary route names, log APIs, normalized error shapes, or hosted Prompt and Tool configuration. List each dependency and decide whether it is an intentional benefit or accidental lock-in.

A practical exit drill should:

export route, key-policy, price, and audit configuration in a documented format;
switch one workload to a provider-native test endpoint;
replace gateway-only headers or aliases with explicit application configuration;
preserve request correlation and usage reconciliation during the switch;
document functionality that cannot move without redesign;
estimate exit engineering from the observed work, not a sales claim.

An exit path does not require the gateway to be interchangeable with every provider. It requires the team to know what it owns, what the gateway owns, and how to recover the underlying protocol contract.

Score only after every mandatory gate passes

Use a worksheet with pass/fail for hard boundaries and a small evidence score for operational quality:

Score	Meaning
0	Unsupported or contradicted by the test
1	Claimed or manually demonstrated once, with weak evidence
2	Repeatably demonstrated with request-level evidence
3	Repeatably demonstrated, monitored, and recoverable through a tested control

Suggested scored areas are protocol coverage, route reliability, attempt evidence, latency diagnostics, usage accuracy, cost reconciliation, key isolation, auditability, configuration rollback, supportability, and exit effort. Weight them according to the workload, but keep the raw evidence beside every score.

Avoid a false-precision total such as 87.4/100 when several rows are subjective. Record the decision as:

mandatory gates and outcomes;
score by area with evidence links;
accepted gaps and owner;
remediation deadline;
cost and contract assumptions;
selected candidate and rejected alternatives;
review date after the first production month.

Compare build versus buy by ownership

The right question is not whether an internal gateway has a license fee. It is which responsibilities the team can own continuously.

Responsibility	If built internally	If purchased or managed
Protocol updates	Track provider schemas and regressions	Verify vendor updates and route compatibility
Routing and retry	Design state machine and failure evidence	Configure policy and audit actual attempts
Usage and billing	Normalize usage and maintain pricing logic	Reconcile vendor records with internal finance truth
Security	Store secrets, enforce tenancy, audit access	Validate vendor boundary and configure least privilege
Reliability	Operate data plane, control plane, and on-call	Monitor vendor plus your integration and retain an exit path
Product support	Diagnose every application/provider interaction	Triage gateway versus provider versus application failures

Do not insert generic salary or “engineering time saved” figures. Estimate from your own on-call load, protocol-change history, incident frequency, finance requirements, and compliance work. A managed product still requires an accountable internal owner.

Apply the checklist to Modelflare accurately

For Modelflare, the current evaluation boundary should be explicit. It provides workload API Keys, requested-model routing through eligible groups and channels, ordered group fallback for regular keys, strategy-based group selection for Smart API Keys, pre-upstream group RPM admission, and request-level usage, cost, status, and timing records.

GPT, Codex, and OpenAI traffic is the fully adapted compatibility target. Other OpenAI-compatible model families should be evaluated as raw Chat Completions pass-through unless a capability has been separately verified. A shared base URL is not proof that Responses, hosted tools, Structured Outputs, or Function Calling behave identically on every route.

Modelflare’s fallback should search for an eligible path for the requested model rather than silently selecting another model. Channel failover stops after downstream output starts. Those claims can be evaluated with the protocol corpus and failure drills above instead of accepted as marketing statements.

Use Models & Pricing to identify the current model and group surface and Modelflare Docs to configure an isolated test key. Keep the evaluation requests non-sensitive, pin the exact model, and retain the Request IDs needed to inspect each attempt.

The final decision should be reproducible: another engineer should be able to run the same corpus, inspect the same categories of evidence, and understand why the candidate passed. That process is slower than reading a comparison page, but it is much faster than discovering an incompatible Tool Contract, untraceable bill, or unsafe fallback after the gateway owns production traffic.

Originally published at modelflare.dev.

Disclosure: I'm part of the Modelflare team, which builds the AI API gateway mentioned in this article.

Claude Opus 5.5: Specs, Price, and Place in the Field

Magnus — Thu, 22 Oct 2026 02:09:59 +0000

A source-backed account of Claude Opus 5.5: published specs, official token prices, how to read the benchmark table against Fable 5.1, Opus 5, and GPT-6 Astra, and the Modelflare groups and user prices returned on 30 September 2026.

Claude Opus 5.5 was released on 22 September 2026. The public contract is on the model page and in the what's-new note: API id claude-opus-5-5, a 1M-token context window, 128K max output, adaptive thinking that stays on, default effort medium, and a list price of $4 per million input tokens and $20 per million output tokens. Anthropic's pages do not publish a parameter count or a training-compute figure. This article stays with the published spec, the published price card, and the published benchmark table.

Checked conclusion: on Anthropic's list price, Opus 5.5 is the current Opus tier. Per token it is cheaper than Opus 5, and much cheaper than Fable 5.1. On the coding and knowledge-work rows Anthropic printed, its scores lead Fable 5.1, Opus 5, and, on most of those rows, GPT-6 Astra. Two printed rows go the other way: Astra leads Terminal-Bench-Science 0.1 and Zapier's AutomationBench. Modelflare's pricing API, read on 30 September 2026, includes claude-opus-5-5 at the official $4 / $20 unit prices. That row enables claude-award 0.25, claude-stable 0.315, and claude-premium 0.5. A key for this tier sends claude-opus-5-5. Live rates are on the pricing page.

Official behavior below is cited from the Opus 5.5 model page, what's new, and the launch page. The earlier Fable 5.1 review is the place for that model's migration notes.

Published specifications

Item	Claude Opus 5.5
Model id, Claude API	`claude-opus-5-5`
Amazon Bedrock id	`anthropic.claude-opus-5-5`
Google Cloud and Microsoft Foundry id	`claude-opus-5-5`
Release date	22 September 2026
Context window	1M tokens, both the default and the maximum
Max output	128K tokens
Thinking	Adaptive, always on
Default effort	`medium`
Latency class on the model page	Moderate
Reliable knowledge cutoff	June 2026
Parameter count	Not on the public model page

"Reliable knowledge cutoff" is Anthropic's name for the date through which the model's stored knowledge is most extensive. It is separate from the context window. A file you attach is still in the request.

Fast mode is a speed switch on the same id, documented as a research preview on the Claude API only. The request sets speed: "fast" with the fast-mode-2026-02-01 beta header. Anthropic prices that mode at $8 / $40 per million tokens and describes output as up to 2.5× faster. The what's-new page says this fast mode is absent on Amazon Bedrock, Claude Platform on AWS, Google Cloud, and Microsoft Foundry.

The same page lists the effort ladder as the control for thinking depth: low, medium, high, xhigh, and max. A manual thinking budget is rejected. The minimum cacheable prompt is 512 tokens.

Prices

Official list prices

Figures are US dollars per million tokens. Opus 5.5's cache, batch, and fast-mode rows come from the what's-new pricing section. Opus 5's cache-write cell is the single "cache writes" number on the launch comparison table, which does not split the 5-minute and 1-hour tiers. Fable 5.1's cache read is the official figure already used in this blog's Fable 5.1 review. Sonnet 5 and Haiku 4.5 input and output prices are the comparison row on the Opus 5.5 model page. An em dash means this article did not take that cell from a primary price row.

Price per 1M tokens	Opus 5.5	Opus 5	Fable 5.1	Sonnet 5	Haiku 4.5
Input	$4	$5	$10	$2	$1
Output	$20	$25	$50	$10	$5
Cache read	$0.20	$0.50	$0.25	—	—
Cache write	$5 for 5 minutes; $8 for 1 hour	$6.25	—	—	—
Batch input / output	$2 / $10	—	—	—	—
Fast mode input / output	$8 / $40	—	—	—	—

The list-price move from Opus 5 is 20% on fresh input and output, from $5 / $25 to $4 / $20. Cache reads, which dominate long agent loops, move from $0.50 to $0.20. That is 60% off the Opus 5 cache-read price, and it also sits under Fable 5.1's $0.25 cache read. Anthropic's separate workload claim is that a typical run at default settings costs about 40% less than Opus 5, because the model also uses fewer tokens. That 40% describes their harness. A real bill still follows effort, cache hit rate, and tool rounds.

Batch processing is half the new list price. Fast mode is double the new list price, on the Claude API only.

Modelflare catalog on 30 September 2026

GET https://modelflare.dev/api/pricing on 30 September 2026 includes claude-opus-5-5. The same response also includes claude-opus-5, claude-fable-5-1, claude-sonnet-5, claude-sonnet-5-5, claude-haiku-4-5, gpt-6-astra, and gpt-5.6-sol. It does not include a Haiku 5.5 id.

Unit prices below are that response's token prices before a group ratio. For the Claude rows they match the official list. Astra and Sol are tiered: the second pair applies when input context is above 272,000 tokens, and the long-context output price is the tier's own output price. The example column multiplies Claude rows by claude-stable 0.315 and OpenAI rows by openai-stable 0.08. Claude ratios on this read are claude-premium 0.5, claude-stable 0.315, and claude-award 0.25. OpenAI ratios are openai-premium 0.13, openai-stable 0.08, and openai-award 0.03. Opus 5.5 enables only the three Claude groups above. claude-tmp-cheap is a public Claude group, and this model row does not include it.

Model	Catalog id	Unit input / output	User price on the stable group
Opus 5.5	`claude-opus-5-5`	$4 / $20	$1.26 / $6.30
Opus 5	`claude-opus-5`	$5 / $25	$1.575 / $7.875
Fable 5.1	`claude-fable-5-1`	$10 / $50	$3.15 / $15.75
Sonnet 5	`claude-sonnet-5`	$2 / $10	$0.63 / $3.15
GPT-6 Astra	`gpt-6-astra`	$10 / $50, then $20 / $75	$0.80 / $4.00, then $1.60 / $6.00
GPT-5.6 Sol	`gpt-5.6-sol`	$5 / $30, then $10 / $45	$0.40 / $2.40, then $0.80 / $3.60

Opus 5.5 on claude-stable is $4 × 0.315 and $20 × 0.315. Use the official table to compare labs. Use the catalog table to read a Modelflare bill you can send today.

Haiku 4.5 is in the same catalog at a $1 / $5 unit price. On claude-stable that is $0.315 / $1.575. The model-page comparison still gives Haiku a 200K context window and 64K max output. Haiku 5.5 is not in the 30 September response. Haiku 4.5 remains the low end of the Claude line that is actually listed.

Cache ratios on the catalog rows are a fraction of the input unit price: 0.05 for Opus 5.5, so the catalog cache read is $0.20 and the catalog cache write is $5. That write matches the official 5-minute tier. The official 1-hour write is $8 and is not a separate catalog price. Opus 5 and Sonnet 5 use 0.1, and Fable 5.1 uses 0.025. Settle cache, long context, and priority tiers from the pricing page and from the finished usage record.

What changed versus Opus 5 and Fable 5.1

Behavior	Opus 5.5	Opus 5	Fable 5.1
API id	`claude-opus-5-5`	`claude-opus-5`	`claude-fable-5-1`
Context / max output	1M / 128K	1M / 128K	1M / 128K
Thinking	Always on. `disabled` and a manual budget return 400	On by default. `disabled` is accepted only at effort `high` or below	Always on
Default effort when the field is omitted	`medium`	`high`	`high`
Reliable knowledge cutoff	June 2026	May 2026	June 2026
List input / output	$4 / $20	$5 / $25	$10 / $50
Cache read	$0.20	$0.50	$0.25
Forced tool choice	`any` and `tool` return 400	Listed as a breaking change away from Opus 5	Same 400 as Opus 5.5
Cybersecurity safeguard	Hands the task to Opus 4.8	The launch note does not put this fallback on Opus 5	Fable-class safeguards
Biology and frontier-LLM safeguard	Hands the task to Opus 5	The launch note pairs this fallback with Opus 5.5	Same safeguard class

Opus 5.5 reads thinking blocks from Opus 5 and from earlier Opus, Sonnet, and Haiku models. It does not read thinking blocks from Fable or Mythos. On the Claude API, Fable 5.1 and Mythos 5.1 read Opus 5.5 thinking blocks, and the what's-new page says no other current model does. A block the target model cannot read is dropped before the model sees it, and a dropped block is not billed.

Accounts created on or after 31 August 2026 get a stricter check: if the system prompt, the tool list, or an earlier message changed after an Opus 5.5 thinking block was produced, replaying that block returns 400. The migration guide's path is an append-only history, or a mid-conversation system message when instructions have to change. The launch page also says Opus 5.5 ships with the preserved-thinking control introduced on Fable 5.1, with EU AI Act watermarking, and with zero data retention, in line with earlier Opus models.

How to read the intelligence scores

The table is the one printed on the launch page. Anthropic says that, unless a footnote says otherwise, Opus 5.5 ran with adaptive thinking at max effort. Terminal-Bench 4.0 is the footnote: Opus 5.5 at xhigh, GPT-6 Astra at high, each as that model's highest reported score. Astra and Sol numbers are quoted as reported by OpenAI. Production safeguards were on. When they fired, cybersecurity tasks were completed by Opus 4.8, and biology plus frontier LLM-development tasks were completed by Opus 5. Anthropic says that pulls Opus 5.5's scores down.

Benchmark	Opus 5.5	Fable 5.1	Opus 5	GPT-6 Astra	GPT-5.6 Sol
Terminal-Bench 4.0	66.4%	55.8%	52.3%	57.9%	37.3%
FrontierCode v1.1 (Main)	54.4%	50.3%	48.0%	53.3%	47.5%
CursorBench 4.0	57.8%	51.8%	46.6%	—	41.7%
GDPval-AA v2.1	1846	1735	1708	1542	1588
AutomationBench	40.0%	31.4%	26.9%	41.4%	28.8%
Humanity's Last Exam, with tools	67.7%	65.6%	63.6%	57.2%	—
Terminal-Bench-Science 0.1	58.7%	52.6%	29.0%	64.6%	22.4%
OSWorld 2.0, partial	81.8%	80.7%	74.0%	—	—
Chartography, with tools	89.0%	88.4%	83.4%	—	—

Put the gaps next to the error bars they published. Terminal-Bench 4.0 is ±2.6 points for Opus 5.5 and ±1.6 to ±2 points for the other Claude models. The gap from 66.4% to Astra's 57.9%, and to Fable's 55.8%, is wider than those bars. Terminal-Bench-Science is ±3.5 to ±5 points per model. Astra's 64.6% against Opus 5.5's 58.7% sits near the width of those bars, with Astra ahead in the printed point estimate.

AutomationBench was run by Zapier. Safeguard interventions counted as failures because the run had no fallback model. Anthropic says the practical score would be higher. The printed 40.0% against Astra's 41.4% is a close published result, with Astra ahead on the point estimate. GDPval-AA v2.1 is an Elo on real professional tasks across 44 occupations: 1846 for Opus 5.5, 1735 for Fable 5.1, 1708 for Opus 5, 1588 for Sol, 1542 for Astra.

The same launch page prints a second pair of scores on the cost charts, taken at default effort medium: FrontierCode 54.6% and CursorBench 52.5%. The comparison table prints 54.4% and 57.8%. Keep them as two Anthropic figures at two effort settings. The chart caption also says that, at medium effort, FrontierCode beats Astra's top score of 53.3% at about a fifth of the cost per task, and that Terminal-Bench matches Astra at about 40% of the cost. Those are cost-per-task claims about Anthropic's harness.

The sentence under the table is the one to keep: in Anthropic's own use, the gap between Opus 5.5 and Fable 5.1 is narrower than the scores, and benchmark margins at this level are a weaker guide to a real task. Customer quotes on the launch page are selected testimonials. They can suggest what to replay. They do not belong in the score table.

Where it sits among current models

Three checkable shifts landed on the same day.

The work Anthropic describes as Fable 5.1 level for most tasks now has an Opus price card. List price is $4 / $20 against Fable 5.1 at $10 / $50. Cache reads are $0.20 against Fable's $0.25 and Opus 5's $0.50. Where a team's own replay agrees, Fable 5.1 stays the model for the tasks that still fail, and Opus 5.5 is the model that should run the loop. Where the replay disagrees, the table already shows the published exceptions: science-agent tasks and Zapier's business workflows, where Astra leads the printed numbers.

The default moved with the price. Claude Code's model configuration, updated with this release, sets Opus 5.5 as the default for Pro, Max, Team, Enterprise, and the Anthropic API, and also for Claude Platform on AWS, Amazon Bedrock, and Google Cloud's Agent Platform. A default inside the agent product moves more traffic than a new row in a picker.

The safeguard boundary moved down onto Opus. Earlier Opus models were where traffic went when a Fable classifier would not answer. Opus 5.5 launches with a similar set: cybersecurity tasks are completed by Opus 4.8, and biology plus frontier LLM-development tasks are completed by Opus 5. A log line that records the requested id claude-opus-5-5 can sit on a completion produced by a different model. Refusals return HTTP 200 with stop_reason: "refusal".

Sonnet 5.5 is already in the 30 September catalog at the same $2 / $10 unit price as Sonnet 5. Haiku 5.5 is not in that response. The cheaper listed Claude cards are Sonnet 5.5, Sonnet 5 at $2 / $10, and Haiku 4.5 at $1 / $5. Haiku 4.5 remains on a 200K window. Sonnet 5.5's groups are in the Sonnet 5.5 review.

This is a position you can audit from documents: list price, the Claude Code default, which model answers when a safeguard fires, and which rows of the vendor table Astra still leads. It is not a market-share claim.

Integration constraints

Requests that return 400

A client that only replaces the model string still hits four breaks documented for code that was running on Opus 5. The first three also apply to Fable 5.1. The shape below is the one the migration note tells you to send. The comments mark the variants that return invalid_request_error.

model: claude-opus-5-5
thinking: {"type": "adaptive"}
tool_choice: {"type": "auto"}
output_config.effort: medium
# 400: thinking type disabled, or enabled with budget_tokens
# 400: tool_choice any, or tool_choice tool
# 400 on the Claude API and Google Cloud: tool type computer_20251124

Depth replaces the old on/off thinking switch. Omit thinking, or send adaptive. Set effort explicitly. tool_choice auto and none remain valid, including on the token-counting endpoint. For schema-valid JSON, the migration note points to strict tool use or structured outputs, plus a prompt line that says when the tool applies.

On the Claude API and Google Cloud, replace computer_20251124 with the computer_toolset_20260801 toolset and drop the old computer-use beta header. Amazon Bedrock still accepts computer_20251124 on Opus 5.5. Browser-use integrations that already use the toolset need no change for this break.

For accounts created on or after 31 August 2026, a replay that edits the prefix in front of an Opus 5.5 thinking block is the fourth 400. Keep the conversation append-only.

Calls that succeed and still change behavior

Omitting effort now runs medium. The same omission on Opus 5 ran high. At a matched effort, Opus 5.5 tends to think more per turn than Opus 5, especially at xhigh and max. An effort value copied from an Opus 5 config is a new cost point. Leave room in max_tokens for the thinking blocks.

Text written between tool calls comes back inside thinking blocks. At the default display value omitted, those blocks are empty. A UI that used to stream that text goes quiet, and the request still succeeds. Set a display value when the product needs the progress lines.

A biology classifier runs beside the cybersecurity classifier. A prompt that pushes the model to copy its reasoning into the visible answer can be declined under reasoning_extraction. Chart and screenshot reading is sharper before any vision tool is added, according to the prompting guide, so older prompts that used to compensate for weak chart reading are worth replaying on their own.

Which workloads to evaluate first

Replay Opus 5.5 on the jobs the launch page is specific about: a multi-file change, a migration with the repository's own tests, a research memo graded against its sources, and a computer-use flow you can run twice. Record the requested model, the model that actually answered, effort, input tokens, cache-read tokens, output tokens, wall time, retries, and human edits. Anthropic's "about 40% less" and "about a fifth of the cost" are claims about their harness. They do not replace that sheet.

Keep Fable 5.1, or an explicit fallback, on tasks the replay still loses. Plan for Opus 4.8 on traffic the cybersecurity safeguard will hand off, and for Opus 5 on biology and frontier-model-development traffic the safeguard will hand off. Keep Sonnet 5.5 or Sonnet 5 where the task fits a $2 / $10 card and does not need the Opus loop.

On Modelflare, send claude-opus-5-5 for this tier. The 30 September row enables claude-premium at 0.5 when a failed call is expensive, claude-stable at 0.315 for ordinary production, and claude-award at 0.25 when the call can be retried. claude-tmp-cheap is not enabled on this model. On claude-stable the user price is $1.26 per million input tokens and $6.30 per million output tokens.

Sources

Launch page, 22 September 2026: https://www.anthropic.com/claude-opus-5-5
Model page, specs and the current comparison row: https://platform.claude.com/docs/en/models/opus-5-5/overview
What's new, breaking changes and the price card: https://platform.claude.com/docs/en/models/opus-5-5/whats-new-opus-5-5
Claude Code model configuration, default model: https://code.claude.com/docs/en/model-config
Modelflare pricing API, read 30 September 2026: https://modelflare.dev/api/pricing
Fable 5.1 review on this blog: Fable 5.1 review

Originally published at modelflare.dev.

Disclosure: I'm part of the Modelflare team, which builds the AI API gateway mentioned in this article.

AI API Latency Metrics: TTFT, First Response, and Output Speed

Magnus — Tue, 20 Oct 2026 02:09:35 +0000

A request-timeline guide to upstream headers, first SSE event, first effective response, first visible text, end-to-end latency, and output speed.

AI API latency is not one number. For a streaming request, record when the upstream headers arrive, when the first non-empty SSE event arrives, when the first meaningful content or action appears, when the first visible text appears, and when the response completes. These milestones answer different questions and should not all be labeled “TTFT.”

For a text-only chat, time to first visible text is usually the user-facing latency. For a reasoning or tool-using workflow, the first effective output can occur earlier as reasoning metadata or function-call arguments. Generation speed is a separate measurement taken after output starts.

Build one request timeline before naming a metric

Use a monotonic clock and one origin for all client- or gateway-level timestamps. A streaming request can be represented as follows:

t0  gateway receives request
t1  gateway starts upstream request
t2  upstream response headers arrive
t3  first non-empty SSE event arrives
t4  first effective content or action arrives
t5  first visible text delta arrives
t6  upstream body completes or closes
t7  gateway handler finishes

Not every request has every milestone. A non-streaming response has no useful SSE timeline. A function-call-only response may never produce visible text. A cancelled stream may have a start but no normal completion. Store absent values as absent rather than as zero, because zero would falsely imply that the event happened immediately.

Modelflare records request-level metadata under other.timing. The fields map to the timeline like this:

Field	What it measures	Important boundary
`auth_ms`	Token and user authentication duration	Local phase duration, not upstream time
`distribution_ms`	Channel selection and routing-policy duration	Local phase duration
`body_read_ms`	Time spent reading the downstream request body	Can expose a slow upload before routing
`upstream_headers_ms`	Upstream request start to response headers	Its origin is upstream start, not initial gateway receipt
`first_sse_event_ms`	Gateway receipt to first non-empty upstream SSE event	An event can contain metadata rather than useful output
`first_response_ms`	Gateway receipt to first effective content or action delta	Includes visible text, reasoning summary, function arguments, or custom tool input
`first_text_delta_ms`	Gateway receipt to first visible text delta	Exists only when visible streaming text appears
`upstream_done_ms`	Gateway receipt to upstream body completion or close	A close is not automatically a successful completion
`total_handler_ms`	Gateway receipt to the latest known relay completion point	Closest gateway-side end-to-end duration
`visible_output_tps`	Output tokens divided by time from first visible text to upstream completion	A visible-output diagnostic, not system-wide TPS

This mapping prevents a common error: subtracting timestamps that do not share an origin. For example, upstream_headers_ms is a duration beginning when the upstream request starts, while first_response_ms begins when the gateway receives the request.

TTFT has more than one practical meaning

The conventional definition of time to first token is the duration from query submission to the first received output token. The NVIDIA NIM benchmarking guide notes that this generally includes network latency, queueing, and prompt prefill, and that empty initial responses should not count.

That definition works for a text-generation benchmark, but modern API streams can emit several things before visible text:

a response-created or metadata event;
a reasoning summary delta;
function-call argument fragments;
custom tool input;
an empty heartbeat or provider-specific envelope.

Use explicit names in dashboards and incident reports:

Name	Recommended meaning	Best use
Time to headers	Request start to upstream headers	Network, proxy, and upstream admission diagnosis
Time to first event	Request start to first non-empty SSE event	Transport liveness only
Time to first effective response	Request start to useful content or action	Agent and reasoning workflow responsiveness
Time to first visible text	Request start to text a user can render	Chat and user-perceived responsiveness

If a chart says only “TTFT,” document which row it means. Otherwise two teams can report different values for the same request and both appear correct.

Separate responsiveness from generation speed

End-to-end latency measures the whole request from submission until the full response is received. It includes queueing, prefill, decoding, network transfer, tool pauses represented within the upstream stream, and gateway work visible inside the chosen boundary.

For a simple completed text stream:

end_to_end_latency = final_response_time - request_start_time
visible_generation_window = final_response_time - first_visible_text_time
visible_output_tps = output_tokens / visible_generation_window_seconds

Inter-token latency, also called time per output token, is normally the average spacing between output tokens after the first token. NVIDIA’s AIPerf definition excludes TTFT and divides the remaining duration by output_tokens - 1. Its reciprocal approximates per-user decode throughput for sufficiently long outputs.

Do not calculate token-level ITL by counting SSE events or text Delta objects. One event can contain zero, one, or several tokens, and token boundaries depend on the model tokenizer. Modelflare’s visible_output_tps uses the recorded output-token count and the visible generation window; it does not claim to be a per-token event-spacing measurement or total system throughput.

These metrics answer separate questions:

low first-text latency with slow output speed feels responsive at first and then sluggish;
high first-text latency with fast output speed feels stalled and then finishes quickly;
low per-user output speed does not prove low system throughput under concurrency;
high aggregate TPS does not guarantee good latency for one user.

Reasoning and tools can precede visible text

In a text-only flow, first_response_ms and first_text_delta_ms may be nearly identical. In a reasoning or function-calling flow, the gap can be meaningful.

Suppose a model emits function-call arguments at 1.8 seconds, the application or provider-side tool flow continues, and visible text begins at 6.4 seconds. The system has produced an actionable result at 1.8 seconds, but the user has not seen prose until 6.4 seconds. Calling both values TTFT hides whether the delay occurred before planning, during a tool step, or before answer rendering.

Interpret the fields according to the product experience:

for a terminal agent, first effective action may be the best responsiveness signal;
for a chat UI that displays only prose, first visible text is the relevant user signal;
for a tool-call API consumed by code, visible text may be absent and should not be required;
for a reasoning model, an early SSE envelope proves connection progress but not useful progress.

The AI API Streaming Guide covers SSE parsing, cancellation, and idle-timeout behavior. Latency metrics should be computed only after the parser distinguishes envelopes from effective output.

Diagnose the slow stage in a fixed order

Start at the upstream boundary, then move toward local phases. This avoids blaming authentication or routing when the request spent most of its time waiting for the model.

Symptom	Primary metric	Likely layer to inspect	Next check
Slow before headers	`upstream_headers_ms`	Network path, upstream admission, provider queue, proxy route	Compare channel, region, status, and concurrency
Headers are fast, useful output is late	`first_response_ms` minus early phases	Model queueing, prompt prefill, reasoning, upstream scheduling	Compare input tokens, model, route, and inflight count
First event is fast, effective output is late	Gap from `first_sse_event_ms` to `first_response_ms`	Metadata-only events, heartbeats, reasoning startup	Inspect safe event types, not raw content
Effective action is fast, visible text is late	Gap from `first_response_ms` to `first_text_delta_ms`	Tool/reasoning phase or response composition	Check requested tools and output mode
Text begins quickly, then crawls	`visible_output_tps` and `upstream_done_ms`	Decode throughput, contention, long context, network backpressure	Compare output length and channel inflight count
All upstream metrics are normal	`auth_ms`, `distribution_ms`, `body_read_ms`	Local auth, policy selection, client upload	Inspect only the phase that is elevated

No single metric proves a root cause. High upstream_headers_ms, for example, groups several possibilities that require route, region, provider, and concurrent-load evidence to separate.

Read three synthetic traces

The following values are illustrative diagnostics, not Modelflare production averages or provider benchmarks.

Trace A: waiting before upstream headers

Metric	Value
`upstream_headers_ms`	6,100 ms
`first_sse_event_ms`	6,300 ms
`first_response_ms`	6,350 ms
`first_text_delta_ms`	6,400 ms
`total_handler_ms`	9,200 ms
`visible_output_tps`	42

Most of the wait happens before headers. Generation after visible text is comparatively healthy. Check the selected upstream route, provider admission, network path, region, and concurrent load before optimizing the client renderer.

Trace B: useful action before visible prose

Metric	Value
`upstream_headers_ms`	240 ms
`first_sse_event_ms`	310 ms
`first_response_ms`	2,900 ms
`first_text_delta_ms`	8,700 ms
`total_handler_ms`	10,200 ms
`visible_output_tps`	55

Transport is alive early and an effective action appears at 2.9 seconds, but visible text waits another 5.8 seconds. If the request uses reasoning or tools, inspect that phase. Increasing a header timeout would not address this pattern.

Trace C: fast start, slow generation

Metric	Value
`upstream_headers_ms`	260 ms
`first_sse_event_ms`	330 ms
`first_response_ms`	420 ms
`first_text_delta_ms`	430 ms
`total_handler_ms`	20,430 ms
`visible_output_tps`	9.8

The request becomes visible quickly but spends about 20 seconds generating. Compare output length, context length, selected channel, inflight count, and provider behavior. A first-output timeout would pass and therefore cannot detect this failure mode.

Compare latency only under controlled conditions

A fair comparison holds the workload contract stable. At minimum, record:

exact model identifier and route or group;
input-token and output-token distributions, not just averages;
streaming versus non-streaming mode;
enabled reasoning effort and tools;
region and client-to-gateway network path;
concurrency or arrival rate;
sampling settings and maximum output length;
warmup policy, retry policy, and excluded failures;
sample size, time window, and percentile method.

Compare p50, p95, and p99 rather than presenting one average. Keep failures visible: removing timeouts and errors can make a route look faster precisely when it is less reliable. Never compare two providers with different prompts, output lengths, concurrency, or endpoint behavior and present the result as a model-speed ranking.

Retain diagnostic metadata without retaining content

Latency diagnosis does not require storing prompts, responses, API keys, or plaintext client identities. A useful request record can contain:

a generated request ID and timestamp;
model, group, and selected channel reference;
status and terminal outcome;
input, output, and cached-token counts when available;
the timing fields above;
stream event and text-delta counts;
channel inflight count;
coarse region and a privacy-reviewed network trace identifier;
a coarse classifier such as upstream_headers_slow or generation_slow_tps.

Apply retention limits and access controls even to metadata. Request IDs, route choices, timing patterns, and token counts can still reveal operational behavior when combined.

Begin an investigation with the request timeline, not a single TTFT label. Then use Reliable AI API Routing to compare the exact route selected for each attempt and AI API Error Troubleshooting to pair timing with the terminal status. This produces evidence that can separate admission delay, model startup, reasoning or tool work, visible generation, and local gateway overhead.

Originally published at modelflare.dev.

Disclosure: I'm part of the Modelflare team, which builds the AI API gateway mentioned in this article.

AI API Fallback Strategy: Build a Provider Failure Matrix

Magnus — Mon, 19 Oct 2026 03:18:55 +0000

A phase-aware policy for deciding when to retry, use a same-contract route fallback, stop, reconcile side effects, or investigate a provider path.

An AI API retry, a route fallback, and a model substitution are three different actions. A retry repeats an attempt under the same contract. A route fallback sends the requested model and protocol to another eligible channel or group. A model substitution deliberately changes the requested model and may change quality, price, latency, tool behavior, context limits, and output format.

A reliable policy decides which action is allowed before an incident. The decision must consider the failure class, whether any response has already reached the caller, whether replay is idempotent, and how many attempts fit within the user’s deadline.

Separate retry, fallback, and model substitution

Use distinct names in configuration, logs, and runbooks.

Action	What changes	Appropriate purpose	Main risk
Same-route retry	Time and attempt number	Recover from a short transient failure on the same route	Repeating load against an unhealthy dependency
Same-contract fallback	Upstream channel, account, or explicitly ordered group	Preserve the requested model and protocol when one path fails	Hidden incompatibility between supposedly equivalent routes
Model substitution	Model identifier or a named model policy	A product-approved quality, cost, or availability tradeoff	Silent behavior and billing changes

Do not label all three as “retry.” Operators need to know whether a request was replayed, moved to another path, or answered by a different model. Users need model substitution to be an explicit product contract, not an invisible recovery shortcut.

Some gateways support ordered model or provider steps. For example, Cloudflare’s fallback documentation exposes which step produced the successful response. The important design lesson is not to copy a vendor’s exact policy; it is to preserve attempt-level evidence whenever a route changes.

Apply four gates before replaying a request

A status code alone is not a retry policy. Evaluate four gates:

Failure class: Is the failure transient, permanent, caller-caused, or ambiguous?
Response phase: Did it happen before headers, before effective output, or after output was delivered?
Idempotency: Can the complete application operation be repeated without a duplicate side effect?
Attempt budget: Is there enough remaining wall-clock time and an unused attempt?

The Google Cloud retry strategy makes the same two foundational distinctions for general APIs: the response indicates whether retry may be useful, and idempotency determines whether replay is safe. It lists 408, 429, 5xx, socket timeouts, and disconnects as commonly transient, while warning that non-idempotent operations need stronger conditions.

For an AI workflow, idempotency extends beyond the model HTTP request. A replayed prompt can propose the same email, refund, deployment, or database write again. Tool execution therefore needs its own stable idempotency key and persisted result, even if the inference call itself is read-only.

Start from a failure matrix

The matrix below is a conservative application policy. A gateway may perform an internal same-contract channel failover before the application sees the terminal result, so coordinate the two layers rather than duplicating them.

Failure or phase	Same-route retry	Same-contract fallback	Stop or investigate	Reason
Client validation error, unsupported field, or malformed request	No	No	Fix request	Replaying the same invalid contract cannot succeed
Gateway authentication, authorization, quota, or policy denial	No	No	Stop and correct account or policy	Another provider route must not bypass the gateway decision
Upstream credential or account failure before output	No on the failed route	Yes, if another verified channel is eligible	Quarantine and investigate failed channel	The user contract can remain the same while an unhealthy credential is removed
Network failure or `408` before response output	At most one bounded attempt if idempotent	Yes	Stop at deadline	The result is transient, but completion may be ambiguous after a disconnect
`429` before output	Honor `Retry-After` for delayed retry	Yes when another same-contract route has capacity	Stop when budget is exhausted	Immediate repeated retries amplify rate limiting
Upstream `500`, `502`, `503`, or `504` before output	Bounded retry with backoff	Yes	Investigate repeated route failures	Usually transient, but not evidence that every route is safe
Schema-invalid or malformed provider response before downstream output	Usually no	Only to a route verified for the same schema	Quarantine or investigate compatibility	Repeating the same incompatible implementation is unlikely to help
Model refusal or policy-safe completion	No	No	Return the modeled outcome	A valid refusal is not an infrastructure outage to route around
Caller cancellation or downstream `499`	No	No	Stop immediately	The caller no longer wants work; retries waste capacity and cost
Partial stream after visible content or tool arguments reached the caller	No transparent replay	No transparent fallback	Mark partial and let the application decide	A second stream can duplicate or contradict already delivered output
Tool side effect has unknown completion state	No until reconciled	No until reconciled	Query the idempotency record or downstream system	Re-inference can propose the same side effect again

This matrix is deliberately phase-aware. A 503 received before any downstream output is different from a connection that closes after 400 text tokens have already been rendered.

Treat a started stream as a commit boundary

Before downstream output starts, a gateway can often discard a failed attempt and try another eligible route without exposing two answers. After the first byte of meaningful output reaches the caller, transparent replay becomes unsafe.

A restarted stream may:

repeat the beginning of an answer;
produce a different continuation;
issue a duplicate function call with a new Call ID;
change usage and cost without a clear boundary;
leave the client unable to tell which events belong together.

When a stream breaks after output has started, return a terminal partial or transport error with the original request identity. The application can offer an explicit “try again” action, continue from a safe application checkpoint, or discard the partial output. It should not splice a new model stream onto the old stream as if nothing happened.

For Function Calling, the safest boundary is stricter: persist accepted Tool Call identities and side-effect results before any retry can recreate them. The Function Calling comparison shows how Call IDs and application idempotency fit together.

Bound backoff by attempts and wall-clock time

Exponential backoff spreads repeated attempts over time. Jitter prevents many clients from retrying in lockstep after a shared outage.

delay_cap = min(max_delay, base_delay * 2^retry_index)
sleep_for = random_between(0, delay_cap)

Honor a valid Retry-After signal when it fits the user’s deadline. Backoff is not permission to retry: the failure and idempotency gates must pass first.

Define a total budget, not only a retry count:

maximum attempts for one user action;
maximum elapsed time including queueing and backoff;
maximum attempts before and after selecting a fallback group;
minimum time left to produce a useful answer;
cancellation propagation from caller to every active attempt.

For an interactive request with a 15-second deadline, three 10-second attempts are not a real policy. The later attempts cannot finish within the user contract. Batch workloads can use a longer budget, but should still have a terminal deadline and a durable job identity.

Prevent retry amplification across layers

Assume an SDK makes one request plus two retries: three client attempts. The gateway makes one primary attempt plus two channel fallbacks for each client attempt: three gateway attempts. An upstream proxy makes one retry per attempt: two provider calls.

3 client attempts × 3 gateway attempts × 2 upstream attempts = 18 provider calls

One user action has now become 18 provider calls. During an outage, this increases queueing, rate limiting, cost, and recovery time.

Assign retry ownership deliberately:

the gateway owns immediate same-contract channel failover;
the application owns whether a complete user action may be repeated;
an SDK’s automatic retries are disabled or bounded when the gateway already retries;
asynchronous jobs use one durable job ID and an attempt ledger;
no layer starts a new attempt after caller cancellation.

Log both “attempt number at this layer” and a stable end-to-end request identity. Otherwise each layer appears to have made only two or three attempts while the combined amplification remains hidden.

Verify that a fallback really preserves the contract

The same public model name does not prove two routes behave identically. Before adding a channel to a same-contract fallback set, test:

Contract area	Evidence required
Model identity	Requested model remains available without silent mapping
Endpoint	Responses or Chat Completions request is accepted as configured
Streaming	Event types, termination, usage, and cancellation are handled
Structured output	Required JSON Schema subset and strict behavior work
Function calling	Tool definitions, Call IDs, argument streaming, and results round-trip
Limits	Context, output, rate, and concurrency limits fit the workload
Errors	Status and error bodies remain classifiable without exposing secrets
Usage and cost	Token accounting, cache fields, service tier, and price policy are understood
Safety and region	Required policy, data path, and residency constraints remain satisfied

If a candidate route fails one required area, it is not a transparent fallback for that workload. It may still be usable under a separate, explicit product policy.

Changing to another model is always such a policy decision. Define the permitted model, quality floor, price ceiling, Tool Contract, and user-visible disclosure. Do not use a cheaper or weaker model merely because a route returned an error.

Understand Modelflare’s current fallback semantics

Modelflare searches eligible paths for the model requested by the API client. A regular API Key has one primary group and can have an ordered list of fallback groups. A Smart API Key evaluates groups available to the account using its configured routing strategy. Neither mechanism should silently replace the requested model with another model.

Group-level RPM admission occurs before billing and before an upstream request. If the selected group is full, an ordered fallback group or Smart Routing candidate can be evaluated; if no eligible group remains, the request returns 429.

Within a selected group, channel priority defines account failover order. After an upstream error, the failed channel is excluded and selection continues through remaining eligible channels. Current channel failover is independent of RetryTimes and AutomaticRetryStatusCodes. It stops on success, eligible-route exhaustion, caller cancellation, or after a downstream response has started and can no longer be transparently replayed.

These are internal same-model path decisions, not permission for a client to add another unlimited retry loop. Use Reliable AI API Routing for group and channel design, and AI API Error Troubleshooting to distinguish gateway policy errors from upstream failures.

Preserve evidence for every attempt

A final 200 does not prove that the first route succeeded. A final channel ID does not describe failed attempts. Retain enough metadata to reconstruct the path:

stable request ID and caller-visible correlation ID;
attempt sequence and selected group/channel reference;
model and endpoint contract used for each attempt;
failure status, error class, and stream phase;
whether downstream output had started;
timing milestones and cancellation state;
input, output, and cached-token usage when available;
cost attribution by completed or billable attempt;
final terminal reason: success, exhausted, cancelled, partial, or policy stop.

Do not retain API keys, raw prompts, raw responses, or provider credentials merely to diagnose fallback. Redacted error classes and timing metadata are usually sufficient, with restricted short-lived request archives reserved for explicitly configured failure investigation.

Drill the policy before production traffic depends on it

Run a staging exercise using the real protocol boundary and safe deterministic inputs:

make the primary channel unavailable before headers and verify the next eligible same-model path;
return a rate limit and verify attempt limits and any Retry-After handling;
cancel the caller and prove that no later attempt starts;
break a stream after output and prove that no transparent replay occurs;
send an invalid request and prove that fallback does not hide it;
repeat a tool workflow and prove that one side effect is recorded;
exhaust every route and verify one clear terminal error;
inspect the attempt ledger and reconcile usage and cost.

Roll out the policy to a small workload, monitor attempt count and success-after-fallback separately from raw success rate, and keep a fast way to remove an unhealthy route. The goal is not to maximize fallback frequency. It is to recover only when the original model contract can be preserved safely, within a bounded deadline, with evidence for every attempt.

Originally published at modelflare.dev.

Disclosure: I'm part of the Modelflare team, which builds the AI API gateway mentioned in this article.

Structured Outputs with OpenAI-Compatible APIs: A JSON Schema Guide

Magnus — Fri, 16 Oct 2026 01:45:19 +0000

A field-level guide to strict JSON Schema outputs across Responses and Chat Completions, including validation layers, failure handling, and route compatibility tests.

Structured Outputs let an application request JSON that follows a defined schema instead of merely asking the model to “return JSON.” With an OpenAI-compatible API, the same schema can be used through Responses or Chat Completions, but the request fields are different—and compatibility must be verified for the selected model and route.

The safe production pattern has three layers: use strict schema output when the model supports it, parse and validate the returned JSON in the application, and then apply business rules to the values. Schema compliance removes many formatting failures; it does not prove that the model’s answer is factually or semantically correct.

Structured Outputs are not the same as JSON mode

There are three common ways to ask a model for JSON-like output.

Method	Produces valid JSON	Enforces your schema	Typical use
Prompt only	Not guaranteed	No	Prototypes where parsing failures are acceptable
JSON mode	Yes, when supported and completed	No	Flexible JSON where the application validates the shape
Structured Outputs with `strict: true`	Yes, subject to completion and refusal handling	Yes, within the supported JSON Schema subset	Typed extraction and application workflows

The OpenAI Structured Outputs guide recommends Structured Outputs instead of JSON mode when the selected model and endpoint support it. The same guide also distinguishes two different jobs:

Use a structured response format when the model should answer the user with a predictable data object.
Use function calling when the model should request an action from your application.

This article focuses on the first job. A later tool loop can reuse strict argument schemas, but its call IDs and result messages follow a different contract.

Define one schema before choosing the endpoint

Consider a support-triage workflow. The application needs four fields:

category: one of billing, technical, or account;
priority: an integer from 1 through 3;
requires_human: a boolean;
summary: a short description for the support queue.

The JSON Schema is:

{
  "type": "object",
  "properties": {
    "category": {
      "type": "string",
      "enum": [
        "billing",
        "technical",
        "account"
      ]
    },
    "priority": {
      "type": "integer",
      "minimum": 1,
      "maximum": 3
    },
    "requires_human": {
      "type": "boolean"
    },
    "summary": {
      "type": "string"
    }
  },
  "required": [
    "category",
    "priority",
    "requires_human",
    "summary"
  ],
  "additionalProperties": false
}

Making every property required and setting additionalProperties to false produces a stable object for downstream code. It does not make every possible JSON Schema keyword portable. OpenAI supports a documented subset, and another OpenAI-compatible provider may support a different subset or no strict schema mode at all.

Send the schema with the Responses API

The Responses API places the response schema under text.format. Set a Modelflare API key and choose a model that is currently listed for your API key and has been verified for Structured Outputs.

export MODELFLARE_API_KEY="<YOUR_API_KEY>"
export MODEL_ID="<MODEL_WITH_VERIFIED_STRUCTURED_OUTPUT_SUPPORT>"

curl -sS https://modelflare.dev/v1/responses \
  -H "Authorization: Bearer $MODELFLARE_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"$MODEL_ID\",
    \"input\": \"The customer was charged twice and wants a refund.\",
    \"text\": {
      \"format\": {
        \"type\": \"json_schema\",
        \"name\": \"support_ticket\",
        \"schema\": {
          \"type\": \"object\",
          \"properties\": {
            \"category\": {
              \"type\": \"string\",
              \"enum\": [\"billing\", \"technical\", \"account\"]
            },
            \"priority\": {
              \"type\": \"integer\",
              \"minimum\": 1,
              \"maximum\": 3
            },
            \"requires_human\": { \"type\": \"boolean\" },
            \"summary\": { \"type\": \"string\" }
          },
          \"required\": [\"category\", \"priority\", \"requires_human\", \"summary\"],
          \"additionalProperties\": false
        },
        \"strict\": true
      }
    }
  }"

In a Responses response, generated content appears in typed output items. SDKs may expose convenience helpers such as aggregated output text or parsed output, but an HTTP client should not assume that the structured object is a top-level field. Read the completed response object, locate its text output, then parse that text as JSON.

Send the same schema with Chat Completions

Chat Completions places the response schema under response_format.json_schema. The schema body is unchanged; only the wrapper differs.

curl -sS https://modelflare.dev/v1/chat/completions \
  -H "Authorization: Bearer $MODELFLARE_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"$MODEL_ID\",
    \"messages\": [
      {
        \"role\": \"user\",
        \"content\": \"The customer was charged twice and wants a refund.\"
      }
    ],
    \"response_format\": {
      \"type\": \"json_schema\",
      \"json_schema\": {
        \"name\": \"support_ticket\",
        \"schema\": {
          \"type\": \"object\",
          \"properties\": {
            \"category\": {
              \"type\": \"string\",
              \"enum\": [\"billing\", \"technical\", \"account\"]
            },
            \"priority\": {
              \"type\": \"integer\",
              \"minimum\": 1,
              \"maximum\": 3
            },
            \"requires_human\": { \"type\": \"boolean\" },
            \"summary\": { \"type\": \"string\" }
          },
          \"required\": [\"category\", \"priority\", \"requires_human\", \"summary\"],
          \"additionalProperties\": false
        },
        \"strict\": true
      }
    }
  }"

For Chat Completions, the JSON text is normally returned in choices[0].message.content. Parse the content rather than treating the string as an already typed object.

The two wrappers can be summarized as follows:

Purpose	Responses API	Chat Completions
Schema container	`text.format`	`response_format.json_schema`
Format type	`text.format.type`	`response_format.type`
Schema name	`text.format.name`	`response_format.json_schema.name`
Schema body	`text.format.schema`	`response_format.json_schema.schema`
Strict flag	`text.format.strict`	`response_format.json_schema.strict`
Result location	Typed output items / output text helper	`choices[0].message.content`

Modelflare’s OpenAI compatibility layer has explicit conversion logic for these two wrappers on supported OpenAI/Codex routes. That conversion does not create Structured Outputs support in an upstream model that lacks it. For other model families served as raw Chat Completions pass-through, the upstream provider’s exact request contract remains the source of truth.

Validate structure and meaning separately

Suppose the model returns:

{
  "category": "billing",
  "priority": 2,
  "requires_human": true,
  "summary": "Customer reports a duplicate charge and requests a refund."
}

This object satisfies the schema, but the application should still perform two distinct checks.

Structural validation

Use a JSON Schema validator or a typed SDK helper to confirm:

the object contains every required property;
no unexpected property is present;
each value has the required type;
the category is in the allowed enum;
the priority is within the permitted range.

Even when the provider promises strict adherence, application-side validation protects the rest of the system from unsupported routes, integration mistakes, truncated content, and future contract changes.

Semantic and business validation

The schema cannot determine whether the customer was actually charged twice, whether priority 2 is correct, or whether a human must approve a refund. Those decisions require source data and application policy.

For high-impact workflows, treat the model object as a proposed classification. Compare it with authoritative records, enforce permissions and financial limits in deterministic code, and retain a safe audit reference. Never let a structurally valid model object bypass an authorization, billing, or security boundary.

Handle incomplete and exceptional output

A production parser needs more than a happy-path JSON.parse call.

Refusals

A model may refuse a request for safety reasons. OpenAI’s structured-output contract makes refusals distinguishable from normal schema output. Check the response status and refusal representation before looking for the JSON object; do not report a refusal as a schema parsing error.

Truncation and output limits

If generation stops before the object is complete, no schema can repair the missing suffix. Check the endpoint’s completion status and stop reason. Set an output limit that is sufficient for the largest valid object, while keeping the schema and requested content bounded.

Unsupported schema keywords

Strict output implementations commonly support only a subset of JSON Schema. Start with objects, arrays, primitive types, enums, required fields, and explicit additional-property behavior. Check the current provider documentation before using references, recursive shapes, complex unions, or advanced validation keywords.

First-use schema latency

Some providers preprocess and cache a new schema. The first request for a new schema can therefore take longer than later requests. Reuse stable, versioned schemas instead of generating a unique schema for every request, and separate first-use measurements from steady-state latency comparisons.

Model or route incompatibility

An OpenAI-compatible endpoint can accept ordinary Chat Completions while rejecting json_schema, ignoring strict, or forwarding the fields to a model that does not support them. A 200 response containing JSON is still not proof of strict adherence. Test invalid and edge-case inputs and validate the resulting object.

Run a compatibility test before production

Use a small matrix for every model and route you intend to enable.

Test	Evidence to record
Minimal required object	Status, model, route, parsed object, validation result
Every enum value	Whether each allowed value can be produced and parsed
Missing information	Whether the model uses a bounded fallback value or asks for clarification
Safety-triggering input	Refusal representation and application handling
Low output limit	Completion status and truncation handling
Unsupported schema feature	Explicit error versus silently ignored constraint
Responses and Chat wrappers	Whether both produce equivalent application objects
Repeated stable schema	First-use versus steady-state timing

Do not compare models with different schemas, prompts, regions, streaming modes, or output limits and then attribute the result only to model quality. Record the exact test date because provider behavior and model availability can change.

Choose the endpoint after testing the workflow

Use Responses when the application already works with typed output items, Responses streaming events, or a broader Responses tool workflow. Use Chat Completions when the application has a stable message-based integration and the selected route supports its response_format contract. The broader decision is covered in Responses API vs Chat Completions.

Whichever endpoint you select:

keep one versioned source for the JSON Schema;
translate that schema into the endpoint’s wrapper without changing its meaning;
validate the completed response before using it;
apply deterministic business rules after structural validation;
monitor parsing, refusal, truncation, and compatibility failures separately.

For initial client migration, start with the OpenAI-Compatible API Guide. For event parsing and incomplete streams, use the AI API Streaming Guide. Check Models & Pricing for the current model and group surface, then verify Structured Outputs on the exact route your API key will use.

Structured Outputs are most useful when they narrow a model response into a dependable application interface. They are not a substitute for factual verification, authorization, or billing logic—and “OpenAI-compatible” remains a claim to test feature by feature, not a guarantee to infer from the Base URL.

Originally published at modelflare.dev.

Disclosure: I'm part of the Modelflare team, which builds the AI API gateway mentioned in this article.

One Key, a Dozen-Plus Models: Switching LLMs in VS Code Chat with Modelflare

Magnus — Wed, 14 Oct 2026 14:32:00 +0000

How to use the Modelflare VS Code extension to put every text model your account can reach into the VS Code chat model picker, behind a single Smart API Key.

Most developers no longer use one model. A fast, cheap model is fine for "rename this variable everywhere." A frontier model earns its price on a gnarly refactor. A third model is the one you trust for reviewing a migration. The problem is that each of those usually lives behind its own extension, its own account, and its own key.

That setup has real costs:

Configuration sprawl. Three providers means three keys in three places, three billing dashboards, and three sets of rate limits.
Switching is slow. If changing models means changing extensions, you stop doing it, and you end up paying frontier prices for trivial edits.
New models lag. Every release means waiting for a provider extension update or editing config by hand.
Outages are your problem. When one provider has a bad hour, the chat panel just errors.

VS Code already has the right UI for this: the chat model picker. What it needs is one provider that can fill that picker with models from many vendors. That is what the Modelflare extension does.

What the extension does

Modelflare for VS Code registers Modelflare as a language model provider for VS Code chat. According to its README:

Text models available to your key are loaded from GET /v1/models, so new models appear without editing the key.
Responses stream, and tool calling works for agent mode.
Routing across groups, fallback and the routing strategy are handled server-side by the Smart API Key.
The key lives in VS Code's secret storage, never in settings.json.
It is bring-your-own-key: no Copilot subscription is required for BYOK models.

The extension ID is modelflare.modelflare-vscode, and it requires VS Code 1.104 or newer.

Step 1: Create a Smart API Key

This is the piece that makes "one key" true. An ordinary Modelflare key is bound to a single routing group and can only call that group's models. A Smart API Key is bound to auto. On every request Modelflare:

collects every routing group your account may use,
keeps only the groups that currently have a channel for the model you asked for,
orders them by the key's strategy, and
sends to the first candidate, moving to the next group if that one runs out of usable channels.

So a model that only one group serves still works, and a group that is momentarily unusable is skipped instead of failing your chat.

To create one:

Sign up or open the console.
Go to API Keys â†’ Create Smart API Key.
Pick a routing strategy:

Strategy	Order of candidate groups
Balanced (default)	Starts in the middle of the price range, works up to the most expensive candidate, then back down through the cheaper ones
Stability first	Highest group multiplier first
Low price first	Lowest group multiplier first

You can change the strategy on the existing key at any time; the next request uses the new order. Quota, expiry date and IP allowlist work as on any other key.

Step 2: Install the extension

From the VS Code Marketplace, search for Modelflare in the Extensions view, or install from the terminal:

code --install-extension modelflare.modelflare-vscode

The same extension is also published on Open VSX, the registry used by VS Code-compatible builds such as VSCodium: open-vsx.org/extension/modelflare/modelflare-vscode. It needs a build based on VS Code 1.104 or newer, the release line with the chat model provider API the extension uses.

A note for early adopters: version 0.1.0 shipped under the ID modelflare.modelflare. From 0.1.1 the ID is modelflare.modelflare-vscode, because the shorter name is taken on the VS Code Marketplace. Commands and settings are unchanged, but VS Code keeps secrets per extension ID, so if you installed 0.1.0, run Manage Modelflare API Key once after updating.

Step 3: Add your key

Either:

open the chat model picker â†’ Manage Models â†’ Modelflare, or
run Modelflare: Manage Modelflare API Key from the Command Palette.

Paste the Smart API Key. It goes into VS Code's secret storage. Then open the model picker again and choose a Modelflare model.

The extension has exactly one setting:

{
  "modelflare.baseUrl": "global"
}

global (the default) points at https://modelflare.dev/v1; china points at https://origin.modelflare.dev/v1; you can also give a custom URL. Most readers never need to touch it.

Worked example: three models, one task

To show why switching matters, here is a realistic loop on a small TypeScript service. All three models were picked from the same dropdown, on the same key, without leaving the chat panel.

The model IDs below are real entries in the Modelflare catalog. Each one was also returned by GET /v1/models for a Smart API Key on the day we wrote this, and that is the same list the extension loads. Which ones appear in your picker depends on what your account can reach.

1. Explore with a fast model. Pick gemini-3.5-flash and ask:

Summarize what src/billing/ does and list every place we compute a rounded price.

Cheap, quick, and good enough to map a folder.

2. Refactor with a frontier model in agent mode. Switch the picker to claude-opus-5-5, switch the chat to agent mode, and ask:

Extract the rounding logic you found into src/billing/round.ts, update every caller, and add unit tests.

This is where streaming plus tool calling matters: the model reads files, proposes edits, and runs through the change set the way agent mode normally does.

3. Get a second opinion. Switch to gpt-6.1-sol and ask:

Review the diff in this workspace. Look for behavior changes in rounding at .5 boundaries.

A different model family reviewing another one's work catches a surprising number of edge cases.

Three vendors, one key, one bill. Every request shows up in the usage logs, billed with the multiplier of the group that actually served it, and the log records the resolved group and strategy (Usage and cost).

For reference, here are a dozen-plus text model IDs that were both in the live catalog and in a Smart API Key's GET /v1/models response on the day we wrote this:

Family	Example IDs
OpenAI GPT	`gpt-6.1-sol`, `gpt-6-sol`, `gpt-5.5`
Anthropic Claude	`claude-opus-5-5`, `claude-sonnet-5-5`, `claude-haiku-5-5`
Google Gemini	`gemini-3.8-flash`, `gemini-3.5-flash`, `gemini-3.1-pro-preview`
xAI Grok	`grok-4.7`
DeepSeek	`deepseek-v4-pro`, `deepseek-v4-flash`
Others	`kimi-k3`, `glm-5.3`, `qwen3.8-max`

Treat this as a snapshot. The picker is the source of truth for your account, and Models & Pricing is the source of truth for prices.

Tips

Refresh after a release. When a new model lands, run Modelflare: Refresh Modelflare Models. The key itself needs no edit: a Smart API Key has no per-model allowlist and always follows your account's current access.

Use the strategy, not a second key. If you want cheaper defaults for a side project and maximum stability for work, change the strategy, or create two Smart API Keys with different strategies. You do not need provider-specific keys.

Do not build your own fallback list. A Smart API Key already walks every group your account can use. Ordered fallback chains are for ordinary keys; see Group failover.

Pick the model per task, not per day. The cost difference between a flash-class model and a frontier model is large. Defaulting to a fast model and switching up only for the hard step is the habit this setup makes cheap.

Read errors literally.

401: the key is missing or invalid. Create a Smart API Key at modelflare.dev/keys.
model_not_found: a regular key only calls its own group's models. If you already use a Smart API Key, that model is not available to your account right now.
403 insufficient_user_quota: top up at modelflare.dev/billing.

Slow model list from mainland China? Set modelflare.baseUrl to china. Since 0.1.2, the extension's public catalog request times out after 8 seconds and falls back to the key's own model list, so the picker no longer hangs.

Want images and video too? The VS Code extension covers text models. For media generation in an agent, see the Media Skill or the @modelflare/mcp-server package.

Where to go next

Install: VS Code Marketplace Â· Open VSX
Source and changelog: github.com/castaly-ai/modelflare-vscode
Smart API Key: docs.modelflare.dev/guides/features/smart-api-key
Models and groups: docs.modelflare.dev/guides/models-and-groups
Text models: docs.modelflare.dev/models/text Â· Live catalog: modelflare.dev/pricing
Platform features: docs.modelflare.dev/guides/features

The model picker was always the right place to choose a model. With one key behind it, choosing is finally cheap enough to do on every task.

Disclosure: I'm part of the Modelflare team, which builds the AI API gateway mentioned in this article.

Generate Images and Videos in Claude Code with One Sentence

Magnus — Wed, 14 Oct 2026 14:31:41 +0000

How to give Claude Code image and video generation with the Modelflare Claude Code plugin, the Media Skill, or the @modelflare/mcp-server MCP server, all on one Smart API Key.

Claude Code is good at writing the landing page. It is not good at producing the hero image for that page, the 5-second product loop in the header, or the twelve icon variants the designer asked for. Those normally mean leaving the terminal: open an image tool, sign in to a video provider, download files, rename them, drag them into public/, and come back.

The friction is not the generation itself. It is everything around it:

Every media provider has its own contract. OpenAI-compatible image models use POST /v1/images/generations; Gemini image models use native generateContent. Sending a model to the wrong endpoint is rejected with unsupported_endpoint (Image models).
Video is asynchronous. You create a task, poll it, and download the result before the URL expires.
Prices change. "Which video model under 0.10 USD per second can do image-to-video?" is a question you should not answer from memory.
Keys multiply. A separate key per vendor means separate billing, separate quotas, and separate places to leak a secret.

This guide removes that loop. By the end, a single sentence in Claude Code produces a file on disk.

Three ways in, one key

There are three Modelflare integrations that reach Claude Code. They overlap on purpose, so pick the one that matches how you work.

Integration	What it is	Best for
Claude Code plugin	A Claude Code plugin marketplace that ships the Modelflare Media Skill	Claude Code users who want a two-command install and `claude plugin update`
Media Skill	A folder with a `SKILL.md`, a model reference and one helper script	Any `SKILL.md`-aware agent (Claude Code, Codex and others), shared machines, CI
`@modelflare/mcp-server`	An MCP server with explicit `list_models`, `generate_image`, `create_video`, `get_video_status` and `download_video` tools	Any MCP client (Claude Desktop, Cursor, Gemini CLI, Claude Code) when you want typed tools instead of a skill

All three use the same credential: a Modelflare Smart API Key in the MODELFLARE_API_KEY environment variable.

Step 1: Create a Smart API Key

An ordinary Modelflare key is bound to one routing group and can only call the models that group serves. A Smart API Key is bound to auto: it can call every model your account has access to and resolves the routing group per request. For media work this matters, because image and video models are rarely all served by the same group.

Sign up or open the console.
Go to API Keys and choose Create Smart API Key.
Pick a routing strategy: Balanced (default), Stability first, or Low price first. You can change it on the existing key later; the next request uses the new order.

A request is billed with the multiplier of the group that actually served it, and the consume log records the resolved group and strategy (Usage and cost).

Then expose the key to the environment Claude Code runs in:

# macOS / Linux: add this to your shell profile to keep it
export MODELFLARE_API_KEY="sk-..."

# Windows PowerShell: the second line makes it persistent
$env:MODELFLARE_API_KEY = "sk-..."
setx MODELFLARE_API_KEY "sk-..."

The Media Skill never writes the key into files or prompts and never prints it. Keep it that way on your side: do not paste it into a chat, a commit, or a ticket.

If Claude Code itself does not run through Modelflare yet and you want it to, the Claude Code setup guide covers the installer (curl -fsSL https://modelflare.dev/install/claude.sh | sh) and the manual ~/.claude/settings.json. That is optional: media generation works whatever model Claude Code is using for text.

Step 2a: Install the Claude Code plugin (recommended)

The plugin is a Claude Code marketplace hosted on GitHub. Two commands install it:

claude plugin marketplace add castaly-ai/modelflare-claude-code
claude plugin install modelflare@modelflare

Both commands print a confirmation (âœ” Successfully added marketplace: modelflare and âœ” Successfully installed plugin: modelflare@modelflare (scope: user)). The plugin carries the modelflare-media skill, so after installing there is nothing else to configure besides MODELFLARE_API_KEY. To pick up a newer version later:

claude plugin update modelflare@modelflare

Step 2b: Or install the Media Skill directly

If you would rather not use the plugin system, or you also want the skill in Codex, the docs offer an install that works for any agent. The simplest version is to ask the agent itself:

Install this skill: https://modelflare.dev/skills/modelflare-media/modelflare-media.zip (or use the attached zip), then read its SKILL.md. If you do not support skills, extract it anywhere and follow SKILL.md. My key is in the environment variable MODELFLARE_API_KEY.

Or place the files yourself:

curl -fsSL https://modelflare.dev/skills/modelflare-media/install.sh | sh

irm https://modelflare.dev/skills/modelflare-media/install.ps1 | iex

Both install into ~/.claude/skills/modelflare-media, which Claude Code reads as a user-level skill. A project-level copy goes into .claude/skills/ inside the repository. For Codex, set SKILL_DIR first:

SKILL_DIR=~/.codex/skills curl -fsSL https://modelflare.dev/skills/modelflare-media/install.sh | sh

The helper script needs only Node.js 18 or newer. You can read the SKILL.md before installing to see exactly what the agent will follow.

Install either the plugin or the standalone skill, not both, so Claude Code does not see two copies of the same skill.

Step 2c: Or add the MCP server

The MCP server exposes media generation as explicit tools. Its README gives this client configuration:

{
  "mcpServers": {
    "modelflare": {
      "command": "npx",
      "args": ["-y", "@modelflare/mcp-server"],
      "env": {
        "MODELFLARE_API_KEY": "<your Smart API Key>"
      }
    }
  }
}

In Claude Code you can put that block in a project .mcp.json, or register it from the terminal with Claude Code's own claude mcp add command:

claude mcp add --env MODELFLARE_API_KEY="$MODELFLARE_API_KEY" \
  --transport stdio modelflare -- npx -y @modelflare/mcp-server

Claude Code confirms the registration and health-checks the server:

$ claude mcp add --env MODELFLARE_API_KEY="$MODELFLARE_API_KEY" \
    --transport stdio modelflare -- npx -y @modelflare/mcp-server
Added stdio MCP server modelflare with command: npx -y @modelflare/mcp-server to local config

$ claude mcp list
Checking MCP server healthâ€¦

modelflare: npx -y @modelflare/mcp-server - âœ” Connected

By default this is a local-scope entry, private to you in the current project. Remove it with claude mcp remove modelflare -s local.

The npm package (current version 0.1.1) declares node >=20.3. It reads four environment variables:

Variable	Value
`MODELFLARE_API_KEY`	Your Smart API Key (required)
`MODELFLARE_BASE_URL`	`global` â†’ `https://modelflare.dev/v1` (default), `china` â†’ `https://origin.modelflare.dev/v1`, or a custom URL
`MODELFLARE_OUTPUT_DIR`	Where images and videos are saved (default `./modelflare-output`)
`MODELFLARE_LOCALE`	`en` (default) or `zh` for error messages

For Claude Desktop, the GitHub releases include a modelflare-mcp-<version>.mcpb bundle that asks for the key, base URL and output folder on open.

Worked example: a hero image and a looping header video

Here is the workflow we use for a small product landing page, with Claude Code started in the project root and the plugin installed. Under it is a real run of the MCP tool behind prompt 1.

Prompt 1:

Generate a 16:9 poster of a coastal observatory at dusk with gpt-image-2 and save it to public/hero.png.

With the skill, Claude Code reads SKILL.md, fetches the live catalog from https://modelflare.dev/api/pricing to confirm the key can use gpt-image-2 and what it costs, opens the gpt-image-2 model page for the exact request body and limits, and calls POST /v1/images/generations with its own HTTP client. Every call carries the header X-Modelflare-Skill: modelflare-media/1.0, so you can filter those requests in the usage logs.

With the MCP server, the same sentence becomes a generate_image tool call. Model-specific fields go in parameters and are sent unchanged, and the result is saved immediately because result URLs expire. Here is a real generate_image call from our test run, at the cheapest quality setting the gpt-image-2 page documents:

tool: generate_image
arguments:
  model: gpt-image-2
  prompt: A 16:9 poster of a coastal observatory at dusk, flat illustration
  parameters: { "size": "1536x1024", "quality": "low" }

result:
  Saved 1 image(s):
  ./modelflare-output/gpt-image-2-2026-10-14T14-22-22-789Z.png

It took about 35 seconds and produced a 1.9 MB PNG. list_models on the same key returned image and video models, each with its docs URL:

[
  { "id": "gpt-image-2", "kind": "image", "docs": "https://docs.modelflare.dev/models/image/gpt-image-2/" },
  { "id": "seedance-2.0-fast", "kind": "video", "docs": "https://docs.modelflare.dev/models/video/seedance-2-0-fast/" },
  { "id": "wan2.7-image", "kind": "image", "docs": "https://docs.modelflare.dev/models/image/wan2-7-image/" }
]

(Trimmed to three entries; the full list follows your account's access.)

If the upstream is briefly unavailable, the tool returns the error with a request ID rather than failing silently. On our first attempt we got exactly that, and the retry a minute later succeeded. Quote the request ID if you contact support.

Prompt 2:

Turn public/hero.png into a 5 second 1080p video with Seedance and save it as public/hero-loop.mp4.

Video is asynchronous. With the MCP server you can watch the three steps happen: create_video starts the task and returns its ID, get_video_status reports progress, and download_video saves the finished file (it can wait up to 300 seconds first). With the skill, Claude Code performs the same create-poll-save cycle itself, using the contract from the model page, for example seedance-2.0 or seedance-2.0-fast.

The local start frame is embedded in the request as a data URL. The skill does not upload files to any storage, so keep reference images small (a few MB).

Prompt 3, before committing to a model:

Which video models under 0.10 USD per second can do image-to-video? Show the price per 5-second clip.

Because the catalog is read live, the answer reflects today's prices and today's models, not what was true when the skill was installed.

Tips

Name the model when it matters. "With gpt-image-2" or "with Seedance" pins the choice. Leave it out and the agent picks from what your key can reach, which is fine for drafts but makes results harder to reproduce. Use exact IDs from Models & Pricing; the docs list 14 image models and 28 video models.

Ask for a quote first on video. Video is billed per second or per token depending on the model (see the video models table). A one-line "quote this before running it" saves surprises.

Keep output out of git by default. The MCP server writes to ./modelflare-output unless you set MODELFLARE_OUTPUT_DIR. Add that folder to .gitignore and move only the assets you keep into public/.

Reproduce work from the browser. Every generation in the Castaly workbench has a "Reproduce this in your own agent" panel with the install message and a task brief. Paste the brief into Claude Code and it repeats the same model and parameters against your key.

Read errors literally.

401: the key is missing or invalid. Create a Smart API Key at modelflare.dev/keys.
model_not_found / model_unavailable: a regular key only calls its own group's models. With a Smart API Key, the model is not available to your account right now. The skill cannot unlock model groups.
403 insufficient_user_quota: top up at modelflare.dev/billing.
unsupported_endpoint: the request went to the wrong contract (for example, a Gemini image model sent to /v1/images/generations).

Know what the skill does not do. It does not decide retries, batching or what to report back; that is between you and your agent. It only documents the API.

Where to go next

Media Skill guide: docs.modelflare.dev/guides/clients/media-skill
Claude Code setup: docs.modelflare.dev/guides/clients/claude-code
Smart API Key: docs.modelflare.dev/guides/features/smart-api-key
Image models: docs.modelflare.dev/models/image Â· Video models: docs.modelflare.dev/models/video
Claude Code plugin: github.com/castaly-ai/modelflare-claude-code
MCP server: npmjs.com/package/@modelflare/mcp-server Â· github.com/castaly-ai/modelflare-mcp

The point is not that Claude Code can now draw. It is that the asset step stops being a context switch: one key, one sentence, one file in the place your code expects it.

Disclosure: I'm part of the Modelflare team, which builds the AI API gateway mentioned in this article.

Function Calling: Responses API vs Chat Completions

Magnus — Tue, 13 Oct 2026 05:38:42 +0000

A wire-level comparison of function definitions, Call IDs, result messages, streaming arguments, authorization, idempotency, and route compatibility.

Responses and Chat Completions can both ask an application to run a function, but they represent the tool loop differently. Chat Completions puts tool definitions under tools[].function, returns message.tool_calls, and accepts results as role: "tool" messages. Responses uses flat function definitions, returns typed function_call output items, and accepts function_call_output items linked by call_id.

The model does not execute your application function. Your code must validate the proposed arguments, authorize the operation, execute it exactly as intended, return a result to the model, and prevent duplicate side effects when a request is retried.

The tool loop has four steps

The endpoint-specific JSON differs, but the application state machine is the same:

1. Declare an allowed function and its argument schema
2. Receive one or more model-proposed function calls
3. Validate, authorize, and execute each call in application code
4. Return each result using the call's correlation ID

Only after step four can the model produce an answer grounded in the tool result. If the model requests another tool, the loop repeats with a new call ID.

The OpenAI Function Calling guide describes this as a multi-step exchange. Treating the first tool call as a final answer is a common integration error.

Compare the wire contracts

The main differences are structural rather than conceptual.

Concern	Responses API	Chat Completions
Function definition	Flat `tools[]` item with `name`, `description`, `parameters`, `strict`	`tools[]` item with those fields nested under `function`
Model-proposed call	Typed `function_call` output item	Assistant message `tool_calls[]` entry
Correlation identifier	`call_id`	Tool call `id`, returned as `tool_call_id`
Function name	`function_call.name`	`tool_calls[].function.name`
Arguments	JSON string in `function_call.arguments`	JSON string in `tool_calls[].function.arguments`
Tool result	`function_call_output` input item	Message with `role: "tool"`
Common streamed arguments	Typed function-argument delta events	Indexed `delta.tool_calls[]` fragments
Final text	Typed output items and output-text helpers	`choices[0].message.content`

Do not correlate calls by array position. Parallel calls and streamed chunks can arrive in an order that differs from the order in which your application finishes. The explicit call ID is the stable join key.

Define one strict function schema

The following examples expose a read-only get_delivery_status function. The model may request an order status, but it cannot change an order, issue a refund, or access arbitrary storage.

The argument schema is deliberately narrow:

{
  "type": "object",
  "properties": {
    "order_id": {
      "type": "string",
      "pattern": "^ORDER-[0-9]{4}$"
    }
  },
  "required": [
    "order_id"
  ],
  "additionalProperties": false
}

strict: true asks the supported model to adhere to the function argument schema. The application must still parse and validate the JSON before execution. Pattern support and other JSON Schema features can vary by model or provider, so the same compatibility gate used for Structured Outputs also applies to function arguments.

Implement the loop with the Responses API

This JavaScript example keeps the exchange stateless by returning the model’s output items with the function result. It uses a deterministic local function so the control flow is visible without introducing another external API.

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.MODELFLARE_API_KEY,
  baseURL: "https://modelflare.dev/v1",
});

const model = process.env.MODEL_ID;
if (!model) throw new Error("MODEL_ID is required");

const orders = new Map([
  ["ORDER-1001", { status: "in_transit", eta: "2026-08-06" }],
]);

const tools = [
  {
    type: "function",
    name: "get_delivery_status",
    description: "Return the current delivery status for one order.",
    parameters: {
      type: "object",
      properties: {
        order_id: { type: "string", pattern: "^ORDER-[0-9]{4}$" },
      },
      required: ["order_id"],
      additionalProperties: false,
    },
    strict: true,
  },
];

function executeTool(name, rawArguments) {
  if (name !== "get_delivery_status") {
    throw new Error(`Tool is not allowed: ${name}`);
  }

  const args = JSON.parse(rawArguments);
  if (!/^ORDER-[0-9]{4}$/.test(args.order_id)) {
    throw new Error("Invalid order_id");
  }

  return orders.get(args.order_id) ?? { status: "not_found" };
}

const first = await client.responses.create({
  model,
  input: "Where is ORDER-1001?",
  tools,
  parallel_tool_calls: false,
});

const calls = first.output.filter((item) => item.type === "function_call");
if (calls.length === 0) {
  console.log(first.output_text);
  process.exit(0);
}

const toolOutputs = calls.map((call) => ({
  type: "function_call_output",
  call_id: call.call_id,
  output: JSON.stringify(executeTool(call.name, call.arguments)),
}));

const final = await client.responses.create({
  model,
  input: [...first.output, ...toolOutputs],
  tools,
  parallel_tool_calls: false,
});

console.log(final.output_text);

For a stateful integration, Responses also supports continuing from a prior response where the route and storage policy permit it. A stateless example makes the complete transcript explicit and avoids assuming that every OpenAI-compatible route stores response state.

The critical fields are call.call_id on the model output and the matching call_id on function_call_output. Returning a result with the wrong ID disconnects the data from the model’s request.

Implement the loop with Chat Completions

The same application function can be used with Chat Completions. The tool definition and result message use the Chat wrapper.

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.MODELFLARE_API_KEY,
  baseURL: "https://modelflare.dev/v1",
});

const model = process.env.MODEL_ID;
if (!model) throw new Error("MODEL_ID is required");

const orders = new Map([
  ["ORDER-1001", { status: "in_transit", eta: "2026-08-06" }],
]);

const tools = [
  {
    type: "function",
    function: {
      name: "get_delivery_status",
      description: "Return the current delivery status for one order.",
      parameters: {
        type: "object",
        properties: {
          order_id: { type: "string", pattern: "^ORDER-[0-9]{4}$" },
        },
        required: ["order_id"],
        additionalProperties: false,
      },
      strict: true,
    },
  },
];

function executeTool(name, rawArguments) {
  if (name !== "get_delivery_status") {
    throw new Error(`Tool is not allowed: ${name}`);
  }

  const args = JSON.parse(rawArguments);
  if (!/^ORDER-[0-9]{4}$/.test(args.order_id)) {
    throw new Error("Invalid order_id");
  }

  return orders.get(args.order_id) ?? { status: "not_found" };
}

const messages = [
  { role: "user", content: "Where is ORDER-1001?" },
];

const first = await client.chat.completions.create({
  model,
  messages,
  tools,
  parallel_tool_calls: false,
});

const assistant = first.choices[0].message;
const calls = assistant.tool_calls ?? [];
if (calls.length === 0) {
  console.log(assistant.content);
  process.exit(0);
}

messages.push(assistant);
for (const call of calls) {
  const result = executeTool(call.function.name, call.function.arguments);
  messages.push({
    role: "tool",
    tool_call_id: call.id,
    content: JSON.stringify(result),
  });
}

const final = await client.chat.completions.create({
  model,
  messages,
  tools,
  parallel_tool_calls: false,
});

console.log(final.choices[0].message.content);

Here, the first assistant message must be appended to messages before its tool results. Each result uses tool_call_id: call.id. Omitting the assistant tool-call message or returning an unmatched ID produces an invalid conversation history.

Stream arguments as fragments, not complete JSON

In streaming mode, function arguments can arrive in fragments. A chunk that contains {"order is not invalid JSON; it is incomplete JSON.

The safe parser keeps one buffer per call identity:

identify the call or indexed call slot from the event;
append each argument delta to that call’s buffer;
wait for the endpoint’s argument-done or completed signal;
parse the complete string once;
validate and execute only after completion.

Responses uses typed events for function-call arguments and output items. Chat Completions emits fragments in choices[].delta.tool_calls[], where the index identifies the in-progress entry and the final tool call carries its stable ID. Do not execute a function when the first fragment arrives.

The AI API Streaming Guide explains why connection, first event, first effective output, idle timeout, and completion should be measured separately.

Make tool execution safe and idempotent

Strict arguments improve structure, not authorization. Before executing any call:

allow only registered function names;
parse arguments with a size limit;
validate the complete schema in application code;
authorize the current user or workload for the requested resource;
separate read-only tools from tools with side effects;
redact secrets and sensitive output before returning it to the model;
use a deadline and bounded downstream retries;
record a safe operation reference, not raw secrets or private content.

For a side-effecting tool, use an idempotency key derived from a stable application request and call identity. Store the completed result before returning it to the model. If a network retry replays the same call, return the stored result rather than issuing a second refund, message, deployment, or database mutation.

The model-generated function name and arguments are untrusted input. A valid schema cannot grant permission, prove ownership, or replace a transaction boundary.

Decide how to handle multiple calls

The examples set parallel_tool_calls: false to keep the state machine easy to inspect. If parallel calls are enabled:

correlate every result by call ID, never by completion order;
enforce a maximum number of calls per response;
apply a concurrency limit in the application;
define whether one failed tool cancels, blocks, or coexists with successful tools;
return one result for every accepted call;
keep side-effecting operations serialized when their order matters.

Parallel execution can reduce latency for independent reads, but it increases authorization, ordering, retry, and partial-failure complexity. Enable it because the workload benefits, not because the parameter exists.

Understand the Modelflare compatibility boundary

Modelflare’s OpenAI compatibility code preserves strict function definitions and maps the supported function-call shapes between Responses and Chat Completions on eligible OpenAI/Codex routes. That mapping is deliberately narrower than the complete Responses tool surface.

Application-defined function tools can be represented in both formats. Responses-only hosted tools—such as provider-run search, code execution, or other built-in tool types—are not equivalent to application function calling and cannot be assumed to convert into Chat Completions.

Other OpenAI-compatible model families are exposed through raw Chat Completions pass-through unless separately verified. For those routes, the upstream provider decides whether tools, strict, parallel calls, streaming argument fragments, and specific tool_choice shapes are supported.

Before enabling a route, run the complete loop rather than checking only whether the first request returns 200:

Test	Required evidence
One read-only call	Function name, parsed arguments, call ID, result linkage, final text
Invalid arguments	Explicit validation failure with no tool execution
Unknown function	Rejected by the application allowlist
No tool required	Normal final text without a fabricated call
Streaming call	Complete reconstructed arguments and correct call ID
Repeated request	One side effect or one cached result, not duplicate execution
Multiple calls	Correct correlation independent of completion order
Route fallback	Same model, protocol, tool schema, and result contract preserved

Use Reliable AI API Routing to keep route changes explicit, and AI API Key Security to separate workloads and permissions.

Choose the format by the application contract

Use Responses when the application benefits from typed output items, the Responses event model, state-continuation options, or other verified Responses capabilities. Use Chat Completions when the application already owns a stable message transcript and the selected provider’s tool contract is verified. The broader endpoint decision is covered in Responses API vs Chat Completions.

The implementation standard should be the same in both cases: explicit tool allowlists, strict argument schemas, application-side validation, authorization before execution, correlation by call ID, idempotency for side effects, and a complete result returned to the model. Those controls—not the endpoint name—make a function-calling workflow reliable.

Originally published at modelflare.dev.

Disclosure: I'm part of the Modelflare team, which builds the AI API gateway mentioned in this article.