DEV Community: llmbrake

Per-customer budget caps for Anthropic API calls (with prompt caching priced right)

llmbrake — Mon, 02 Nov 2026 14:56:30 +0000

If your product calls Claude on behalf of your customers, one heavy account can spend a month's margin in a weekend. Anthropic's console limits protect your organization. They don't know which of your customers sent the traffic, so they can't stop one of them without stopping everyone.

This post builds a per-customer monthly cap for the Anthropic Messages API in TypeScript:

Price each call correctly, including prompt-cache reads and writes.
Check the budget before the call, in a way that holds up under concurrent requests.
Switch to a cheaper model near the cap, and refuse at the cap.

It uses @anthropic-ai/sdk and Redis (ioredis). The ideas carry over to any store with atomic operations.

1. Pricing a Messages API call

A Claude response's usage has four token counts that matter for cost:

Field	What it is	Billed at
`input_tokens`	input after the last cache breakpoint	base input price
`cache_read_input_tokens`	prefix read from the prompt cache	0.1x base for most models (0.05x for Opus 5.5 and Sonnet 5.5)
`cache_creation_input_tokens`	prefix written to the cache this call	1.25x base (5-minute TTL), 2x (1-hour TTL)
`output_tokens`	generated tokens, including thinking	output price

The trap is that input_tokens is not the total input. Total input is input_tokens + cache_read_input_tokens + cache_creation_input_tokens. If you only price input_tokens, a cached chat app looks almost free. And if you ignore cache writes, you under-count the first call of every conversation by 25% or more on its prefix.

When the 1-hour TTL is in play, usage.cache_creation splits writes into ephemeral_5m_input_tokens and ephemeral_1h_input_tokens, so you can price them separately.

// prices.ts: USD per 1M tokens, standard tier.
// From https://docs.anthropic.com/en/docs/about-claude/pricing, read 2026-11-02. Check before relying on these.
type Price = { input: number; output: number; cacheRead: number; cacheWrite5m: number; cacheWrite1h: number };

export const PRICES: Record<string, Price> = {
  "claude-sonnet-5-5": { input: 2.0, output: 10.0, cacheRead: 0.1,  cacheWrite5m: 2.5,   cacheWrite1h: 4.0 },
  // Haiku 5.5 has a higher price for prompts over 100K tokens; this is the up-to-100K tier.
  "claude-haiku-5-5":  { input: 0.1, output: 0.5,  cacheRead: 0.01, cacheWrite5m: 0.125, cacheWrite1h: 0.2 },
};

export interface ClaudeUsage {
  input_tokens: number;
  output_tokens: number;
  cache_read_input_tokens?: number | null;
  cache_creation_input_tokens?: number | null;
  cache_creation?: { ephemeral_5m_input_tokens?: number; ephemeral_1h_input_tokens?: number } | null;
}

export function costUsd(model: string, u: ClaudeUsage): number {
  const p = PRICES[model];
  if (!p) throw new Error(`No price for ${model}; add it to PRICES`);
  const write1h = u.cache_creation?.ephemeral_1h_input_tokens ?? 0;
  const write5m = u.cache_creation?.ephemeral_5m_input_tokens ?? (u.cache_creation_input_tokens ?? 0) - write1h;
  return (
    u.input_tokens * p.input +
    (u.cache_read_input_tokens ?? 0) * p.cacheRead +
    write5m * p.cacheWrite5m +
    write1h * p.cacheWrite1h +
    u.output_tokens * p.output
  ) / 1_000_000;
}

Not modeled here: batch discounts, long-context tiers, data-residency uplifts and server-tool fees (web search and the like). If you use them, add them; otherwise your cap is a floor, not a ceiling.

Throwing on an unknown model is deliberate. Silently pricing a new model at $0 is how caps stop working without anyone noticing.

2. Why "check, then record" isn't enough

The simple version reads the customer's spend, compares it to the budget, makes the call, and adds the cost afterwards. That's fine for one request at a time. But an agent loop or a user with three tabs open can have ten requests in flight. All ten read the same "under budget" total, all ten go through, and the customer ends up well past their cap.

The fix is to reserve an estimate before the call, atomically with the check, then settle with the real cost afterwards:

before:  if spent + reserved + estimate > budget â†’ refuse
         else reserved += estimate                  (one atomic step)
call Claude
after:   reserved -= estimate; spent += actual

Parallel requests now see each other's holds, so the overshoot is bounded by how wrong one estimate is, instead of by everything in flight.

Estimating a call before you make it

The upper bound on a call's cost is its input at full price plus max_tokens of output. You can get the input size from the token-counting endpoint:

const { input_tokens } = await anthropic.messages.countTokens({ model, system, messages });
const estimate = (input_tokens * price.input + maxTokens * price.output) / 1_000_000;

That's an extra round trip per call. If it's too slow for you, JSON.stringify(messages).length / 4 is a rough stand-in. Either way, pricing all input at the full rate over-estimates cached prompts, which is the safe direction for a hold.

Atomic reserve and settle in Redis

Two keys per customer per month: spent and held. A Lua script makes the check and the hold one step:

// budget.ts
import Redis from "ioredis";

const redis = new Redis(process.env.REDIS_URL!);
const TTL = 60 * 60 * 24 * 40; // let old months expire on their own

const period = () => new Date().toISOString().slice(0, 7); // "2026-11", UTC
const keys = (customerId: string) => [
  `budget:${period()}:${customerId}:spent`,
  `budget:${period()}:${customerId}:held`,
];

const RESERVE = `
local spent = tonumber(redis.call('GET', KEYS[1]) or '0')
local held  = tonumber(redis.call('GET', KEYS[2]) or '0')
if spent + held + tonumber(ARGV[1]) > tonumber(ARGV[2]) then return 0 end
redis.call('INCRBYFLOAT', KEYS[2], ARGV[1])
redis.call('EXPIRE', KEYS[2], ARGV[3])
return 1`;

const SETTLE = `
redis.call('INCRBYFLOAT', KEYS[2], ARGV[1])
redis.call('INCRBYFLOAT', KEYS[1], ARGV[2])
redis.call('EXPIRE', KEYS[1], ARGV[3])
redis.call('EXPIRE', KEYS[2], ARGV[3])
return 1`;

export async function usedFraction(customerId: string, budgetUsd: number): Promise<number> {
  const [spent, held] = await redis.mget(...keys(customerId));
  return (Number(spent ?? 0) + Number(held ?? 0)) / budgetUsd;
}

export async function reserve(customerId: string, estimateUsd: number, budgetUsd: number): Promise<boolean> {
  const ok = await redis.eval(RESERVE, 2, ...keys(customerId), estimateUsd, budgetUsd, TTL);
  return ok === 1;
}

export async function settle(customerId: string, estimateUsd: number, actualUsd: number): Promise<void> {
  await redis.eval(SETTLE, 2, ...keys(customerId), -estimateUsd, actualUsd, TTL);
}

(keys() is computed per call, so a request that starts at 23:59 UTC on the 31st and settles after midnight will settle into the new month. If that bothers you, compute the keys once and pass them through.)

3. Putting it around the call

// claude.ts
import Anthropic from "@anthropic-ai/sdk";
import { PRICES, costUsd } from "./prices";
import { reserve, settle, usedFraction } from "./budget";

const anthropic = new Anthropic();

export class BudgetExceededError extends Error {
  constructor(public customerId: string) {
    super(`AI budget reached for ${customerId}`);
  }
}

const CHEAPER: Record<string, string> = { "claude-sonnet-5-5": "claude-haiku-5-5" };

export async function createForCustomer(
  customerId: string,
  budgetUsd: number,
  params: Anthropic.MessageCreateParamsNonStreaming,
): Promise<Anthropic.Message> {
  // Near the cap: same request, cheaper model.
  let model = params.model;
  if (CHEAPER[model] && (await usedFraction(customerId, budgetUsd)) >= 0.8) model = CHEAPER[model];

  const price = PRICES[model];
  if (!price) throw new Error(`No price for ${model}`);
  const { input_tokens } = await anthropic.messages.countTokens({
    model, system: params.system, messages: params.messages, tools: params.tools,
  });
  const estimate = (input_tokens * price.input + params.max_tokens * price.output) / 1_000_000;

  if (!(await reserve(customerId, estimate, budgetUsd))) throw new BudgetExceededError(customerId);

  let actual = 0;
  try {
    const msg = await anthropic.messages.create({ ...params, model });
    actual = costUsd(msg.model, msg.usage);
    return msg;
  } finally {
    // Failed calls release their hold at $0.
    await settle(customerId, estimate, actual);
  }
}

And in a route handler:

try {
  const msg = await createForCustomer(user.id, user.plan === "pro" ? 20 : 2, {
    model: "claude-sonnet-5-5",
    max_tokens: 1024,
    messages: [{ role: "user", content: prompt }],
  });
  res.json(msg);
} catch (e) {
  if (e instanceof BudgetExceededError) {
    return res.status(402).json({ error: "monthly_ai_limit", message: "You've reached this month's AI limit." });
  }
  throw e;
}

A specific error code lets your UI show "upgrade or wait until the 1st" instead of a generic failure. That's the difference between a cap and an outage.

Streaming

With anthropic.messages.stream(...), input and cache counts arrive in the message_start event and the final output count in message_delta. The SDK's await stream.finalMessage() gives you the assembled usage, so price it the same way and call settle in a finally. If the client disconnects and you abort the stream, you may not get final usage; settling at the estimate in that case errs on the side of the cap.

Decisions you still have to make

Fail open or closed? If Redis is down, do you let calls through (unmetered spend) or refuse them (feature outage)? Most products should fail open and alert. Wrap reserve in a try/catch and make it a conscious choice.
Leaked holds. If a process dies between reserve and settle, its hold stays until the month's keys expire. For most apps that's a few cents. If it isn't for you, store holds individually with a timestamp and sweep old ones.
Estimate size. A large max_tokens makes big holds, which can refuse a customer who has a little budget left. Set max_tokens to what the feature really needs.
What "cheaper model" means. Test the fallback model on the feature before wiring it in. For extraction and short answers it's usually fine; for long reasoning it may not be.

If you'd rather not write this

Disclosure: I maintain llmbrake, an MIT-licensed TypeScript library for this pattern. It wraps OpenAI-compatible chat.completions calls, so with Claude it works through Anthropic's OpenAI-compatible endpoint rather than the native Messages API shown above, and its default price table doesn't price cache writes separately. If you use the native SDK and prompt caching heavily, the code in this post is the more accurate option.

Tracking LLM cost per customer in Node.js with AsyncLocalStorage

llmbrake — Mon, 02 Nov 2026 14:56:08 +0000

Your OpenAI invoice tells you what you spent. It doesn't tell you which customer you spent it on. If you sell a flat-price plan on top of an LLM, that second number is the one that decides whether a customer is profitable.

The usual first attempt is to add a customerId parameter to every function that might end up calling the model. That works until the call sits five layers down, inside a retry helper, a summarizer, and a background job. This post shows a smaller approach:

Put the customer on the request's async context once, in middleware.
Wrap the OpenAI client once, so every call reads that context and logs its cost.
Store one row per call and use SQL for the questions you actually have.

Everything here is plain Node 18+ and the official openai package. No proxy, no extra service.

1. Put the customer on the async context

Node's AsyncLocalStorage carries a value through every await, timer and promise that starts inside a run() call. That's exactly the scope of "this HTTP request".

// context.ts
import { AsyncLocalStorage } from "node:async_hooks";

export interface CostContext {
  customerId: string;
  plan?: string;
  feature?: string; // "chat", "summarize", "search"... useful later
}

export const costContext = new AsyncLocalStorage<CostContext>();

export function currentCostContext(): CostContext | undefined {
  return costContext.getStore();
}

In Express (any framework with middleware works the same way):

app.use((req, res, next) => {
  const user = req.user; // whatever your auth sets
  if (!user) return next();
  costContext.run({ customerId: user.id, plan: user.plan }, next);
});

To tag a feature, nest another run() around the code that implements it:

export function withFeature<T>(feature: string, fn: () => Promise<T>): Promise<T> {
  const parent = currentCostContext();
  if (!parent) return fn();
  return costContext.run({ ...parent, feature }, fn);
}

// in a route handler
const summary = await withFeature("summarize", () => summarizeThread(thread));

Background jobs don't go through your middleware, so start a context there too, using the customer the job belongs to:

queue.process(async (job) =>
  costContext.run({ customerId: job.data.customerId, feature: "nightly-digest" }, () =>
    runDigest(job.data),
  ),
);

2. Price each call

Token counts aren't comparable across models, so convert to dollars at write time. Keep prices in one table with the date you read them, because they change.

// prices.ts: USD per 1M tokens. Check https://developers.openai.com/api/docs/pricing before relying on these.
export const PRICES: Record<string, { input: number; cachedInput: number; output: number }> = {
  "gpt-5.5":      { input: 5.0,  cachedInput: 0.5,   output: 30.0 },
  "gpt-5.4-mini": { input: 0.75, cachedInput: 0.075, output: 4.5 },
};

export function priceFor(model: string) {
  if (PRICES[model]) return PRICES[model];
  // Responses report dated snapshots like "gpt-5.4-mini-2026-03-17"; fall back to the base name.
  const base = model.replace(/-\d{4}-\d{2}-\d{2}$/, "");
  return PRICES[base];
}

export function costUsd(model: string, usage: {
  prompt_tokens: number;
  completion_tokens: number;
  prompt_tokens_details?: { cached_tokens?: number } | null;
}): number | null {
  const p = priceFor(model);
  if (!p) return null; // unknown model: log it, don't guess
  const cached = usage.prompt_tokens_details?.cached_tokens ?? 0;
  const uncached = usage.prompt_tokens - cached;
  return (uncached * p.input + cached * p.cachedInput + usage.completion_tokens * p.output) / 1_000_000;
}

Two details that cause most wrong numbers:

Cached input tokens are included in prompt_tokens and billed at a lower rate. If you price all of prompt_tokens at the full rate, you'll over-count, sometimes by a lot for chat apps with long system prompts.
Unknown models should produce null, not zero. A zero looks like a free call and hides the gap; a null shows up in a query.

3. Wrap the client once

// openai.ts
import OpenAI from "openai";
import { currentCostContext } from "./context";
import { costUsd } from "./prices";
import { recordUsage } from "./usage-log";

const client = new OpenAI();

export async function chat(params: OpenAI.Chat.ChatCompletionCreateParamsNonStreaming) {
  const res = await client.chat.completions.create(params);
  const ctx = currentCostContext();
  if (res.usage) {
    // Don't let logging failures break the user's request.
    recordUsage({
      customerId: ctx?.customerId ?? "unattributed",
      plan: ctx?.plan,
      feature: ctx?.feature,
      model: res.model,
      promptTokens: res.usage.prompt_tokens,
      cachedTokens: res.usage.prompt_tokens_details?.cached_tokens ?? 0,
      completionTokens: res.usage.completion_tokens,
      costUsd: costUsd(res.model, res.usage),
    }).catch((err) => console.error("usage log failed", err));
  }
  return res;
}

Note res.model, not params.model: the response tells you the model that actually ran, which is what you're billed for.

Calls with no context get logged as "unattributed" rather than dropped. Watching that bucket is the quickest way to find code paths that bypass your middleware.

Streaming

With stream: true, usage is only sent if you ask for it, and it arrives in the last chunk:

export async function* chatStream(params: OpenAI.Chat.ChatCompletionCreateParamsStreaming) {
  const ctx = currentCostContext(); // read it before the first yield
  const stream = await client.chat.completions.create({
    ...params,
    stream_options: { ...params.stream_options, include_usage: true },
  });
  let usage: OpenAI.CompletionUsage | undefined;
  let model = params.model;
  try {
    for await (const chunk of stream) {
      if (chunk.usage) usage = chunk.usage;
      if (chunk.model) model = chunk.model;
      yield chunk;
    }
  } finally {
    // runs on normal completion *and* when the consumer stops reading early
    if (usage) {
      recordUsage({
        customerId: ctx?.customerId ?? "unattributed",
        plan: ctx?.plan,
        feature: ctx?.feature,
        model,
        promptTokens: usage.prompt_tokens,
        cachedTokens: usage.prompt_tokens_details?.cached_tokens ?? 0,
        completionTokens: usage.completion_tokens,
        costUsd: costUsd(model, usage),
      }).catch((err) => console.error("usage log failed", err));
    }
  }
}

If the client disconnects before the final chunk, there's no usage to record. Decide whether that matters for you; for most apps it's a small, known gap.

4. Store one row per call

create table llm_usage (
  id               bigserial primary key,
  at               timestamptz not null default now(),
  customer_id      text not null,
  plan             text,
  feature          text,
  model            text not null,
  prompt_tokens    int not null,
  cached_tokens    int not null,
  completion_tokens int not null,
  cost_usd         numeric(12, 6) -- null = model missing from the price table
);
create index on llm_usage (customer_id, at);

// usage-log.ts
import { pool } from "./db";

export async function recordUsage(u: {
  customerId: string; plan?: string; feature?: string; model: string;
  promptTokens: number; cachedTokens: number; completionTokens: number; costUsd: number | null;
}) {
  await pool.query(
    `insert into llm_usage (customer_id, plan, feature, model, prompt_tokens, cached_tokens, completion_tokens, cost_usd)
     values ($1, $2, $3, $4, $5, $6, $7, $8)`,
    [u.customerId, u.plan ?? null, u.feature ?? null, u.model, u.promptTokens, u.cachedTokens, u.completionTokens, u.costUsd],
  );
}

Store only metadata. You don't need prompts or completions to answer cost questions, and keeping them makes this table a privacy problem.

5. The queries you'll actually run

Top customers this month:

select customer_id, plan, sum(cost_usd) as cost, count(*) as calls
from llm_usage
where at >= date_trunc('month', now())
group by customer_id, plan
order by cost desc nulls last
limit 20;

Which feature is expensive:

select feature, model, sum(cost_usd) as cost, avg(prompt_tokens)::int as avg_prompt
from llm_usage
where at >= now() - interval '7 days'
group by feature, model
order by cost desc nulls last;

A high avg_prompt on one feature usually means it's sending a whole document or conversation history when a summary would do.

Cost vs. revenue, if you keep each customer's monthly price in a subscriptions table:

select u.customer_id, s.monthly_price_usd as revenue, sum(u.cost_usd) as ai_cost,
       s.monthly_price_usd - sum(u.cost_usd) as margin
from llm_usage u
join subscriptions s on s.customer_id = u.customer_id
where u.at >= date_trunc('month', now())
group by u.customer_id, s.monthly_price_usd
order by margin asc
limit 20;

The top rows of that last query are the conversation you need to have with your pricing page.

Models missing from your price table:

select model, count(*) from llm_usage where cost_usd is null group by model;

Where this stops

This setup answers "who cost what", after the fact. It doesn't stop a customer mid-month. For that you need to read the running total before each call and refuse, or switch to a cheaper model, when it crosses a budget. That check has to be fast and safe under concurrent requests, which is a different problem from logging (I wrote about it in How to stop one customer from blowing up your OpenAI bill).

Disclosure: I maintain llmbrake, a small MIT-licensed TypeScript library that does the pricing and the before-call budget check for OpenAI-compatible chat.completions calls. You don't need it for anything in this post; the code above is self-contained.

How to stop one customer from blowing up your OpenAI bill

llmbrake — Tue, 13 Oct 2026 00:24:26 +0000

If you run a SaaS on top of an LLM API, your provider bill is one number. OpenAI and Anthropic let you set limits per organization or project. Those limits protect your account, not your margins: they can't tell you which customer drove the spend, and they can't stop one customer without stopping everyone.

There are two ways this goes wrong:

The runaway user. Someone scripts your feature, or a client bug retries in a loop, and one account burns real money overnight.
The quietly unprofitable account. A customer on a $20 plan sends long contexts every day and costs you more in tokens than they pay. You can't see it because cost isn't attributed per customer.

Here's a pattern that fixes both, and the edge cases to watch for.

1. Price every call in dollars, per customer

Every chat completion returns token usage. Convert it to dollars immediately, because token counts aren't comparable across models:

cost = (uncachedInput * inputPrice + cachedInput * cachedInputPrice + output * outputPrice) / 1_000_000

Cached input tokens: OpenAI reports them in usage.prompt_tokens_details.cached_tokens, and they're billed at a lower rate. Ignore them and you'll over-count.
Dated snapshots: model names like gpt-5.4-mini-2026-03-17 should resolve to the base model's price.
Price table: keep it in one place, with a source link and the date you read it. Prices change. When I checked OpenAI's pricing page on 2026-10-09, it listed gpt-5.5 at $5 input / $30 output per 1M tokens and gpt-5.4-mini at $0.75 / $4.50. Verify before you rely on that.

2. Check before the call, record after it

before call:  spent = store.get(customer, month)
              if spent >= budget        → block
              if spent >= 0.8 * budget  → swap to a cheaper model
call provider
after call:   store.add(customer, month, cost)

The check has to sit in the request path, before the call. A nightly job that emails you the damage is forensics, not protection.

3. Handle streaming properly

With stream: true, OpenAI only reports usage if you set stream_options: { include_usage: true }, and it arrives in the final chunk. Wrap the async iterator, pass chunks through unchanged, and record the cost in a finally block so it still runs when the client stops reading early.

4. Degrade before you block

A hard cutoff is a bad experience for a paying customer. A softer first step works better: past 80% of the budget, route the same task to a cheaper model. Many users won't notice, and your margin is protected. At 100%, block with a specific error so your UI can say "You've reached this month's AI limit. Upgrade?" instead of showing a generic failure.

5. Use a store that handles concurrency

In-memory counters are fine for one process. With several instances or serverless functions, use an atomic increment such as Redis INCRBYFLOAT llmbrake:{month}:{customer} {cost}. Key it by month and budgets reset without a cron job; add an EXPIRE so old months clean themselves up.

Two honest caveats:

Overshoot: concurrent in-flight requests can each pass the check and then overshoot together, usually by the cost of those few calls. If you need a strict ceiling, reserve an estimated cost before the call and settle afterwards.
Store outages: decide whether to fail open (keep serving and risk some unmetered spend) or fail closed (protect spend and break the feature). For most products, failing open plus an alert is the right call.

6. Compare cost to revenue

Once spend is attributed per customer, join it with what they pay you, for example Stripe subscriptions. A table of customer, AI cost, revenue and margin is usually enough to spot plans that need repricing or a lower cap.

The library version

I packaged this pattern as LLMBrake, a small MIT-licensed TypeScript library (about 500 lines, zero runtime dependencies):

npm install llmbrake

import { createGuard, isBudgetExceededError } from "llmbrake";

const guard = createGuard({
  budgetUsd: (ctx) => (ctx.plan === "pro" ? 20 : 2),
  downgradeAt: 0.8,
  downgradeMap: { "gpt-5.5": "gpt-5.4-mini" },
});

const chat = openai.chat.completions;
const create = guard.wrap(chat.create.bind(chat));

const res = await create(
  { model: "gpt-5.5", messages: [{ role: "user", content: "Hello" }] },
  { customerId: user.id, plan: user.plan, feature: "chat" },
);
// at the cap → throws BudgetExceededError

It covers cached-token pricing, streaming usage, the downgrade step, a typed block error, metadata-only usage events, and a pluggable store with a fail-open default. Use it, or copy the pattern. There's an in-browser demo (real library, simulated provider): https://llmbrake.com/demo?ref=devto

Disclosure: I build LLMBrake. A hosted dashboard (shared counters, alerts, AI cost vs. revenue per customer) is in early access at app.llmbrake.com.