DEV Community: Quinn Li

Find the Concurrency Where Free Stops Being Cheaper

Quinn Li — Tue, 27 Oct 2026 13:20:52 +0000

Free capacity has no price, but it does have a shape. That shape bends at some concurrency level you have probably never measured, and everything above the bend is queue time you paid for with tokens you thought you saved.

The usual math is volume math. You multiply your monthly tokens by zero, compare it to a paid invoice, and declare a winner. The problem is that the comparison only holds while requests arrive one at a time. Put sixteen of them in flight and the free path stops being a discount and starts being a latency budget you are spending down.

I keep a small harness for exactly this question. It ramps concurrency against any OpenAI-compatible endpoint and prints one row per level: latency percentiles, retry rate, and an effective cost per thousand completions that charges you for waiting as well as for tokens. You run it, you get your numbers. Nobody can hand you these numbers from the outside.

# sweep.py — run with: BASE_URL=... API_KEY=... MODEL=... python sweep.py
import asyncio, os, statistics, time
import httpx

BASE = os.environ["BASE_URL"].rstrip("/")
KEY = os.environ["API_KEY"]
MODEL = os.environ["MODEL"]
PROMPT = os.environ.get("PROMPT", "Reply with one sentence about queueing.")
MAX_TOK = int(os.environ.get("MAX_TOK", "64"))
LEVELS = [1, 2, 4, 8, 16, 32]
PER_LEVEL = 64
HOURLY_RATE = float(os.environ.get("HOURLY_RATE", "60"))   # loaded cost of an engineer-hour
PRICE_IN = float(os.environ.get("PRICE_IN", "0"))          # per 1M tokens, pa
PRICE_OUT = float(os.environ.get("PRICE_OUT", "0"))        # per 1M tokens, paid path

async def one(client, sem):
    async with sem:
        attempts, t0 = 0, time.perf_counter()
        while True:
            attempts += 1
            try:
                r = await client.post(
                    f"{BASE}/chat/completions",
                    headers={"Authorization": f"Bearer {KEY}"},
                    json={"model": MODEL, "max_tokens": MAX_TOK,
                          "messages": [{"role": "user", "content": PROMPT}]},
                )
                if r.status_code in (429, 500, 502, 503, 504) and attempts < 3:
                    await asyncio.sleep(0.5 * attempts)
                    continue
                r.raise_for_status()
                u = r.json().get("usage") or {}
                return {"s": time.perf_counter() - t0, "attempts": attempts,
                        "tin": u.get("prompt_tokens"), "tout": u.get("completion_tokens")}
            except Exception:
                if attempts >= 3:
                    raise
                await asyncio.sleep(0.5 * attempts)

async def level(n):
    sem = asyncio.Semaphore(n)
    async with httpx.AsyncClient(timeout=120) as client:
        out = await asyncio.gather(*(one(client, sem) for _ in range(PER_LEVEL)),
                                   return_exceptions=True)
    ok = [o for o in out if isinstance(o, dict)]
    if not ok:
        return n, 0, None, None, None, None
    lat = sorted(o["s"] for o in ok)
    p50, p95 = statistics.median(lat), lat[max(0, int(len(lat) * 0.95) - 1)]
    retries = sum(o["attempts"] - 1 for o in ok) / len(ok)
    tin = sum(o["tin"] or 0 for o in ok)
    tout = sum(o["tout"] or 0 for o in ok)
    token_cost = ((tin / 1e6) * PRICE_IN + (tout / 1e6) * PRICE_OUT) * (1000 / len(ok))
    wait_cost = (sum(lat) / len(lat)) * (HOURLY_RATE / 3600) * 1000
    return n, len(ok) / len(out), p50, p95, retries, token_cost + wait_cost

async def main():
    print("conc,ok_rate,p50_s,p95_s,retries_per_req,eff_cost_per_1k")
    for n in LEVELS:
        print("%.2f" % n if False else n, *[("—" if v is None else round(v, 4)) for v in (await level(n))[1:]], sep=",")

asyncio.run(main())

Run it twice. Once against the free path, once against the paid path with the same prompt and the same PER_LEVEL. The absolute latencies matter less than the slope. If p95 grows roughly linearly with concurrency, you are queueing behind other people and the queue is not yours to size. If p95 stays flat until some level and then jumps, you have found the ceiling of the shared pool and you now know it by number instead of by vibe.

The eff_cost_per_1k column is where the argument actually gets settled. It charges token spend at your paid rates and wait time at your loaded hourly rate, which is the honest trade: a request that finishes in 0.4 seconds and one that finishes in nine seconds are not the same product, even when both cost zero tokens. Set PRICE_IN and PRICE_OUT to zero for the free path and you will see the crossover anyway, because wait_cost does not go to zero when the price does.

Three readings of the table change what you ship. If the crossover sits above your real peak concurrency, free is the right default and you should stop worrying about it. If the crossover sits below your peak, free is fine for batch and background work and wrong for anything a human is waiting on. If retries per request climbs with concurrency, stop reading the table as a cost question — that is a correctness question wearing a cost question's clothes, and a retry storm on a shared pool can outlast your deadline.

This is the part the volume math hides. Retries are not free capacity doubled; they are the same request billed twice in time, and time is the resource that free tiers are actually short of. A pool that answers every request eventually is not the same as a pool that answers every request in time for your build to pass.

Where does MonkeyCode fit in this? The project is described by its operator as open source, and it offers two things that make a sweep like the one above cheap to run: free model access and a free server option. That combination matters for the harness specifically — you can put sweep.py on the free box and point it at the free endpoint, so the measurement costs you an afternoon rather than an invoice.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

But run the sweep before you bet a deadline on it, not after. The interesting number is not "what does a token cost today" — it is "at what concurrency does this stop being the right tool", and that number belongs to your workload, your prompt lengths, and your latency tolerance. If you want a place to start measuring, the project's own docs are the place to confirm current terms, because availability and model rosters move faster than any article about them.

Some honest limits on all of this. The harness measures one prompt shape at one size; long prompts and long completions will bend the curve differently. Token counts come back in usage on most compatible endpoints but not all, and when they are missing the script silently prices them at zero — check that column before you trust the total. Your HOURLY_RATE is a judgment call, and picking it too high will make every free option look bad. And a shared free pool is shared: your 3 a.m. sweep is not your 3 p.m. reality.

Do not use this approach for interactive latency budgets you have already promised to users, for anything where a mid-transaction failure costs real money, or for compliance workloads where you need a contract rather than a convenience. Free capacity is a good default for batch jobs, internal tooling, and the measurement harness itself. It is a bad bet the moment a human is staring at a spinner, or the moment a retry loop can eat the margin the free tier just gave you.

Measure the bend. Then choose the side of it you can defend.

Cost Regression Should Fail the Build

Quinn Li — Mon, 26 Oct 2026 11:51:31 +0000

You will not notice a cost regression by reading an invoice. Invoices arrive after the traffic, after the incident review, and after the person who wrote the expensive loop has moved to another team. You need the build to tell you, on the commit that caused it, with a number and a threshold and someone's name on it.

That means treating cost per successful outcome as a test assertion. Same as a snapshot test, same as a schema check: a fixture, a threshold, a deterministic runner, and a red build when it moves.

The usual objection is that cost tests are expensive to run. They are — if you point them at a live endpoint. So don't measure live. Record once, replay forever, and put the gate behind the replay.

One run, five numbers

Start with a frozen record of a single run. Five fields matter, and wall clock is two of them.

# costgate/model.py
from dataclasses import dataclass
from fractions import Fraction

@dataclass(frozen=True)
class Run:
    id: str
    usd_model: Fraction     # metered model spend for this run
    usd_infra: Fraction     # server minutes, billed or amortized
    tokens_in: int
    tokens_out: int
    retries: int
    wall_ms: int
    queued_ms: int
    ok: bool

    @property
    def spend(self) -> Fraction:
        return self.usd_model + self.usd_infra

Two things matter here. Money is a Fraction, not a float, because a gate that compares 0.041000000000000004 to 0.041 will fail on a Tuesday for no reason. And ok is present because the denominator of your metric is not "runs" — it is "runs that actually produced the thing you sold."

The divisor is the whole test

Dividing spend by successes is the part people skip, and skipping it is why retry storms look free. When a call fails twice and succeeds on the third attempt, the numerator grows by three attempts of spend while the denominator grows by one. Nothing else in your observability captures that shape.

# costgate/gate.py
from fractions import Fraction

def cost_per_success(runs):
    spend = sum((r.spend for r in runs), Fraction(0))
    wins = sum(1 for r in runs if r.ok)
    return None if wins == 0 else spend / wins

def retry_share(runs):
    attempts = sum(r.retries + 1 for r in runs)
    return Fraction(sum(r.retries for r in runs), attempts)

def p95(values):
    if not values:
        return 0
    s = sorted(values)
    return s[max(0, int(round(0.95 * (len(s) - 1))))]   # nearest-rank

Note that cost_per_success returns None instead of raising when nothing succeeded. A cassette that fails every call should fail the gate with a readable message, not a ZeroDivisionError buried in a traceback.

Wire it to a budget file

Keep thresholds in version control next to the code they constrain, in a format a reviewer can diff.

# costgate/budget.toml
[gate]
max_cost_per_success_usd = "0.041"
max_retry_share          = "0.15"
max_p95_wall_ms          = 24000

Fraction("0.041") parses that string exactly, which is the reason for the quotes. Then the check itself stays boring:

# costgate/check.py
def check(runs, budget):
    failures = []
    cps = cost_per_success(runs)
    if cps is None:
        failures.append("no successful runs: cost/success undefined")
    elif cps > Fraction(budget["max_cost_per_success_usd"]):
        failures.append(f"cost/success {float(cps):.4f} over budget")
    if retry_share(runs) > Fraction(budget["max_retry_share"]):
        failures.append("retry share regressed")
    if p95([r.wall_ms for r in runs]) > int(budget["max_p95_wall_ms"]):
        failures.append("p95 wall clock regressed")
    return failures

Run it from CI with one command and let the exit code do the talking.

python -m costgate check --cassette fixtures/nightly.jsonl --budget costgate/budget.toml

The replay transport is the trick

A cassette is a recorded JSONL file: one row per model call, holding a hash of the prompt and the response that came back. Replay asserts the hash before returning the response, so a prompt change shows up as cassette drift instead of a silently passing test.

# costgate/replay.py
import hashlib, json

def sha(text):
    return hashlib.sha256(text.encode()).hexdigest()

class Replay:
    """Deterministic stand-in for a model call, backed by a recorded cassette."""
    def __init__(self, path):
        with open(path, encoding="utf-8") as fh:
            self.rows = [json.loads(line) for line in fh if line.strip()]
        self.i = 0

    def call(self, prompt):
        row = self.rows[self.i]
        assert row["prompt_sha"] == sha(prompt), f"cassette drift at row {self.i}"
        self.i += 1
        return row["response"]

Now the gate is deterministic, runs in seconds, and costs nothing. Refresh the cassette on a schedule so drift is a decision someone made, not a surprise. This reference implementation is not executed against your stack — treat it as a shape to copy, not a library to depend on.

Calibrate before you enforce

The first week of this gate will be red, and that is calibration, not a bug. Take the median cost per success across your last twenty nightlies, add a margin you can defend, and only then make the gate blocking. Until then, run it in report mode and write down who owns each threshold.

This is also where a free model tier and a free server option genuinely participate in the method rather than decorate it. Running a replay gate needs almost no throughput and almost no money; what it needs is somewhere to run that isn't your metered build fleet, and a way to re-record cassettes without watching a spend graph while you wait. MonkeyCode's free model access and free server option cover exactly those two gaps. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Check the current limits and data-handling terms in their docs before you point anything sensitive at it — free tiers change, and your cassette may contain real prompts.

Where free capacity is the right tool, and where it isn't

Situation	Free capacity	Why
Pinning cost logic in CI	Good fit	Replayed cassettes need tiny, predictable throughput
Cold-path dev loops and demos	Good fit	Low stakes, low volume, easy to re-run
A contractual p95 latency SLO	Wrong bet	You don't control a shared queue, so you can't promise a tail
Regulated or customer-identifiable data	Wrong bet	A free tier is the wrong place to discover a residency clause
One nightly batch dominating spend	Wrong bet	Per-success is the wrong metric; you want a hard per-job cap

The last row is the one people get wrong. Cost per success is a ratio, and ratios hide fixed cost. An idle server you keep warm all month contributes almost nothing to any single run and everything to the invoice.

Limitations worth stating out loud

A replay gate tests your cost logic, not your vendor's pricing. If the price per token moves, your cassette doesn't know, and your threshold silently drifts out of date — put an expiry date on the budget file. The divisor also punishes a flaky test corpus: make your fixture suite more failure-prone and cost per success rises even though production is fine. Keep the cassette small and curated, twenty to fifty runs, and review it like a fixture, not a log.

Don't adopt this if your real constraint is a latency SLO, if your data can't leave a controlled environment, or if your spend is dominated by a single large batch job rather than many small calls. And don't adopt it if nobody will own the thresholds — a red build with no owner trains the team to ignore red builds, which is worse than no gate at all.

Start with one command in report mode, one budget file, and one person who reads the failure message. If you want to run that command somewhere other than your metered build fleet, the free server option on MonkeyCode is a reasonable place to try it.

Count Minutes and Tokens in the Same Ledger

Quinn Li — Tue, 20 Oct 2026 09:59:11 +0000

A run that succeeds on the third attempt after a long queue wait can cost more than a paid run that succeeds once — even when the tokens are free.

That sounds obvious until you look at how most teams measure. They count tokens because tokens are the thing on the invoice. They ignore minutes because minutes never appear anywhere. Then they ship a pipeline whose real cost per accepted output is two or three times what the dashboard claims, and nobody can explain why the week felt expensive.

So here is the only rule worth carrying into free capacity: put tokens and minutes in the same ledger, then divide by outputs you actually kept. Not outputs you generated. Outputs you accepted.

The two currencies

Tokens are the cheap currency and the easy one to see. Minutes are the expensive one and the easy one to hide behind the word "waiting".

You need a number for your own minute. Use your loaded hourly rate divided by sixty. If you are not opportunity-cost constrained — a solo developer on a quiet week — pick a floor value anyway, because a zero here silently guarantees the ledger will always favor free.

The interesting subtlety is that queue wait is not fully lost. If you submit a job and go review a pull request, the wait is overlap, not cost. So the ledger carries an idle weight: how much of the wait actually blocked you. Set it to 0.0 for a fire-and-forget batch, 1.0 for a synchronous call in a request path, and something honest in between for interactive work.

The harness

The script below is deliberately boring. One dataclass, one cost function, one JSONL file you append to on every run. It is unexecuted on your provider — you supply the numbers.

# runledger.py — append-only run accounting for AI workloads
import json, time
from dataclasses import dataclass, asdict

@dataclass
class Run:
    task_id: str
    lane: str                 # "free" or "paid"
    attempts: int = 0
    tokens_in: int = 0
    tokens_out: int = 0
    queue_s: float = 0.0      # submit -> first byte
    compute_s: float = 0.0    # first byte -> last byte
    review_s: float = 0.0     # your minutes reading and fixing
    accepted: bool = False

IDLE_WEIGHT = 0.5             # fraction of queue wait that actually blocked you


def cost_of(run, price_in, price_out, minute_rate):
    token_cost = (run.tokens_in / 1e6) * price_in
    token_cost += (run.tokens_out / 1e6) * price_out
    review_cost = (run.review_s / 60.0) * minute_rate
    wait_cost = (run.queue_s / 60.0) * minute_rate * IDLE_WEIGHT
    return token_cost + review_cost + wait_cost


def log(run, path="runs.jsonl"):
    with open(path, "a") as f:
        f.write(json.dumps(asdict(run)) + "\n")

The number you actually decide on is not the sum. It is the sum divided by accepted runs.

def cost_per_accepted(path, price_in, price_out, minute_rate):
    total, kept = 0.0, 0
    for line in open(path):
        run = Run(**json.loads(line))
        total += cost_of(run, price_in, price_out, minute_rate)
        kept += run.accepted
    return total / max(kept, 1)

One line of arithmetic, and the retries stop being invisible. Three attempts on the same prompt re-bill the input tokens and re-bill your review minutes. A left-joined cost model hides that entirely, because it prices the successful call and forgets the two funerals before it.

A measurement plan that survives scrutiny

Take twenty tasks you have already solved by hand. Same prompts, same acceptance criteria, both lanes, run in the same hour so you are not comparing a quiet morning to a busy evening. Log accepted by hand — that flag is the whole measurement, and it is the one thing you cannot automate honestly.

Twenty is a smell test, not a benchmark. If the gap is under thirty percent, your sample is too small to conclude anything and you should decide on constraints instead of arithmetic. If the gap is a factor of two, you have a real answer and you can stop measuring.

The illustrative arithmetic, with the numbers left blank on purpose: suppose a free lane needs 1.6 attempts per accepted output at zero token price but an extra nine minutes of queue wait and review per attempt, while a paid lane needs 1.1 attempts at a nonzero price. Whether free wins depends entirely on your minute rate and your idle weight. That is not a dodge — it is the actual shape of the problem, and any vendor claiming otherwise is quoting their numbers at you.

When free capacity is the wrong bet

Situation	Why free loses	What to do
Deadline-bound run	Queue variance is the dominant term, not price	Use capacity with a queue you can predict
High retry amplification	Each retry re-bills input tokens plus your review minutes	Fix the prompt before scaling the lane
Review-heavy output	Your minutes dwarf token cost by an order of magnitude	Optimize the acceptance loop first
Synchronous request path	`IDLE_WEIGHT` is 1.0 and the wait is pure latency	Free capacity belongs in batch, not in the path

The last row is the one people get wrong most often. Free capacity is a batch instrument. Putting it behind a user-facing request converts a cost saving into a latency budget you did not agree to spend.

I ran this harness shape against free model access and the free server option in MonkeyCode to see whether the ledger changes a decision I had already made. It did, in one place: batch work moved, interactive work did not. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Verify the current limits yourself before you plan around them — availability and quotas are exactly the kind of fact that should never be taken from an article, including this one.

Limitations, and who should skip this

Your minute rate is a fiction if you are not actually constrained by time. The ledger does not capture model drift, so numbers older than a few weeks are archaeology. And it only measures the runs you remembered to log, which is why the append happens in code rather than in your head.

Skip this if you run fewer than ten jobs a day — eyeballing two runs will teach you the same lesson faster. Skip it if a paid lane already fits inside your latency budget at a price you do not think about. And skip it if your pipeline cannot be instrumented at all, because a cost model you cannot feed is just a spreadsheet with opinions.

If you want to run the harness against a free lane before you spend anything, try the free model access and the free server option, log twenty runs, and let cost_per_accepted argue with you.

The Free Tier Is an Instrument, Not a Budget

Quinn Li — Mon, 19 Oct 2026 08:41:43 +0000

Before you move a workload onto free capacity, run it there once and write down what it costs you in wall-clock and attention. Tokens are the only line item that goes to zero. Queue wait, retries, and review minutes all survive the move — and they usually get bigger.

You already know the price per token. What you don't know is tokens per accepted output, attempts per accepted output, and how long a job sits before anyone touches it. Budget decisions made without those three numbers are guesses wearing a spreadsheet.

So the useful question about a free tier isn't "how much can I run for nothing". It's "what can I learn for nothing". Free model access and a free server are excellent for the second question and misleading for the first, because a free host and a free endpoint let you observe the shape of your pipeline at zero marginal price while telling you almost nothing about throughput you can promise someone.

Record every attempt, not every run

The artifact below is deliberately boring. One JSON line per attempt, written before anything else can fail. Python 3.10+, standard library only.

# ledger.py - one row per attempt, nothing clever
import json, os, time
from datetime import datetime, timezone

LEDGER = os.environ.get("LEDGER", "ledger.jsonl")

def _now() -> str:
    return datetime.now(timezone.utc).isoformat(timespec="milliseconds")

def run_job(job_id: str, prompt: str, call, max_attempts: int = 3):
    """call(prompt) -> object with .prompt_tokens/.completion_tokens, or raises."""
    for attempt in range(1, max_attempts + 1):
        queued = _now()
        t0 = time.monotonic()
        started = _now()
        try:
            out = call(prompt)
            status, err = "ok", None
        except Exception as exc:
            out, status, err = None, "error", type(exc).__name__
        with open(LEDGER, "a", encoding="utf-8") as fh:
            fh.write(json.dumps({
                "job_id": job_id,
                "attempt": attempt,
                "queued_at": queued,
                "started_at": started,
                "ended_at": _now(),
                "wall_s": round(time.monotonic() - t0, 3),
                "prompt_tokens": getattr(out, "prompt_tokens", None),
                "completion_tokens": getattr(out, "completion_tokens", None),
                "status": status,
                "error": err,
            }) + "\n")
        if status == "ok":
            return {"job_id": job_id, "attempts": attempt}
        time.sleep(2 ** attempt)
    return None

One rule matters more than the code: turn off your SDK's automatic retries before you run this. If the client retries behind your back, the ledger records one attempt where four happened, and your retry amplification silently reads 1.00 while your bill disagrees.

Four numbers, then a decision

# report.py - the numbers that actually move a budget
import json, statistics
from collections import defaultdict
from datetime import datetime

rows = [json.loads(l) for l in open("ledger.jsonl", encoding="utf-8") if l.strip()]
jobs = defaultdict(list)
for r in rows:
    jobs[r["job_id"]].append(r)

accepted = [j for j, rs in jobs.items() if any(r["status"] == "ok" for r in rs)]
n = max(len(accepted), 1)
tokens = sum((r["prompt_tokens"] or 0) + (r["completion_tokens"] or 0) for r in rows)
waits = [(datetime.fromisoformat(r["started_at"])
          - datetime.fromisoformat(r["queued_at"])).total_seconds() for r in rows]
walls = [r["wall_s"] for r in rows]
p = lambda xs, q: sorted(xs)[min(int(len(xs) * q), len(xs) - 1)] if xs else float("nan")

print(f"accepted            {len(accepted)} / {len(jobs)}")
print(f"attempts/accepted   {len(rows) / n:.2f}")
print(f"tokens/accepted     {tokens / n:,.0f}")
print(f"queue wait p50/p95  {p(waits, 0.5):.2f}s / {p(waits, 0.95):.2f}s")
print(f"wall time p50/p95   {p(walls, 0.5):.2f}s / {p(walls, 0.95):.2f}s")

Now the arithmetic. Let attention cost one dollar per minute. If a free run needs three extra minutes per accepted output for re-driving and review, you have spent three dollars of attention to save whatever the tokens cost. If your paid cost per accepted output sits under that number, free capacity is the more expensive option and the invoice simply arrives as a calendar instead of a statement.

That comparison is the whole method. You are not comparing token prices; you are comparing tokens-into-attention against money-into-tokens.

Reading the calibration

What the ledger says	Reading	Move
attempts/accepted ≈ 1.0, under 5 min attention per output	free tier is a fine development loop	stay, re-measure monthly
attempts/accepted > 1.5, driven by rate-limit errors	attention scales with attempts, not outputs	move volume to paid, keep free for tests
p95 wall time swings between runs of the same job	noise bigger than signal	measure longer, commit nothing yet
free host CPU pegged near 100%	you measured the host, not the model	split host cost out before deciding
over 15 min human review per accepted output	attention dominates the bill	free tokens save the smallest line

If you want to run that first pass without paying for tokens while you're still measuring, MonkeyCode's free model access and free server option are a reasonable place to point it — bring your own ledger and keep your own numbers.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Where this measurement lies to you

Quotas and rate limits move. Check the current terms on the day you measure, not a figure you read in a post — including this one. A number that was true last quarter is a budget error with good intentions.

Your own timestamps cannot see server-side queueing. started_at is when you handed the request over; if the endpoint held it for forty seconds first, that time lands in wall_s and nowhere else. Report it, don't explain it away. A quiet measurement window also tells you nothing about your Monday burst, where rate limits change your effective concurrency and you end up measuring a different system than the one you ship.

And if the free server is CPU- or memory-bound, the calibration is about that host. Model latency and host latency are two bills; don't merge them into one conclusion.

Who should skip this

Anyone with a hard latency SLO, data that cannot leave your boundary under the provider's terms, or a workload whose retry behavior is driven by a client you don't control. In those cases the measurement itself is the risk, and a short paid pilot with a known ceiling is the cheaper way to learn the same thing.

For everyone else: run the free tier as an instrument once. Then decide with a number instead of a feeling.

Do the Deadline Math Before You Enqueue

Quinn Li — Sun, 18 Oct 2026 07:20:41 +0000

Do the Deadline Math Before You Enqueue

Free capacity does not fail on price. It fails on the tail. The median request comes back fast enough that you stop worrying, and then one job in twenty sits in a slot for six minutes while your downstream step holds a lock and a human waits.

So stop pricing free capacity in tokens. Price it in the only currency the queue actually charges you: wall-clock hours of pipeline occupancy per accepted result. Once you have that number, admission control becomes arithmetic instead of vibes.

The rule I use is short: a job may enter free capacity only if its measured p90 completion time, inflated by its retry rate, still fits inside the deadline with a margin. Everything else gets routed somewhere with a contract. Here is how to get those three numbers without guessing.

Log attempts, not averages

Averages hide the tail, and the tail is the whole argument. You need one log line per attempt, with monotonic timestamps taken at submit, first byte, and completion, plus whether the result was actually accepted by your validator.

# measure.py - one JSONL line per attempt, no dependencies beyond stdlib
import json, os, time, urllib.request, uuid

ENDPOINT = os.environ["MC_ENDPOINT"]
KEY = os.environ["MC_KEY"]
LOG = "attempts.jsonl"

def call(prompt, timeout=180):
    body = json.dumps({"prompt": prompt}).encode("utf-8")
    req = urllib.request.Request(
        ENDPOINT,
        data=body,
        headers={"content-type": "application/json", "authorization": f"Bearer {KEY}"},
    )
    t_submit = time.monotonic()
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        t_first = time.monotonic()
        payload = json.loads(resp.read().decode("utf-8"))
    return t_submit, t_first, time.monotonic(), payload

def run(job_id, prompt, attempt, deadline_s):
    rec = {"job_id": job_id, "attempt": attempt, "deadline_s": deadline_s, "ok": False}
    try:
        t0, t1, t2, payload = call(prompt)
        rec.update(ttft_s=round(t1 - t0, 3), total_s=round(t2 - t0, 3),
                   tokens=payload.get("usage", {}).get("total_tokens"), ok=True)
    except Exception as exc:
        rec.update(error=type(exc).__name__, total_s=None)
    with open(LOG, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(rec) + "\n")
    return rec

Two details matter more than they look. deadline_s is written at submit time, not reconstructed later, so you can count missed deadlines honestly. And ok means your validator accepted the output, not that HTTP returned 200. A 200 with a truncated answer is a failure that still burned a slot.

Run the harness against the prompts you actually send, not a synthetic list. Fifty to a hundred samples is enough to see the shape; a handful is enough to fool yourself.

Turn the log into three numbers

You want p90 end-to-end time, retry amplification, and tokens per accepted result. Retry amplification is just total attempts divided by accepted results. If a fifth of your jobs need a second try, every slot you book is 20 percent larger than the job.

import json, statistics

rows = [json.loads(l) for l in open("attempts.jsonl")]
done = sorted(r["total_s"] for r in rows if r.get("total_s") is not None)
ak = sum(1 for r in rows if r["ok"])

def pct(xs, p):
    if not xs:
        return float("inf")
    k = max(0, min(len(xs) - 1, int(round((p / 100) * len(xs) + 0.5)) - 1))
    return xs[k]

print("attempts      ", len(rows))
print("accepted      ", ak)
print("p50_s         ", round(pct(done, 50), 2))
print("p90_s         ", round(pct(done, 90), 2))
print("amplification ", round(len(rows) / ak, 2) if ak else "n/a")
print("tok/accepted  ", round(sum(r.get("tokens") or 0 for r in rows) / ak, 1) if ak else "n/a")

That last line is the honest free-tier cost. Free tokens are not zero cost if you spend three attempts to get one accepted answer; the grant is consumed by the retries, and the clock is consumed by all three attempts whether they succeed or not. This is the same reason a retry is rent rather than insurance: you pay it every time the job runs, including the runs that work.

Where the free tier earns its place

This is the part where MonkeyCode is genuinely relevant to the method rather than bolted onto it. The harness above is worthless if sampling it costs more than the pipeline it measures, and the project advertises a free model tier plus a free server option, currently promoted with a large token grant (the exact figure moves, so read the current terms before you budget against it). That makes it a reasonable place to run the first hundred samples and to host the collector that writes attempts.jsonl.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Host the collector there, keep the log on a volume you control, and re-run the sampler whenever the free tier changes. If you would rather not run a second machine, the free server is enough for a cron job that posts fifty prompts a day and appends to a file.

One caution: a shared free server measures the vendor's contention as well as your code. Compare its p90 against a small paid box before you conclude anything about your own application.

The gate itself

With p90 and amplification in hand, admission is four lines. Ask for the job's deadline, compute the inflated p90, and refuse anything that does not fit inside 80 percent of the remaining budget. The missing 20 percent is your margin for the day the tail is worse than the sample.

def admit(deadline_s, p90_s, amplification, margin=0.20, in_flight=0, max_slots=4):
    if in_flight >= max_slots:
        return False, "no slot"
    expected = p90_s * amplification
    budget = deadline_s * (1 - margin)
    return expected <= budget, round(expected, 1)

Refusals are not failures. They are the queue telling you the truth before you have paid for it. Route the refused jobs to committed capacity, or shrink the job until it fits, or move the deadline. What you must not do is enqueue optimistically and discover the overrun at the downstream lock.

A useful side effect: the ratio of refused to admitted jobs is a leading indicator. When refusals climb, either your prompts grew, or the free tier's tail widened, or both. Either way you learn it from a log line instead of an incident.

When free capacity is the wrong bet

It is the wrong bet when the deadline is hard and the consequence of missing it is not yours to absorb: payment capture windows, anything a human is actively waiting on, and jobs whose side effects are not idempotent so a retry is not a retry but a duplicate. It is also wrong when the tail is invisible to you, because then no admission rule can fire.

Skip this whole approach if your traffic is steady and small enough that a fixed committed plan is cheaper than the engineering time to build the gate. A ledger nobody reads is more expensive than the overrun it was meant to catch.

For everyone else, the sequence is unglamorous and it works: log attempts with deadlines attached, compute p90 and amplification, admit only what fits, and treat the free grant as a place to learn the shape of the tail rather than a place to park production work. If you want to start measuring without paying for the first batch of samples, the free model access and free server are a fair place to begin.

Queue Time Is a Line Item

Quinn Li — Sat, 17 Oct 2026 06:23:00 +0000

You already meter what your jobs cost. Almost nobody meters how long each job sat still before anything happened — and in most AI pipelines that wait is the largest cost a single afternoon of instrumentation can find. This is a measurement recipe and a routing rule, not another retry story.

Token meters answer "what did the work cost". They do not answer "what did waiting cost", and for queued workloads the second number is usually bigger. A job that waits 40 seconds and runs for 3 spends 93% of its life in a queue. You will not see that in a token dashboard; you will see it in a timestamps file nobody is writing. Think of a restaurant kitchen: ingredient cost per plate is a fine metric, but if you never count how long each table sat empty, you can optimise the wrong number all the way to closing time.

Record four timestamps, nothing else

You need enqueued_at, started_at, finished_at, deadline_at, plus a route label and a token count if you have one. That is it. If you use a queue, three of those already exist somewhere; enqueued_at is the one people forget to propagate, and it is the only one that reveals the bill you are not looking at.

Write one JSONL record per job:

{"id":"j1","route":"free","enqueued_at":0.0,"started_at":12.5,"finished_at":40.1,"deadline_at":60.0,"tokens":8200}

Then a script that turns timestamps into a bill. This one is about forty lines, has no dependencies, and does one thing.

#!/usr/bin/env python3
"""wait_bill.py - turn job timestamps into a queue-time bill.

Reads JSONL, one record per job, all times in epoch seconds.
  {"id":"j1","route":"free","enqueued_at":0.0,"started_at":12.5,
   "finished_at":40.1,"deadline_at":60.0,"tokens":8200}

Usage: python wait_bill.py jobs.jsonl
"""
import json
import sys
from collections import defaultdict


def quantile(values, p):
    if not values:
        return float("nan")
    xs = sorted(values)
    idx = min(len(xs) - 1, max(0, int(round(p / 100 * (len(xs) - 1)))))
    return xs[idx]


def rows(path):
    with open(path, encoding="utf-8") as fh:
        for line in fh:
            line = line.strip()
            if line:
                yield json.loads(line)


def bill(jobs):
    lanes = defaultdict(list)
    for j in jobs:
        wait = j["started_at"] - j["enqueued_at"]
        service = j["finished_at"] - j["started_at"]
        lanes[j["route"]].append(
            {
                "wait": wait,
                "wall": wait + service,
                "on_time": j["finished_at"] <= j["deadline_at"],
                "tokens": j.get("tokens", 0),
            }
        )
    return lanes


def report(lanes):
    cols = ("lane", "n", "wait p50", "wait p90", "wait%", "late%", "tok/ok")
    header = "{:<12}{:>6}{:>10}{:>10}{:>8}{:>8}{:>10}".format(*cols)
    print(header)
    print("-" * len(header))
    for name, rs in sorted(lanes.items()):
        waits = [r["wait"] for r in rs]
        wall = sum(r["wall"] for r in rs) or 1.0
        on_time = [r for r in rs if r["on_time"]]
        tok_ok = sum(r["tokens"] for r in on_time) / len(on_time) if on_time else float("nan")
        late = 100 * (1 - len(on_time) / len(rs))
        print(
            "{:<12}{:>6}{:>10.1f}{:>10.1f}{:>7.0f}%{:>7.0f}%{:>10.0f}".format(
                name, len(rs), quantile(waits, 50), quantile(waits, 90),
                100 * sum(waits) / wall, late, tok_ok,
            )
        )


if __name__ == "__main__":
    report(bill(rows(sys.argv[1])))

If you have no production data yet, generate a synthetic file so you can prove the script runs end to end. The numbers it produces are fabricated on purpose and say nothing about any real system:

import json, random


def synth(path, n=2000, seed=7):
    rng = random.Random(seed)
    with open(path, "w", encoding="utf-8") as fh:
        for i in range(n):
            free = rng.random() < 0.7
            enq = i * 0.8
            wait = rng.expovariate(1 / (18 if free else 2))
            service = rng.expovariate(1 / (6 if free else 5))
            fh.write(json.dumps({
                "id": f"j{i}",
                "route": "free" if free else "committed",
                "enqueued_at": enq,
                "started_at": enq + wait,
                "finished_at": enq + wait + service,
                "deadline_at": enq + (45 if free else 30),
                "tokens": rng.randint(2000, 20000),
            }) + "\n")


synth("jobs.jsonl")

Read the bill before you optimise anything

wait% is the share of wall-clock time your jobs spend queued on that lane. late% is how often they miss the deadline. tok/ok is tokens spent per job that still landed on time, which is the only token ratio worth arguing about, because tokens burned on a job that missed its deadline are tokens you will spend again.

Two patterns deserve names. If wait% is high on a lane and late% is near zero, the lane is queue-bound but healthy and the work is genuinely deferrable. If wait% is high and late% is also high, you are not buying throughput, you are buying surprises, and no amount of retry logic repairs that. Measure first, because the fix differs.

Route cheapest-feasible, and let reject be a route

Once you have measured p90 wait per lane, routing becomes a cheap predicate instead of a vibe:

def route(job, lanes, now, safety=1.5):
    """Cheapest lane that can still make the deadline. Reject is a lane."""
    for name in ("free", "committed"):
        p90_wait = lanes[name]["p90_wait"]      # from wait_bill.py output
        p90_service = lanes[name]["p90_service"]
        if now + safety * p90_wait + p90_service <= job["deadline_at"]:
            return name
    return "reject"

The interesting branch is the last one. A job that cannot make its deadline on any affordable lane should fail fast and be re-planned by a scheduler or a human, not quietly occupy capacity another job could have used. reject is a cost control, not an error path, and treating it as one is the difference between a budget and a wish.

Where a free lane actually earns its place

Free model access and a free server option — both operator-supplied availability claims for MonkeyCode — fit the free slot in the predicate above: batch summarisation, overnight re-indexing, evaluation sweeps, anything whose deadline is soft and whose wait is genuinely cheap. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What free capacity does not do is shorten a queue tail. If your p90 wait is dominated by burst concurrency, a free lane absorbs the burst and hands you a worse tail than a capped queue would, so cap concurrency and measure again. Current limits, quotas and server details belong in the project's own documentation rather than in a blog post. I have therefore quoted no number here that I could not verify today, and you should treat any quota you read elsewhere the same way.

When free capacity is the wrong bet

Job class	Can it wait?	Cheapest sane lane	Why
Nightly batch summarisation	hours	free	soft deadline; wait costs almost nothing
PR review bot, 10-minute SLA	minutes	free first, escalate on measured p90	you need the escalation path, not a faster default
Interactive assistant	no	committed	the user experiences every queued second
5k-job fan-out burst	yes, but	free with a concurrency cap	uncapped, the tail eats the burst
Hard external deadline	no	committed plus reject	a missed deadline costs more than the tokens

Limitations, and who should not do this

Timestamps from different hosts drift; sync clocks or accept a few hundred milliseconds of noise. A p90 computed from thirty jobs is not a p90. This method tells you nothing about output quality, and nothing useful about capacity you already pay for up front, where wait is a sunk cost rather than a marginal one. If your workload is a single interactive request with a sub-second expectation, the queue is not your problem and this whole exercise is overhead. Same if you cannot yet emit enqueued_at — fix that first, because it is one line and it unlocks everything above.

Start by counting the wait you already have. Then decide which jobs deserve committed capacity and which can sit in a free lane until the queue reaches them. If you want somewhere to point the script while you calibrate, MonkeyCode's free model access and free server option are a reasonable place to run the batch-shaped half of a workload — verify the current details on the project's own pages, and let the measured p90 decide the rest.

Retries Are Rent

Quinn Li — Fri, 16 Oct 2026 05:21:41 +0000

Your retry loop does not bill you in tokens. It bills you in seconds you cannot buy back, and free capacity is the one lane where the second invoice is the only one that arrives. That is the whole argument. Everything below is a way to put a number on it before you hand a week of batch work to a queue you do not control.

Tokens are the visible line on the receipt. Wall clock is the line the vendor never prints. When you move a workload onto a free lane, the first line shrinks and the second one grows, and most teams never notice the trade because only one of the two numbers shows up in a dashboard.

The ledger you are not keeping

Every agent framework I read now ships a retry policy, and almost none of them log what a retry costs in time. So write the ledger yourself. The harness below is a proposal, not something I ran against a hosted endpoint, so treat the output as a placeholder until you point it at your own client.

# retry_ledger.py — proposal; run it against your own client before trusting it
from dataclasses import dataclass, field
from time import monotonic

RETRYABLE = {"rate_limited", "timeout", "upstream_5xx"}

@dataclass
class Attempt:
    queue_wait_s: float
    work_s: float
    tokens_in: int
    tokens_out: int
    outcome: str

@dataclass
class Task:
    name: str
    attempts: list = field(default_factory=list)

    @property
    def succeeded(self):
        return bool(self.attempts) and self.attempts[-1].outcome == "ok"

    @property
    def wall_s(self):
        return sum(a.queue_wait_s + a.work_s for a in self.attempts)

    @property
    def queue_share(self):
        return sum(a.queue_wait_s for a in self.attempts) / max(self.wall_s, 1e-9)

    @property
    def tokens(self):
        return sum(a.tokens_in + a.tokens_out for a in self.attempts)


def run_task(name, call, max_attempts=4, backoff=1.5):
    task = Task(name)
    delay = 1.0
    for _ in range(max_attempts):
        t0 = monotonic()
        result = call(name)          # your client returns outcome + timing + usage
        t1 = monotonic()
        task.attempts.append(Attempt(
            queue_wait_s=getattr(result, "queue_wait_s", 0.0),
            work_s=t1 - t0 - getattr(result, "queue_wait_s", 0.0),
            tokens_in=getattr(result, "tokens_in", 0),
            tokens_out=getattr(result, "tokens_out", 0),
            outcome=result.outcome,  # "ok" | "rate_limited" | "timeout" | "malformed" | "bad_input"
        ))
        if result.outcome == "ok":
            return task
        if result.outcome not in RETRYABLE:
            return task              # do not buy rent on a bug you can fix
        delay *= backoff
    return task

Two lines in that dataclass matter more than the rest. queue_share tells you how much of your latency was someone else's scheduling, and tokens counts every attempt, not just the successful one. Log both and the free-lane decision stops being a vibe.

Failure classes decide who pays

Retrying a bad_input error is the most expensive habit in this whole space. You pay queue wait, you pay tokens, and the answer cannot change. Treat malformed the same way until you have a repair step that actually alters the request.

Retries are rent because the landlord is the queue, not your code. A three-attempt success at 12 seconds of queue wait each costs you 36 seconds of clock whether or not a single token was billed. On an interactive path that is a product failure. On an overnight batch it is a rounding error.

Where the free lane actually wins

Free capacity is a scheduling choice, not a discount. It wins when your deadline is loose, your tasks are idempotent, and your failures are retryable. It loses when a human is watching the spinner, or when retry amplification pushes you past a wall-clock deadline that has money attached.

Workload shape	Free lane	Committed lane
Interactive edit, p99 under 2s	no	yes
Overnight batch, morning deadline	yes, if queue_share stays under ~50%	yes if it does not
Fan-out where 30%+ attempts are rate_limited	yes, with a jittered cap	yes
Anything with `bad_input` in the loop	fix the input first	fix the input first
Long cross-repo refactor you must restart often	risky: restarts multiply queue wait	yes

That table is a starting point, not a benchmark. Fill in your own columns from the ledger after a few hundred tasks.

Where MonkeyCode fits in the ledger

The project currently advertises free model access and a free server option, which makes it a reasonable candidate for the slow lane in the harness above: batch jobs where you care about the total and not the tail. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not going to quote a quota number here. Allowances, model availability, and server terms are set by the project and they move; read the current terms instead of a blog post, including this one. What matters for the ledger is only that you can point call() at it, capture queue wait if the client exposes timing, and count attempt-level tokens.

One natural way to start: take a batch you already run, cap max_attempts at two, run it twice — once on the lane you pay for, once on the free lane — and compare tokens / successes and wall_s / successes side by side. Two numbers, one afternoon.

Test plan you can run this week

Write a stub call() that returns rate_limited 30% of the time and ok otherwise, with a fixed 8-second simulated queue wait. Run 200 tasks and assert the ledger reports an amplification near 1.43 and a queue_share above 0.6. Then flip the stub to 2 seconds of queue wait and rerun. The amplification barely moves; the wall clock collapses.

That experiment is the point. Free capacity changes the constant, not the shape. If your retry temperature is wrong, a faster queue just gets you to the wrong answer sooner.

Limitations and who should skip this

Do not use the free lane for anything with a human on the other end, for regulated data whose residency rules you have not confirmed, or for workloads you must restart often. Do not use it when the deadline penalty exceeds the token savings by an order of magnitude — usually the case for paid client work.

And do not use any of this if you cannot state your per-success budget. A ledger without a target is a diary. Pick the number first, then let the queue wait tell you which lane you can afford.

Price the Run, Not the Token

Quinn Li — Thu, 15 Oct 2026 11:31:27 +0000

Price the Run, Not the Token

Your spend graph has two meters and the free one only covers half of them. Tokens are the meter you can see on a dashboard. Seconds are the meter your deadline actually reads. Cap the first and ignore the second, and you will ship late while the chart looks perfectly healthy.

This is the failure mode that shows up whenever free capacity is in the mix. Free tokens make attempts cheap, so people stop counting attempts. Attempts are not free in time.

Every attempt re-buys the context

A retry is not a replay from the last good word. In most agent loops you append the failure or the validator output and send the whole thing again. Attempt one costs your context. Attempt two costs your context plus the first error. Attempt three costs context plus two errors, and so on.

The token cost of a run therefore grows faster than the number of attempts, and the wall-clock cost grows with queue wait you do not control. Two retries on a shared free tier might be ninety seconds of generation and four minutes of waiting. That is fine for a nightly job. It is fatal for anything a person is staring at.

Record both meters on every attempt

The cheapest fix is a ledger. One line per attempt, written before you decide anything about the result. No SDK assumptions — if your transport exposes token counts, log them; if it does not, log characters and label the field as an estimate.

# runledger.py — one JSONL line per model attempt.
import json, time, pathlib, hashlib

LEDGER = pathlib.Path("runs/ledger.jsonl")
LEDGER.parent.mkdir(exist_ok=True)

def attempt(task_id, prompt, client, accept, **gen_kw):
    rec = {
        "task": task_id,
        "prompt_sha": hashlib.sha256(prompt.encode()).hexdigest()[:12],
        "prompt_chars": len(prompt),
    }
    started = time.monotonic()
    resp = client.generate(prompt, **gen_kw)      # your transport, free or paid
    gen_s = time.monotonic() - started
    ok = accept(resp)
    rec.update(
        gen_s=round(gen_s, 2),
        wall_s=round(time.monotonic() - started, 2),
        prompt_tokens=getattr(resp.usage, "prompt_tokens", None),
        completion_tokens=getattr(resp.usage, "completion_tokens", None),
        accepted=bool(ok),
    )
    with LEDGER.open("a") as fh:
        fh.write(json.dumps(rec) + "\n")
    return resp, ok

Note what is not in that record: the model output. You are logging cost, not content. Keep the outputs in your own store so the ledger stays small enough to read.

Let the cap fall out of the arithmetic

Most teams pick a retry cap by feel — three, five, whatever the last person typed. You can derive it instead. If each failure appends roughly e tokens to an initial context of c, the cumulative token cost of n attempts is n*c + e*n*(n-1)/2. Invert that against your ceiling.

def attempt_cap(context_tokens, error_tokens, ceiling_tokens):
    """How many attempts fit under the ceiling, including re-sends."""
    spent = 0
    n = 0
    while True:
        spent += context_tokens + n * error_tokens   # attempt n+1 re-sends n errors
        if spent > ceiling_tokens:
            return n
        n += 1

print(attempt_cap(context_tokens=8000, error_tokens=400, ceiling_tokens=120000))

Run that with your real context size and the answer is usually smaller than the number you were using. That is the whole point. The cap is a budget decision, not a personality trait.

Aggregate by accepted task, not by call

A per-call dashboard flatters itself: every call succeeded, therefore the system is efficient. Group by task and charge the failed attempts to the task, because that is where the money went.

import json, collections

runs = collections.defaultdict(lambda: {"tok": 0, "s": 0.0, "n": 0, "ok": False})
for line in open("runs/ledger.jsonl"):
    r = json.loads(line)
    g = runs[r["task"]]
    g["tok"] += (r["prompt_tokens"] or 0) + (r["completion_tokens"] or 0)
    g["s"] += r["wall_s"]
    g["n"] += 1
    g["ok"] = g["ok"] or r["accepted"]

done = [g for g in runs.values() if g["ok"]]
print("accepted:", len(done), "of", len(runs))
print("tokens/accepted:", sum(g["tok"] for g in done) // max(len(done), 1))
print("seconds/accepted:", round(sum(g["s"] for g in done) / max(len(done), 1), 1))
print("attempts/accepted:", round(sum(g["n"] for g in done) / max(len(done), 1), 2))

The third number is the one that tells you whether to raise your acceptance test or lower your cap. If attempts per accepted task drifts above two, your prompts are guessing, not working, and no amount of free capacity will fix a guessing loop.

When free capacity is the wrong bet

Free capacity is a fine trade when you are spending queue time instead of money. It is a bad trade the moment the queue time lands on someone else's critical path.

Job shape	Free capacity	Committed capacity
Someone is watching a spinner	rarely worth it	yes
Overnight batch, due by morning	yes, with a derived cap	only if the deadline is contractual
Long context with deep retries	only if the cap is enforced	yes, because each retry re-sends everything
Burst of one-off jobs	yes, accept the jitter	yes, if jitter breaks a downstream SLA

Read the second column carefully. It is not a downgrade. Trading seconds for tokens is a legitimate engineering choice — as long as you know which currency you spent and you wrote it down.

Limits of this method

This ledger measures only what your transport exposes. If token counts are missing, your totals are estimates and should be labelled that way. Queue wait varies by hour of day, so a single afternoon sample will mislead you; collect for at least a week before you set a cap in production. The numbers are yours and they do not transfer — another team's tokens-per-task tells you nothing about your prompts.

Do not use this approach if you have a hard external deadline, a sub-second interactive surface, or a compliance rule that keeps prompts inside your own network. And if your acceptance test is "looks fine to me," skip the ledger entirely; without a real test, accepted is a constant and the third metric becomes noise.

Where the free tier actually fits

There is a project worth knowing about here: MonkeyCode is an open-source coding agent project, and its operator states that new accounts receive a 10,000,000-token allowance on its free models along with a free server option. Treat that as operator-supplied and check the current project page before you plan a budget around it, because allowances change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The honest way to use it is as the measurement environment, not the production answer. Point the ledger at free models, run your own task set for a week, and you get real tokens-per-accepted-task and seconds-per-accepted-task numbers without paying for the observation. The free server option means the collector and a cron for the aggregator can live somewhere that is not your laptop.

Then read your own table. If seconds per accepted task is fine for your workload, keep spending the cheap currency. If it is not, you now have a number to argue with instead of a feeling. If you want to try it, the free models and the free server are a low-cost place to start — and if you self-host the ledger instead, that works too, because the point was never the provider. The point was pricing the run.

Book the Partial Before You Retry

Quinn Li — Tue, 13 Oct 2026 02:03:13 +0000

A free model call that you abandon is not free. You already spent the queue, and you already spent the tokens the server emitted before you cut the socket. Write that partial off before you retry. If you do not, you are running a loop that looks thrifty and bills you in minutes.

You hit this when a shared free server is slow and your client is optimistic. The first attempt streams a couple hundred tokens, then the deadline you promised a person slips, so you cancel and send the same prompt again. The second attempt waits behind work you did not actually pull out of the pool by closing the socket. A zero price on the token meter does not rewind that queue.

It only hides the wait inside a dashboard that still says free. The retry question is live in developer discussions again. Treat it as a ledger problem, not as a mood your HTTP client is in.

Picture a print shop that takes night jobs at no charge. You submit a chapter, they set type for three pages, then you cancel because the pickup window moved. Those pages stay on the bench. Submitting the chapter again adds a second job behind the first.

What you book

Keep three numbers on the job, and update them after every attempt. spent_s is queue plus generation you already waited through. spent_tokens is what came back, including a partial you will not ship. ceiling_tokens is a cap you chose for this job, not a vendor quota copied out of a post.

The cap exists so a free lane cannot mean an unbounded loop inside your own process. A retry on that lane is allowed only when the next attempt can finish inside the remaining deadline, and only when spent tokens plus a fresh worst-case estimate still fit inside the ceiling. Otherwise you cut over to a paid lane for one clean attempt, or you stop and return what you have.

Stopping is a result. Another hopeful enqueue is not. Pass an empty attempt list for the first call, so the same function admits or refuses before you have a partial and you are not maintaining two policies.

The figures in the fixtures are placeholders you replace with timings you took this week. Nothing here is a measurement of a public pool, a price list, or a limit that will still be true next Monday. If a number is older than your last sample, it is already the wrong input.

A ledger you can run

Save the script as book_partial.py. It is a local decision helper, not a client for any vendor, and it will not open a socket. You feed it history. It returns an action.

#!/usr/bin/env python3
"""Write off partial free-lane attempts before you allow another retry."""

from __future__ import annotations

import json
import sys
from dataclasses import dataclass


@dataclass(frozen=True)
class Attempt:
    queue_s: float
    gen_s: float
    emitted_tokens: int
    shipped: bool


@dataclass(frozen=True)
class Policy:
    deadline_s: float
    ceiling_tokens: int
    next_queue_s: float
    next_gen_s: float
    next_est_tokens: int
    paid_queue_s: float
    paid_usd_per_1k: float


def book(attempts: list[Attempt], policy: Policy) -> dict:
    spent_s = sum(a.queue_s + a.gen_s for a in attempts)
    spent_tokens = sum(a.emitted_tokens for a in attempts)
    remaining_s = policy.deadline_s - spent_s
    next_free_s = policy.next_queue_s + policy.next_gen_s
    next_paid_s = policy.paid_queue_s + policy.next_gen_s
    projected = spent_tokens + policy.next_est_tokens

    if any(a.shipped for a in attempts):
        action, why = "stop", "a previous attempt already shipped"
    elif projected > policy.ceiling_tokens:
        action, why = "stop", "another try would cross your token ceiling"
    elif next_free_s <= remaining_s:
        action, why = "retry_free", "remaining clock covers one more free attempt"
    elif next_paid_s <= remaining_s:
        action, why = "cut_over", "free retry does not fit; one paid attempt does"
    else:
        action, why = "stop", "no lane finishes inside the time you have left"

    sunk_usd = (spent_tokens / 1000.0) * policy.paid_usd_per_1k
    return {
        "action": action,
        "why": why,
        "spent_s": round(spent_s, 2),
        "remaining_s": round(remaining_s, 2),
        "spent_tokens": spent_tokens,
        "projected_tokens": projected,
        "sunk_usd_if_those_tokens_were_paid": round(sunk_usd, 4),
    }


def main() -> int:
    raw = json.load(sys.stdin)
    attempts = [Attempt(**item) for item in raw["attempts"]]
    policy = Policy(**raw["policy"])
    json.dump(book(attempts, policy), sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The first fixture is a night job with a wide clock. One partial came back, nothing has shipped, and you still have room under a ceiling you set yourself.

python3 book_partial.py <<'EOF'
{
  "attempts": [
    {"queue_s": 35, "gen_s": 9, "emitted_tokens": 220, "shipped": false}
  ],
  "policy": {
    "deadline_s": 600,
    "ceiling_tokens": 4000,
    "next_queue_s": 35,
    "next_gen_s": 12,
    "next_est_tokens": 800,
    "paid_queue_s": 2,
    "paid_usd_per_1k": 0.002
  }
}
EOF

You should see retry_free. Spent time is 44 seconds, the remaining window is wide, and 220 plus a guessed 800 stays under 4000. The dollar field is a shadow price so the partial stays visible on the record. It is not an invoice, and the rate is a stand-in you must replace.

If you do not know a current paid rate, set paid_usd_per_1k to 0 and decide on time and tokens alone. A stale price copied from an old thread is worse than a zero, because it looks like evidence.

Now the interactive case. Same partial, a 60 second deadline, and the same next guess. The night job and the inline job are not the same bet, even when they share a prompt.

python3 book_partial.py <<'EOF'
{
  "attempts": [
    {"queue_s": 35, "gen_s": 9, "emitted_tokens": 220, "shipped": false}
  ],
  "policy": {
    "deadline_s": 60,
    "ceiling_tokens": 4000,
    "next_queue_s": 35,
    "next_gen_s": 12,
    "next_est_tokens": 800,
    "paid_queue_s": 2,
    "paid_usd_per_1k": 0.002
  }
}
EOF

Remaining time is 16 seconds. The next free attempt wants 47. One paid attempt wants 14, which still fits, so the action is cut_over. The free server did not become a bad product in that branch; the remaining clock stopped matching the queue you would sit in again.

If you lower the ceiling under the projection, the same function stops instead of retrying, even when the deadline is wide. Run the third fixture and read the action before you argue with it.

python3 book_partial.py <<'EOF'
{
  "attempts": [
    {"queue_s": 35, "gen_s": 9, "emitted_tokens": 220, "shipped": false}
  ],
  "policy": {
    "deadline_s": 600,
    "ceiling_tokens": 500,
    "next_queue_s": 35,
    "next_gen_s": 12,
    "next_est_tokens": 800,
    "paid_queue_s": 2,
    "paid_usd_per_1k": 0.002
  }
}
EOF

Projected tokens are 1020 against a ceiling of 500, so you stop. A long deadline does not authorize an open token loop just because the lane advertises free model access. Your ceiling is how you keep that advertisement from becoming a retry policy. Count tokens the same way on every attempt, or the stop branch fires late and you will blame the wrong line.

Pin the branches

Wire a tiny assertion so the three branches cannot drift while you edit the formula. This is a local test. It does not touch a network, and it fails if you reorder the checks without updating the expected actions.

python3 - <<'PY'
import importlib.util

spec = importlib.util.spec_from_file_location("book_partial", "book_partial.py")
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)

def run(deadline, ceiling, emitted=220):
    attempts = [mod.Attempt(35, 9, emitted, False)]
    policy = mod.Policy(deadline, ceiling, 35, 12, 800, 2, 0.002)
    return mod.book(attempts, policy)["action"]

assert run(600, 4000) == "retry_free"
assert run(60, 4000) == "cut_over"
assert run(600, 500) == "stop"
assert run(60, 500) == "stop"
print("ok")
PY

Run that from the directory where you saved the file, then read the last assert with the cut-over case beside it. A tight deadline and a blown ceiling both stop, and stop wins over cut-over because the ceiling check runs first. If you would rather pay once the free path has already burned a partial, swap those branches and re-run the test until it matches the policy you actually want. A client that retries on its own will not honor either order, so turn that off or the ledger is a story you tell after the loop has already run.

Sample the queue you will sit in

Before you trust next_queue_s, take a sample in the hour you actually run. Send a short fixed prompt on the free lane, record time to first token as queue, and record the rest as generation. Do that three times, keep the worst queue, and discard the probe text. If the probe fails, the lane is not eligible this hour.

Do not retry the probe until it feels fine. That loop is the one the ledger is meant to stop. The command below is a stand-in timer, not a benchmark and not a vendor call. Replace the sleep with your own client, then put the worst sample into the policy.

python3 - <<'PY'
import time

start = time.perf_counter()
# Stand-in only. Replace this sleep with one short probe call.
time.sleep(0.05)
queue_s = time.perf_counter() - start
print(f"sample_queue_s={queue_s:.3f}")
PY

Where a free lane still belongs

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode belongs in this workflow only as a spill lane: free model access and a free server option, as supplied for this note, for jobs the ledger has already marked retry_free. This page does not name models, token grants, hardware, or how long either option lasts, because those facts move and a post should not freeze them. If you still want that spill lane, read the live limits, copy the ones you will depend on into ceiling_tokens and next_queue_s, and run the assertion again when the sample changes.

Free capacity is the wrong bet when a person is blocked on the answer, when the attempt you would abandon is not idempotent, or when you cannot sample queue time and would be typing a wish into next_queue_s. It is also the wrong bet when a retry can double-apply a side effect, since booking tokens will not undo a second write. Stamp your own idempotency key on the job before the first call. Treat shipped: true as final, so a late free response cannot overwrite a paid answer you already returned.

What this leaves out

The function trusts your samples. One quiet afternoon is not a week, so if the queue swings, keep the worst sample from the hour you run and put that number in next_queue_s. It also treats every emitted token as spent, which is the right bias for a ceiling and the wrong bias for a quality score. A partial you can resume is not the same shape as a partial you must throw away.

If your client supports continuation, add only the missing gap to the projection, then re-run the assertion so the new bias is the one you tested. Do not use this for payments, access control, or any path where stop is not an acceptable outcome and you have no contracted fallback. Do not use it as a capacity forecast for a team. It books one job's history, and it will not tell you whether a free server can hold everyone at the top of the hour.

You will still be tempted to skip the write-off, because the dashboard said the call was free and the partial looked small. Book it anyway. The next retry is where a free lane turns into a queue you cannot see the end of.

Write the Prefix Down Before the Slot Dies

Quinn Li — Sun, 11 Oct 2026 21:51:35 +0000

A preempted stream is not a free retry. If the lane already emitted tokens, those tokens are spent, and starting the prompt over buys a prefix you already hold.

You feel the urge anyway. The socket drops, the free slot vanishes, and the editor still shows an unfinished function. The cheap instinct is to send the same messages again and hope the next slot is kinder. That instinct turns a short outage into a second bill for the same answer.

Think of a courier who already walked the parcel to your door, then got pulled off the route. You do not hire a second courier to repeat the walk from the warehouse. You keep what arrived and pay only for the last flight of stairs.

A model stream has that shape, with one wrinkle. Input context may have to be shown again, because a provider often will not hand you a receipt for cached prompt tokens. Output you already received must not be generated again. Those are different costs, and mixing them up is how the bill grows while the answer stays the same.

This note is a small ops habit for that split. It is a proposal, not a measured result, and the sample is unexecuted example code. It does not assume a model name, a quota, a machine size, or a promise that a free lane will exist next week.

Before the call, write a job row. Give it an id, a hash of the prompt, a token budget, and a cursor at zero. The hash is how you refuse to resume against a prompt you have since edited. If the hash does not match, the cursor belongs to a different question, and stitching it on would answer the wrong ask.

While tokens arrive, flush the cursor on a short interval, not only at the end. A flush at the end is a diary of success. A flush every few dozen tokens is a diary of the failure you actually have.

Store the assistant text, the count you treat as spent, and a monotonic revision. If the process dies between flushes, you lose only the gap. That gap is the acceptable loss. The prefix before it is not.

On the way back in, do not rebuild the original request and ask for a fresh completion. Build a continuation, and cap output at budget minus spent. If the API allows an assistant prefill, put the stored prefix there and ask it to continue without repeating itself.

If the API will not accept a prefill, send a short instruction plus the unfinished tail, not the entire original thread plus a demand for a new essay. If the remainder is below a floor you choose, abandon. A continuation that may spend the original budget again is just a replay wearing a new function name.

Here is the gate in plain Python. Save it as budget_cursor.py. Treat every number as a knob, not a measurement. The annotations assume a current Python 3.

import hashlib, json
from pathlib import Path

ROOT = Path(".jobstate")

def prompt_hash(messages: list[dict]) -> str:
    blob = json.dumps(messages, sort_keys=True, ensure_ascii=False)
    return hashlib.sha256(blob.encode()).hexdigest()

def open_job(job_id: str, messages: list[dict], budget_tokens: int) -> dict:
    ROOT.mkdir(exist_ok=True)
    path = ROOT / f"{job_id}.json"
    if path.exists():
        job = json.loads(path.read_text())
        if job["prompt_hash"] != prompt_hash(messages):
            raise SystemExit("prompt changed; refuse stale cursor")
        return job
    job = {
        "job_id": job_id,
        "prompt_hash": prompt_hash(messages),
        "budget_tokens": budget_tokens,
        "spent_tokens": 0,
        "prefix": "",
        "revision": 0,
        "state": "open",
    }
    path.write_text(json.dumps(job))
    return job

def flush(job: dict, piece: str, piece_tokens: int) -> dict:
    job["prefix"] += piece
    job["spent_tokens"] += piece_tokens
    job["revision"] += 1
    if job["spent_tokens"] > job["budget_tokens"]:
        job["state"] = "over_budget"
    path = ROOT / f"{job['job_id']}.json"
    tmp = path.with_suffix(".json.tmp")
    tmp.write_text(json.dumps(job))
    tmp.replace(path)
    return job

def continuation(job: dict, floor: int = 32) -> dict | None:
    remain = job["budget_tokens"] - job["spent_tokens"]
    if job["state"] != "open" or remain < floor or not job["prefix"]:
        return None
    return {
        "assistant_prefill": job["prefix"],
        "max_output_tokens": remain,
        "instruction": "Continue from the prefill. Do not repeat it.",
    }

The temp-file rename avoids a half-written JSON line if the process dies mid-flush on the same filesystem. It is not a distributed lock. Two workers sharing one job id will still clobber the cursor, so give each in-flight call its own id.

You can exercise the refusal path with no network and no key. The 12 below is a stand-in, not a tokenizer result. In a real call you store the usage figure for that chunk, not a guess from string length.

mkdir -p .jobstate
python - <<'PY'
from budget_cursor import open_job, flush, continuation
msgs = [{"role": "user", "content": "draft a migration note"}]
job = open_job("mig-14", msgs, budget_tokens=400)
flush(job, "Step 1. Freeze writes.\n", 12)
print(continuation(job))
print("remain", job["budget_tokens"] - job["spent_tokens"])
PY

If continuation returns None, you stop. You do not send the original call once more just in case. None means the prefix already consumed the useful budget, or there is no prefix worth resuming. A full replay would ignore the ending you just computed.

Watch the file if you want to see the cursor move under a fake stream. This loop is still local. It proves the flush, not a provider.

python - <<'PY'
import json
from budget_cursor import open_job, flush
job = open_job("mig-14", [{"role": "user", "content": "draft a migration note"}], 400)
for piece, n in [("Step 2. Copy rows.\n", 10), ("Step 3. Verify counts.\n", 11)]:
    flush(job, piece, n)
print(json.dumps(job, indent=2))
PY

You have three exits, and you should pick one before the call, not during the panic. Abandon when the remainder is under the floor or the prefix is empty. Resume when the prefix is real and the remainder still fits the budget.

Spill to a lane you are willing to wait on only when the job cannot be dropped and the free slot has already failed. Spilling is not a loop. One spill, then a human, or a queue that can actually reserve the work. The reservation problem is older than this cursor, and this note does not solve it.

A free lane is a decent stage for that rehearsal, and a bad home for work you cannot drop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The useful fit is narrow: MonkeyCode's free model access and a free server option can host a broken-stream drill, after you read the live limits in the product you have, and only for jobs you can abandon. This note does not record a quota, a card, a region, or a duration, because those figures move and no primary source for them was attached.

Free capacity is the wrong bet when the remainder is not optional. A migration note that can be rewritten later may die at a cursor. A payment capture, a mailbox send, an infrastructure apply, or a customer-visible webhook may not. Those need an idempotency key, and the side effect should run after the text is finished, not inside the stream. If a partial answer is worse than silence, do not emit it onto a lane that can vanish mid-sentence.

There is a quieter wrong bet than preemption: input replay. If the provider bills the prompt again on every continuation, a huge system prompt will swallow the savings from not regenerating the prefix. Log input tokens and output tokens separately on the resume call. If resumed input is nearly the original prompt plus the stored prefix, you moved the cost, you did not remove it.

Shrink the resumed context to the tail you still need, or stop using continuation for that prompt shape. The cursor still earns its keep when it blocks a second full completion. It will not make a bloated prompt cheap, and it will not refund tokens a vendor has already metered.

Skip this habit if you cannot persist the prefix outside the process. A cursor that lives only in memory dies with the preempt and leaves the replay you were trying to avoid. Skip it when the client hides token counts and you would be guessing piece_tokens. A guessed cursor is a lie in the ledger.

Skip it when the attempt must be byte-identical to the first sample. Continuation is a new draw, not a restored sampler, unless the provider documents otherwise. Do not claim determinism you did not buy. And skip it for any call that mutates the world before the stream ends, because a resume would be a second chance to mutate it.

The habit is small enough to keep. Open a job, flush while text arrives, and on death ask only for the remainder or ask for nothing. A free slot can host the drill. It cannot refund a prefix you already spent, and it cannot be the commitment under work you are unwilling to lose.

Free Tokens Do Not Pay the Clock

Quinn Li — Sat, 10 Oct 2026 18:28:01 +0000

Free tokens do not pay a deadline. If the seat is slow, each retry spends minutes you cannot refund, and the next call still has to fit inside what is left.

You feel the trap when a repair loop looks harmless. The prompt failed once, the seat is marked free, and another attempt costs nothing on the rate card. Then the request sits. The next attempt starts late, sometimes after the caller has already given up.

You did not buy extra intelligence. You bought a slower no. Treat it like a restaurant that comps the meal and still holds the table for an hour. The food is free, and the evening is not.

Model work splits the same way. A token grant can zero the model line while the clock keeps charging the humans and jobs waiting on the result. Cost work here is four coupled numbers, not a vibe about cheap inference.

What you are actually budgeting

Before you attach a retry policy to any free seat, write down four numbers you can measure without a vendor dashboard. You need a deadline, a token ceiling for the whole job, a maximum number of attempts, and the longest queue wait you will tolerate. If you cannot name those four, you do not have a policy. You have a hope.

The deadline is wall-clock, not a feeling. Pick a duration the caller can still use, such as 45 seconds for an interactive repair or 10 minutes for a batch note. The token ceiling is the most you will let one job read and write, prompt plus completion, across every attempt.

The attempt cap stops a flaky case from eating the evening. The wait cap is the number people skip, and it is the number that makes a free seat expensive. A call that is free on the rate card and late on the clock has already failed the job you claimed to care about.

This ledger is a proposal, not a trace from a production run. It does not call a model. It decides whether another attempt is still admissible given the time and tokens you have left. Run it locally, and label the output as a sketch, before you point any client at a live seat.

from dataclasses import dataclass

@dataclass
class Budget:
    deadline_s: float
    token_ceiling: int
    max_attempts: int
    max_wait_s: float

@dataclass
class Attempt:
    wait_s: float
    elapsed_s: float
    tokens: int

def admit(budget, spent_s, spent_tokens, attempts_used, nxt):
    if attempts_used >= budget.max_attempts:
        return 'refuse: attempt cap'
    if nxt.wait_s > budget.max_wait_s:
        return 'refuse: queue wait'
    remaining = budget.deadline_s - spent_s
    if nxt.wait_s + nxt.elapsed_s > remaining:
        return 'refuse: deadline'
    if spent_tokens + nxt.tokens > budget.token_ceiling:
        return 'refuse: token ceiling'
    return 'admit'

if __name__ == '__main__':
    budget = Budget(deadline_s=45, token_ceiling=8000, max_attempts=2, max_wait_s=5)
    # Unexecuted sketch. Replace the numbers with timings you actually observed.
    rows = [
        (0.0, 0, 0, Attempt(1.2, 6.0, 2100)),
        (7.2, 2100, 1, Attempt(18.0, 9.0, 2400)),
    ]
    for spent_s, spent_tokens, used, nxt in rows:
        print(admit(budget, spent_s, spent_tokens, used, nxt))

The first sketch attempt fits. The second does not. An 18 second queue already blows a 5 second wait cap, and the remaining clock cannot honestly host another full call. That refusal is the result you wanted.

You stop before a complimentary seat turns a short repair into a late one. Walk the same numbers so the function is not a black box. Your caller can wait 45 seconds. Scratch attempt one waits 1.2 seconds and finishes in 6, using about 2100 tokens.

You then have 37.8 seconds and 5900 tokens left, with one attempt remaining. Scratch attempt two sits 18 seconds before the first byte. Even if the model answered instantly, you have already spent more wait than the cap. Admitting it would teach the client that free means late.

Refuse, log the wait, and change the seat or the prompt. You can do the same check on paper, without the script. Remaining seconds must exceed observed wait plus a conservative elapsed time from the last scratch run that actually returned. Remaining tokens must exceed the last prompt size plus the last completion size, and attempts used must sit under the cap.

All three have to pass. One failure refuses the call. That is stricter than a typical retry helper, which often looks only at the HTTP status and then sends the transcript again. Status is not a budget.

How to rehearse it without spending a real job

You can collect the inputs with a stopwatch and a token estimate. Start a scratch prompt you are willing to throw away. Record queue wait as the gap between submit and first byte. Record elapsed time through the last byte.

Record tokens from the response usage field when the API returns one, or from a local tokenizer when it does not. Write those three numbers down beside the decision. Do not average a week of runs into a single comfortable mean. A mean hides the wait that will miss the deadline.

Time the gap in the shell so the log has a clock, not a memory. This snippet only measures pacing you already observed. It does not call a provider, and the offsets are stand-ins until you paste real timestamps.

python3 - <<'PY'
import time
start = time.perf_counter()
submitted = time.perf_counter()
first_byte = submitted + 1.2
done = first_byte + 6.0
print(f'wait_s={first_byte - submitted:.3f} elapsed_s={done - submitted:.3f}')
print(f'rehearsal_wall_s={time.perf_counter() - start:.3f}')
PY

Append each observed row before you raise the attempt cap. Keep the decision next to the inputs so a later reader can see why the gate closed.

python3 ledger.py | tee -a rehearsal.log
printf '%s\n' 'wait_s=18.0 elapsed_s=9.0 tokens=2400 decision=refuse: queue wait' >> rehearsal.log

Read the log cold, a day later if you can. If two scratch runs already show waits near the cap, a third automatic retry is not a tuning idea. It is a plan to miss the deadline on purpose. Fix the prompt, shrink the context, or move the job to a seat you can hold.

There is a second bill hiding in the retry itself. Each new attempt often resends the same transcript. A 2100 token prompt, sent three times, is not 2100 tokens of work. It is 6300, plus three completions, plus three trips through the queue.

A free grant makes that multiplication feel fake. It is not fake. Grants end, and a duplicated transcript is a fast way to spend one on copies of a prompt you already paid for in time. Cap tokens per job, not only per pretty-looking call.

A timeout caused by the queue will tempt the client to send again while the first request is still in line. Now two of your calls are waiting. You doubled the wait, and you may double the tokens if both complete. Keep one in-flight attempt per job until a scratch log shows the seat keeping up.

Do not put keys, customer text, or private logs into the scratch prompt or into rehearsal.log. The gate only needs wait, elapsed time, and a token count. If a row needs a secret to make sense, the rehearsal is pointed at the wrong example.

Where a free seat fits, and where it does not

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Those are useful as a scratch seat for the rehearsal above: you can price wait, tokens, and retries on a call you are allowed to discard.

This note does not claim a quota, a model list, a hardware shape, a duration, or a benchmark. If those details matter to your job, read the current project docs and treat them as time-sensitive. A number you remember from a launch post is not a control you can schedule against.

Use that scratch seat when the work is disposable. A prompt you are shaping, a parser you are testing, a failure you want to see once: that is the fit. The free option earns its place because a thrown-away call should not land on a paid invoice, and because you want queue behavior in the log before you trust it.

You are buying information about the seat, not a result you mean to ship. Once the log says the wait is inside the cap, you still re-check on the seat you will actually use for the real job. A good scratch hour does not transfer.

Do not use a complimentary seat for a committed job. A customer request with a latency promise, a migration step that must finish inside a window, or a nightly close that pages someone if it slips needs a seat you can hold, or a deadline you can move. Leftover room is not a reservation.

You do not get to assume the room will be empty when the retry fires. If the first observed wait already exceeds max_wait_s, refuse the next attempt on that seat. Hoping the queue clears is not a control. Neither is refreshing a dashboard until the line looks short.

The same rule applies when the token line is still at zero and the clock is not. A repair that looks free can still miss a human waiting on a review, a CI job holding a runner, or a teammate who planned to leave. Price those minutes as real cost. If the admit function would refuse, the label on the seat does not overrule it.

Who should skip this

Skip the ledger if you have no deadline and no shared seat. A solo notebook experiment with nobody waiting does not need admission control. Also skip it if you cannot observe wait time, because a token-only policy will green-light a call that arrives after the user has left.

Do not use this sketch as a scheduler for production traffic. It is a local gate for one job's next attempt. It does not know other tenants, it does not retry with backoff magic, and it will not save you from a provider outage. If you need fairness across a fleet, this file is the wrong tool.

There is a quieter failure mode. You can tune the caps until every historical row is admitted, then call the policy safe. That is curve-fitting, not ops. Keep one held-out scratch run that you did not use to pick the numbers.

If that run refuses, believe the refusal. If you cannot explain the refusal in one sentence a teammate understands, the cap is decoration. Raise the deadline in public, or shrink the job, rather than silently widening the gate until it never says no.

If you already keep a scratch budget and a job you can afford to delay, rehearse the gate on MonkeyCode's free model access and free server before you attach the same policy to anything with a pager. The log, not the word free, tells you whether the seat is cheap enough to keep.

Stamp a Deadline Before You Pick a Lane

Quinn Li — Fri, 09 Oct 2026 14:57:14 +0000

A zero token price does not make a job free. If the work has a deadline, the queue wait and the retry fan-out are the invoice, and a free lane is the wrong place to send it.

You already know the cheap thrill: a prompt returns, the dashboard shows nothing owed, and the next attempt feels harmless. Then the job sits, a worker restarts, and the same prompt goes out again because nobody stamped a stop. The token counter stays polite. The clock does not.

Treat that gap as an admission problem, not a model problem. Before you pick a lane, write down the deadline, the worst wait you will tolerate, and how many attempts you will allow. If those three numbers cannot coexist, do not send the job. A later, smarter model will not repair a queue you never bounded.

Picture a ferry that boards for nothing and a taxi that prints a fare. The ferry is fine when you are carrying a sample and you can miss the evening, and it is a bad vehicle when someone is waiting on the other shore and the boat has no posted departure. A zero-price lane works like that ferry, while a lane you can actually hold works like the taxi. You match the vehicle to the promise you already made, not to whichever ride looks cheaper.

What you are actually buying

Time, tokens, and retries are one budget wearing three names. Queue time is time you do not control. Generation time is time you half control. Tokens are the residue of both, because a retry sends the prompt again and often sends a longer context than the first try.

A free price zeroes only the third name. It does not zero the first two. If the machine you did not hold is busy, preempted, or simply slow to schedule, you can miss a deadline while spending no money at all. That is still a failed job, because your caller does not grade you on the invoice.

Retries make the math worse. Two attempts are not twice as innocent as one when each attempt waits in line. Four attempts with a cold start can cost more wall clock than one attempt on a lane you held, even when the held lane has a nonzero token rate. Compute that comparison before you feel loyal to the zero.

This is also how a casual rerun becomes an incident. The second run is not a copy of the first. It is a new wait, a new context, and sometimes a new failure mode. If you cannot point at a hold, or at an explicit decision that this job may miss its window, you are guessing with someone else's afternoon.

A smaller model can still be the right model. That choice is about output quality per token, and it belongs in a different column from lane admission. A cheap model that starts immediately can beat a free model that sits. Do not collapse those columns, or you will celebrate a price you never paid while the queue eats the window.

A ledger you can run locally

The snippet below is a proposal. It is not a benchmark, and it was not executed against a live provider for this article. It only encodes the rule: deadline work does not ride an unreserved lane, and the retry fan-out must fit inside the window.

from dataclasses import dataclass

@dataclass(frozen=True)
class Attempt:
    lane: str
    queue_wait_s: float
    gen_s: float
    prompt_tokens: int
    completion_tokens: int
    succeeded: bool

def summarize(attempts, prompt_rate, completion_rate):
    tokens = 0
    wall = 0.0
    money = 0.0
    for a in attempts:
        tokens += a.prompt_tokens + a.completion_tokens
        wall += a.queue_wait_s + a.gen_s
        if a.lane != "free":
            money += a.prompt_tokens * prompt_rate
            money += a.completion_tokens * completion_rate
    return {"tokens": tokens, "wall_s": wall, "money": money}

def admit(deadline_s, predicted_wait_s, max_attempts, predicted_gen_s, lane):
    if deadline_s is not None and lane == "free":
        return False, "deadline jobs do not ride the free queue"
    worst = max_attempts * (predicted_wait_s + predicted_gen_s)
    if deadline_s is not None and worst > deadline_s:
        return False, "retry fan-out breaks the deadline"
    if max_attempts < 1:
        return False, "nothing to run"
    return True, "ok"

Save it as lane_admit.py. Then check the branches with the standard library. No network, no key, and no account are required.

python - <<'PY'
from lane_admit import Attempt, summarize, admit

probe = [
    Attempt(lane="free", queue_wait_s=40, gen_s=6,
            prompt_tokens=900, completion_tokens=180, succeeded=False),
    Attempt(lane="free", queue_wait_s=55, gen_s=7,
            prompt_tokens=900, completion_tokens=40, succeeded=False),
]
held = [
    Attempt(lane="reserved", queue_wait_s=1.5, gen_s=5,
            prompt_tokens=900, completion_tokens=180, succeeded=True),
]
assert summarize(probe, 0.000002, 0.000008)["money"] == 0.0
assert summarize(probe, 0.000002, 0.000008)["wall_s"] > summarize(held, 0.000002, 0.000008)["wall_s"]
assert admit(30, 40, 2, 6, "free")[0] is False
assert admit(12, 8, 2, 5, "reserved")[0] is False
assert admit(None, 40, 1, 6, "free")[0] is True
print("lane checks passed")
PY

Read the three decisions in order. The free lane is refused because a deadline was set, and the held lane is refused too, in this made-up example, because two attempts still overflow twelve seconds. Only the last call is admitted, because you passed deadline_s=None, and that call is a probe. It may wait or be dropped, but it must not sit on the path that pages a human.

The rates in the command are placeholders so the arithmetic is visible. Do not treat them as a vendor price. Swap in the figures from your current billing page when you care about money. When the lane is free, the function already forces the money term to zero, so you can still lose on wall_s.

Keep the inputs honest. predicted_wait_s should be a high percentile from your own recent attempts, not a number you liked in a demo. If you have no samples, pass a wait you are willing to be wrong about, and treat an admit as provisional. A function cannot discover a queue it has never seen.

Failed rows still count

You will want to delete the failures before you summarize. Don't, because a timeout still occupied a slot and still attached every prompt token you sent. If the server returned a partial completion, those tokens belong in the row. Drop them and the two charts disagree, so you fix the wrong one.

When you debug a slow evening, sort by queue_wait_s before you sort by answer length. A long answer is a generation problem, and a long wait is a lane problem. Mixing them is how a team swaps models for a week and never touches scheduling. The log has to separate those, because a dashboard that reports no token spend will not.

python - <<'PY'
import json
rows = [json.loads(line) for line in open("attempts.jsonl")]
rows.sort(key=lambda r: r["queue_wait_s"], reverse=True)
for r in rows[:5]:
    print(r["job_id"], r["lane"], r["queue_wait_s"], r["succeeded"])
PY

Give each attempt an idempotency key before it leaves your process. A retry must be the same job, not a new job with a fresh context and a second copy of the prompt. Without that key, this ledger under-counts, and your notebook will disagree with the platform about what you owe in time. Store the key, the lane, the wait, the token estimates, and whether you intended a deadline.

Run the zero-price path on purpose, as a bounded sample, not as the default route. Cap it at a count you can read, such as one hundred attempts that carry no deadline, and record wait, generation time, and whether the call returned. After that sample, put a high percentile wait into predicted_wait_s. If that percentile already exceeds the tightest deadline you owe a caller, that lane is disqualified for that caller no matter what the token price says.

Where a zero price earns a place

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode, as the operator described it for this note, includes a free model path and a free server option. Those are availability claims, not a measured quota, a hardware class, or a promise that the queue is short. Use them for the probe half of the workflow: prompt diffs, parser checks, and replaying recorded waits through admit.

Keep caller-visible work, deploy gates, and anything you will retry inside a window on a lane you can hold, or shed the job in your own code. Do not paste a token ceiling into the runbook from memory. Limits, model names, and windows move, and a stale figure is worse than a blank. Read the project docs on the day you enroll, write down what you observed, and attach that note to the job id.

If the docs are silent, record the free lane as zero price and unknown wait. Unknown wait is not a hold. If your attempt logs already exist, replay a week of them through admit before you move a user-facing job onto that free path. That replay is the only next step this note asks for.

What this will not do

The function does not know why a queue grew. It will not see a provider incident, a bad prompt, or a client that resubmits on every non-200. You still own the stop condition. A refusal inside admit is only useful if the caller actually stops.

It will not turn yesterday's samples into today's capacity. A quiet Tuesday does not hold Wednesday. If predicted_wait_s is a wish, the function will approve a lane you merely labeled reserved. Feed it measurements, or feed it a ceiling you can afford to miss.

Skip this approach if you need a contractual latency target and you have no lane you can hold. A refusal cannot manufacture capacity. Skip it if your team will treat a false result as advice and send the job anyway. Skip it, too, if the plan is to drop secrets, customer text, or production identifiers into a free probe because the token price is zero.

A free process is still a process you must account for, so check the current docs before you decide what it is allowed to see. Label every number you have not measured. The waits in the sample command are illustrations, picked so the branch is obvious, and your queue will not match them. If you publish a chart from this ledger, keep estimated tokens apart from invoiced tokens, or the fiction will win the meeting.

Stamp the deadline first. If the wait and the retries cannot fit inside that stamp, do not pick a lane. Learn the wait on the free path, and hold a lane or shed the job when a person is actually waiting on you.