DEV Community: Jordan Huang

Your Timeout Is Local: Five Myths About Model Calls You Retry

Jordan Huang — Wed, 28 Oct 2026 01:14:22 +0000

Your Timeout Is Local: Five Myths About Model Calls You Retry

My batch job died at 03:40. The box was up. The model was reachable. My retry logic was wrong.

I keep seeing the same five claims in code review. Each one sounds sensible. Each one quietly produces duplicate work, empty output, or a worker stuck forever.

So I stopped arguing and built a probe. One script, one verdict table, one ledger. No benchmarks here. Free-tier conditions change too fast for my numbers to help you.

Myth 1: "A timeout means the call did not happen"

Your deadline is measured on your machine. The request may already sit in a queue. Or it may already be done.

A local clock expiring tells you one thing only. You stopped listening. It says nothing about remote execution.

Corrected model: DEADLINE means unknown, not failed.

outcome = with_deadline(lambda: call_model(model, prompt), seconds=20)
if outcome.verdict is Verdict.DEADLINE:
    # remote state unknown: do not assume "no work done"
    log.warning("stopped listening fp=%s", fp)

This matters when the call has side effects. A tool call that writes a row, or sends a message, may have landed. Retry blind and you write twice.

Myth 2: "HTTP 200 means I got an answer"

A 200 says the transport worked. It does not say the text is usable.

I have seen three failure shapes arrive with a healthy status code:

Empty — whitespace only, or a blank field where text should be.
Truncated — ends mid-phrase because a cap was hit.
Off-target — fluent text that ignores the instruction.

Only the first two are cheap to detect. The third needs a check you wrote yourself, like a required field or a schema parse.

Corrected model: success has two gates. Transport success, then content success. Your retry policy should see both.

Myth 3: "Retries are free"

Every retry is another call. Another unit of time. Another chance to hit a limit.

Ask yourself a question. What is your worst case, attempts times deadline? If the answer is forty minutes, you do not have a retry policy. You have an outage.

Cap two numbers, not one:

Attempts — a small integer, not a loop.
Total wall clock — the budget the whole job may spend.

And treat THROTTLED differently from TRANSPORT. Backpressure wants a longer pause. A dropped connection wants a short jitter.

Myth 4: "Same prompt, same answer, so I can cache by prompt"

Outputs are snapshots, not constants. The same prompt can return different text on two runs.

That is fine for chat. It is a bug for a cache key.

Corrected model: the key is a fingerprint of everything that shapes the output.

def fingerprint(model: str, prompt: str, params: dict) -> str:
    blob = json.dumps({"m": model, "p": prompt, "params": params}, sort_keys=True)
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()[:16]

Model id, prompt, and parameters all belong in the hash. Swap the model and your cache must miss. Otherwise you serve yesterday's answer under a new label.

Myth 5: "Streaming means progress, so I can drop the timeout"

Streaming gives you bytes sooner. It gives you no guarantee of finishing.

A stalled stream is worse than a slow response. It holds a worker slot and looks alive. Your total timeout never fires because data keeps trickling.

Use two clocks:

Idle timeout — no bytes for N seconds, give up.
Total timeout — hard ceiling for the whole call.

If you only have one, you have the wrong one.

Build the probe

Here is the harness I actually run. It is provider-agnostic on purpose. Inject your client and keep your own numbers.

"""probe_call.py - classify model-call outcomes before you retry them."""
from __future__ import annotations

import argparse
import hashlib
import json
import os
import threading
import time
from dataclasses import dataclass
from enum import Enum
from pathlib import Path

LEDGER = Path(os.environ.get("LEDGER", "calls.jsonl"))


class Verdict(str, Enum):
    OK = "ok"
    TRANSPORT = "transport"
    DEADLINE = "deadline"
    THROTTLED = "throttled"
    EMPTY = "empty"
    TRUNCATED = "truncated"


@dataclass
class Outcome:
    verdict: Verdict
    text: str = ""
    detail: str = ""
    seconds: float = 0.0


def fingerprint(model: str, prompt: str, params: dict) -> str:
    blob = json.dumps({"m": model, "p": prompt, "params": params}, sort_keys=True)
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()[:16]


def with_deadline(fn, seconds: float) -> Outcome:
    """Run fn in a thread. Return DEADLINE on overrun. Never block forever."""
    box: dict = {}

    def runner() -> None:
        try:
            box["text"] = fn()
        except Exception as exc:  # classify, do not swallow
            box["error"] = exc

    t = threading.Thread(target=runner, daemon=True)
    started = time.monotonic()
    t.start()
    t.join(seconds)
    elapsed = time.monotonic() - started

    if t.is_alive():
        return Outcome(Verdict.DEADLINE, detail="local clock expired", seconds=elapsed)
    if "error" in box:
        return Outcome(Verdict.TRANSPORT, detail=repr(box["error"]), seconds=elapsed)

    text = (box.get("text") or "").strip()
    if not text:
        return Outcome(Verdict.EMPTY, detail="status ok, no usable text", seconds=elapsed)
    if text.endswith((",", " and", " the")):
        return Outcome(Verdict.TRUNCATED, text=text, detail="ends mid-phrase", seconds=elapsed)
    return Outcome(Verdict.OK, text=text, seconds=elapsed)


def record(fp: str, out: Outcome) -> None:
    row = {"fp": fp, "verdict": out.verdict.value, "seconds": round(out.seconds, 3)}
    with LEDGER.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(row) + "\n")


def seen(fp: str) -> str | None:
    if not LEDGER.exists():
        return None
    for line in LEDGER.read_text(encoding="utf-8").splitlines():
        row = json.loads(line)
        if row["fp"] == fp:
            return row["verdict"]
    return None


RETRYABLE = {Verdict.TRANSPORT, Verdict.THROTTLED, Verdict.DEADLINE}


def call_model(model: str, prompt: str) -> str:
    """Wire your client here. Return raw text. Raise on transport errors."""
    raise NotImplementedError("plug in your client, then rerun")


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--prompt", required=True)
    ap.add_argument("--model", default=os.environ.get("MODEL_ID", "unset"))
    ap.add_argument("--attempts", type=int, default=1)
    ap.add_argument("--deadline", type=float, default=20.0)
    args = ap.parse_args()

    fp = fingerprint(args.model, args.prompt, {})
    if seen(fp) == Verdict.OK.value:
        print(f"fp={fp} already OK; skipping on ledger hit")
        return

    for attempt in range(1, args.attempts + 1):
        out = with_deadline(lambda: call_model(args.model, args.prompt), args.deadline)
        record(fp, out)
        print(f"attempt={attempt} verdict={out.verdict.value} t={out.seconds:.2f}s")
        if out.verdict is Verdict.OK:
            print(out.text)
            return
        if out.verdict not in RETRYABLE:
            break
        time.sleep(min(2 ** attempt, 10) * 0.5)
    raise SystemExit(f"fp={fp} gave up; read {LEDGER}")


if __name__ == "__main__":
    main()

Two things make this useful beyond a toy. The ledger is append-only, so a restarted job can skip finished work. And the verdict is written before any retry decision.

Run the kill tests first

Do not test your retry path on a good day. Force the bad ones.

# 1) force the local-clock path
python probe_call.py --prompt "reply with: ok" --deadline 0.01

# 2) force empty and truncated shapes
#    point call_model at a stub that returns "" and "the"
python probe_call.py --prompt "reply with: ok" --attempts 1

# 3) force transport errors, then restore
#    drop your network briefly, or point at a closed port
python probe_call.py --prompt "reply with: ok" --attempts 3

# 4) prove the ledger gate
python probe_call.py --prompt "reply with: ok" --attempts 1

Step 4 is the one people skip. If you run the same fingerprint twice, you should see a skip, not a second call.

The decision table

Verdict	Retry?	Why	What to log
`OK`	No	You have text	Fingerprint, seconds
`TRANSPORT`	Yes, short jitter	You never got a reply	Error class
`THROTTLED`	Yes, long pause	The remote asked you to slow down	Pause length
`DEADLINE`	Only if side-effect free	Remote state is unknown	Fingerprint, seconds
`EMPTY`	No, inspect first	Often a prompt or filter problem	Raw body
`TRUNCATED`	Yes, with a bigger cap	Output hit a ceiling	Last 40 chars

The DEADLINE row is the one that needs judgement. Retrying the same fingerprint is safe when the call only reads. It is not safe when the call writes.

How I run this on a free host

I run the probe matrix on resources that cost me nothing, so a bad loop cannot turn into a bill. That is where MonkeyCode comes in for me: it offers free model access and a free server option, which is enough to run the kill tests above end to end.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server is exactly the kind of environment that exposes these myths. It can be slow, it can sleep, and it can be reclaimed. That is a feature for this experiment. It is not a latency baseline.

So read the verdicts. Ignore the wall-clock numbers. Rerun on your own host before you trust any of them.

What this harness does not prove

Be honest about the edges. Mine are sharp.

It does not measure model quality. OK means text arrived, not that the text is right.
It does not make calls idempotent. The ledger records attempts. Only your code can prevent double writes.
The truncation check is a heuristic. Three suffix patterns is a demo, not a parser.
It does not model your rate limits. Any specific attempt count I print would be a guess about your account.
Threads are not cancellation. The DEADLINE branch abandons a thread, it does not kill the remote work.

That last one is the whole article in a sentence. You can stop waiting. You cannot un-ask.

Who should skip this

Some teams should not adopt this pattern. I would rather say so up front.

Hard latency SLOs. A free host and a sleeping box make latency claims meaningless.
Strict data residency. Read the terms of whatever host and model you pick. Do not take my word.
High fan-out pipelines. A per-call thread plus ledger append gets clumsy fast. Use a real queue.
Anything with money attached. If a duplicate write costs real money, build an idempotency key at the API layer first.

The one question to keep

Before you add a retry, ask it. If this call already ran, what breaks?

If you cannot answer, the retry is not the fix. The missing ledger is. Start with the probe, run the kill tests, and let your own verdicts shape the policy.

If you want a zero-cost place to run those kill tests, the free models and free server on MonkeyCode will do the job.

Your App Is Up, Your Model Call Is Down: Five Coupling Myths and a Two-Probe Check

Jordan Huang — Tue, 27 Oct 2026 22:09:05 +0000

Your App Is Up, Your Model Call Is Down: Five Coupling Myths and a Two-Probe Check

A 200 from your own health endpoint proves one thing. Your process is alive. It says nothing about the network path to your model provider.

I keep seeing the same five claims in code review. Each one quietly merges two failure domains into one. Then a green dashboard hides a red dependency.

Here is the myth list, the evidence you can collect yourself, and a small probe script that keeps the domains separate.

Myth 1: "The app is up, so the model path is up."

Your process health and your outbound dependency health are different signals. They fail on different clocks, for different reasons.

A sleeping or restarted host can still answer a local health route. A DNS blip can break outbound calls while every local route returns 200.

Corrected model: you have two probes, not one. One asks "is my process serving?" The other asks "can I complete one minimal model round trip right now?"

If you only have budget for one dashboard row, split it into two rows. Merging them back is how you get the classic "everything is green and nothing works" screen.

Myth 2: "Retrying a failed call is free and safe."

Retries cost wall-clock time and quota. That part is obvious. The non-obvious part is side effects.

If your model call sits inside a loop that writes files, posts comments, or opens merge requests, a blind retry can do the work twice. The second write looks like a new event to everything downstream.

Corrected model: every attempt gets a key, and every side effect checks that key.

import hashlib, json, uuid

def attempt_key(prompt: str, params: dict, job_id: str) -> str:
    # Stable for a logical unit of work, not for a single network attempt.
    blob = json.dumps({"p": prompt, "params": params, "job": job_id}, sort_keys=True)
    return hashlib.sha256(blob.encode()).hexdigest()[:16]

Log the key before the call. Log the outcome after. If the key already has a success row, skip the side effect.

Is that extra bookkeeping annoying? Yes. Is it cheaper than explaining a duplicate write? Almost always.

Myth 3: "temperature=0 gives me the same answer twice."

It gives you a lower-variance sample. It does not give you a contract.

Even with fixed parameters, other things move: the served model revision, batching, hardware assignment, and any routing in front of the endpoint. Your prompt hash did not change. The thing answering it might have.

Corrected model: treat a model response as an observation with metadata, not a pure function output.

Record at minimum:

The model identifier string you sent
Any identifier string the response returned
Parameter set (temperature, max tokens, stop sequences)
Prompt hash
Timestamp, attempt number, latency

If those five fields are missing, you cannot reproduce the bug. You can only re-enact it and hope.

Myth 4: "The model name is the model."

A name in a config file is a pointer. Pointers get repointed.

I am not claiming a specific provider moved a specific alias on a specific date. I am claiming you cannot prove they did not, because you never logged the response-side identifier.

Corrected model: the name you send is intent. The identifier you receive is evidence. Store both, and diff them across runs.

A one-line check catches this cheaply:

# Two probes an hour apart, same config, different response-side id?
grep -o '"served_id": *"[^"]*"' probe.jsonl | sort -u | wc -l

If that count is greater than one and you did not change anything, you learned something real about your dependency.

Myth 5: "A client timeout means no work happened upstream."

A timeout is a statement about your patience. It is not a statement about their side.

The request may have been queued, started, or finished after you stopped listening. Retrying immediately is how you pay twice for one answer you never read.

Corrected model: a timeout is unknown, not failed. Route it to a separate branch with a longer backoff and a duplicate-suppression check.

What you observed	What it proves	Safe next action
Connection refused	Path is down now	Backoff, then re-probe
429	You are over a limit	Stop, wait, do not hammer
5xx	Their side errored	Bounded retry with the same key
Client timeout	Unknown	Same key, longer wait, check ledger first
200 with odd output	Call succeeded	Validate output, log metadata

Note what is missing from that table: "retry five times fast." That row does not exist for a reason.

The artifact: a two-probe check you can run today

This is a reference implementation. I have not run it against your account, so adapt the endpoint shape before trusting it.

MonkeyCode's free model access covers the model probe, and the free server option is enough to host the script itself. Both are operator-supplied availability claims, not measured numbers from me.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

# probe.py - two independent checks, one JSONL ledger
import datetime, json, os, time, uuid

MODEL_ID = os.environ["MODEL_ID"]        # from your account, never hardcoded
BASE_URL = os.environ["MODEL_BASE_URL"]
API_KEY  = os.environ["MODEL_API_KEY"]

def now():
    return datetime.datetime.now(datetime.timezone.utc).isoformat()

def record(kind, ok, detail, **extra):
    row = {"ts": now(), "kind": kind, "ok": ok, "detail": detail, **extra}
    with open("probe.jsonl", "a") as f:
        f.write(json.dumps(row) + "\n")
    return row

def probe_local():
    path = "probe_local.tmp"
    token = str(uuid.uuid4())
    with open(path, "w") as f:
        f.write(token)
    with open(path) as f:
        readback = f.read()
    return record("local", readback == token, "disk round trip")

def probe_model():
    started = time.time()
    prompt = "Reply with the single word: ok"
    payload = {
        "model": MODEL_ID,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
        "max_tokens": 8,
    }
    req = __import__("urllib.request").request.Request(
        BASE_URL,
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json", "Authorization": f"Bearer {API_KEY}"},
        method="POST",
    )
    try:
        with __import__("urllib.request").request.urlopen(req, timeout=30) as r:
            body = json.loads(r.read().decode())
        return record("model", True, "call completed",
                      latency=round(time.time() - started, 3),
                      served_id=body.get("model"),
                      attempt=1)
    except Exception as e:
        # Timeout lands here too. Treat it as unknown, not as failure.
        return record("model", False, type(e).__name__,
                      latency=round(time.time() - started, 3), attempt=1)

if __name__ == "__main__":
    print(probe_local())
    print(probe_model())

Run it twice a day for a week. Then read probe.jsonl and count the rows where local.ok is true and model.ok is false.

That count is the number of times a single health check would have lied to you.

What this does not prove

It does not measure provider reliability. Two probes a day is a smoke test, not a study.
It does not catch slow degradation. A 30-second call that succeeds still hurts.
It does not validate output quality. A round trip can succeed and return nonsense.
It does not survive a full host wipe. The ledger lives on the same disk as the script.

Ship the ledger somewhere else before you depend on it. A file you can lose is a file you will lose.

Who should skip this

If you have a real queue, real tracing, and an SLO on dependency latency, you already own this problem in a better tool. Do not build a probe script to duplicate it.

If your work is regulated, or needs a contractual uptime number, a free tier is the wrong foundation. Use it to learn the failure shape, then move.

And if you only run the model call interactively, in a chat window, you do not need a two-probe check. You have a human watching every call. That human is your monitor.

The short version

Five claims, one fix: stop letting one green light speak for two systems.

Process health is not dependency health
Retries need keys, not courage
Fixed parameters are not a reproducibility contract
A model name is intent, a served id is evidence
A timeout is unknown, not failed

If you want to watch the two domains split on your own box, the free model access and free server option are enough to run probe.py for a week. Then count the mismatched rows and decide what your dashboard has been hiding.

A 200 OK Is Not a Finished Answer: Classifying Model-Call Failures Before You Retry

Jordan Huang — Tue, 27 Oct 2026 19:03:43 +0000

Last week one generation came back with a 200. My parser died at character 812. So was that a failure? Or a good answer that stopped early?

I keep finding the same bug in client code. Every non-200 becomes 'retry'. Every 200 becomes 'done'. Both assumptions burn tokens and hide the real problem.

Here is the mental model I use now. A model call has three outcomes, not two. It fails, it succeeds, or it stops partway. Most clients only have buckets for the first two.

Myth 1: A 200 means the model answered

A 200 means the server accepted the request. That is all. The body can be empty. The body can be an error envelope dressed up as a 200.

Streaming makes this worse. The 200 arrives before the first token exists. The status line says nothing about the content.

So what does your code check after the status line? If the answer is 'nothing', your parser is doing a validator's job.

Myth 2: The stream ended, so the answer is complete

A stream ends when the connection closes. That is not a finished generation.

The evidence you want is finish_reason. Missing means truncated. A value of length means a token cap cut the answer mid-sentence.

Partial JSON often still parses, if you slice carefully. That is the nasty case. You get a valid object with a missing field and zero errors anywhere.

Myth 3: Retrying a call is free

Retries cost tokens, wall time, and sometimes duplicate side effects. If the call also runs a tool, a retry can run that tool twice.

Retry storms also make rate limits worse. A tight loop on a 429 is you building your own outage. Stop the run instead.

Myth 4: If the JSON parses, the schema is fine

Parsing is not validation. Parsing asks 'is this a JSON document?' Validation asks 'does it have the fields I need?'

Check the keys you depend on. Check their types. Reject early, loudly, with the raw text attached.

Myth 5: A slow call is a broken call

Slow and failed are two different axes. A nine-second success is still a success. Collapsing both into one 'unhealthy' gauge hides which one you actually have.

Record elapsed time per call. Decide about it later, with data instead of vibes.

A small probe that classifies outcomes

Here is the artifact I run before trusting a new endpoint. One file, one log, one row per call.

#!/usr/bin/env python3
"""probe.py - classify model-call outcomes instead of guessing.

    python probe.py --url "$MODEL_URL" --key "$MODEL_KEY" --n 12 --out probe.jsonl

Keep --n small. This is an evidence log, not a load test.
The request body is a chat-completions example. Adapt it to your API.
"""
import argparse
import json
import time
import urllib.error
import urllib.request
from collections import Counter

PROMPT = 'Reply with a small JSON object that has keys ok and n.'
REQUIRED = {'ok', 'n'}


def extract(raw):
    """Pull text and finish_reason out of an unknown envelope."""
    try:
        env = json.loads(raw)
    except json.JSONDecodeError:
        return '', None
    choice = (env.get('choices') or [{}])[0]
    msg = choice.get('message') or {}
    text = msg.get('content') or choice.get('text') or ''
    return text, choice.get('finish_reason')


def classify(status, text, finish_reason, err):
    if err:
        return 'TIMEOUT' if err == 'timeout' else 'NETWORK'
    if status == 429:
        return 'RATE_LIMITED'
    if status >= 500:
        return 'SERVER_ERROR'
    if 400 <= status < 500:
        return 'CLIENT_ERROR'
    if not text.strip():
        return 'EMPTY_CONTENT'
    if finish_reason in (None, '', 'length'):
        return 'TRUNCATED'
    try:
        obj = json.loads(text)
    except json.JSONDecodeError:
        return 'NOT_JSON'
    if not REQUIRED.issubset(obj):
        return 'SCHEMA_MISSING_KEYS'
    return 'OK'


def one_call(url, key, timeout):
    body = json.dumps({
        'messages': [{'role': 'user', 'content': PROMPT}],
        'stream': False,
    }).encode('utf-8')
    req = urllib.request.Request(url, data=body, method='POST')
    req.add_header('Content-Type', 'application/json')
    if key:
        req.add_header('Authorization', 'Bearer ' + key)

    start, status, raw, err = time.monotonic(), None, '', None
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            status = resp.status
            raw = resp.read().decode('utf-8', 'replace')
    except urllib.error.HTTPError as e:
        status = e.code
        raw = e.read().decode('utf-8', 'replace')
    except (urllib.error.URLError, OSError) as e:
        err = 'timeout' if 'timed out' in str(e) else 'network'

    text, finish = extract(raw)
    return {
        'status': status,
        'elapsed_s': round(time.monotonic() - start, 3),
        'finish_reason': finish,
        'chars': len(text),
        'outcome': classify(status, text, finish, err),
        'raw_head': raw[:200],
    }


def main():
    p = argparse.ArgumentParser()
    p.add_argument('--url', required=True)
    p.add_argument('--key', default='')
    p.add_argument('--n', type=int, default=10)
    p.add_argument('--sleep', type=float, default=2.0)
    p.add_argument('--timeout', type=float, default=60.0)
    p.add_argument('--out', default='probe.jsonl')
    args = p.parse_args()

    rows = []
    with open(args.out, 'w', encoding='utf-8') as fh:
        for i in range(args.n):
            row = one_call(args.url, args.key, args.timeout)
            row['i'] = i
            fh.write(json.dumps(row) + '\n')
            fh.flush()
            rows.append(row)
            print(i, row['outcome'], row['status'], row['elapsed_s'])
            if row['outcome'] == 'RATE_LIMITED':
                print('rate limited: stop, do not tighten the loop')
                break
            time.sleep(args.sleep)

    print('tally:', dict(Counter(r['outcome'] for r in rows)))
    print('log:', args.out)


if __name__ == '__main__':
    main()

Run it, then summarize without eyeballing twelve rows.

python probe.py --url "$MODEL_URL" --key "$MODEL_KEY" --n 12 --out probe.jsonl
jq -r '.outcome' probe.jsonl | sort | uniq -c | sort -rn
jq -r 'select(.outcome=="TRUNCATED") | [.i,.status,.chars,.finish_reason] | @tsv' probe.jsonl

Turn each outcome into a retry rule

Do not write one except. Write one rule per class.

Outcome	Retry?	Wait	Log
`OK`	no	none	elapsed, chars
`EMPTY_CONTENT`	once	2s	raw_head
`TRUNCATED`	once, smaller cap	2s	finish_reason, chars
`NOT_JSON`	no auto-retry	none	full text
`SCHEMA_MISSING_KEYS`	no auto-retry	none	parsed keys
`RATE_LIMITED`	no	stop the run	status, time
`SERVER_ERROR`	once	5s + jitter	status, raw_head
`TIMEOUT`	once	5s	elapsed, timeout value
`NETWORK`	once	5s	error string
`CLIENT_ERROR`	never	none	status, raw_head

Two rules matter more than the rest. Never retry a 4xx that came from your own bad request. Never auto-retry a schema failure, because it usually repeats.

What the tally actually tells you

Twelve calls in under a minute is a smoke test. It does not measure p50, and you should not report it as one.

It answers a smaller question. Which failure classes does this endpoint produce, for my request shape, right now? That is exactly what your retry code needs to know.

If you see four classes in twelve calls, your old except was hiding all of them. If you see only OK, you learned the happy path and nothing else.

Where I run the probe

My laptop lid closes. That kills a one-minute run halfway through.

For this I want a small box that stays awake for the run and has its own network path. MonkeyCode offers a free server option, and free model access for the model side, so I can keep the whole loop off my own machine. Both are availability claims from the product, not benchmarks from me.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I treat that box as 'somewhere that is not my laptop'. Not as a scheduler, not as durable storage. The run writes probe.jsonl, and I copy the log out when it finishes.

Keep runs short and spaced. Free access is not a licence to hammer an endpoint, and I do not assume it stays identical forever.

What this probe does not prove

It is not a benchmark. Twelve calls, one hour, one network path.
It does not test long prompts, tool calls, or image inputs.
It cannot tell you an endpoint changed; it only shows what happened today.
It says nothing about cost, quota, or whether free access continues.

Who should skip this

If you need an SLA number, this is the wrong tool. Use a real load test with a large sample, a fixed schedule, and a proper percentile report.

If you are running at volume, add tracing and per-class metrics in production instead. A probe script is for the day before you trust an endpoint.

If a duplicate tool call would hurt you, fix idempotency first. Retry policy comes second.

The corrected mental model

A response is not one bit. It is a status, a body, a finish reason, and a duration.

Read all four before you decide anything. Then pick a retry rule per class, not per exception type.

If you run the probe, keep --n small, keep the log, and change REQUIRED to match your own schema.

Same Prompt, Different Answer: Five Reproducibility Myths in Free-Model Pipelines

Jordan Huang — Tue, 27 Oct 2026 15:58:38 +0000

Two runs. Same prompt. Different output. Is that a bug, or is that just how the endpoint works?

I keep seeing the same claims in review threads. The model is deterministic. A retry is harmless. A 200 means it ran once. Let's test those claims instead of repeating them.

Below is a small harness you can run in ten minutes. It uses stdlib Python only. It gives you evidence, not vibes.

The setup: a stub you control

First, a local server that is deliberately nondeterministic. Real endpoints vary too. This one just makes the variation obvious.

# stub_model_server.py
import json, random
from http.server import BaseHTTPRequestHandler, HTTPServer

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        n = int(self.headers.get("Content-Length", 0))
        body = json.loads(self.rfile.read(n) or b"{}")
        prompt = body.get("prompt", "")
        answer = f"answer-{random.randint(1000, 9999)} for: {prompt[:20]}"
        payload = json.dumps({"output": answer}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

HTTPServer(("127.0.0.1", 8765), H).serve_forever()

Run it in one terminal: python3 stub_model_server.py

Now the runner. It writes a two-phase receipt: one row before the call, one row after the side effect.

# job_runner.py
import hashlib, json, os, time, urllib.request, uuid

BASE_URL = os.environ.get("MODEL_BASE_URL", "http://127.0.0.1:8765/generate")
LEDGER = os.environ.get("LEDGER_PATH", "receipts.jsonl")

def key_for(prompt, params):
    blob = json.dumps({"prompt": prompt, "params": params}, sort_keys=True)
    return hashlib.sha256(blob.encode()).hexdigest()[:32]

def call_model(prompt, params):
    req = urllib.request.Request(
        BASE_URL,
        data=json.dumps({"prompt": prompt, **params}).encode(),
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.loads(r.read())["output"]

def record(row):
    with open(LEDGER, "a") as f:
        f.write(json.dumps(row) + "\n")

def applied(key):
    if not os.path.exists(LEDGER):
        return False
    with open(LEDGER) as f:
        return any(
            json.loads(l)["key"] == key and json.loads(l)["status"] == "applied"
            for l in f if l.strip()
        )

def run(prompt, params):
    key = key_for(prompt, params)
    if applied(key):
        print("replay detected, skipping side effect:", key)
        return key
    record({"key": key, "status": "started", "ts": time.time()})
    out = call_model(prompt, params)
    record({
        "key": key,
        "status": "applied",
        "attempt": str(uuid.uuid4()),
        "output_sha": hashlib.sha256(out.encode()).hexdigest()[:16],
        "ts": time.time(),
    })
    print("output_sha:", hashlib.sha256(out.encode()).hexdigest()[:16])
    return key

if __name__ == "__main__":
    run("summarize ticket 42", {"temperature": 0})

Run it twice: python3 job_runner.py. Compare the two output_sha values. That is myth one, dead.

Myth 1: "Temperature 0 makes it repeatable"

Temperature 0 narrows sampling. It does not promise an identical string.

My two runs above differ. Same prompt, same params, same server. Different hash.

So what do you actually need? A fingerprint of the output, stored next to the request. Not a promise of sameness. Then you can detect drift instead of arguing about it.

What to record instead

Request key: hash of prompt plus params, sorted.
Output hash: short SHA of the returned text.
Attempt id: a UUID per call, so duplicates are visible.
Status: started and applied, never one row.
Timestamp, in UTC, from the runner.

Five fields. That's the whole trick.

Myth 2: "A retry only costs time"

Does it, though? A retry can be a second side effect.

Picture this. Your worker calls the model, gets a timeout, retries. Both calls actually succeeded. Now you have two summaries, two emails, two tickets.

The guard is an idempotency key. Mine is key_for(prompt, params). Same input, same key. The ledger refuses the second apply.

But look at my code. The side effect runs after the model call. If the process dies in between, you get an orphan: started, never applied. That's myth two's real cost — not the call, the ambiguity.

# audit.py
import json
rows = [json.loads(l) for l in open("receipts.jsonl") if l.strip()]
started = {r["key"] for r in rows if r["status"] == "started"}
applied = {r["key"] for r in rows if r["status"] == "applied"}
print("orphaned keys:", sorted(started - applied))

Run python3 audit.py and kill the runner mid-flight. You'll see the gap. An orphan is not a failure. It's a question your logs now ask out loud.

Myth 3: "HTTP 200 means it ran exactly once"

A 200 tells you one response arrived. It says nothing about how many attempts happened upstream.

Timeouts, proxy retries, and client reconnects all break the one-to-one story. So treat every response as at least once. Design your write path for duplicates.

Ask yourself: if this job ran twice, what breaks? If the answer is "nothing," good. If it's "the invoice," you have work to do.

Myth 4: "The model name pins the behavior"

Model names are labels. Endpoints get updated behind labels.

I'm not claiming a specific provider changed anything on a specific date — check their changelog today, not my memory. The general lesson holds: a name is not a hash.

So log a response fingerprint on a fixed canary prompt. Run it daily. When the hash shifts, you learn about it from your ledger, not from a user.

This is where free access gets useful. MonkeyCode's free model access gives you a low-pressure place to run that canary (operator-supplied availability; confirm current terms yourself). Cheap enough to test daily, which is the point.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Myth 5: "Local logs are a ledger"

The ledger must outlive the process that wrote it. Files on ephemeral disk are not a ledger. They are a rumor.

MonkeyCode also offers a free server option. If you run the runner there, keep LEDGER_PATH on durable external storage — object storage, a managed database, or a volume the provider documents as persistent. Otherwise audit.py finds nothing after a restart.

I learned this the boring way, so you don't have to.

Who should not use this

Be honest about the fit.

Throwaway scripts. If a duplicate costs nothing, skip the ledger. It's overhead.
Strict exactly-once needs. A file ledger gives at-least-once plus detection. It is not a transaction. Use a database with unique constraints.
Multi-writer queues. Two writers appending to one file will interleave rows. Move to SQL first.
Teams without cleanup. Orphans pile up. If nobody runs the audit, you built a diary, not a control.

Run it in ten minutes

Start the stub. Run the job twice. Diff the hashes. Kill it once. Run the audit.

That's four commands and one honest conclusion: reproducibility is something you measure, not something you assume.

Which myth breaks first in your logs? I'd bet on the retry.

Your Free Server Forgot Your Files: Four Restart Myths and a Survival Check

Jordan Huang — Tue, 27 Oct 2026 13:54:16 +0000

Your job wrote the file. You redeployed. The file was gone.

So which part failed: the write, or your assumption about the disk?

The bug shape you will hit

A scheduled job appends rows to state.jsonl.
The container restarts for an unrelated reason.
The file comes back empty, missing, or half-written.
The job re-runs and double-sends, double-charges, or double-builds.

Nothing in that list is exotic. Free compute just makes it routine.

Myth 1: "A restart is a clean reset"

Restarting a process clears memory. It does not clear disk.

That cuts both ways. Your cache and in-memory locks disappear, which is fine. Your stale worker.lock, your half-finished .part file, and yesterday's build artifact may not.

A restart is a partial reset you never specified. Good to know before you debug it at 2am.

Evidence: write a marker file, restart, read it back. That is the whole test. We automate it below.

Myth 2: "/tmp is temporary, the app dir is permanent"

Folder names are not durability policies.

On plenty of container hosts, the app directory is an image layer. A redeploy rebuilds it. Your "permanent" file was never on a volume; it was on a layer that gets replaced.

/tmp may be a tmpfs. It may also be the same overlay as everything else. The name tells you nothing.

Corrected model: ask where the bytes physically land, not what the folder is called.

Myth 3: "If the write returned, the data is safe"

Three boundaries exist here, and they fail separately.

write() plus flush() moves bytes from your process to the OS page cache.
os.fsync() pushes them toward the storage device.
The volume itself can be discarded on the next deploy or scale event.

Steps 1 and 2 protect you from a crash. Nothing in your code protects you from step 3.

Myth 4: "Free compute is only for demos"

Wrong in one direction, right in another.

Free compute can run real jobs, as long as state lives outside it and every job is idempotent. What it cannot be is your database, your queue, or your source of truth.

Treat it as disposable compute and it behaves predictably. Treat it as a server and it will surprise you.

The probe: a survival check

Two modes. write appends a checksummed record. verify re-reads and re-hashes every line.

#!/usr/bin/env python3
"""survival_check.py -- does this path survive a restart?"""
import argparse, hashlib, json, os, time, uuid
from pathlib import Path


def target(state_dir: Path) -> Path:
    return state_dir / "survival.jsonl"


def digest(rec: dict) -> str:
    body = json.dumps(rec, sort_keys=True)
    return hashlib.sha256(body.encode("utf-8")).hexdigest()


def cmd_write(state_dir: Path, note: str) -> None:
    state_dir.mkdir(parents=True, exist_ok=True)
    rec = {"id": uuid.uuid4().hex[:12], "note": note, "ts": round(time.time(), 3)}
    rec["sha256"] = digest(rec)
    with open(target(state_dir), "a", encoding="utf-8") as f:
        f.write(json.dumps(rec, sort_keys=True) + "\n")
        f.flush()
        os.fsync(f.fileno())   # survives a crash, not a redeploy
    print(f"wrote id={rec['id']} to {target(state_dir)}")


def cmd_verify(state_dir: Path) -> int:
    path = target(state_dir)
    if not path.exists():
        print(f"MISSING {path}")          # the path did not survive
        return 2
    bad, n = 0, 0
    for n, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
        rec = json.loads(line)
        claimed = rec.pop("sha256", None)
        ok = claimed == digest(rec)
        bad += 0 if ok else 1
        print(f"line={n} id={rec['id']} checksum={'ok' if ok else 'MISMATCH'}")
    print(f"lines_checked={n} mismatches={bad}")
    return 0 if bad == 0 else 1


if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("mode", choices=["write", "verify"])
    ap.add_argument("--dir", required=True, type=Path)
    ap.add_argument("--note", default="probe")
    a = ap.parse_args()
    raise SystemExit(cmd_verify(a.dir) if a.mode == "verify" else cmd_write(a.dir, a.note))

Write to three candidate paths, then restart the process or container:

for d in /tmp/survival ./state /data/survival; do
  python3 survival_check.py write --dir "$d" --note before-restart \
    || echo "WRITE FAILED $d"
done

# restart the process, container, or instance here

for d in /tmp/survival ./state /data/survival; do
  echo "== $d"
  python3 survival_check.py verify --dir "$d"
done

Reading the results

verify output	what it means	what to do
`MISSING`	the path vanished	move state to a real volume or external store
`MISSING` but `write` succeeded	write went to a layer that was discarded	stop trusting that path
`mismatches > 0`	bytes changed or truncated	suspect the storage layer, not your hash
all `ok`	survived this restart	repeat after a redeploy before believing it

One clean pass proves very little. A restart is not a redeploy. A redeploy is not a scale event. Run the probe after each one you can trigger on purpose.

Then make the job idempotent

Persistence is half the fix. The other half is a key.

def run_once(job_key: str, store) -> bool:
    if store.seen(job_key):      # store must live outside the free box
        return False
    store.mark(job_key)
    do_the_work(job_key)
    return True

If store is a file on the ephemeral disk, you renamed the bug. You did not fix it.

Where free model access fits

I use a model for two narrow jobs here: drafting the checksum logic, and reviewing idempotency keys for collisions. Both are text problems with testable answers, which is where a model earns its keep.

MonkeyCode is an open-source project offering free model access and a free server option. That is enough to run the write → restart → verify loop above without paid infrastructure. The exact free allowance changes over time, so read the current terms on the project page instead of trusting a number quoted in any article, including this one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Do not read "it ran on free compute" as a durability claim. Read it as evidence about one boundary, on one day.

Who should skip this

You need a written durability or uptime guarantee. Use a managed database and a paid runtime.
Your data carries residency or audit requirements a shared free host cannot meet.
Your job moves money or sends messages. Get a real queue first.
You cannot redeploy on demand. Then you cannot test the boundary that matters.

The one-line version

A restart is a memory reset, not a disk reset. Probe the disk, externalize the state, key the job. Then trust it, after you have re-run the probe once more.

Want to try the loop yourself? The free server option is a reasonable place to start.

Same URL, Different Machine: Four State Myths on a Free Server

Jordan Huang — Tue, 27 Oct 2026 12:53:33 +0000

You deploy to a free server. The first request works. The second one forgets everything.

Is your code wrong? Was it a restart? Or is the URL just a label in front of a machine you never met?

I keep seeing the same three guesses in issue threads. All three are myths. Here is how I check, with a probe you can run in ten minutes.

The setup: one route, two questions

I want two answers from any free box. Which process am I talking to? And what state can that process honestly promise?

So I add a fingerprint route. It is boring on purpose. Stdlib only, no dependencies.

# app.py - stdlib only, no frameworks
import json, os, time, uuid
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

NONCE = uuid.uuid4().hex[:12]   # new value per process
COUNT = 0                        # a lie we are about to test

def boot_id():
    try:
        return open("/proc/sys/kernel/random/boot_id").read().strip()
    except OSError:
        return "unavailable"

def proc_start_ticks():
    # /proc/self/stat field 22 = starttime, in clock ticks since boot
    try:
        raw = open("/proc/self/stat").read().rsplit(") ", 1)[1].split()
        return int(raw[19])
    except (OSError, IndexError, ValueError):
        return -1

class H(BaseHTTPRequestHandler):
    def do_GET(self):
        global COUNT
        COUNT += 1
        body = json.dumps({
            "nonce": NONCE,
            "pid": os.getpid(),
            "boot_id": boot_id(),
            "proc_start": proc_start_ticks(),
            "count": COUNT,
            "uptime_s": round(time.monotonic(), 1),
            "host": os.uname().nodename,
        }).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *a):
        pass

ThreadingHTTPServer(("0.0.0.0", 8080), H).serve_forever()

Run it, expose port 8080, then poll it from your laptop:

# probe.py  ->  python probe.py https://your-app.example.com
import json, sys, time, urllib.request

url = sys.argv[1].rstrip("/") + "/"
prev = None
for i in range(12):
    with urllib.request.urlopen(url, timeout=10) as r:
        d = json.load(r)
    key = (d["nonce"], d["boot_id"], d["proc_start"])
    print(f"{i:02d} {'same' if key == prev else 'NEW':4s} "
          f"count={d['count']:3d} up={d['uptime_s']:7.1f}s "
          f"pid={d['pid']} host={d['host']}")
    prev = key
    time.sleep(5)

That is the whole harness. Twelve requests, five seconds apart. Cheap enough that you will not fight a rate limit.

Myth 1: "One URL means one process."

This is the most expensive myth on the list. A URL is routing, not identity.

Look at the nonce. If it survives every poll, you have one process. If it changes, you are talking to someone new.

Now compare boot_id. Same nonce, same boot, same process. Different boot means a different kernel booted, so a different machine answered you.

Same boot but a new proc_start? Your process restarted on the same host. That is a crash, a redeploy, or an idle shutdown waking back up.

Illustrative output, not a measurement from your box:

00 same count=  1 up=    0.4s pid=41 host=box-a
01 same count=  2 up=   5.4s pid=41 host=box-a
02 NEW  count=  1 up=    0.4s pid=77 host=box-a

That jump at index 02 is the whole story. Same host, new PID, counter back to one.

Myth 2: "In-memory state is cheap, so keep it."

In-memory state is cheap. It is also the first thing you lose.

Ask a sharper question. What does this counter actually guarantee? Nothing beyond the life of the process.

So I sort state by what breaks when it disappears:

Liveness flags, caches, warm handles: fine in memory. Losing them costs latency, not truth.
Job progress, retry budgets, "already emailed": never in memory. Losing them costs correctness.
Counters a user paid for: never on the box at all. The box is not your ledger.

On a free server the suspend/resume behaviour is the thing to verify, not assume. I would rather measure it once than argue about it in a thread.

Myth 3: "A 200 with a body means the model finished."

This one hides inside free model access. You get a clean response, a tidy JSON body, and a stop reason you never read.

Then someone asks why the last paragraph is cut. The response was never complete. It was just delivered.

So I check the reason field on every call in the path I care about. Field names vary by provider, so read yours:

# shape varies - adapt, do not copy blindly
choice = resp.json()["choices"][0]
reason = choice.get("finish_reason")
if reason == "length":
    raise RuntimeError(f"truncated, not complete: {reason}")

Now the failure is loud. A truncated answer becomes an exception, not a mystery.

This is the workflow I actually run: the fingerprint probe on a free server to see what my process is, and a free model call to check whether my completion is real. That pairing is why I use MonkeyCode's free model access and free server option, both operator-supplied availability rather than anything I can promise for you.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Myth 4: "If the write returned, the data landed."

A 200 on open(...).write() means the kernel accepted bytes. It does not mean the disk kept them past the next restart.

On ephemeral filesystems, files vanish with the instance. So I test the claim instead of trusting it. Write a marker, force a restart, poll again.

curl -s $URL >/dev/null; echo "marker-$(date +%s)" > /tmp/marker
# restart the instance from your dashboard, then:
curl -s $URL/marker   # 404 here means: treat /tmp as cache, not storage

If that 404 happens, you learned something real in one restart instead of one incident.

The decision table I keep next to the probe

Signal you saw	Likely cause	Correct response
`nonce` changed, `boot_id` same	Process restart or idle recycle	Move durable state out of memory
`boot_id` changed	Different host or fresh boot	Never assume local disk persists
`count` resets between polls	New process answered	Check your retry and dedup keys
`count` jumps by more than your polls	More than one instance served	Read replicas cannot own a counter
`finish_reason == "length"`	Output cap hit	Treat as failure, retry with a plan

Five rows. That is the entire mental model. State belongs where instance identity does not matter.

Limitations, and who should skip this

This probe measures identity, not performance. It says nothing about latency, throughput, or cost.

Skip it if all four of these are true for you: paid always-on compute, one instance, a real database, and no idle suspend. You already have the guarantees you need.

The free-server route also is not for everything. If you need predictable cold start under a second, a suspend-on-idle box will fight you. If you need a signed uptime guarantee, read the terms before writing code around it.

One more warning. Do not loop the probe forever. Twelve requests is a test. Twelve thousand is a quota incident you caused yourself.

The corrected mental model

A URL is a promise to route bytes. It is not a promise about which process receives them.

A 200 is a delivery receipt. It is not a completion proof. A file write is a syscall. It is not a durability guarantee.

So fingerprint the process, read the stop reason, and move durable state somewhere that survives a reboot. Which of those three are you skipping right now?

nohup Is Not a Checkpoint: An FAQ on Batches That Outlive Your SSH Session

Jordan Huang — Tue, 27 Oct 2026 09:39:18 +0000

You start a 200-item batch on a free server. You close the laptop. You come back to 40 results and a dead prompt.

Why does this keep happening? Because we repeat five claims about long jobs. Most of them are wrong.

This is a myth-busting FAQ. Each answer has a cheap test you can run. The artifact is a small journal-first runner in plain Python.

The setup: a free box, a batch, and one bad assumption

The job is boring. Read 200 prompts, call a model, write the answer down.

MonkeyCode offers free model access and a free server option, both operator-supplied. That combination is exactly where this failure shows up. Cheap compute plus a remote shell invites long jobs.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

So let me be precise about what the box gives you and what it does not.

FAQ: five claims, five corrections

1. "nohup keeps my batch running."

No. nohup only detaches your process from a terminal hangup.

It changes signal delivery. It says nothing about the machine staying up.

If the box is recycled, all PIDs die together. Detaching cannot save you.

Run this test. Start a loop that writes a timestamp every second. Kill your SSH connection, wait ten minutes, reconnect.

nohup bash -c 'while true; do date -u +%FT%TZ >> /tmp/hb.log; sleep 1; done' \
  >/dev/null 2>&1 &
disown

If the file keeps growing, you learned one thing. The box survived ten minutes. That is all you learned.

Corrected model: your process is not your progress. The durable unit is the file you append to.

2. "My client printed the whole answer, so it is saved."

Did you write it to disk before you saw it? That is the only question that matters.

A streamed response lives in a socket buffer and then in your memory. Both vanish on SIGKILL.

Corrected model: a result exists when write() plus fsync() returned. Print statements are not persistence.

That is why the runner below writes one line per item and flushes it.

3. "Resuming means starting over."

Only if you never recorded what finished.

A journal turns resume into a set difference. Read the keys you already have. Skip them. The rest is your queue.

No database. One append-only JSONL file is enough.

4. "Long jobs just need a bigger box or a longer timeout."

Timeouts move the wall. They do not move the crash.

Long-running work is fragile because each unit is large. Shrink the unit and the fragility drops.

Corrected model: durability comes from small, committed steps. Not from generous limits.

5. "A journal makes retries exactly-once."

Careful. It usually gives you at-least-once execution plus dedupe on read.

If you append after the call, a crash between call and append repeats that item. If you append before, a crash loses it.

Pick your failure. For model calls, repeating is usually cheaper than losing.

The artifact: a journal-first batch runner

Standard library only. No frameworks. Read it before you trust it.

The endpoint, model, and key come from environment variables. Nothing is hardcoded.

#!/usr/bin/env python3
"""Journal-first batch runner. One writer. Append, fsync, resume."""
import hashlib, json, os, sys, time, urllib.request
from pathlib import Path

JOURNAL = Path(os.environ.get("JOURNAL", "results.jsonl"))

def key_for(task: dict) -> str:
    raw = json.dumps(task, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(raw).hexdigest()[:16]

def load_done(path: Path) -> set:
    done = set()
    if not path.exists():
        return done
    with path.open("rb") as fh:
        for line in fh:
            line = line.strip()
            if not line:
                continue
            try:
                rec = json.loads(line)
            except json.JSONDecodeError:
                continue  # torn final line after a hard kill
            done.add(rec["key"])
    return done

def append_record(path: Path, rec: dict) -> None:
    with path.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(rec, sort_keys=True) + "\n")
        fh.flush()
        os.fsync(fh.fileno())

def call_model(prompt: str) -> str:
    # OpenAI-compatible chat endpoint. You supply BASE_URL and MODEL.
    body = json.dumps({
        "model": os.environ["MODEL"],
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(
        os.environ["BASE_URL"].rstrip("/") + "/chat/completions",
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + os.environ.get("API_KEY", ""),
        },
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        payload = json.loads(resp.read())
    return payload["choices"][0]["message"]["content"]

def main() -> int:
    src = Path(sys.argv[1])
    tasks = [json.loads(l) for l in src.read_text().splitlines() if l.strip()]
    dry = "--dry-run" in sys.argv
    done = load_done(JOURNAL)
    todo = [t for t in tasks if key_for(t) not in done]
    print(f"{len(tasks)} tasks, {len(done)} journaled, {len(todo)} to run")
    if dry:
        for t in todo[:5]:
            print("would run", key_for(t))
        return 0
    for task in todo:
        k = key_for(task)
        out = call_model(task["prompt"])
        append_record(JOURNAL, {
            "key": k,
            "prompt": task["prompt"],
            "out": out,
            "ts": int(time.time()),
        })
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Step 1: plan before you spend

BASE_URL=... MODEL=... python runner.py tasks.jsonl --dry-run

You get counts and the first five keys. No calls. No surprises.

Step 2: run, then kill it on purpose

Start the batch. After a few records land, send kill -9 to the process group.

BASE_URL=... MODEL=... nohup python runner.py tasks.jsonl >run.log 2>&1 &
kill -9 -$!   # or: pkill -9 -f runner.py

Now rerun the same command. Watch the header line.

Step 3: audit the journal, not the log

jq -r .key results.jsonl | sort | uniq -d | head   # duplicates: expect empty
jq -r .key results.jsonl | sort -u | wc -l         # unique keys: expect 20

Twenty unique keys after a hard kill is the whole point. A torn last line is fine. The loader skips it.

When this is worth the extra file

Situation	Journal?	Why
Single one-shot call	No	Nothing to resume
Batch under ~1 minute	No	Rerun is cheaper than the code
50+ external calls	Yes	Partial progress is real value
Paid per call	Yes	Losing work costs money
Side effects outside the box	Yes, plus dedupe	At-least-once will repeat

If a row says "No", skip this. Simplicity wins.

Limitations, stated plainly

The journal only survives if the disk survives. Copy it off the box.

A git push, an rsync, or an object upload after every N records is enough.

Keep one writer per journal file. Do not lean on O_APPEND for multi-process safety.

fsync per record costs time. Batch it if your item rate is high.

And this pattern does not give you exactly-once. It gives you resumable at-least-once. Design downstream effects accordingly.

Who should not use this

Anyone whose whole job fits in one model call. Stop here.

Anyone with a real job queue and workers. Use that instead.

Anyone who needs strict ordering across items. This runner has none.

The code above is a teaching artifact, not a reviewed library. Check the timeout, error handling, and retry behaviour against your endpoint before a real run.

What changed in my head

I stopped treating the shell session as state. It is a window, not a ledger.

I stopped measuring progress by log lines. I measure it by unique journal keys.

And I stopped asking "how long will this take?". I ask "what happens if it dies at item 41?".

Try the kill -9 test once. If your rerun starts at item 1, you found your bug.

If you want to run that test without a bill, MonkeyCode's free model access and free server option are one place to start. Point BASE_URL and MODEL at whatever endpoint you actually use.

Your Health Check Was Green. The Server Was Asleep.

Jordan Huang — Tue, 27 Oct 2026 06:33:52 +0000

You pinged `/health`. It said 200. Then the page took four seconds.

You check a health endpoint before a demo. It answers in 40 ms. Then you load the real page.

Four seconds of nothing. Then everything works, until you go idle again.

So which request did you actually measure? Not the one your users make.

Why a free box is even in this conversation

I run small probes and demo apps on a free server option. I also run throwaway evaluations against free model access.

That is exactly where cold-start confusion bites. You get real infrastructure. You do not get a warm one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two rules keep me honest:

Measure the first request after idle, not the steady state.
Treat every free box as disposable until a probe says otherwise.

Here are the myths I hear most, and what actually holds up.

Myth 1: "200 OK means the app is warm"

The claim

If /health returns 200, the service is ready for traffic.

The evidence against it

Readiness and warmth are different properties:

Readiness answers: is the process accepting connections?
Warmth answers: how long does a real request take right now?
A trivially cheap endpoint can be ready while everything expensive is cold.

Your /health handler may not touch the template engine, the DB pool, or the auth cache. The real route touches all three.

Corrected model

Warmth is a latency distribution, not a boolean. You do not "have" it. You measure it per request.

Myth 2: "Idle time is free time"

The claim

An unused instance costs nothing and behaves the same when you return.

The evidence against it

The bill arrives on the first request after idle. That request pays for process start, imports, and socket setup.

If you average latency over a day of traffic, the cold request disappears. Your users never see averages. They see the first one.

Corrected model

Bucket your latency by idle gap. A p50 over all requests hides the only number you care about.

Ask yourself: when did I last measure a request after 30 minutes of silence?

Myth 3: "The filesystem remembers"

The claim

You wrote a file yesterday, so it is there today.

The evidence against it

Many free hosting options give you an ephemeral filesystem. Anything outside your deployed artifact can vanish on restart, redeploy, or scale event.

Test it with a sentinel file. If it disappears, that was never storage. That was a cache with extra steps.

Corrected model

State belongs in an external store or in your deploy artifact. The box holds neither.

Myth 4: "`uptime` tells you how long your app has run"

The claim

uptime printed 12 days, so the app has been running for 12 days.

The evidence against it

In a container, /proc/uptime reports the host kernel's uptime. It does not report your container's age.

Your process may have restarted five times under a host that never rebooted. The clock stays calm. Your app does not.

Corrected model

Use three separate signals:

boot_id from /proc/sys/kernel/random/boot_id, to detect host or VM resets.
A sentinel file with an mtime, to detect filesystem resets.
Your app's own start timestamp, written at import time.

Myth 5: "A retry fixes it"

The claim

If the first call is slow or fails, just try again.

The evidence against it

Retries help with a cold start. They do not help with lost state. Retrying a stateful call after a filesystem reset can double-apply work.

And on a model endpoint, a blind retry can double your token spend on a request that already succeeded slowly.

Corrected model

Retry only idempotent reads. For writes, check the sentinel first. For model calls, log the outcome before you decide to resend.

The probe: measure warmth, not liveness

The script below uses only the Python standard library. It records one JSON line per run.

#!/usr/bin/env python3
# warmth_probe.py - cold start vs steady state, stdlib only
import json, os, sys, time, uuid, urllib.request
from datetime import datetime, timezone

SENTINEL = os.environ.get('SENTINEL', '/tmp/warmth_sentinel')
URL = sys.argv[1] if len(sys.argv) > 1 else 'http://127.0.0.1:8080/health'

def read(path, cast=str):
    try:
        with open(path) as f:
            return cast(f.read().split()[0])
    except (OSError, ValueError):
        return None

def probe():
    started = time.monotonic()
    try:
        with urllib.request.urlopen(URL, timeout=30) as r:
            status, body = r.status, r.read(64)
    except Exception as e:
        status, body = -1, str(e).encode()
    elapsed_ms = round((time.monotonic() - started) * 1000, 1)

    sentinel_age = None
    if os.path.exists(SENTINEL):
        sentinel_age = round(time.time() - os.path.getmtime(SENTINEL), 1)
    else:
        with open(SENTINEL, 'w') as f:
            f.write(uuid.uuid4().hex)

    return {
        'ts': datetime.now(timezone.utc).isoformat(),
        'status': status,
        'elapsed_ms': elapsed_ms,
        'boot_id': read('/proc/sys/kernel/random/boot_id') or 'unknown',
        'host_uptime_s': read('/proc/uptime', float),
        'sentinel_age_s': sentinel_age,   # null means this filesystem is new
        'body_prefix': body[:32].decode('utf-8', 'replace'),
    }

print(json.dumps(probe()))

Run it, append, repeat:

chmod +x warmth_probe.py
./warmth_probe.py 'http://127.0.0.1:8080/health' >> warmth.jsonl

Schedule it unevenly on purpose

A fixed 5-minute cron never sees long idle. You will measure warm requests forever and learn nothing.

Use two schedules instead:

# frequent sampling
*/5 * * * * cd /app && python3 warmth_probe.py 'http://127.0.0.1:8080/health' >> warmth.jsonl

Then run one manual probe after a long gap, such as the first request of your morning. That single line is the interesting one.

Reading the log

import json
from datetime import datetime

rows = [json.loads(line) for line in open('warmth.jsonl')]
rows.sort(key=lambda r: r['ts'])

def pct(xs, p):
    xs = sorted(xs)
    return xs[max(0, min(len(xs) - 1, round(p / 100 * (len(xs) - 1))))]

fresh = [r['elapsed_ms'] for r in rows if r['sentinel_age_s'] is None]
warm = [r['elapsed_ms'] for r in rows if r['sentinel_age_s'] is not None]

print('filesystem resets:', len(fresh))
print('boot_id values:', len({r['boot_id'] for r in rows}))
print('warm p50:', pct(warm, 50), 'warm p95:', pct(warm, 95))

buckets, prev = {}, None
for r in rows:
    t = datetime.fromisoformat(r['ts'])
    if prev:
        gap = (t - prev).total_seconds()
        key = 'under 1m' if gap < 60 else 'under 15m' if gap < 900 else '15m plus'
        buckets.setdefault(key, []).append(r['elapsed_ms'])
    prev = t

for key, xs in sorted(buckets.items()):
    print(key, 'n =', len(xs), 'p50 =', pct(xs, 50))

Example output shape, not a benchmark:

filesystem resets: 3
boot_id values: 2
warm p50: 41.0 warm p95: 88.2
under 1m n = 96 p50 = 44.0
15m plus n = 5 p50 = 2140.5

If the last line looks like that, your health check was never lying. It was answering a different question.

A decision table

Workload	Free box OK?	Because
Static docs, demo app	Yes	Cold start is tolerable, state is trivial
Nightly batch job	Usually	Runs long enough to amortize startup
User uploads	No	Ephemeral filesystem eats them
Anything with an SLA	No	You cannot promise someone else's idle policy
Model evaluation runs	Yes, with logging	Token cost is per call, not per hour

Limitations, and who should skip this

This probe has honest limits:

It measures from inside the box. Platform queueing before your process starts is invisible.
/proc is Linux-only. On other systems, check boot_id and fall back to your own start timestamp.
A single sentinel file cannot detect partial resets in subdirectories.
Free offerings change. A probe log from last month proves last month.
The probe itself sends traffic. Do not loop it against a rate-limited endpoint.

Skip this approach if you need contractual uptime, handle regulated data, or run a stateful service. A free box is a lab bench, not a stage.

What to do in the next ten minutes

Drop the probe next to your app and point it at the real route, not /health.
Run it once. Then wait. Run it again after you go get coffee.
Compare the two lines. Bucket by gap, never by average.
Delete every piece of state that is not in your deploy artifact.

The health check was green the whole time. The question is whether your measurement was.

If you try this against a free model endpoint as well, log the outcome of each call before you log the latency. That is where the surprises hide. Running the probe on a free server option is a cheap way to see your own idle policy instead of guessing it.

The Free Server Slept Overnight: Four State Claims Your App Can't Back Up

Jordan Huang — Tue, 27 Oct 2026 03:28:57 +0000

Your app worked at 6pm. At 9am it half-worked.

Nothing changed in git. So what changed?

The process died and came back. That is the whole story.

Most "free server" bugs are not code bugs. They are state bugs wearing a costume.

This post is not about portability. Portability asks "does it run there?" Durability asks "does it still run tomorrow?" Different question, different probe.

Below is a small app, a restart probe, and four claims I keep hearing. Each claim gets a test you can run in about ten minutes.

What this post does and does not claim

I am not claiming your free server is unreliable. Many are fine.

I am claiming you have not measured which state survives a restart. That gap is where the surprises live.

MonkeyCode advertises free model access and a free server option. Those are operator-supplied availability claims, not guarantees I verified. Treat them as a place to run the probe, not as proof of anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The test app: five state surfaces, one file

Keep it boring. Dependencies stay minimal.

# app.py  (FastAPI; pip install fastapi uvicorn)
import os, uuid
from pathlib import Path
from fastapi import FastAPI, Response
from typing import Optional

app = FastAPI()
BOOT_ID = uuid.uuid4().hex
DATA = Path(os.environ.get("DATA_DIR", "./data"))
DATA.mkdir(parents=True, exist_ok=True)
DISK_COUNTER = DATA / "counter.txt"
TMP_MARKER = Path("/tmp/restart_probe_marker")
MEM = {"n": 0}

@app.get("/bump")
def bump(response: Response, sid: Optional[str] = Cookie(default=None)):
    MEM["n"] += 1
    disk = int(DISK_COUNTER.read_text()) if DISK_COUNTER.exists() else 0
    DISK_COUNTER.write_text(str(disk + 1))
    TMP_MARKER.write_text(BOOT_ID)
    sid = sid or "sess-" + uuid.uuid4().hex[:8]
    response.set_cookie("sid", sid, httponly=True)
    return {"boot_id": BOOT_ID, "mem": MEM["n"], "sid": sid}

@app.get("/state")
def state(sid: Optional[str] = Cookie(default=None)):
    return {
        "boot_id": BOOT_ID,
        "mem": MEM["n"],
        "disk": int(DISK_COUNTER.read_text()) if DISK_COUNTER.exists() else None,
        "tmp_marker": TMP_MARKER.exists(),
        "sid": sid,
        "env_probe": os.environ.get("PROBE_TOKEN"),
    }

Five surfaces: process memory, the data directory, /tmp, a client cookie, and an env var.

That covers most of what people assume is "persisted."

The probe: snapshot, restart, snapshot

Run it against local Docker first. Then point it at the free server.

# probe.py  (python probe.py http://127.0.0.1:8000 "docker compose restart app")
import json, subprocess, sys, time, urllib.request, http.cookiejar

BASE = sys.argv[1]
RESTART_CMD = sys.argv[2]

jar = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(jar))

def call(path):
    with opener.open(BASE + path, timeout=30) as r:
        return json.loads(r.read())

before = call("/bump")
state_before = call("/state")
print("restart:", RESTART_CMD)
subprocess.run(RESTART_CMD, shell=True, check=True)

for _ in range(30):
    try:
        state_after = call("/state")
        break
    except Exception:
        time.sleep(2)
else:
    sys.exit("server never came back")

print(json.dumps({"before": state_before, "after": state_after}, indent=2))

rows = [
    ("boot_id changed (control)", state_before["boot_id"] != state_after["boot_id"]),
    ("mem counter kept",         state_after["mem"] >= state_before["mem"]),
    ("data dir kept",            state_after["disk"] == state_before["disk"]),
    ("/tmp marker kept",         state_after["tmp_marker"] is True),
    ("cookie kept",              state_after["sid"] == state_before["sid"]),
    ("env var kept",             state_after["env_probe"] == state_before["env_probe"]),
]
for name, ok in rows:
    print(("PASS  " if ok else "DRIFT " ) + name)

Read the first line before anything else. If boot_id did not change, you did not restart anything.

A probe against a process that never died proves nothing. This is the control, not a nice-to-have.

Claim 1: "If I write a file, it stays."

It stays until the container is replaced. Replacement is not the same as restart.

Some platforms restart the process but keep the filesystem. Others rebuild the unit from the image.

Your counter.txt tells you which one you got.

Expected result: DRIFT on /tmp, and DRIFT on ./data unless you mounted a volume.

Corrected model: decide where durability is required, then test that exact path. Do not infer it from a different path that happened to survive.

Claim 2: "My in-memory cache and counters are fine."

Fine until the process exits. Then they are zero.

This bites hardest in three places: rate-limit counters, feature flags loaded at boot, and idempotency keys for retried jobs.

Ask the awkward question. If a retry arrives two seconds after a crash, does your dedupe still know?

Expected result: DRIFT on mem. That is correct behaviour, not a bug.

Corrected model: memory is a cache, never a source of truth. Move anything that must outlive a boot to a store that survives it.

Claim 3: "It survived a restart, so it is persistent."

One survival is an anecdote. You need a second class of death.

A process restart and an idle eviction are not the same event. Neither is a redeploy.

Test all three if your platform documents all three. Otherwise you are testing the easy one.

Expected result: a mix. Files may survive restarts and vanish on redeploy.

Corrected model: enumerate the death types, then map which state survives which. A two-column table beats intuition.

Claim 4: "My session is server-side."

The cookie came back, so the login must have survived. Right?

No. The cookie is client state. It proves your HTTP client keeps a jar.

Server-side session data lives in memory or in a store. Restart the process and read it again.

Expected result: PASS on the cookie, DRIFT on whatever backs the session.

Corrected model: separate the token from the record it points at. Probe each one.

Letting a free model label the drift

Now you have a JSON diff and a pile of DRIFT lines. Which ones matter?

That is a policy question, and it is repetitive. Good candidate for a cheap classifier.

Here is the workflow I would use with MonkeyCode's free model access:

Redact the snapshot JSON. Drop hostnames, IDs, and anything user-specific.
Paste a short policy: "DRIFT is breaking if it affects auth, money, or job idempotency."
Ask for one label per line, with a one-sentence reason.
Keep the labels in the repo next to the probe output.

Do not let a model decide what is acceptable. Let it sort lines you already understand.

The value is triage speed, not authority.

Where each kind of state belongs

State	Ephemeral OK?	Safer home
Build output, caches	Yes	Anywhere
Rate-limit counters	No, if you bill on them	Shared store
Idempotency keys	No	Durable key-value store
Session records	Depends on logout cost	Durable store
Uploaded files	No	Object storage
Feature flags	Yes	Boot-time load

The right column is a design decision. The probe only tells you where you stand today.

Limitations, and who should skip this

This probe is deliberately shallow. It will not catch partial writes, silent truncation, or clock skew across nodes.

A local docker compose restart is not a platform idle eviction. Read your platform's documented lifecycle before you trust a PASS.

The env var check is weak. Values baked at build time and read from a secrets manager look identical here.

Skip this approach if you already run a managed database with documented durability. Skip it if your compliance model requires an audit trail rather than a probe.

Never run a restart script against a shared host without written permission. And keep PII out of the probe payload entirely.

The checklist I would keep

Before you trust any free server with real users:

Confirm boot_id changes on restart. Otherwise the test is void.
List the death types your platform documents.
Name every piece of state and where it lives.
Test the paths that matter, not the convenient ones.
Re-run the probe after every platform change.

One green run is a data point. Two runs with different death types is evidence.

If you want to try the classification step without a bill, MonkeyCode's free model option is a reasonable place to start — but the probe is the part that will save your weekend.

A Bigger Context Window Is Not More Memory: Five Token Myths and a Budget Gate

Jordan Huang — Tue, 27 Oct 2026 00:23:25 +0000

Your counter says 12k tokens. The API returns a 400. Which one is lying?

Probably neither. They are counting different things. One of them is counting your string. The other is counting what the model actually receives.

I keep seeing the same five assumptions in code review. Each one feels safe. Each one fails quietly, or fails loudly at 2 a.m.

Here is the FAQ, the evidence you can collect yourself, and a small gate you can run in CI.

Myth 1: "One token is about four characters"

This is a rule of thumb for English prose. Vendors describe it that way. It is not a unit conversion.

The ratio moves with content. Code, JSON, CJK text, emoji, and long identifiers all drift. Whitespace and punctuation get their own pieces. So does anything the tokenizer has never seen.

Corrected model: characters are a hint. Tokens are what the model bills and truncates.

You do not need to trust my ratio. Measure yours. Grab 20 real prompts from your logs. Count them with the tokenizer your provider actually ships. Then compare against len(text) / 4.

Different tokenizers disagree with each other too. tiktoken encodings are one family (see the tiktoken repo). Other model families use other vocabularies. A prompt that fits under one tokenizer can overflow under another.

So: which tokenizer does your budget check use?

Myth 2: "The context window is one number"

It is at least three numbers wearing a trench coat.

Input budget. What the prompt may consume.
Output budget. What the answer may consume. Often a separate parameter with its own ceiling.
Template overhead. Role markers, separators, and tool schemas. Not free. Not zero.

Input plus output usually share one window. Reserve the answer before you count the prompt. Most overflow bugs I read are really "nobody subtracted max_tokens".

Tool schemas are the sneaky part. A generous function signature block can cost more than your whole user message. It is sent on every turn.

Corrected model: budget = window − reserved output − template overhead. Then subtract tool schema cost, every turn.

Myth 3: "More context means better recall"

Long-context evaluations keep reporting uneven recall by position. The well-known result is "Lost in the Middle". Details placed mid-prompt get retrieved less reliably than details at the edges.

That is not a reason to panic. It is a reason to stop stuffing.

Corrected model: context is capacity, not attention. A 200k window does not mean 200k of useful signal. Placement and redundancy matter more than volume.

Practical translation: put the constraint you cannot afford to lose near the top. Put the task near the bottom. Do not bury the schema between two chatty tool results.

Myth 4: "Dropping old messages is always safe"

Old messages carry two things. Conversation, and contract.

When you slice messages[2:], you may delete the system constraints. You may also orphan a tool call. Many APIs reject an assistant tool_calls block that has no matching tool result, or the reverse.

That is the 400 you cannot reproduce locally. Your unit test never had the tool pair in the middle.

Corrected model: truncation is a structural edit, not a string edit. Validate pairing after every cut.

Myth 5: "If the API accepted it, the model saw all of it"

Sometimes something upstream trims. A proxy, a gateway, a retry wrapper, an SDK default. The request succeeds. The answer misses the paragraph you cared about.

You cannot detect that from the response alone. You can detect it from the request you sent.

Corrected model: log the estimate at send time. If the answer looks blind, diff the budget against the prompt you thought you sent.

The gate: one file, standard library only

Here is the artifact. It estimates prompt cost, reserves answer space, and checks tool-call pairing. It exits non-zero when the prompt does not fit.

Treat the built-in counter as a conservative estimate. Swap in a real tokenizer when you have one.

#!/usr/bin/env python3
"""context_gate.py - fail fast when a prompt no longer fits the budget."""
import json
import re
import sys
from dataclasses import dataclass


def estimate_tokens(text: str) -> int:
    # Deliberately conservative placeholder. Never exact. Swap in a real tokenizer.
    words = len(re.findall(r"\w+|[^\w\s]", text))
    chars = (len(text) + 3) // 4
    return max(words, chars)


@dataclass
class Budget:
    context: int       # total window, from the provider docs
    reserve: int       # space kept for the answer
    overhead: int = 8  # role markers / separators per message


def count_prompt(messages, budget: Budget, counter=estimate_tokens) -> int:
    total = 0
    for m in messages:
        total += counter(str(m.get("content") or "")) + budget.overhead
        for call in m.get("tool_calls") or []:
            total += counter(json.dumps(call.get("function", {}))) + budget.overhead
    return total


def unpaired_tool_calls(messages):
    pending, problems = [], []
    for i, m in enumerate(messages):
        for call in m.get("tool_calls") or []:
            pending.append(call.get("id"))
        if m.get("role") == "tool":
            cid = m.get("tool_call_id")
            if cid in pending:
                pending.remove(cid)
            else:
                problems.append(f"message {i}: result for unknown call {cid}")
    problems += [f"call {c}: no result" for c in pending]
    return problems


def check(messages, budget: Budget, counter=estimate_tokens):
    used = count_prompt(messages, budget, counter)
    allowed = budget.context - budget.reserve
    return {
        "prompt_tokens_est": used,
        "prompt_budget": allowed,
        "answer_reserve": budget.reserve,
        "fits": used <= allowed,
        "pairing_problems": unpaired_tool_calls(messages),
    }


if __name__ == "__main__":
    path, context, reserve = sys.argv[1], int(sys.argv[2]), int(sys.argv[3])
    with open(path) as f:
        payload = json.load(f)
    report = check(payload["messages"], Budget(context=context, reserve=reserve))
    print(json.dumps(report, indent=2))
    sys.exit(0 if report["fits"] and not report["pairing_problems"] else 1)

Run it against a captured request:

python3 context_gate.py prompt.json 32768 1024

A failing run looks like this:

{
  "prompt_tokens_est": 31895,
  "prompt_budget": 31744,
  "answer_reserve": 1024,
  "fits": false,
  "pairing_problems": []
}

The 32768 is a flag you supply. Read the real number from your provider's docs. Do not copy mine.

Run it before the request, not after

Capture one real request body as JSON. Keep it as a fixture.
Add your two hardest cases: a large tool schema, and a sliced history.
Run the gate on both. Fix the budget, not the fixture.
Wire it into CI as a preflight step.
Log prompt_tokens_est next to every response you store.

Step 5 is the one people skip. Then the incident review has no numbers.

I run this preflight on MonkeyCode's free server option so the check travels with the job instead of living on my laptop. I also point it at free model access to compare estimates across more than one tokenizer family. That comparison is the whole point — one counter proves nothing.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Cut-first decision table

When the gate says no, do not just slice. Pick the cheapest effective edit.

Symptom	First cut	Why
Tool schema dominates	Prune optional params, shorten descriptions	Paid every turn, not once
Long history, short task	Summarize turns older than N	Keeps the contract intact
One huge document	Move to retrieval, pass top-k	Placement beats volume
Duplicated instructions	Keep one canonical system block	Redundancy is not safety
Answer keeps getting cut	Raise reserve before trimming prompt	Reserve is the real ceiling

Limitations and who should skip this

The built-in counter is an estimate. It will not match provider billing. Use the provider's tokenizer when precision matters.
Windows, reserves, and limits change. Read current docs. This gate reads numbers you pass in, never assumptions.
For multi-megabyte corpora, this is the wrong tool. Use retrieval instead.
If your prompts are short and single-turn, a gate is overhead. Skip it.
This checks shape and budget, not answer quality. A prompt can fit and still be wrong.

The takeaway

Context is a budget with a shape. Tokens vary by tokenizer. Output competes with input. Placement beats stuffing. Truncation is structural. And anything upstream can trim silently.

So the next time a 400 lands on your desk, what will you show the reviewer: a guess about characters, or a gate that printed the number?

200 OK Proves Nothing About Tool Calls: A Free-Model Capability FAQ With a Probe

Jordan Huang — Mon, 26 Oct 2026 21:18:18 +0000

I wired up a free model endpoint last month. I sent one request. It returned 200 OK. I shipped the feature. Then the tool calls never happened. The endpoint accepted the request. It just ignored half of it. Sound familiar?

Here is my question. What does a 200 actually prove? My answer: almost nothing about capabilities. It proves the transport worked. That is it.

So I stopped trusting status codes and started probing the contract. This post is the FAQ I wish I had read first. It ends with a script you can run in about two minutes.

Myth 1: A 200 means the model supports my feature

The server can accept a tools array and silently ignore it. The model then answers in prose. No error. No warning. Your code sees a normal response and skips the branch. Is that a bug in the model? No. It is a bug in your assumption.

The corrected model: support is a property of the endpoint, not of the model name. Probe it. Record it. Re-probe when anything changes.

Myth 2: `finish_reason` is optional metadata

It is the most important field in the response. length means the model ran out of output budget. Your answer looks fine and is probably cut in half. I have watched a parser happily eat a truncated JSON object.

{"choices":[{"finish_reason":"length","message":{"content":"{\"items\": [1, 2, 3, 4"}}]}

If you do not branch on finish_reason, you are parsing whatever bytes arrived. That is not a feature. That is a coin flip.

Myth 3: Streaming and non-streaming are interchangeable

They are two different code paths. In streaming, finish_reason arrives in the final chunk, not the first. Some servers only report token usage when you ask for it explicitly. Some omit it entirely.

Ask yourself: does my client read the last event? Mine did not, for a week. I was logging None as the finish reason every single time.

Myth 4: JSON mode means my schema

It means valid JSON. It does not mean your keys. It does not mean your types. Validate every response against a schema before it touches business logic. Treat model output like user input, because that is what it is.

Myth 5: Retries are always safe

Retry a failed read, fine. Retry a tool call that already wrote a row, and now you have two rows. I learned this from a duplicate charge, not a blog post.

Add an idempotency key your tool honors. Make the retry decision depend on the operation, not on the exception class.

The artifact: a five-minute capability probe

This script uses only the Python standard library. It assumes a chat-completions-shaped endpoint. Adjust the paths if yours differs.

# probe_capabilities.py — smoke test only, not a benchmark
import json, os, time, urllib.request, urllib.error, concurrent.futures as cf

BASE  = os.environ["BASE_URL"].rstrip("/")
KEY   = os.environ["API_KEY"]
MODEL = os.environ["MODEL"]

def call(payload, stream=False, timeout=60):
    req = urllib.request.Request(
        BASE + "/chat/completions",
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}",
                 "Accept": "text/event-stream" if stream else "application/json"},
        method="POST")
    t0 = time.time()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            return {"status": r.status, "ms": int((time.time()-t0)*1000),
                    "raw": r.read().decode("utf-8", "replace")}
    except urllib.error.HTTPError as e:
        return {"status": e.code, "ms": int((time.time()-t0)*1000),
                "raw": e.read().decode("utf-8", "replace")}

def probe_truncation():
    p = {"model": MODEL, "max_tokens": 8,
         "messages": [{"role": "user", "content": "List ten colors."}]}
    r = call(p)
    if r["status"] != 200:
        return {"probe": "truncation", "ok": False, "why": r["status"]}
    ch = json.loads(r["raw"])["choices"][0]
    return {"probe": "truncation", "ok": True,
            "finish_reason": ch.get("finish_reason"),
            "reports_usage": "usage" in json.loads(r["raw"])}

def probe_tools():
    p = {"model": MODEL, "max_tokens": 128,
         "messages": [{"role": "user", "content": "Read ./README.md and report its first line."}],
         "tools": [{"type": "function", "function": {
             "name": "read_file",
             "description": "Read a file from disk.",
             "parameters": {"type": "object",
                            "properties": {"path": {"type": "string"}},
                            "required": ["path"]}}}]}
    r = call(p)
    if r["status"] != 200:
        return {"probe": "tools", "ok": False, "why": r["status"]}
    msg = json.loads(r["raw"])["choices"][0]["message"]
    return {"probe": "tools", "ok": True,
            "returned_tool_calls": bool(msg.get("tool_calls"))}

def probe_json():
    p = {"model": MODEL, "max_tokens": 128,
         "messages": [{"role": "user",
                       "content": 'Reply with only JSON: {"status":"ok","count":3}'}]}
    r = call(p)
    if r["status"] != 200:
        return {"probe": "json", "ok": False, "why": r["status"]}
    text = json.loads(r["raw"])["choices"][0]["message"]["content"]
    try:
        obj = json.loads(text)
        return {"probe": "json", "ok": True,
                "keys_match": sorted(obj) == ["count", "status"]}
    except json.JSONDecodeError:
        return {"probe": "json", "ok": True, "keys_match": False,
                "note": "not parseable as JSON"}

def probe_concurrency(n=2):
    p = {"model": MODEL, "max_tokens": 16,
         "messages": [{"role": "user", "content": "Say ok."}]}
    with cf.ThreadPoolExecutor(max_workers=n) as ex:
        codes = list(ex.map(lambda _: call(p)["status"], range(n)))
    return {"probe": "concurrency", "ok": True, "statuses": codes,
            "saw_429": 429 in codes}

if __name__ == "__main__":
    for fn in (probe_truncation, probe_tools, probe_json, probe_concurrency):
        print(json.dumps(fn(), sort_keys=True))

Run it like this. Keep the key in the environment, never in the file.

export BASE_URL="$YOUR_ENDPOINT"
export API_KEY="$YOUR_KEY"
export MODEL="$YOUR_MODEL"
python3 probe_capabilities.py

Reading the output: a decision table

Probe result	What it means	What I do next
`returned_tool_calls: false`	Tools accepted, not acted on	Drop the agent branch, or change endpoint
`finish_reason: "length"` on an 8-token cap	Normal, expected	Keep the check; raise budget in real calls
`finish_reason: null` in streaming	Client is not reading the final event	Fix the parser before anything else
`reports_usage: false`	No token accounting from server	Count tokens client-side, or accept blind cost
`keys_match: false`	Valid JSON, wrong shape	Add schema validation at the boundary
`saw_429: true` at n=2	Very low concurrency ceiling	Queue work; never fan out blindly

Nothing here measures quality. Nothing here measures speed. Those are different experiments with different controls.

Where I run this

I run probes from a throwaway box, not my laptop. Same reason I do not test migrations on my main database. The probe should not touch anything I care about.

MonkeyCode's free model access and free server option are the two pieces I use for that. The free model access gives the probe something to talk to. The free server option gives it a disposable place to run. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not claiming the free tier is fast, generous, or permanent. I do not know its limits, and I am not going to guess at them. Run the concurrency probe and read your own numbers. That is the whole point of this post.

Limitations and who should skip this

This is a smoke test. It is not a benchmark and not a load test.
Results describe one endpoint, one model, one moment. They expire.
Paid tiers can behave differently from free ones. Re-probe after switching.
Only probe endpoints you own or are authorized to test.
Do not use this to grade answer quality. It checks contract shape only.

Skip this approach if you need contractual latency SLOs, deterministic output, or guaranteed tool-call behavior. Buy a plan with a written contract. A probe will not save you from a missing SLA.

The corrected mental model

Treat every free endpoint as a set of separate promises. Transport, truncation, tool calls, JSON shape, and concurrency are five different promises. A 200 only confirms the first one.

Which promise broke on your last integration? I keep a one-page probe log per endpoint now. It has saved me two weekends so far.

It Passed CI. That Still Doesn't Make the Patch Safe.

Jordan Huang — Mon, 26 Oct 2026 18:14:00 +0000

Last week I merged a model-written patch in four minutes.

CI was green. Review said 'lgtm'. The diff was twelve lines.

Two days later a background job stopped retrying. Nobody noticed for a while.

That is the whole problem in one story. A green check feels like a verdict. It is not one.

I generate patches with models every day now. It changed how I merge them.

Below is the FAQ I keep answering, a table you can steal, and a gate script you can run before lunch.

The short version

Claim	What the check actually proves	Corrected model
'CI is green'	the tests you already had still pass	green means no regression in covered paths
'The diff is small'	few lines moved	risk lives in the files touched, not the line count
'I added tests'	assertions exist	a test written by the same author grades itself
'It looks idiomatic'	style matched the codebase	idiom hides semantics, and semantics fail

Keep that table open. Now the details.

Claim 1: 'Green CI means the change works.'

No. CI proves that your existing suite still passes on the new commit.

So ask yourself a sharper question. Does any test actually execute the lines you added?

Most repos cannot answer that quickly. A coverage percentage is not a diff view.

Two commands get you closer in ten seconds.

git diff --stat origin/main...HEAD
git diff --name-only origin/main...HEAD | grep -v -E '(^|/)(tests?|__tests__)/'

Read the second list carefully. Those are changed files with no obvious test neighbor.

If one of them touches retries, auth, or money, slow down.

The corrected model

Green means 'nothing I already test broke'. It never means 'the new path is correct'.

Treat CI as a regression alarm, not a correctness proof.

Claim 2: 'Small diff, small risk.'

Line count is a bad proxy for danger. Blast radius is the real measure.

A three-line change to a default timeout can take down every worker you have.

Ask these four questions instead.

Which files does the diff touch?
Do those files sit on a hot path?
Does it change a default, a limit, or an error branch?
What happens on the second failure, not the first?

Failure paths are where model patches go soft. Retries, fallbacks, except blocks, silent skips.

The corrected model

Risk scales with the semantics you touched. Count branches, not lines.

Claim 3: 'I added a test, so it is tested.'

Who wrote the test? If it was the same model, you have a self-graded exam.

The test asserts what the patch does. It does not assert what the patch should do.

I use one boring rule here. The test must fail on the parent commit.

git switch -c check/parent origin/main
# apply ONLY the new test file, then run it
git checkout patch/retry-fix -- tests/test_retry.py
pytest tests/test_retry.py   # expect: FAIL

If the test passes without the patch, it is not testing the patch.

That single check has caught more bad patches for me than any linter.

The corrected model

A test is evidence only if it can fail for the right reason.

Claim 4: 'Idiomatic code is safe code.'

Python has a few shapes that read fine and behave badly.

except Exception: pass — silent failure, forever.
def f(x, acc=[]) — one list, shared across every call.
if x == None — works, then breaks on __eq__ overrides.
for i in range(len(items)) — index math where you wanted items.

None of these are syntax errors. All of them survive code review.

Models reproduce them because the training data is full of them.

The corrected model

Idiom is about reading. Safety is about failure behavior. Check them separately.

The artifact: a patch gate that runs in one second

I keep this script in tools/. It needs plain Python 3 and git, nothing else.

It does five jobs.

Lists changed files against a base ref.
Flags lines that mention auth, retries, schema, or outbound I/O.
Flags four common smell patterns.
Blocks risky changes with no test file in the diff.
Exits non-zero, so it can sit in front of merge.

#!/usr/bin/env python3
# patch_gate.py - deterministic checks before merging a patch.
from __future__ import annotations

import re
import subprocess
import sys
from dataclasses import dataclass, field

BASE = sys.argv[1] if len(sys.argv) > 1 else 'origin/main'

TEST_HINT = re.compile(r'(^|/)(tests?|__tests__)/|(_test\.py$|\.test\.(js|ts)$|_spec\.rb$)')

RISK = {
    'auth': re.compile(r'password|secret|token|session|oauth', re.I),
    'retry': re.compile(r'retry|backoff|timeout|idempot', re.I),
    'schema': re.compile(r'ALTER TABLE|DROP COLUMN|CREATE INDEX|migrat', re.I),
    'io': re.compile(r'requests\.|httpx\.|fetch\(|socket\.', re.I),
}

SMELL = [
    (re.compile(r'except\s*:\s*(#.*)?$'), 'bare except'),
    (re.compile(r'except\s+Exception\s*:\s*pass'), 'swallowed exception'),
    (re.compile(r'def\s+\w+\([^)]*=\s*(\[\]|\{\})'), 'mutable default arg'),
    (re.compile(r'==\s*None|!=\s*None'), 'equality check against None'),
]


def git(*args: str) -> str:
    result = subprocess.run(['git', *args], capture_output=True, text=True, check=True)
    return result.stdout


def added_lines(path: str) -> list:
    diff = git('diff', '-U0', f'{BASE}...HEAD', '--', path)
    return [ln[1:] for ln in diff.splitlines() if ln.startswith('+') and not ln.startswith('+++')]


@dataclass
class Finding:
    path: str
    risks: set = field(default_factory=set)
    flags: list = field(default_factory=list)


def main() -> int:
    files = [f for f in git('diff', '--name-only', f'{BASE}...HEAD').splitlines() if f.strip()]
    tests = [f for f in files if TEST_HINT.search(f)]
    print(f'base={BASE} files={len(files)} test_files={len(tests)}')

    blocking = 0
    for path in files:
        if TEST_HINT.search(path):
            continue
        finding = Finding(path)
        for line in added_lines(path):
            for name, pattern in RISK.items():
                if pattern.search(line):
                    finding.risks.add(name)
            for pattern, label in SMELL:
                if pattern.search(line):
                    finding.flags.append(label)
        if finding.risks and not tests:
            finding.flags.append('risky change, no test file in diff')
            blocking += 1
        risks = ','.join(sorted(finding.risks)) or '-'
        flags = '; '.join(finding.flags) or '-'
        print(f'{path}: risk={risks} flags={flags}')

    return 1 if blocking else 0


if __name__ == '__main__':
    raise SystemExit(main())

Run it on the branch, not on main.

git switch -c patch/retry-fix origin/main
# ... apply the model patch ...
python3 tools/patch_gate.py origin/main; echo "exit=$?"

The sample output below is illustrative, not a benchmark.

base=origin/main files=3 test_files=0
src/worker/retry.py: risk=retry,timeout flags=risky change, no test file in diff
src/worker/queue.py: risk=- flags=-
docs/retries.md: risk=retry flags=-
exit=1

Running the gate inside a free-model loop

The gate is half the loop. The other half is what you send back to the model.

My sequence is short.

Generate the patch against a fresh branch.
Apply it, then run the gate and save the output.
Paste the gate output back verbatim, with a constraint.
Re-run the gate on the new diff.
Only then read the diff as a human.

python3 tools/patch_gate.py origin/main > gate.txt 2>&1
echo "exit=$?" >> gate.txt

The follow-up prompt I use, unchanged:

Fix only the flagged issues. Do not refactor untouched code.
Add or update one test that fails on origin/main.
Return a unified diff. No prose.

I run that loop with MonkeyCode. Its free model access covers the patch and repair steps. The free server option holds the scratch clone, so the gate runs next to the code it inspects. Both are availability claims from the product team, not numbers I measured.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Where this gate is wrong

Be honest about the edges. The script is regex, so it reads text, not meaning.

A docs page mentioning tokens will trip the auth rule. Expect noise.
It cannot see runtime behavior, feature flags, or production data.
The smell list is Python-shaped. Port it before trusting it elsewhere.
If origin/main is not your real base, the diff is wrong.
Exit 1 is a prompt to look, not an automatic merge veto.

Who should not use this

Teams where any automated gate becomes a rubber stamp within a month.
Repos dominated by generated or vendored code, where every file looks risky.
Codebases whose real risk sits in performance, migrations, or infrastructure.
Anyone who cannot fetch the base ref in CI.

If that is you, keep the parent-commit test check. It needs no script.

What actually changed in my habits

Two habits, both small.

First, I read the file list before the diff. Every time.

Second, I run the parent-commit check on every model patch. No exceptions.

Green CI stopped being a finish line for me. Now it is a starting gun.

If you run the gate on your repo, which flag fires first?