DEV Community: Vikas Srivastava

When more agents produce less work

Vikas Srivastava — Mon, 02 Nov 2026 00:02:06 +0000

When more agents produce less work

A team of language-model agents can fail even when every individual agent is capable. The concrete problem is coordination: once several agents share a world, they must decide what to do, who is allowed to do it, how progress is recorded, and how disagreements are resolved. Those control tasks can consume the system’s available attention before it builds anything useful.

The problem becomes more severe when agents can change the same environment. One agent may reserve materials, alter a file, or move to a location while another is still planning from an earlier description of the world. Even correct local decisions can conflict when they are made from inconsistent views. The resulting failures are architectural, not necessarily evidence that the underlying models cannot reason.

That is the lesson suggested by reports of AI agent swarms in Minecraft becoming trapped in bureaucracy. Minecraft is a particularly revealing test case because progress requires long-horizon planning, resource management, physical action, and a shared understanding of changing state. A swarm that cannot maintain those basics will not be rescued by adding more conversational agents.

The architectural mistake is treating “many agents” as a scaling law. A language model produces a useful action from context; a swarm adds multiple contexts plus a protocol for reconciling them. If the protocol is poorly designed, the system scales its overhead faster than it scales its productive work.

How it works

A multi-agent system usually has four layers. Individual agents generate plans or actions, a communication layer passes messages between them, a shared state records facts and tasks, and an orchestration layer decides which agent runs next. The underlying language model may be identical across all roles; the behavior comes largely from prompts, tools, memory, and scheduling. These layers interact: a good plan is still useless if the communication layer loses it, and an accurate observation is still dangerous if the shared state treats it as current forever.

The simplest design gives every agent access to a common transcript. This is easy to implement but expensive and noisy. Each new message expands the context that every later agent must interpret, while stale plans and contradictory claims remain visible unless some component explicitly summarizes or removes them. Agents can then spend their turns reconciling conversation history rather than examining the environment or executing a task.

A more structured design assigns roles such as explorer, builder, resource manager, or reviewer. That can reduce ambiguity, but it introduces authority questions: does the builder trust the explorer’s map, can the resource manager reject a request, and who resolves two agents claiming the same materials? Role boundaries also need handoff rules. An explorer’s discovery should become a usable fact, not merely another message that a builder may or may not notice.

The orchestrator is therefore not just a router. It is a control system. It must convert a global objective into bounded tasks, assign ownership, detect completion, and recover when an action changes the world in an unexpected way. A useful mental model is a transaction system rather than a group chat: tasks need an owner, resources need reservations, observations need timestamps, and actions need verification. Without those properties, agents repeatedly rediscover the same facts or act on state that is no longer true.

from dataclasses import dataclass

@dataclass
class Task:
    name: str
    owner: str | None = None
    status: str = "queued"

queue = [Task("map nearby resources"), Task("choose a build site")]

for task in queue:
    if task.owner is None:
        task.owner = "explorer" if "map" in task.name else "planner"
    task.status = "running"
    # The assigned agent performs the task and returns verified results.

This pattern is deliberately mundane. Explicit ownership and state transitions often matter more than adding another persona to the prompt. The same principle applies whether the shared world is a game, a codebase, a research workspace, or a set of business tools.

What the evidence actually shows

The reported Minecraft outcome is evidence against a simplistic claim: that a sufficiently large collection of autonomous agents will spontaneously organize into a productive society. It does not show that all multi-agent systems are ineffective, nor that a single model is always superior. It shows that open-ended environments expose coordination costs that short benchmark tasks can hide.

The failure mode is technically plausible because the environment has delayed rewards. An agent can spend many steps discussing plans, appointing roles, or negotiating procedures without immediately receiving a strong penalty. In contrast, a human player recognizes quickly that gathering materials or placing blocks is progress, while procedural activity is not. A system needs an equivalent way to distinguish actions that change the world from messages that merely describe intentions.

The result also highlights the difference between verbal agreement and operational alignment. Agents may produce compatible-sounding plans while maintaining different maps, assumptions about inventory, or definitions of “finished.” Shared language is not shared state. If one agent believes a structure is complete because its design was approved, while another expects the blocks to be placed, both can report progress honestly and still leave the task unfinished.

For practitioners, the important observation is what appears on day one: more agents may increase latency, token usage, duplicate work, and contradictory actions before they improve task completion. If a single agent already lacks reliable tool use or state tracking, multiplying it creates more opportunities to express the same weakness. Collaboration should therefore be treated as a hypothesis to test, not as an automatic upgrade.

Using it

Start with one agent and a measurable task boundary. Add another agent only when a distinct capability—such as visual exploration, code review, or independent verification—cannot be handled reliably by the first. The second agent should have a narrower input and a clearly defined output, not a general invitation to participate. Narrow interfaces make it easier to determine whether the additional agent contributed or merely added another stream of text.

Keep shared state structured and external to the chat transcript. Store facts, resource counts, task status, ownership, and timestamps in a database or typed document. Let agents read the subset relevant to their task, and require them to write back observations separately from proposals. Separating “what was observed” from “what should happen next” helps later agents judge whether a recommendation is still valid.

Use leases or reservations for scarce resources. An agent that claims a tool, file, or game item should have an expiry and a release condition; otherwise a failed agent can block the whole system indefinitely. Reservations should also identify the task they support, so an orchestrator can reclaim resources when the task is cancelled or superseded.

Make progress observable through intermediate checks. After an agent reports that a task is complete, another component should verify the artifact or world state rather than accepting the report as truth. For long-running work, a simple event log is often more useful than a larger conversational memory because it records what changed, when it changed, and which agent was responsible.

Measure the system against a single-agent baseline. Track successful task completion, wall-clock latency, model calls, context length, duplicate actions, recovery time, and the fraction of calls spent on coordination. If the swarm improves only an informal impression of sophistication, it is not yet an architecture.

Limitations and failure modes

Role prompts do not create genuine specialization. If all agents use the same model, tools, and evidence, assigning titles such as “architect” and “manager” may mostly produce stylistic variation. Specialization becomes meaningful when agents have different permissions, views, tools, or evaluation criteria. A reviewer who can inspect but not modify an artifact has a different function from a builder, even if both use the same underlying model.

Central orchestration can become a bottleneck or a single point of failure. A coordinator that must inspect every observation and approve every action may serialize the workflow, while a decentralized design can lose consistency. The right balance depends on how costly conflicting actions are. Central control is more attractive when actions are difficult to undo; looser coordination can be appropriate when tasks are independent and failures are cheap to recover from.

Memory is another trap. A larger context window does not guarantee better collective memory; it can make retrieval harder by mixing durable facts with transient conversation. Summaries can help, but a wrong summary becomes a shared false belief and may be repeated by every downstream agent. Memory therefore needs provenance and a way to revise or invalidate facts when new observations contradict them.

Finally, open-ended environments make evaluation difficult. A swarm may appear busy, cooperative, and creative while making no durable progress. Without explicit success criteria and state-based verification, teams can optimize for conversation quality rather than outcomes. The more freedom agents have to define their own process, the more important it becomes to define completion outside that process.

What to watch

The next meaningful advances will likely come from coordination primitives rather than additional personas: typed shared state, verifiable tool results, bounded delegation, and schedulers that learn when collaboration is worth its cost. Watch whether new systems report work saved per successful task, not merely the number of agents involved. A useful scheduler should be able to decline collaboration when the expected benefit is smaller than the cost of another model call and another handoff.

The strongest architectures may also be selectively collaborative. Most steps can remain with one agent, with parallel agents activated only for uncertainty, independent checking, or genuinely separable work. That makes multi-agent behavior a controlled resource allocation problem instead of a permanent meeting.

This design also leaves room for collaboration to stop cleanly. Once an uncertainty is resolved or an artifact is verified, temporary agents should release their resources and return control to the main workflow. The goal is not to keep every agent involved; it is to use additional views only when they produce more reliable progress than a simpler system would.

https://dev.to/vikassri/why-trillion-parameter-models-need-a-new-serving-architecture-6gk

Vikas Srivastava — Thu, 29 Oct 2026 04:50:04 +0000

Vikas Srivastava

Oct 26

Why Trillion-Parameter Models Need a New Serving Architecture

#inference #cerebras #modelarchitecture

7 min read

Why Trillion-Parameter Models Need a New Serving Architecture

Vikas Srivastava — Mon, 26 Oct 2026 00:02:22 +0000

The problem: model size has become a serving constraint

A trillion-parameter model is not merely a larger checkpoint. It changes the engineering problem from “fit a model on accelerators” to “move, store, schedule, and decode an enormous computation without making every token wait on a distributed system.” The model’s size affects not only how much arithmetic is required, but also where weights reside, how they reach compute units, and how much state must remain available while requests are active.

For practitioners, the visible failure mode is latency. A model may produce a strong answer in batch evaluation, yet become impractical in an agent loop, code completion tool, or retrieval pipeline if each generated token requires synchronization across many devices. A delay that seems modest in one request can become expensive when an agent makes several model calls, waits for tools, and then continues generation.

Cerebras’s deployment of trillion-parameter models is notable because it targets that serving problem directly. The important question is not whether parameter count alone produces better outputs, but whether a different compute architecture makes models at this scale usable. That means examining the full path from model partitioning and memory placement to runtime scheduling and the latency experienced by an actual application.

How it works

Large-model inference has two distinct phases. During the prompt or “prefill” phase, the system processes many input tokens in parallel and builds attention state; during “decode,” it generates one token at a time, repeatedly reading model weights and the accumulated key-value cache. These phases stress different parts of the system, so a deployment optimized for prompt ingestion may not deliver the same advantage during generation.

Decode is especially difficult to accelerate. The next token depends on the previous token, so a server cannot parallelize the sequence in the same way it parallelizes independent examples. It can batch requests together, but batching increases memory pressure and may hurt the latency of interactive users. A service must balance keeping compute units busy against avoiding queues that make each user wait longer.

A conventional deployment spreads layers or tensor shards across multiple accelerator cards. Every layer transition can require communication, and tensor parallelism can require collective operations between devices. As parameter counts rise, communication, memory capacity, and weight movement become as important as arithmetic throughput. The arithmetic may be highly optimized while the request still spends time waiting for data or for synchronization at a device boundary.

Cerebras approaches the problem with wafer-scale hardware: a very large processor containing many compute cores and local memory on one wafer, connected through a high-bandwidth on-wafer fabric. A compiler partitions the model across those cores, placing pieces of tensors and operations close to where they are consumed rather than treating each accelerator as an isolated island. This makes the placement of data and operations part of the execution plan instead of leaving every transfer to a conventional device-to-device network.

That does not eliminate the model’s computation. It changes where the coordination happens. Instead of repeatedly crossing the network between discrete cards, much of the movement occurs through the wafer’s internal interconnect, while external systems provide additional memory or coordinate larger jobs when the model exceeds on-wafer capacity. The benefit therefore depends on how well the compiler and runtime map the model’s actual dependencies onto the available hardware.

The result is most relevant to autoregressive decoding. If the hardware can keep weights and intermediate state available with less inter-device overhead, the system can reduce the time between generated tokens. A trillion parameters still represents substantial work; the architectural change is that the work is organized around data movement and synchronization rather than only peak matrix multiplication. For interactive applications, that distinction can matter as much as raw throughput.

What the evidence actually shows

The deployment signal demonstrates that trillion-parameter models can be integrated into a production-oriented inference stack rather than remaining purely theoretical training artifacts. That is meaningful systems evidence: model partitioning, compilation, memory management, runtime scheduling, and operational interfaces all have to work together. A model that exists only as a checkpoint has not yet solved the practical problems of loading, serving, monitoring, and handling concurrent requests.

It does not, by itself, establish that a trillion-parameter model is more accurate than a smaller model, cheaper per token, or faster for every workload. Those claims require comparable measurements with the same precision, prompt lengths, output lengths, batching policy, and service-level target. Hardware and model comparisons can otherwise mix several variables and attribute an improvement to parameter count or architecture when it came from a different serving configuration.

The most informative evidence will therefore be workload-specific. A single-token latency measurement says something different from sustained throughput; prefill speed says something different from decode speed; and a high-throughput batch service is not equivalent to an interactive coding assistant. Tail latency and behavior under realistic concurrency are also important because a system that is fast on average may still create unacceptable waits for some requests.

There is also a distinction between deploying a model with a trillion total parameters and executing all of them for every token. Sparse or mixture-of-experts architectures can contain a large total parameter count while routing each token through only a subset of experts. Any comparison should disclose whether the model is dense, sparse, quantized, or using another form of conditional computation. Without that information, parameter count is an incomplete description of the work performed per token.

Using it

Start by measuring your actual serving shape before choosing hardware. Record prompt-token distributions, output lengths, concurrency, time-to-first-token, inter-token latency, and the fraction of requests that use long context. Aggregate throughput can hide a poor interactive experience, especially when a workload mixes short user prompts with long retrieval inputs or extended agent trajectories.

For a rough first pass, inspect token counts separately from generation behavior:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")
prompt = open("prompt.txt", encoding="utf-8").read()

print("prompt tokens:", len(enc.encode(prompt)))
print("characters:", len(prompt))

This tokenizer is only an approximation unless it matches the deployed model, but it can reveal whether your workload is dominated by prompt processing or generation. Then benchmark the target model with representative prompts, fixed output limits, and several concurrency levels instead of relying on a vendor’s single headline number. Keep prompt and output measurements separate so that a change in time-to-first-token is not confused with a change in inter-token latency.

Treat context as a memory budget. Long prompts increase prefill work and enlarge the key-value cache during decoding; repeated retrieval documents can therefore damage latency even when the model’s advertised context window supports them. Summarize, retrieve selectively, cache stable prefixes where the serving stack supports it, and avoid sending tool schemas or conversation history that the model does not need. These choices reduce both immediate computation and the state that must remain available for active requests.

For model selection, compare quality at equal latency and cost, not equal parameter count. A smaller dense model, a sparse model, or a quantized checkpoint may be the better engineering choice if it meets the task’s accuracy target with simpler operations and predictable capacity. The right comparison should include the complete application path, including retries, tool calls, context construction, and the amount of generated text needed to complete a task.

Limitations and failure modes

Wafer-scale execution does not make memory free. Trillion-parameter checkpoints still require storage, loading, checkpoint management, and often external memory systems. If those paths are slow, startup time or model swapping can dominate the user experience even when steady-state decoding is fast. Operational concerns therefore remain part of the serving design rather than being solved by processor layout alone.

Communication is reduced, not abolished. Very large models may still span multiple systems, and the boundary between wafer, memory appliance, host CPU, and network becomes a new source of latency. Runtime behavior can also change sharply with sequence length, batch size, and the number of concurrently active requests. A configuration that works for one context distribution may behave differently when requests become longer or arrive in bursts.

Sparse models introduce their own failure modes. Expert routing can create load imbalance, with some experts receiving disproportionate traffic; capacity limits may drop or reroute tokens; and total parameter count becomes a poor proxy for actual compute. Quantization can improve efficiency while changing calibration, rare-token behavior, or tool-call reliability. Both techniques require evaluation on the application’s failure-sensitive paths, not only on aggregate model quality.

Finally, faster decoding can increase demand rather than reduce total consumption. If lower latency encourages longer agent trajectories, more retries, or larger prompts, the system may process more tokens overall. Capacity planning should track tokens per completed task, not just tokens per second. A faster individual response is useful only if the resulting workload remains affordable and reliable at the application level.

What to watch

The next useful comparisons will separate prefill from decode and report tail latency under realistic concurrency. Pay particular attention to how performance changes with long contexts, because that is where memory traffic and cache management expose architectural trade-offs. Measurements should also make clear whether they describe dense execution, sparse routing, quantization, or another form of conditional computation.

Also watch whether software ecosystems make wafer-scale deployment portable. If model conversion, compiler support, quantization, observability, and fallback serving remain specialized, the hardware advantage may be real but difficult to operationalize. Portability affects more than convenience: it determines how easily teams can test models, handle failures, change workloads, and maintain a service over time.

The broader shift is toward treating inference architecture as part of model design. Parameter count, sparsity, context strategy, and decoding hardware are increasingly coupled; the best model for a production system will be the one whose computation matches the system’s latency and memory constraints, not necessarily the one with the largest headline size.

SELFTEST Production Document Ingestion and Structure-Aware Chunking

Vikas Srivastava — Sun, 25 Oct 2026 06:32:42 +0000

Build a deterministic ingestion pipeline that preserves document structure and produces retrieval-ready chunks with traceable metadata.

Engineers building document-QA systems who already understand Python, embeddings, and basic vector search.

What you'll be able to do

Normalize PDF, Markdown, HTML, and plain-text inputs into a common document representation.
Detect headings and preserve section context during chunking.
Emit deterministic JSONL records with source, page, section, offsets, and chunk identifiers.
Validate chunk size, metadata completeness, and deterministic output before indexing.

The problem

A PDF, Markdown file, or HTML page can be split into arbitrary character windows that separate headings from content, duplicate context across chunks, and make citations impossible to trace back to the source.

How it works

Central question

How should an ingestion pipeline transform heterogeneous documents into chunks that retrieval can use without destroying the document structure needed for ranking, citations, and access control?

The answer is to treat chunking as a data-contract problem, not as a string-splitting operation. The pipeline should produce records with stable identifiers, source provenance, section context, normalized text, and explicit size estimates. Later retrieval and generation stages should consume this contract rather than reimplementing parsing assumptions.

The ingestion contract

Each emitted chunk should contain at least:

{
  "chunk_id": "sha256-prefix:ordinal",
  "document_id": "sha256-prefix",
  "source": "relative/path.pdf",
  "source_type": "pdf",
  "page": 3,
  "section": "Installation > Configuration",
  "text": "...",
  "char_start": 1204,
  "char_end": 1890,
  "token_estimate": 172,
  "metadata": {}
}

The important properties are:

document_id identifies the source content, not its filename alone.
chunk_id is stable when the source and chunking configuration are unchanged.
source and page support citations and debugging.
section preserves hierarchy that a flat embedding cannot recover reliably.
char_start and char_end make it possible to inspect exactly what was indexed.
metadata is the place to carry tenant, ACL, retention, and classification attributes. It must be populated before indexing, not added after retrieval.

Parse first, chunk second

Parsing and chunking should be separate stages:

Discover supported files.
Extract text and structural hints.
Normalize whitespace without removing meaningful boundaries.
Detect headings and maintain a section stack.
Group paragraphs under the current section.
Split only oversized groups.
Emit records and validate the contract.

This separation makes failures diagnosable. An empty PDF extraction is a parser failure. A chunk that exceeds the configured limit is a chunker failure. A retrieved record without tenant metadata is a metadata-contract failure.

Structure-aware boundaries

Prefer boundaries in this order:

Document or page boundary.
Heading boundary.
Paragraph boundary.
Sentence boundary.
Character boundary as a final fallback.

A heading should be copied into the chunk's section metadata and, where useful, into the chunk text. Keeping it in both places helps lexical retrieval match terms such as Retries or Authentication while allowing downstream systems to display the hierarchy.

The example implementation uses a conservative paragraph-oriented splitter. It does not attempt to infer tables, code blocks, or semantic boundaries from arbitrary PDFs. Those need format-specific handlers and validation fixtures.

Size is a budget, not a universal constant

A character limit is only a proxy for model tokens. Tokenization varies by language, markup, code, and model vocabulary. Use a conservative character limit during ingestion, then measure the actual tokenizer distribution used by the embedding and generation models.

Record at least p50, p95, and maximum token counts per source type. If p95 is acceptable but the maximum is not, investigate outliers instead of raising the global limit. Large chunks increase embedding input cost and can reduce retrieval precision by combining unrelated facts.

Overlap is selective context duplication

Overlap can help when a fact crosses a paragraph boundary, but it also duplicates vectors, increases index size, and can cause answer generation to see repeated evidence. Start with zero overlap at clean heading and paragraph boundaries. Add a measured overlap only for formats where boundary splitting causes a recall regression.

Determinism and reindexing

Hash normalized source content and the chunking configuration. A changed parser version, normalization rule, or maximum size should produce a new ingestion version. Do not silently overwrite records while retaining old vectors; otherwise a query can retrieve chunks from incompatible generations of the same document.

A production record should therefore also carry fields such as ingestion_version, parser_version, and acl_version. The sample keeps the implementation small but leaves a metadata object for those fields.

flowchart LR
    A[Source files] --> B[Discovery]
    B --> C[Format parser]
    C --> D[Normalized blocks]
    D --> E[Heading and section tracking]
    E --> F[Paragraph-aware chunker]
    F --> G[Schema validation]
    G --> H[JSONL artifact]
    H --> I[Embedding and indexing]
    G --> J[Rejects and diagnostics]

Parsing, structural chunking, validation, and indexing are separate stages so each failure has a distinct diagnostic location.

Implementation

Install

The example uses PyMuPDF for PDFs and Beautiful Soup for HTML. Markdown and text use the standard library.

python -m venv .venv
. .venv/bin/activate
pip install pymupdf==1.25.5 beautifulsoup4==4.13.3

Set the input and output locations without embedding credentials:

export RAG_INPUT_DIR=./documents
export RAG_OUTPUT_FILE=./artifacts/chunks.jsonl

The program walks the input directory, extracts blocks, tracks Markdown and HTML headings, groups paragraphs, splits oversized paragraphs by characters, and writes one JSON object per line. It also computes document and chunk identifiers from content so the output is reproducible.

The output JSONL is intentionally independent of a vector database. An embedding/indexing stage can read it later and attach vectors without changing provenance fields.

Operational extensions

For production, add parser-specific fixtures for:

PDFs containing tables, headers, footers, scanned pages, and multi-column layouts.
Markdown code blocks, lists, and nested headings.
HTML navigation, boilerplate, hidden content, and repeated templates.
Documents with explicit tenant and access-control metadata.

Reject or quarantine documents when extraction yields no text, when the source type is unsupported, or when required security metadata is absent.

Code 1: Ingest PDF, HTML, Markdown, and text files; preserve section metadata; emit deterministic JSONL chunks.

# Install:
# python -m venv .venv
# . .venv/bin/activate
# pip install pymupdf==1.25.5 beautifulsoup4==4.13.3

from __future__ import annotations

import hashlib
import json
import os
import re
from dataclasses import dataclass
from pathlib import Path
from typing import Iterable

import fitz  # PyMuPDF
from bs4 import BeautifulSoup


MAX_CHARS = 2200
HEADING_RE = re.compile(r"^(#{1,6})\\s+(.+?)\\s*$")


@dataclass
class Block:
    text: str
    page: int | None = None
    heading_level: int | None = None
    is_heading: bool = False


def normalize(text: str) -> str:
    text = text.replace("\\r\\n", "\\n").replace("\\r", "\\n")
    text = re.sub(r"[ \\t]+", " ", text)
    text = re.sub(r"\\n{3,}", "\\n\\n", text)
    return text.strip()


def read_text_blocks(path: Path) -> list[Block]:
    raw = path.read_text(encoding="utf-8", errors="replace")
    suffix = path.suffix.lower()
    if suffix == ".html" or suffix == ".htm":
        soup = BeautifulSoup(raw, "html.parser")
        blocks: list[Block] = []
        for node in soup.find_all(["h1", "h2", "h3", "h4", "h5", "h6", "p", "li"]):
            text = normalize(node.get_text(" ", strip=True))
            if not text:
                continue
            if node.name.startswith("h"):
                blocks.append(Block(text, heading_level=int(node.name[1]), is_heading=True))
            else:
                blocks.append(Block(text))
        return blocks

    blocks = []
    for part in re.split(r"\\n\\s*\\n", raw):
        text = normalize(part)
        if not text:
            continue
        match = HEADING_RE.match(text)
        if match and "\\n" not in text:
            blocks.append(Block(match.group(2), heading_level=len(match.group(1)), is_heading=True))
        else:
            blocks.append(Block(text))
    return blocks


def read_pdf_blocks(path: Path) -> list[Block]:
    blocks: list[Block] = []
    with fitz.open(path) as document:
        for page_number, page in enumerate(document, start=1):
            text = normalize(page.get_text("text"))
            if not text:
                continue
            for paragraph in re.split(r"\\n\\s*\\n", text):
                paragraph = normalize(paragraph)
                if paragraph:
                    blocks.append(Block(paragraph, page=page_number))
    return blocks


def source_blocks(path: Path) -> list[Block]:
    if path.suffix.lower() == ".pdf":
        return read_pdf_blocks(path)
    if path.suffix.lower() in {".txt", ".md", ".markdown", ".html", ".htm"}:
        return read_text_blocks(path)
    return []


def split_oversized(text: str, limit: int) -> Iterable[str]:
    if len(text) <= limit:
        yield text
        return
    start = 0
    while start < len(text):
        end = min(start + limit, len(text))
        if end < len(text):
            cut = text.rfind(" ", start, end)
            if cut > start + limit // 2:
                end = cut
        piece = text[start:end].strip()
        if piece:
            yield piece
        start = end


def digest(value: str, length: int = 16) -> str:
    return hashlib.sha256(value.encode("utf-8")).hexdigest()[:length]


def make_chunks(path: Path, root: Path, limit: int = MAX_CHARS) -> list[dict]:
    blocks = source_blocks(path)
    if not blocks:
        raise ValueError(f"no extractable text: {path}")

    source = path.relative_to(root).as_posix()
    document_id = digest("\\n".join(block.text for block in blocks))
    section_stack: list[tuple[int, str]] = []
    output: list[dict] = []
    pending: list[Block] = []

    def flush() -> None:
        if not pending:
            return
        text = "\\n\\n".join(block.text for block in pending).strip()
        if not text:
            pending.clear()
            return
        page_values = [block.page for block in pending if block.page is not None]
        page = page_values[0] if page_values else None
        section = " > ".join(item[1] for item in section_stack)
        ordinal = len(output)
        chunk_key = f"{document_id}|{ordinal}|{section}|{text}"
        output.append({
            "chunk_id": f"{document_id}:{digest(chunk_key, 12)}",
            "document_id": document_id,
            "source": source,
            "source_type": path.suffix.lower().lstrip("."),
            "page": page,
            "section": section,
            "text": text,
            "char_start": None,
            "char_end": None,
            "token_estimate": max(1, round(len(text) / 4)),
            "metadata": {"ingestion_version": "1"},
        })
        pending.clear()

    for block in blocks:
        if block.is_heading:
            flush()
            level = block.heading_level or 1
            section_stack = [(n, title) for n, title in section_stack if n < level]
            section_stack.append((level, block.text))
            continue
        for piece in split_oversized(block.text, limit):
            candidate = "\\n\\n".join(item.text for item in pending + [Block(piece, block.page)])
            if pending and len(candidate) > limit:
                flush()
            pending.append(Block(piece, block.page))
    flush()
    return output


def main() -> None:
    root = Path(os.environ.get("RAG_INPUT_DIR", "./documents")).resolve()
    output = Path(os.environ.get("RAG_OUTPUT_FILE", "./artifacts/chunks.jsonl"))
    output.parent.mkdir(parents=True, exist_ok=True)
    supported = {".pdf", ".txt", ".md", ".markdown", ".html", ".htm"}
    files = sorted(path for path in root.rglob("*") if path.is_file() and path.suffix.lower() in supported)
    if not files:
        raise SystemExit(f"no supported files found below {root}")

    count = 0
    with output.open("w", encoding="utf-8") as handle:
        for path in files:
            for record in make_chunks(path, root):
                if not record["text"] or not record["document_id"] or not record["chunk_id"]:
                    raise ValueError(f"invalid record from {path}")
                handle.write(json.dumps(record, ensure_ascii=False) + "\\n")
                count += 1
    print(f"wrote {count} chunks to {output}")


if __name__ == "__main__":
    main()

Verifying it

Create at least one Markdown file with # Authentication, two paragraphs, and ## Token expiry, then run python ingest.py. Expected output is a line such as wrote N chunks to ..., where N is greater than zero. Inspect the artifact with python -c "import json, pathlib; p=pathlib.Path('artifacts/chunks.jsonl'); rows=[json.loads(x) for x in p.read_text().splitlines()]; assert rows; assert all(r['chunk_id'] and r['document_id'] and r['text'] for r in rows); print(rows[0].keys())". Add a deterministic-output check by running the command twice and comparing normalized JSONL files with sha256sum; identical source files and configuration should produce identical output. For real documents, separately measure empty extraction rate, chunk character and tokenizer-length distributions, chunks per document, and duplicate-content rate.

Performance and cost

Ingestion cost is primarily disk I/O and parser CPU; PDF and HTML parsing do not require model tokens or API dollars in this implementation. Chunking increases downstream cost approximately in proportion to the number of emitted chunks: each chunk generally requires one embedding request or local embedding inference. Measure total characters, actual tokenizer tokens, embedding batch size, indexing latency, and storage bytes. Increasing overlap or reducing the chunk limit increases vector count and embedding cost. The token_estimate field is only a screening estimate; use the exact embedding-model tokenizer before budgeting.

What breaks

Scanned PDFs may produce empty text because OCR is not included; detect this through the no-extractable-text rejection and route those files to an OCR stage. Multi-column PDFs may interleave reading order; detect it with page-level fixtures and visual review. Headers and footers may repeat in every page; detect repeated n-grams across pages and remove them with a format-specific rule. Large tables may be flattened into unusable prose; detect low row or column fidelity and use table extraction. Heading detection can fail when a PDF encodes headings as ordinary text; compare parser output with expected section fixtures. Oversized chunks indicate a splitter or normalization bug; monitor maximum actual tokenizer length rather than only character count.

Security

Treat every source file and extracted string as untrusted input. Do not execute embedded macros, HTML, or code. Store tenant, document, and ACL metadata with every chunk and enforce authorization before retrieval results reach the model; filtering only after generation is too late. Avoid logging full document text by default. Restrict input and output directories, validate file types using parser behavior rather than filename alone, cap file size and page count, and quarantine decompression or parser failures. If documents contain personal or regulated data, define retention and deletion propagation for source files, chunks, embeddings, caches, and evaluation traces. Keep document_id opaque and avoid placing sensitive content in identifiers.

In production

Replace the local directory walk with an object-store or queue-driven ingestion service. Make ingestion idempotent using a source version or content hash, and maintain states such as discovered, parsed, chunked, indexed, failed, and quarantined. Pin parser versions and retain parser and chunker configuration with each batch. Add golden fixtures for each document family and compare extraction, section paths, chunk counts, and token distributions in CI. Carry ACL and tenant metadata into the index, support deletion by document version, and expose metrics for throughput, failure rate, empty extraction, chunk-size percentiles, duplicate rate, and reprocessing lag. Use bounded concurrency and backpressure so a large upload cannot starve query traffic. Keep the JSONL artifact or an equivalent manifest for replay and audit.

Limitations

The sample does not perform OCR, semantic table extraction, language-specific tokenization, sentence-boundary detection, or embedding. Character limits are not token limits. PDF reading order and heading recovery are inherently parser-dependent. The implementation sets page metadata but leaves character offsets null because offsets from heterogeneous parsers are not comparable without a source-specific coordinate model. It also treats HTML list items as paragraphs and does not remove boilerplate. These trade-offs make the example deterministic and inspectable, not universally correct.

Conclusion

Define a chunk schema with provenance, section context, stable IDs, and security metadata before choosing a vector store.
Parse structure before splitting; prefer heading and paragraph boundaries and use character splitting only as a fallback.
Validate extraction and chunk distributions with representative fixtures, including malformed and scanned documents.
Version parser and chunking configuration so reindexing is reproducible and deletions can follow a document version.

In this series

RAG architecture and the five places it fails
Building a document ingestion and chunking pipeline
Hybrid retrieval and reranking
Evaluating retrieval quality and answer faithfulness separately
Deploying RAG with monitoring and access control
Benchmarking RAG architectures under realistic workloads
D01/01 — RAG architecture and the five places it fails
D01/03 — Hybrid retrieval and reranking
Production-Grade Retrieval-Augmented Generation: From Fundamentals to Operations

Try it yourself

Add a fixture-based quality gate for one PDF, one HTML page, and one Markdown document. Assert that expected headings appear in section metadata, no emitted chunk exceeds the configured character limit, every chunk has tenant and ACL metadata, and rerunning ingestion produces byte-identical JSONL. Then add an OCR adapter for scanned PDFs and compare empty-extraction rate before and after it.

Originally published in the LLM engineering blog series.

Why Compute Is Difficult to Trade

Vikas Srivastava — Fri, 23 Oct 2026 23:31:43 +0000

The problem: compute is difficult to trade

AI teams often describe compute shortages as a simple capacity problem: there are not enough GPUs, so more GPUs must be built. The deeper problem is that compute is not a homogeneous commodity. A machine-learning workload needs a particular kind of accelerator, in a particular topology, with enough memory, network bandwidth, power, storage, and uptime, at a time when the software stack can actually use it.

That makes “available capacity” an ambiguous number. An idle accelerator in another region may be useless for a latency-sensitive service. A cluster with enough chips may still fail a distributed training job if its interconnect is slow or if instances are preempted. Even a machine that satisfies a hardware specification can be impractical if the required drivers, kernels, or libraries are unavailable.

The constraints interact. A buyer may accept a lower price in exchange for interruptibility, but only if the job can checkpoint and resume. A provider may advertise a large pool of devices, but the useful capacity depends on whether those devices can be assembled into the required placement at the same time. Network, storage, and power limits can turn nominal accelerator supply into much less usable capacity.

The argument behind treating compute as a new kind of resource is therefore not that GPUs resemble barrels of oil. It is that AI systems need a tradable input whose value depends heavily on location, timing, technical properties, and reliability. The unit being exchanged must describe the conditions under which computation can actually be completed.

How it works

A compute market has several layers. At the bottom is physical capacity: accelerators, host CPUs, memory, networking, cooling, electricity, and data-center space. Above that is an allocation layer that turns this capacity into jobs, reservations, or services. The allocation layer must match a workload’s requirements against machines that satisfy them, while accounting for when the resources will be available and how long they can remain available.

For inference, the important variables may be accelerator memory, tokens per second, batching behavior, and tail latency. A system that is inexpensive at full utilization may perform poorly when requests arrive unevenly or when batching increases response time. For training, the problem is more constrained: a job may require many similar devices connected with low-latency links, and its cost can rise sharply when workers wait for the slowest participant. The scheduler is not merely selecting cheap machines; it is selecting a topology and execution plan.

A market can represent a supply offer as a multidimensional contract rather than a single price:

offer = {
    "accelerator": "GPU",
    "memory_gb": 80,
    "count": 64,
    "region": "eu-west",
    "network": "high_bandwidth",
    "start": "2026-11-01T00:00:00Z",
    "duration_hours": 12,
    "interruptible": False,
}

A buyer can then submit constraints and preferences: minimum memory, acceptable regions, deadline, interruption tolerance, and maximum price. The matching engine can reserve capacity, split a job across providers, or reject the request when the constraints cannot be met. Splitting may be useful for loosely coupled workloads, but it is not automatically valid for tightly synchronized training, where placement and communication can dominate the result.

Settlement is another part of the mechanism. A reservation should specify whether the buyer pays for wall-clock time, successful work, or a committed block of capacity. It should also define what happens when a host fails, power is curtailed, a network link degrades, or a provider returns fewer usable devices than promised. Without those terms, a market can appear liquid while leaving the buyer to absorb nearly all operational risk.

What the evidence actually shows

The strongest evidence is operational rather than rhetorical: teams routinely struggle to turn nominal capacity into dependable training and serving capacity. The existence of specialized offerings, high-speed variants, and provider-specific configurations indicates that users care about more than the number of accelerators. They care about performance characteristics and the conditions under which that performance is delivered.

Those conditions affect the economics of a job. A lower-priced machine may deliver less useful work if communication stalls, if data must be moved across regions, or if a failure forces a restart. Conversely, a more expensive reservation can be valuable when it provides stable placement and makes completion time predictable. The relevant comparison is therefore between end-to-end outcomes, not just listed rates.

Energy is part of the same evidence. The carbon intensity of a workload depends on where and when it runs, not only on how much computation it performs. A job moved between grids can have a materially different environmental cost even when the model, batch size, and accelerator count remain unchanged. Location and timing are consequently operational attributes as well as environmental ones.

These observations support a market with location- and time-aware pricing. They do not prove that a single global exchange is the right design, nor that open trading will automatically reduce costs. They show that the abstraction of one universal “GPU-hour” is too coarse for many real workloads.

Using it

Start by describing workloads in terms that a scheduler or provider can verify. Record accelerator type, memory use, interconnect needs, expected utilization, checkpoint behavior, data location, latency targets, and whether interruption is acceptable. Separate hard constraints from preferences; otherwise a matching system will either overconstrain requests or make unsafe substitutions.

The distinction matters during procurement and scheduling. A data-location rule may be non-negotiable, while a preference for one accelerator family may be relaxed if an equivalent option passes a performance test. Similarly, an inference service may tolerate changes in capacity if routing can move traffic, whereas a synchronized training job may require a fixed group of devices for the duration of a run.

Measure effective work, not just rented time. For training, track useful steps per hour, restart time, checkpoint transfer cost, and synchronization stalls. For inference, track tokens per second at the required percentile latency, including queueing and cold-start effects. These measurements let you compare offers that look different on paper and identify which parts of a contract are creating real value.

Use flexible capacity only when the application is designed for it. Training jobs should checkpoint frequently enough to tolerate eviction, and inference systems should have routing, warm spares, or graceful degradation. A portable container is not sufficient portability if kernels, drivers, model-parallel libraries, or network assumptions tie the workload to one provider.

Keep an internal record of the full cost:

effective_cost =
    instance_cost
  + storage_and_egress
  + orchestration_overhead
  + failed_work
  + migration_or_restart_cost

This makes market quotes comparable and exposes the value of reliability. A cheaper interruptible instance can be more expensive if it repeatedly destroys long-running work. It also gives engineering and finance teams a shared way to evaluate a reservation: the best offer is the one that completes the required work under the required conditions.

Limitations and failure modes

Compute is difficult to standardize because hardware performance depends on software and workload shape. Two systems with the same accelerator count can produce different results because of compiler versions, kernel availability, memory pressure, communication libraries, or placement. A market may therefore need executable tests or performance attestations, not just provider-supplied specifications.

Attestations themselves need context. A result measured on one model, batch size, or communication pattern may not predict performance for another workload. Buyers should be able to distinguish a guaranteed property, such as a location or device count, from an estimate based on a benchmark. Otherwise precise-looking listings can create false confidence without making offers genuinely comparable.

There are also serious coordination risks. Providers may report capacity optimistically, reserve the best machines for internal customers, or make offers hard to compare. Buyers can create instability by canceling reservations, flooding the market with speculative requests, or concentrating demand in one region. A matching system must account for these behaviors rather than assuming every listing represents firm, independent supply.

A compute exchange cannot remove physical scarcity. Power interconnection, cooling, construction time, and chip supply remain bottlenecks. Nor does geographic optimization automatically solve emissions: moving work to a cleaner grid may conflict with data-residency rules, latency requirements, or available network capacity.

Finally, fragmentation can worsen portability. If every provider exposes a different scheduling model, billing contract, and reliability definition, users may become dependent on intermediaries rather than gaining a liquid market. The interface may look standardized while the underlying operational assumptions remain provider-specific.

What to watch

The most revealing developments will be the interfaces between providers and schedulers: portable workload descriptions, verifiable performance claims, and contracts that price interruption and failure explicitly. Watch whether buyers can compare effective completed work across heterogeneous systems, rather than merely comparing hourly rental rates.

It will also matter whether contracts make capacity commitments enforceable. A useful offer should communicate what is reserved, what can change during execution, and how compensation or replacement works when the promised service is unavailable. Those details determine whether a buyer can plan a training run or serving deployment around a market purchase.

Carbon-aware scheduling will test whether environmental attributes can be treated as first-class constraints without making jobs unpredictable. A scheduler may need to balance cleaner execution against data location, deadline, latency, and capacity availability rather than optimizing any one variable in isolation.

The practical outcome will depend less on a slogan about compute as a resource than on whether these details become machine-readable, enforceable, and useful to engineers.

What Microsoft’s AI Diagnostic System Actually Shows

Vikas Srivastava — Thu, 22 Oct 2026 23:31:49 +0000

The concrete problem

Clinical diagnosis is not simply a test of whether a model can recognize an image or retrieve a fact. A difficult case may contain incomplete symptoms, contradictory test results, an unusual disease, and several plausible explanations. The practical problem is deciding which evidence matters, what to ask next, and how confidently to recommend an action.

That makes diagnosis a sequential decision problem rather than a one-shot classification task. A clinician must separate observations from interpretations, preserve uncertainty, and decide whether the next useful step is another question, a test, a consultation, or immediate treatment. A system that names the correct condition but fails to identify an urgent action can still be unsafe.

Microsoft says its new AI diagnostic system outperformed generalist doctors on hard cases in a study published in Science. That is a consequential claim, but “outperformed doctors” is not the same as “can safely replace doctors.” It describes performance under a particular evaluation design, and the design determines what the result means.

The system is better understood as a reasoning workflow built around large language models. It attempts to organize clinical evidence, generate and test hypotheses, use tools or references, and produce a differential diagnosis rather than immediately predicting one label. Its value therefore depends both on the model’s capabilities and on how the workflow controls, exposes, and evaluates those capabilities.

How it works

A conventional classifier maps an input to an output: an image to a finding, or a set of measurements to a risk score. A diagnostic reasoning system instead treats the case as a sequential decision problem. It must maintain a working state containing observations, candidate diagnoses, unresolved questions, and the evidence supporting or weakening each candidate.

That state is useful because the same fact can have different significance depending on the competing hypotheses. A symptom may support several diagnoses, while a negative test may eliminate only some of them. Recording the candidates and the reasons for changing their ranking makes it possible to inspect whether the system is responding to evidence or merely producing a plausible narrative after the fact.

A language model supplies the basic capabilities: interpreting the case, proposing hypotheses, and generating a next step. The surrounding system can improve reliability by asking for explicit intermediate artifacts, such as a summary of the patient, a ranked differential, discriminating evidence, and a final recommendation. Those artifacts also create checkpoints where a clinician or evaluator can catch an error before it reaches the final answer.

The core loop resembles hypothesis testing:

Extract the facts without adding assumptions.
Generate several plausible explanations.
Identify evidence that distinguishes them.
Seek information, through a tool or the case record, that reduces uncertainty.
Update the ranking and repeat.
Report the diagnosis, alternatives, rationale, and confidence.

This structure matters because a single unconstrained model response tends to mix observation with interpretation. A staged workflow makes those categories inspectable and gives the evaluator more than a final answer to score. It also clarifies where a failure occurred: in extracting a fact, proposing a candidate, selecting a next question, interpreting evidence, or communicating uncertainty.

The system may also use multiple model calls or agents with different roles. One call can propose a diagnosis, another can challenge it, and a final call can synthesize the disagreement. That can produce better answers, but it does not create independent clinical evidence: all agents may share the same training biases and blind spots. Debate can improve coverage while still reinforcing a shared false premise.

What the evidence actually shows

The reported result supports a narrower conclusion than the headline suggests. On the study’s selected difficult cases and under its scoring procedure, the AI system produced answers judged better than those from the generalist physicians who participated. This is evidence that structured model-assisted reasoning can be effective on challenging written cases.

It does not establish superiority across ordinary clinical practice. A case-based evaluation usually provides a bounded description of the patient, while real clinicians can examine the patient, observe changes over time, consult colleagues, and order tests. The evaluation also normally rewards diagnostic correctness or explanation quality, whereas clinical care additionally requires triage, communication, consent, safety-netting, and responsibility for action.

The distinction is especially important when the system is presented as a complete product. A written answer can appear clinically sophisticated while omitting a practical constraint, such as what information is missing or what must happen immediately. Those omissions may not be fully represented by a score that focuses on whether the named diagnosis and rationale match a reference answer.

The signal that most of the improvement came from the underlying models is technically important. If replacing the model changes performance substantially while keeping the workflow fixed, then the orchestration layer is not the main source of capability. Conversely, if the workflow is compared only as a complete product, it is difficult to tell whether gains came from prompting, tool use, model scale, case selection, or better presentation.

A strong result should therefore include ablations: the base model alone, the workflow without tools, the workflow with fewer reasoning steps, and alternative models under the same conditions. It should also report errors, calibration, and performance by case type—not only the aggregate winner. These comparisons reveal whether the method is robust or whether its advantage depends on a particular combination of model and evaluation setup.

Using it

For engineering teams, the immediate use is decision support, not autonomous diagnosis. Give the system a constrained task: summarize the record, identify missing information, produce a differential, or flag a dangerous alternative that a clinician should consider. A narrower task makes the intended role easier to communicate and the failure modes easier to test.

Keep the original evidence separate from generated interpretation. Store the model’s extracted facts, citations, intermediate hypotheses, final answer, and clinician disposition so that an error can be reconstructed later. This separation also prevents a generated statement from silently becoming part of the source record and allows reviewers to compare what the system saw with what it inferred.

A simple evaluation harness can begin with expert-labeled cases:

for case in test_cases:
    facts = system.extract_facts(case)
    differential = system.rank_diagnoses(facts)
    answer = system.explain(differential, facts)

    score = expert_score(answer, case.reference)
    log(case.id, facts, differential, answer, score)

The important engineering work is in expert_score, not in producing a polished response. Define whether the system receives partial credit for including the correct diagnosis, penalize unsafe omissions, and separately measure unsupported claims. Have reviewers assess confidence against correctness; a system that is occasionally right but confidently wrong is unsuitable for high-stakes escalation.

Use retrieval only from sources with known provenance, and make citations visible to the reviewer. Tool calls should be logged and permissioned, especially if the system can access patient records or initiate orders. The workflow should make clear which actions are suggestions and which, if any, require explicit human authorization.

Limitations and failure modes

The largest risk is distribution shift. Rare cases are often represented in medical literature more clearly than in practice, while local populations, documentation styles, comorbidities, and available tests differ. A model can excel on curated puzzles and fail on an ordinary patient whose presentation is noisy or incomplete.

This shift can affect every stage of the workflow. Missing information may be mistaken for a negative finding, an unfamiliar documentation style may distort fact extraction, and a locally unavailable test may be recommended because it is prominent in the retrieved evidence. A system that performs well on the cases used for development therefore needs testing on the conditions in which it will actually be used.

Language models can also manufacture coherence. They may transform an uncertain clue into a confident narrative, overlook a negative finding, or cite a plausible but irrelevant source. Multi-agent debate does not reliably fix this: agents can converge on the same incorrect assumption and make it sound more persuasive. More text and more agreement are not substitutes for independent evidence.

Evaluation leakage is another concern. If case material or near-duplicates appeared in training data, apparent reasoning may partly be recall. Human comparison groups also need careful design; generalists answering written vignettes are not equivalent to specialists managing patients with full access to examinations and longitudinal data.

Finally, diagnostic accuracy is only one component of safety. A system that identifies the right disease but delays emergency escalation, recommends an unavailable test, or exposes sensitive records can still cause harm. Deployment needs human ownership, audit logs, access controls, and a clear mechanism for overriding the model.

What to watch

The next meaningful evidence will be prospective evaluation in real workflows, with outcomes beyond answer quality. Watch for studies that measure time to appropriate care, harmful recommendations, calibration, subgroup performance, and how clinicians behave when the model is wrong.

The human interaction is part of the evaluation, not a detail around it. A useful system might help a clinician notice a dangerous alternative or identify a missing question, while a poorly presented system might encourage premature agreement. Tests should therefore examine whether clinicians inspect the supporting evidence, how they respond to uncertainty, and whether the workflow makes disagreement easy.

Also watch whether model improvements transfer across clinical settings without rebuilding the workflow. If they do, the underlying foundation models are carrying most of the capability; if not, the durable contribution may be the case structure, retrieval layer, and evaluation discipline around them. That distinction matters for deployment: model progress may require repeated validation, while a carefully designed workflow can provide a more stable basis for oversight.

Fix, Not Just Resample

Vikas Srivastava — Wed, 21 Oct 2026 23:31:59 +0000

The problem: proof search wastes its best feedback

Language models generate Lean proofs by proposing tactics or terms, compiling them, and trying again after failure. The common fallback is resampling: if a candidate does not compile, discard it and ask the model for another one. That treats a failed proof as a binary event even though Lean usually explains what went wrong.

A failed step can reveal that a theorem has the wrong argument order, a hypothesis is missing, a type does not unify, or a tactic produced the wrong shape of goal. These are actionable constraints. They can narrow the next proposal without requiring the model to rediscover everything that was already correct.

The transferable technique is to make compilation an online repair signal, while strictly bounding how many edits are allowed before abandoning the candidate. The goal is not to rescue every bad proof. It is to spend the information already produced by verification before paying for an entirely new search branch.

How it works

A Lean proof is a sequence of state transitions. The current state contains one or more goals; a tactic transforms that state, and Lean checks the resulting term before accepting the transition. If the tactic fails, Lean emits a diagnostic tied to the source location and often includes the expected type, available hypotheses, or remaining goal.

A repair loop keeps the current proof context and asks for a local correction. The model does not restart from the original theorem unless the bounded repair budget is exhausted. Conceptually, the loop is:

Generate the next proof step.
Compile the file or command containing that step.
If it succeeds, commit the step and continue.
If it fails, return the diagnostic and current goal to the model.
Accept only a repaired step that compiles.
Stop repairing after a small fixed number of attempts.

The important detail is granularity. Repairing one tactic or one proof fragment preserves all earlier verified work. Rewriting the entire proof after an error throws away useful structure and makes the search distribution much broader. A local loop also gives the model a clearer assignment: explain the reported mismatch by changing the failing fragment, rather than inventing a new proof from scratch.

A minimal compiler check can be driven with ordinary process execution:

import subprocess

result = subprocess.run(
    ["lake", "env", "lean", "Main.lean"],
    text=True,
    capture_output=True,
)

if result.returncode == 0:
    print("accepted")
else:
    print(result.stdout + result.stderr)

In a real agent, the generated fragment should be inserted into a temporary Lean file or an isolated project. The repair prompt should include the exact diagnostic, the active theorem statement, the local hypotheses, and the candidate step—not an unbounded transcript of every previous attempt. This keeps the context focused on the state Lean actually rejected and makes each repair easier to attribute.

What the evidence actually shows

The relevant evidence is mechanistic: Lean rejects incorrect proof steps with structured, precise feedback, while pure sampling discards that information. A bounded compiler-guided repair module is therefore exploiting a signal that ordinary generate-and-resample pipelines already pay to obtain. The compiler is doing more than checking the final answer; it is describing the boundary between the proposed term and the required one.

The reported work, Fix, Not Just Resample, presents this as an online, step-level repair strategy rather than a claim that every failed proof can be salvaged. Its central comparison is between using compiler errors to modify a candidate and treating each error as a reason to sample independently. The distinction is especially important when a candidate is mostly sound and only one local choice has the wrong shape.

That distinction matters when interpreting results. A repair system can improve proof completion without making the language model intrinsically better at theorem proving; it may simply use external verification more efficiently. Conversely, any gains depend on diagnostics being sufficiently informative and on the model being capable of mapping them to a valid local change. If the error is ambiguous or the needed change is global, a local repair loop may offer little advantage.

The strongest general lesson is broader than Lean: whenever a verifier returns structured failure information, feed that information back into a bounded local search. The verifier becomes part of the inference loop, not merely a final gate. This framing also clarifies what to evaluate: not only final success, but how often a diagnostic leads to a valid next step and how often the system recognizes that it should stop repairing.

Using it

Start with a narrow repair budget. Allow perhaps a few attempts for each failed step, and then fall back to a fresh candidate or terminate the theorem. The bound prevents a difficult goal from consuming the entire run and makes latency and cost predictable. It also keeps the agent from repeatedly making cosmetic edits when the underlying proof plan is wrong.

Separate accepted proof state from speculative text. Only update the canonical proof after Lean compiles the candidate; otherwise, the next repair should see the last verified state plus the failed fragment and its diagnostic. This separation prevents one unsuccessful edit from contaminating later prompts and gives backtracking a precise point to return to.

Normalize diagnostics before placing them in a prompt. Remove duplicated compiler banners, preserve line and column information, and keep the goal state intact. A useful repair instruction is explicit: change only the failing step, preserve preceding code, and return Lean code without explanation. The model should know both what it may alter and what must remain invariant.

Use deterministic checks for every proposed repair. Run Lean in the same project environment, with the same imports and options used for evaluation. If possible, compile a small temporary file containing the theorem and its dependencies so syntax and elaboration errors are caught quickly. Log each transition: candidate text, diagnostic, repair attempt, and whether the repaired step compiled. These traces are useful for debugging prompts, measuring where repairs fail, and later training a model on successful error-to-fix transformations.

Limitations and failure modes

Compiler feedback is not always a complete diagnosis. A message may point at the location where elaboration became impossible rather than the earlier mistake that caused it. Multiple fixes may be plausible, and the model can choose one that compiles locally but makes later goals harder. A diagnostic can constrain the next move without identifying the best move.

Local repair also cannot solve every global planning error. If the proof chose the wrong lemma, introduced an unusable intermediate claim, or discarded a necessary hypothesis, repeatedly editing the current tactic may be worse than restarting. The repair budget must therefore coexist with a recovery policy for backtracking or resampling. The system needs a way to recognize that the current prefix is no longer promising.

There is a verification cost as well. Invoking Lean after every token is impractical, while waiting until an entire long proof is generated reduces the value of step-level feedback. Systems need a practical unit of compilation, such as one tactic block or one theorem fragment, and should measure that choice against process startup and elaboration time. Smaller units provide sharper feedback but may require more checks; larger units amortize checking but make failures harder to localize.

Finally, successful compilation is not the same as a useful proof search trajectory. A repair policy can overfit to error messages, produce brittle tactic scripts, or exploit expensive automation. Track proof length, compilation time, and robustness—not only whether the final theorem closes. A repair that succeeds once but leaves an opaque or costly script may be less valuable than a fresh candidate that produces a simpler continuation.

What to watch

The next useful developments will be better representations of proof state and diagnostics, not merely larger generators. In particular, watch for repair policies that learn when to edit locally, when to backtrack, and when to request a new plan. Those decisions determine whether compiler feedback becomes a disciplined search strategy or just another prompt payload.

The interface between verifier and model is also important. A useful representation should distinguish the active goal from surrounding text, identify which fragment is speculative, and preserve the hypotheses that a repair is allowed to use. That structure can help the model make smaller edits and can make logged repair traces useful beyond the run that produced them.

The same pattern should transfer to SQL execution, typed program synthesis, configuration validation, and hardware design: preserve the verified prefix, expose the verifier’s structured failure, and spend a bounded number of attempts on a targeted fix. In each setting, the exact diagnostic differs, but the control problem is similar: decide how much of the current artifact remains trustworthy, apply a constrained correction, and abandon the branch when local repair no longer matches the failure.

When Training Data Makes False Facts Look True

Vikas Srivastava — Wed, 21 Oct 2026 01:39:06 +0000

The problem: a training corpus can amplify false facts

A language model does not know whether a statistic is true because it appeared in a polished report, a forum post, or a fabricated page. During pretraining, all of those sources become token sequences competing for probability mass. A false number repeated often enough can become a plausible continuation, even when no individual document is authoritative.

This creates a specific engineering problem: web-scale collection optimizes coverage and throughput, while factual reliability depends on provenance, independence, and verification. A fabricated statistic can be copied into summaries, scraped into datasets, and quoted by later pages, making duplication look like corroboration. By the time it enters a training corpus, the same claim may appear to have many sources while actually tracing back to one unsupported origin.

The resulting failure may appear at inference time as hallucination, but the cause sits earlier. The model was trained on a mixture whose frequency and structure rewarded the false claim. Prompting may expose the problem, and decoding may shape how confidently it is expressed, but neither necessarily created the underlying association.

How it works

Pretraining usually minimizes next-token cross-entropy. Given a sequence of tokens, the objective increases the probability of the observed next token; it does not include a truth label or a requirement that claims agree with external evidence. A sentence containing a false statistic therefore produces the same training signal as an accurate sentence with similar syntax.

Frequency matters, but not in a simple “most repeated claim wins” way. The model also learns authorial patterns, formatting, surrounding explanations, and associations between entities. A number repeated across near-duplicate pages can become strongly associated with a topic, while a contradictory source may be diluted by deduplication, lower sampling weight, or simply being less common. The claim is learned together with the language used to introduce it, explain it, and present it as settled.

The data mixture determines these pressures. A corpus may combine books, code, news, reference pages, social content, and synthetic text, each with separate quality and duplication profiles. Sampling weights act like an implicit prior: increasing a noisy source’s share gives its patterns more influence, while aggressive filtering can remove rare but valuable evidence. The same filtering decision can therefore reduce a false claim in one domain while removing useful disagreement in another.

Scale makes the issue more subtle. Larger models and more compute generally improve pattern learning, but they also make it easier to memorize distinctive sequences when the data contains repeated or highly specific claims. Regularisation can control overfitting to examples, yet it does not turn an unverified corpus into a verified one; a broadly distributed false statement may remain a well-supported pattern from the optimizer’s perspective.

Fine-tuning changes the balance rather than erasing pretraining. Instruction data can teach a model to qualify uncertain answers or cite sources, but a small supervised set may not overwrite a strongly represented association. Preference optimization can reward useful refusal and citation behavior while still leaving the underlying factual association available for generation. The model may sound more cautious without having acquired a reliable basis for deciding which claim is correct.

What the evidence actually shows

The strongest evidence from the fake-statistics signal is behavioral: language models can reproduce or endorse claims that appear to have been planted or propagated online. That demonstrates a path from public text to model output, but it does not by itself measure how often fabricated claims enter training, which training run consumed them, or whether the model memorized a particular page.

It also does not establish that repetition guarantees acceptance. Models can produce different answers across prompts, distinguish some reputable sources from weak ones, or retrieve current evidence at runtime. The important observation is narrower and more actionable: publication and repetition can increase the chance that a claim becomes available as a fluent answer. Availability is not the same as belief, and fluent expression is not evidence of verification.

A useful evaluation therefore separates three questions. Can the model generate the claim without retrieval? Does it prefer the claim when shown conflicting evidence? And can it identify the claim’s provenance rather than merely attach a plausible citation? These tests distinguish memorization, calibration, and source-grounding. They also reveal whether a retrieval system changes the answer because it supplies evidence or merely gives the model another passage to imitate.

For training teams, corpus audits are more informative than a single factuality score. Measure near-duplicate clusters, claim frequency, source independence, and how often high-impact facts occur without supporting references. Inspecting the data mixture can reveal that a model’s confident error is a predictable consequence of sampling, not an unexplained defect in decoding. It also makes remediation more concrete: teams can decide whether to change collection, weighting, filtering, or fine-tuning.

Using it

Treat factual quality as a data-lineage problem. Preserve source URLs, timestamps, document hashes, extraction methods, and license metadata before normalization; without these fields, later filtering and incident investigation become guesswork. These records allow a team to trace a suspicious output back through transformed documents and identify whether apparently separate pages share an origin.

Hash-based deduplication should be supplemented with near-duplicate detection because copied paragraphs often differ only in introductions or formatting. A practical first pass can identify suspicious repetition without pretending to verify every claim:

from collections import Counter
import re

def normalize(text):
    return re.sub(r"\s+", " ", text.lower()).strip()

docs = [normalize(x) for x in open("documents.txt", encoding="utf-8")]
counts = Counter(docs)

for document, frequency in counts.most_common():
    if frequency > 5:
        print(frequency, document[:160])

Production systems should cluster semantically similar documents, downweight low-independence clusters, and route high-impact claims for human or trusted-source verification. Keep a held-out evaluation set containing conflicts, unattributed numbers, and deliberately repeated false claims; otherwise, a cleaner corpus may look better only because the test set shares the same biases. The audit should record both removed material and retained uncertainty, so later changes can be interpreted rather than guessed at.

Sampling is a safer control than an irreversible deletion rule when uncertainty is high. Retain a small, traceable sample of disputed material for research, but prevent it from dominating training. During supervised fine-tuning, include examples that model uncertainty, request evidence, and refuse to invent citations; evaluate those behaviors under adversarial prompts rather than only with clean questions. The goal is to make uncertainty useful and observable, not to make every answer evasive.

Limitations and failure modes

No filter can infer truth reliably from text alone. Authoritative-looking sources can be wrong, true claims can be rare, and disagreement may reflect legitimate differences in definitions or measurement windows. A system that removes every claim lacking a citation will bias the corpus toward well-documented institutions and away from emerging or marginalized sources. Documentation can support review, but it cannot substitute for understanding what a claim actually measures.

Deduplication also has costs. Two similar documents may be independent reports that should count as corroboration, while many differently worded pages may descend from one original error. Provenance graphs and source ownership help, but they require accurate extraction and maintenance. A cluster is therefore evidence about relationships among documents, not an automatic judgment that every member is false or redundant.

Fine-tuning and retrieval are not complete repairs. A retrieval layer can surface evidence that contradicts the model’s internal association, yet the model may still summarize the wrong passage or privilege a familiar claim. Conversely, forcing citations can produce citation-shaped text whose sources do not support the answer. Verification must examine whether the cited material entails the response, not only whether a citation appears.

Finally, data curation competes with compute and latency budgets. Claim extraction, clustering, verification, and repeated evaluation add pipeline cost. The right target is not a perfectly clean corpus, which is unattainable, but a measurable reduction in correlated, high-impact errors with an audit trail for the remaining uncertainty. That target supports incremental controls instead of promising that one filter will solve factuality.

What to watch

The next useful generation of training evaluations will track provenance through the entire pipeline: collection, filtering, sampling, pretraining, fine-tuning, and deployment. Pay particular attention to whether models can distinguish independent confirmation from copied repetition. A system should be tested not only on whether it gives the right answer, but also on whether it can explain which evidence supports that answer and where uncertainty remains.

Also watch for training recipes that report data-mixture changes alongside objective and compute changes. Without that separation, an apparent factuality improvement may come from a cleaner corpus, a stronger instruction layer, retrieval at evaluation time, or simply a different prompt distribution. Controlled reporting makes it possible to connect an observed improvement to the intervention that produced it.

Finally, monitor the gap between behavior in evaluation and behavior in deployment. New documents can repeat old errors, retrieval sources can conflict, and fine-tuning can change how confidently a model presents an unchanged association. Provenance-aware incident review can show whether a failure came from collection, sampling, model behavior, or the evidence supplied at runtime.

The pipeline problem: false claims are easy to train on

Vikas Srivastava — Tue, 20 Oct 2026 23:32:07 +0000

The pipeline problem: false claims are easy to train on

A language model does not know that a statistic is true because it has a citation, a polished chart, or a confident sentence around it. If a false number appears often enough in material that enters the training mixture, the model can learn it as a plausible continuation—and later reproduce it with the style of an established fact. The model is learning relationships among tokens and contexts, not independently checking the world described by those tokens.

That is the concrete problem behind reports of fabricated statistics being cited by language models. A number can acquire an appearance of authority as it moves through articles, summaries, social posts, and other derivative material. By the time it reaches a model or a retrieval system, the claim may look well established even though the apparent agreement traces back to one unsupported source.

The failure is not located in one bad prompt or one defective retrieval result. It can emerge from the interaction between web collection, deduplication, quality filtering, repeated publication, supervised examples, preference optimization, and evaluation gaps. Each stage can preserve the problem, obscure its origin, or reward the model for expressing it more fluently.

For training teams, “data quality” therefore means more than removing profanity or malformed HTML. It means estimating whether a claim is independently supported, how widely it has propagated, and whether the model is rewarded for presenting uncertainty when support is weak. It also means making it possible to distinguish an original source from the many pages that merely repeat it.

How it works

Pretraining usually minimizes next-token prediction loss over a large mixture of documents. For a sequence of tokens, the model is trained to assign high probability to the observed next token. It is not directly trained to distinguish an accurate claim from a repeated false claim. A sentence that is grammatically complete, stylistically conventional, and frequently observed can therefore be an easy target for the objective even when its content is unsupported.

Suppose several pages contain the same unsupported statistic. Near-duplicate removal may reduce exact copies, but lightly rewritten versions, syndicated articles, scraped summaries, and derivative social posts can remain. Each occurrence supplies another gradient signal for the same wording and its surrounding context. Even when the exact number changes, repeated associations between a topic and a confident numerical claim can make that type of answer easier to produce.

The mixture matters as much as the raw corpus. A small, high-quality reference set can be overwhelmed by a much larger low-quality source, while aggressive filtering can remove useful long-tail evidence. Sampling weights determine how often each source contributes to optimization; they are an implicit statement about what the model should treat as normal knowledge. A document that survives filtering is not merely stored—it is given opportunities to influence the model.

Later stages can amplify the effect. Instruction tuning teaches the model to answer questions directly, and preference optimization may favor an answer that sounds specific and helpful over one that says “I cannot verify that.” If synthetic training data is generated by an earlier model, unsupported claims can also be copied into new examples and fed back into the pipeline. The claim may then appear in training examples that are more directive and easier for the model to imitate than its original web occurrence.

Retrieval changes the path but not the basic risk. A search system may find the original fabricated page, while a language model may summarize it as evidence. If the same claim appears across many copied pages, retrieval ranking can mistake repetition for independent corroboration. Retrieval can improve answers when it surfaces suitable evidence, but it cannot turn a collection of repeated pages into independent verification.

What the evidence actually shows

The source signal is a useful warning because it demonstrates an observable failure mode: fabricated statistics can acquire apparent authority through publication and later appear in model-generated answers. It does not, by itself, establish how frequently this occurs, which model stage caused it, or whether the model learned the claim from pretraining rather than retrieval or prompting. An example is evidence that the pipeline can fail, not a complete diagnosis of why a particular output occurred.

That distinction matters when interpreting examples. A model may have memorized a sentence, inferred a plausible number from a familiar pattern, followed misleading context supplied in the prompt, or retrieved a bad source at inference time. These mechanisms produce similar outputs but require different fixes. Memorization points toward corpus and mixture analysis; retrieval errors require source ranking and evidence handling; prompt sensitivity requires testing how the answer changes when the framing changes.

The broader technical conclusion is narrower and stronger: citation and repetition are weak proxies for truth. A document-level quality classifier can detect spam signals, but it cannot generally verify an empirical claim without access to suitable primary sources, methods, definitions, and time boundaries. A claim may look consistent across documents because the documents share an origin, use different definitions, or describe different periods.

A useful evaluation should therefore test claims, not just prose quality. Construct adversarial sets containing unsupported numbers, conflicting sources, copied claims, and questions where the correct response is uncertainty. Measure attribution quality, calibration, abstention, and sensitivity to source changes rather than only exact-match accuracy. Tests should also examine whether the model changes its answer when a copied summary is replaced with a primary source or when the relevant time boundary is made explicit.

Using it

Start by making the data mixture inspectable. Record source domain, collection time, document lineage, language, deduplication cluster, quality scores, and sampling weight. If a claim later appears in an evaluation or incident report, engineers should be able to trace which documents and training stages could have exposed the model to it. Lineage is especially important when several URLs contain nearly identical material, because URL counts alone can exaggerate the apparent breadth of support.

Use several filters instead of one universal “quality” score. Exact and semantic deduplication reduce repetition; source-level allowlists and blocklists provide coarse controls; claim extraction and contradiction checks can prioritize human review. Keep an untouched holdout so that tuning filters does not turn the evaluation set into another training source. These controls should be treated as complementary: a document can pass a spam filter while still containing an unsupported statistic, and a repeated claim can remain suspicious even when each copy looks clean.

A simple claim-review queue might begin like this:

from collections import Counter

def candidate_claims(records, min_sources=3):
    counts = Counter(
        (r["claim"], r["domain"])
        for r in records
        if r["quality_score"] >= 0.8
    )
    return [
        {"claim": claim, "domain": domain, "count": count}
        for (claim, domain), count in counts.items()
        if count >= min_sources
    ]

This code does not establish truth. It identifies repeated claims within domains for review; the important production feature is preserving provenance and distinguishing independent sources from copies. A practical queue would attach the underlying records, timestamps, lineage information, and any extracted definitions so that a reviewer can investigate why the claim appears repeatedly rather than treating the count as evidence.

During training, compare mixtures through controlled ablations. Hold compute and optimization settings as constant as practical, vary source weights or filtering rules, and evaluate factuality, calibration, and memorization on both common and obscure claims. If preference data rewards confident specificity, add examples where a cautious, source-qualified answer is preferred. The goal is not to make every answer vague; it is to teach the model that the strength of its wording should track the strength of its evidence.

At inference time, require evidence for high-stakes numeric answers. Retrieval should rank independent primary sources above summaries, expose publication dates and definitions, and avoid presenting the number of matching pages as a confidence score. A response should make clear when sources disagree or when the retrieved material does not support the requested precision, rather than smoothing those limitations into a single confident number.

Limitations and failure modes

Truth filtering is expensive and domain-dependent. A claim can be correct but poorly sourced, temporarily true, defined differently across studies, or expressed with rounding that appears contradictory. Automated contradiction detection may discard valid disagreement or preserve a fluent falsehood. Review systems therefore need to retain the context in which a claim was made, including its population, measurement, date, and level of precision.

Source reputation is not enough either. A respected domain can publish an error, while a new domain can contain valuable primary research. Blocking domains can also create blind spots and encourage attackers to move claims to sources that pass the filter. Reputation can help prioritize review, but it should not be treated as a substitute for examining the evidence behind a statistic.

Deduplication has its own trade-off. Removing every semantically similar document may erase legitimate independent confirmations and reduce coverage of minority viewpoints. Keeping them can let a single origin dominate the gradient under many different URLs. The useful distinction is not simply between identical and non-identical text, but between independent evidence and repeated language.

Finally, no training pipeline can guarantee factual answers from next-token prediction alone. Models can hallucinate unseen claims, fail to use retrieved evidence, or follow a user's false premise. Monitoring must cover the full path from corpus to serving, with incident analysis that separates memorization, retrieval, and generation. Without that separation, teams may apply a filter to a problem caused by prompting or change retrieval when the model has already absorbed the claim.

What to watch

The next useful tooling will make claim provenance a first-class training artifact rather than an after-the-fact debugging detail. Watch for mixture specifications that include lineage, independence estimates, temporal cutoffs, and explicit uncertainty policies. Such specifications make it easier to reason about what the model was exposed to and why a source was allowed to contribute at a particular weight.

Also watch evaluation design. The most revealing tests will not ask only whether a model knows a popular fact; they will test whether it resists repeated unsupported claims, updates when primary evidence changes, and declines to manufacture precision when the data does not justify it. They should also vary prompts and retrieval sources so that teams can see whether a failure follows the model, the evidence presented to it, or the framing of the question.

Vector Caching in Production AI Systems

Vikas Srivastava — Tue, 20 Oct 2026 01:31:48 +0000

The production problem: repeated work hides inside every request

Production AI systems rarely process each request only once. The same document chunks are embedded repeatedly, common questions trigger nearly identical retrievals, and application-level prompts often differ only in metadata such as a user ID or timestamp. Agent workflows can amplify the effect: one user request may produce several observations, searches, or follow-up calls that revisit the same material.

That repetition creates three separate costs: compute for embeddings, latency for vector search, and tokens for downstream model calls. A conventional key-value cache handles exact strings well, but retrieval workloads are not exact-string workloads. Two queries can be worded differently while selecting the same context, and one query can require a different result when the corpus changes. Caching the final model response may also be too brittle when only the retrieved evidence is reusable.

The emergence of Redis-compatible vector-cache projects, including the KiviDB signal, points to a practical design question: should retrieval results be treated as temporary application data rather than recomputed on every request? The useful answer is not “cache everything,” but to identify which intermediate results are stable enough to reuse and which must be regenerated. The right boundary depends on how often data changes, how costly an error is, and whether the result is tied to a particular user or permission scope.

How it works

A retrieval pipeline usually has at least four stages: normalize the request, create an embedding, search an index, and assemble context for a model. Each stage has a different cache key and a different invalidation problem. Normalization must be stable enough that harmless formatting differences do not create unnecessary misses, while context assembly may change whenever a prompt template, chunk formatting rule, or citation requirement changes.

An embedding cache maps a canonicalized text representation to a vector. A retrieval cache maps a query representation, collection version, filters, and retrieval parameters to a ranked list of document identifiers and scores. A context cache may go one step further and store the formatted passages, but that layer is more sensitive to prompt-template changes. Keeping identifiers and scores instead of full content usually makes it easier to refresh documents and enforce current access rules.

Approximate reuse is where vector caching differs from ordinary memoization. Instead of asking whether the new query has the same bytes as an old query, the system searches a cache of prior query vectors. If a nearby cached vector is within a chosen distance threshold, its result can be reused. The threshold is a policy decision: a generous threshold improves hit rate while increasing the chance that a semantically different request receives stale context. Thresholds may therefore differ by endpoint, with stricter reuse for policy or account questions than for broad discovery.

A safe cache key therefore includes more than the query. It should encode the tenant or authorization scope, corpus or index version, embedding-model identity, distance metric, top-k value, filters, reranking configuration, and any policy that changes what a caller may see. Without those dimensions, a cache can return technically relevant information to the wrong user or apply yesterday’s retrieval rules to today’s index.

A vector cache is also not a replacement for the source index. It is a shortcut around repeated work. On a miss, the system performs the normal retrieval path, stores a result with an expiry or version, and returns the answer. On a hit, it still needs to verify authorization and possibly fetch the current document payloads by ID. This separation lets the source index remain authoritative while the cache absorbs repeated access patterns.

What the evidence actually shows

The strongest evidence for caching is operational rather than model-centric: production traffic is usually skewed. Frequently asked questions, common support workflows, repeated agent observations, and popular documents create locality that a cache can exploit. The same locality can appear at different layers: repeated text benefits from embedding caching, repeated intent benefits from approximate retrieval reuse, and repeated document IDs benefit from payload caching.

The source signal is a project described as a faster Redis-compatible memory database and vector cache. That description supports investigating a familiar operational shape—low-latency, memory-resident access through Redis-style interfaces—but it does not establish a universal speedup, a particular hit rate, or superiority over an existing database. Those claims require measurements on an organization’s workload, data size, distance function, and hardware. A system that performs well for compact query vectors may behave differently when entries include filters, scores, provenance, and larger metadata.

The important experiment is not a benchmark on random vectors. Measure a replay of real requests with separate counters for embedding hits, approximate retrieval hits, exact retrieval hits, misses, invalidations, and rejected reuse. Record p50 and tail latency, cache memory, downstream model calls, and answer-quality changes against an uncached control. Also distinguish a cache hit that avoids only vector search from one that avoids the entire retrieval-and-context path; otherwise the operational benefit can be overstated.

A cache can make a system cheaper while making it less correct. Evaluation must therefore compare the final response and retrieved evidence, not merely whether the cache returned quickly. Include changed documents, permission changes, ambiguous paraphrases, rare queries, and adversarial requests designed to cross tenant boundaries. Inspect both false reuse, where an old result is accepted incorrectly, and false misses, where the system safely recomputes but loses the expected efficiency.

Using it

Start with the least risky layer: cache deterministic embeddings for immutable text. Canonicalize whitespace and encoding, include the embedding model in the key, and retain the original text hash so that accidental preprocessing changes do not create silent collisions. This layer is comparatively easy to reason about because the output is a function of the text and model identity, rather than of a changing corpus or caller-specific permissions.

Next, cache retrieval results by collection version. A simple version can be a monotonically increasing index generation that changes whenever documents are added, removed, re-embedded, or access policies change. This is often easier to reason about than trying to invalidate individual query results perfectly. It may create extra misses during updates, but those misses are visible and predictable instead of leaving teams to discover stale entries through incorrect answers.

A typical application boundary can look like this:

import hashlib
import json
import redis

cache = redis.Redis(host="localhost", decode_responses=True)

def retrieval_key(query, collection_version, filters, top_k):
    payload = {
        "query": query.strip(),
        "collection": collection_version,
        "filters": filters,
        "top_k": top_k,
    }
    raw = json.dumps(payload, sort_keys=True)
    return "retrieval:" + hashlib.sha256(raw.encode()).hexdigest()

def get_or_retrieve(query, version, filters, top_k, retrieve):
    key = retrieval_key(query, version, filters, top_k)
    cached = cache.get(key)
    if cached:
        return json.loads(cached), "hit"

    result = retrieve(query, filters, top_k)
    cache.setex(key, 300, json.dumps(result))
    return result, "miss"

In a real vector cache, approximate lookup can sit before this exact-key lookup. Treat its result as a candidate, not an unquestionable answer: verify tenant scope, collection version, filters, and distance threshold before reuse. The exact-key path remains useful after approximate matching, because it can provide a deterministic fallback and make behavior easier to inspect.

Keep payloads small by caching document IDs, chunk offsets, scores, and retrieval metadata rather than duplicating large documents. Fetch current content after the cache hit, and make cache writes asynchronous when they would extend request latency. Log the cache decision and provenance alongside the request so an operator can determine why a result was reused.

Limitations and failure modes

The central failure mode is stale evidence. A document can be corrected while its cached retrieval result continues pointing to an old chunk, or a deleted record can remain available through a payload cache. Versioned indexes and short expirations reduce exposure, but neither substitutes for explicit deletion handling. Deletions and permission changes should trigger a clear invalidation path rather than relying solely on eventual expiry.

Semantic similarity is not semantic equivalence. A query about cancelling an account may be close in embedding space to a query about changing billing details while requiring different policies. Use approximate reuse only where the application can tolerate that distinction, and prefer exact caching for high-risk operations. A result can be topically related yet still omit the qualification that makes the answer safe.

Authorization leakage is more serious than a low-quality answer. Never let a shared approximate cache decide access. Scope entries by tenant and security context, recheck permissions after retrieval, and avoid caching responses whose contents depend on private user state unless the isolation boundary is explicit. Even document identifiers can reveal sensitive information if they are exposed across boundaries, so the protected unit is not limited to the final text.

Memory pressure introduces another tradeoff. Eviction can turn a cache into an unpredictable latency layer, while oversized entries can starve the actual retrieval service. Track hit rate by endpoint and tenant, cap entry size, and define behavior when the cache is unavailable: bypass it and continue, rather than making it a hidden single point of failure. Operational controls should also make it possible to disable approximate reuse independently from exact caching.

Finally, evaluate cache poisoning. If untrusted users can cause arbitrary vectors or results to be inserted, they may increase false matches or force expensive eviction. Restrict writes to trusted retrieval paths and attach provenance to every cached result. Provenance should identify the collection generation and retrieval configuration used to create the entry, giving operators a way to distinguish legitimate reuse from an entry produced under an unexpected path.

What to watch

The next useful development is not merely faster vector storage. Watch for cache systems that expose provenance, collection generations, authorization-aware namespaces, distance thresholds, and observability as part of the data model. These features turn caching from an opaque performance layer into a component whose decisions can be inspected and governed.

Also watch whether teams converge on layered policies: exact caches for deterministic work, approximate caches for low-risk retrieval, and no caching for mutable or security-sensitive decisions. That division is likely to matter more than the raw lookup speed, because production reliability depends on knowing when reuse is safe.

The practical signal will be whether these systems fit naturally into existing operational workflows. Teams need version changes to invalidate predictably, cache failures to degrade gracefully, and quality regressions to be visible in the same monitoring loop as latency and cost. A vector cache becomes production infrastructure only when those controls are treated as part of retrieval correctness, not as optional features added after deployment.

Quizzing Developers on AI-Generated Code

Vikas Srivastava — Mon, 19 Oct 2026 01:31:45 +0000

The concrete problem

AI coding agents can produce a plausible patch faster than a human can reconstruct its reasoning. That reverses the usual review problem: the bottleneck is no longer typing code, but determining whether the generated change preserves the system’s actual invariants. The code may look idiomatic and fit the surrounding style while still reflecting a faulty model of the system.

A diff can pass formatting and even a narrow test suite while encoding a misunderstanding about authorization, retries, data ownership, or failure handling. For example, a new retry path may duplicate an external side effect, or a fallback may quietly treat missing data as permission to proceed. A developer who accepts a fluent explanation without checking those assumptions has effectively moved review from code to trust.

The Sphinx concept—a hook that quizzes developers on AI-generated code before shipping—targets this gap. Its important architectural choice is to make explanation a delivery-stage control rather than an optional prompt in the coding session. The question arrives when the patch is concrete, reviewable, and close to becoming part of the system.

That placement matters because an agent’s confidence is not evidence that its assumptions are correct. A delivery-stage check can force the person responsible for approval to articulate what changed, where it can fail, and which existing guarantees it touches. It does not make the patch correct by itself, but it makes unexamined understanding harder to disguise as review.

How it works

A coding agent produces at least two artifacts: a proposed patch and an implicit model of why that patch should work. Conventional tooling validates mostly the first artifact through compilation, linting, tests, and static analysis. A quiz hook attempts to inspect the second by asking the human to state the patch’s behavior and risks. In practice, the answers can reveal whether the reviewer has connected the changed lines to the surrounding system.

The hook belongs at a boundary where the change is already concrete but has not yet become difficult to undo. Depending on the team’s workflow, that might be a commit hook, pull-request check, merge gate, or deployment approval. It can examine the staged diff, identify changed files, and generate questions focused on the affected behavior. Running against the staged or proposed artifact also prevents a mismatch between what was explained and what eventually ships.

Good questions are falsifiable and local. “What does this change do?” invites a vague summary; “What happens when this lookup returns no row?” tests a specific branch. Other useful prompts ask which input is trusted, what happens on retry, which test would fail if the assumption were wrong, and whether the change alters transaction or concurrency behavior. Questions should direct attention toward observable consequences rather than reward polished descriptions.

The mechanism resembles active recall more than documentation. The developer must retrieve an explanation before seeing or accepting a suggested answer, which can expose unfamiliar code paths that passive reading misses. The hook is therefore a human-in-the-loop verifier, not a proof system and not a replacement for tests. A minimal delivery gate can combine deterministic checks with a human checkpoint:

#!/bin/sh
set -eu

git diff --cached --check
python -m pytest
printf '%s\n' "Explain the changed behavior, failure cases, and affected invariants."
read -r explanation

test -n "$explanation" || {
  echo "A non-empty review explanation is required." >&2
  exit 1
}

A production implementation would use the explanation as review input, not treat any non-empty string as evidence of correctness. The key is that the gate runs against the staged or proposed change, so the reviewed artifact is the one that can actually ship.

What the evidence actually shows

The available signal shows substantial developer interest in a hook that quizzes people about AI-written code. That supports the existence of a real workflow concern: generated patches are creating pressure for stronger review habits and better accountability. Interest also suggests that teams recognize a difference between accepting a patch and understanding the assumptions behind it.

It does not establish that quizzes reduce escaped defects, improve code comprehension, or distinguish safe patches from unsafe ones. Attention to a tool idea is evidence of demand for the workflow, not evidence that its particular intervention works. A team may welcome an additional review ritual without using it carefully, and a developer may answer accurately while the implementation contains a separate defect.

The strongest case for the approach is mechanistic. Requiring a developer to predict behavior before approval should reveal some misunderstandings that a quick visual scan misses, especially around edge cases and implicit assumptions. But whether that benefit survives time pressure, repetitive prompts, and experienced developers who learn to answer mechanically requires controlled evaluation. The effect may also differ by patch complexity and by how much repository context the question generator can access.

Useful measurements would compare generated patches with and without the gate, while tracking escaped defects, review time, reverted changes, test additions, and the rate at which explanations identify genuine issues. Teams should also inspect false confidence: a polished but incorrect explanation may be more dangerous than an explicit admission of uncertainty. Answers should be checked against later review findings rather than evaluated only for completeness or tone.

Using it

Start with pull requests or merge queues rather than a local pre-commit hook. Local hooks are easy to bypass, can slow unrelated work, and often run before the developer has enough context to explain a multi-file change. A server-side check provides a stable enforcement point while leaving room for a normal interactive review. It also makes the control visible to the team instead of turning it into an individual developer’s private ritual.

Scope the quiz to risk. Changes involving authentication, permissions, migrations, financial calculations, concurrency, external side effects, or prompt and policy handling deserve targeted questions. A documentation edit or mechanical rename should not incur the same cognitive cost. Risk-based scope keeps the mechanism from becoming background noise and leaves reviewers more attention for changes where a mistaken assumption can have broad consequences.

Generate questions from the diff and repository context, but keep the final set small. Ask about changed control flow, new assumptions, error paths, and tests that provide meaningful coverage. If an answer reveals uncertainty, the useful action is to inspect code, add a test, narrow the patch, or request another review—not to reward confident wording. The gate should create a path toward better evidence, not merely collect prose.

Treat answers as review evidence and retain them with the change when practical. Over time, compare explanations with later defects and refine question templates around recurring misses. The system becomes valuable when it teaches the team which assumptions generated code routinely hides, such as implicit ownership rules or behavior at service boundaries. Those patterns can then inform tests and review guidance as well as future questions.

Limitations and failure modes

A quiz cannot verify behavior that the reviewer does not understand. It may also produce automation theater: developers can memorize expected language, paste an agent-generated explanation, or answer questions without opening the relevant implementation. A required field is especially weak evidence when the process measures whether text exists rather than whether the text predicts actual behavior.

There is a risk of shifting responsibility onto individuals while leaving weak engineering controls untouched. A quiz should supplement tests, static analysis, dependency checks, observability, and domain review. It should never be used to justify shipping code that lacks reproducible validation. Human explanation is useful precisely because it can expose uncertainty; it should not become a substitute for controls that can independently check the system.

Question generation can fail in both directions. Generic questions create fatigue and meaningless approvals; overly specific questions may encode the generator’s own mistaken interpretation of the patch. If the hook asks about only the visible diff, it can miss configuration, database schema, runtime flags, and downstream consumers. Questions therefore need enough repository and operational context to identify affected contracts without pretending that the generated interpretation is authoritative.

Finally, a strict gate can encourage smaller or less transparent patches rather than safer ones. Measure bypasses and review latency, and provide an explicit path for escalation when the change is urgent or the question is wrong. A control that engineers routinely evade is not a control. The team should distinguish a genuine exception from a pattern of avoiding difficult review, then adjust the workflow rather than simply making the gate more punitive.

What to watch

The next useful step is to connect explanation gates to other verification signals. A system could compare a developer’s stated behavior with tests, trace data, type information, and production invariants, while preserving human authority over ambiguous cases. Such comparisons could point reviewers to contradictions—for instance, an explanation that claims a failure is impossible while existing tests or traces show that path occurring.

Watch whether these tools evolve from generic “explain your code” prompts into targeted checks for contracts and failure modes. That shift would make them less like an AI-themed checklist and more like an executable part of software assurance. The important architectural question is not whether an agent can generate another explanation, but whether the workflow helps connect human judgment to evidence about the change.

Also watch where the questions are placed in the delivery process and how they interact with other review signals. A prompt that appears too early may be based on an unstable patch; one that appears too late may expose problems only after the change is expensive to undo. The most useful design will make assumptions visible at the point where a developer can still investigate, test, narrow, or reject the generated change.

Your Agent Benchmark Pauses the World. Production Doesn't.

Vikas Srivastava — Sun, 18 Oct 2026 09:58:10 +0000

Your evals assume the world holds still

Almost every agent benchmark in wide use is turn-based. The harness hands the model an observation, the model emits an action, the harness applies the action, and then — crucially — nothing happens until the model produces its next token. The environment is a pure function of the interaction history. Time is logical, not physical.

That is a deliberate and usually correct simplification. It isolates reasoning from scheduling noise, it makes runs reproducible, and it lets you compare two checkpoints without worrying that one got a slower GPU. It is also a bad model of deployment.

In production, the world keeps moving while your agent thinks. A price changes between the quote and the order. A lease expires. A human answers the email you were drafting a reply to. Another replica of your own agent takes the lock you were about to take. A recent preprint, "The World Won't Wait: Evaluating and Teaching Agents Under a Live Clock," takes this mismatch as its starting point. The title names the problem precisely enough that the engineering consequences are worth spelling out on their own.

What wall-clock time actually breaks

Two distinct failures get conflated under "the agent is slow," and separating them is the first useful step.

Staleness. An observation is a claim about the world at some instant. Turn-based harnesses make it a claim about the world now, forever. Once you add a clock, every observation carries an implicit age, and the agent's context becomes a mix of fresh and expired beliefs with no visual difference between them. A log line, a search result, and a file listing taken ninety seconds apart look identical to a transformer. The model has no way to discount the stale ones unless you tell it.

Validity windows. Some actions are only meaningful within a time band. A session token, a signed upload URL, a pagination cursor, a reservation hold, a one-time code — all of these are affordances that exist at t and are gone at t + Δ. In a frozen environment, a failed tool call means the agent made a mistake. In a live environment, it might just mean the agent deliberated too long.

The concrete failure modes that show up once you stop freezing the world:

Read-modify-write races. The agent reads state, spends seconds reasoning, then writes back a value derived from a snapshot that no longer exists.
Duplicate side effects. A tool call times out. The agent retries. The first call had actually succeeded, and now the customer has two orders.
Expired affordances. The plan is right, but step four uses a handle that expired during step three.
Blocking serialization. The agent waits on a synchronous tool while an independent subtask could have progressed in parallel — the plan is fine, the schedule is not.
Deadline overrun. The answer is correct and arrives after the SLA it was supposed to satisfy. Under a turn-based scorer, this is a perfect score.

What a live-clock evaluation requires

Mechanically, there are three ways to build one, and they trade off differently.

Real time. Run the harness against actual concurrent processes and score on wall-clock. Highest fidelity, lowest reproducibility: your numbers now depend on your serving stack, your network, and your machine's load.

Virtual-but-live time. Introduce a clock abstraction, let the environment advance on its own simulated ticks, and give the agent a monotonic clock it can read through the tool layer. This is the interesting design point. If the agent can query time and always gets a frozen value, you have reverted to turn-based evaluation with extra steps. The clock must be observable to the agent but not controllable by it, exactly like environment state. Observations must carry timestamps and the agent must be able to act on them.

Recorded timelines with replay. Log real interactions with their timestamps, then replay them against a controllable clock for scoring. Cheap, reproducible, and it lets you sweep the agent's own latency without re-running the world.

Two scoring decisions follow. First, a live-clock eval cannot report a single number. It should report a latency-accuracy curve: what fraction of tasks complete correctly within a budget, at several budgets. Second, because the interleaving of agent deliberation and environment change is nondeterministic, you need seeded scheduling and a distribution over seeds rather than one run. Some of those seeds should be adversarial — a change landing just after the agent reads.

Teaching, not just measuring

Training under a live clock changes the objective, not just the test. Latency becomes part of the action space: how long to deliberate, when to re-check before committing, when to act on a stale-but-good-enough belief.

Concretely, that means putting a clock reading into the observation, charging wall-clock time into the reward rather than treating it as free, and being careful about credit assignment. The outcome of an action depends on the state at the moment the action landed, not the state at episode start. Standard advantage estimation over a trajectory silently assumes those are the same, and under a live clock they are not.

What to do before the benchmark exists

You do not need a research harness to get most of the benefit. Start here:

Timestamp every observation you put in context, and state freshness requirements in the tool description ("this cursor is valid for 60 seconds").
Prefer compare-and-swap and idempotency keys over blind writes in your tool implementations, so retries are safe by construction.
Make timeouts return a distinguishable unknown, not a generic failure, and teach the agent to re-check rather than assume failure.
Log wall-clock per step. The histogram usually has a long tail that dominates the total, and it is often a small number of tools.
Add one eval where the environment mutates on a timer independent of your agent, and measure whether the agent notices.

Open questions

Is live-clock performance a capability axis or a robustness axis? It plausibly needs to be reported separately, because a slower model with better staleness handling can beat a faster one on the same tasks. Reproducibility is unresolved: seeded nondeterminism may be the honest default, but the community has no convention for it yet. Transfer is unproven — latency distributions in production are heavy-tailed and machine-specific, so training on one clock may not survive another. And multi-agent settings turn the environment from a Markov decision process into a game, where the other players also own clocks.

DEV Community: Vikas Srivastava

When more agents produce less work

When more agents produce less work

How it works

What the evidence actually shows

Using it

Limitations and failure modes

What to watch

https://dev.to/vikassri/why-trillion-parameter-models-need-a-new-serving-architecture-6gk

Why Trillion-Parameter Models Need a New Serving Architecture

Why Trillion-Parameter Models Need a New Serving Architecture

The problem: model size has become a serving constraint

How it works

What the evidence actually shows

Using it

Limitations and failure modes

What to watch

SELFTEST Production Document Ingestion and Structure-Aware Chunking

What you'll be able to do

The problem

How it works

Central question

The ingestion contract

Parse first, chunk second

Structure-aware boundaries

Size is a budget, not a universal constant

Overlap is selective context duplication

Determinism and reindexing

Implementation

Install

Operational extensions

Verifying it

Performance and cost

What breaks

Security

In production

Limitations

Conclusion

Further reading

In this series

Try it yourself

Why Compute Is Difficult to Trade

The problem: compute is difficult to trade

How it works

What the evidence actually shows

Using it

Limitations and failure modes

What to watch

What Microsoft’s AI Diagnostic System Actually Shows

The concrete problem

How it works

What the evidence actually shows

Using it

Limitations and failure modes

What to watch

Fix, Not Just Resample

The problem: proof search wastes its best feedback

How it works

What the evidence actually shows

Using it

Limitations and failure modes

What to watch

When Training Data Makes False Facts Look True

The problem: a training corpus can amplify false facts

How it works

What the evidence actually shows

Using it

Limitations and failure modes

What to watch

The pipeline problem: false claims are easy to train on

The pipeline problem: false claims are easy to train on

How it works

What the evidence actually shows

Using it

Limitations and failure modes

What to watch

Vector Caching in Production AI Systems

The production problem: repeated work hides inside every request

How it works

What the evidence actually shows