When an AI agent commits a change to code, it makes a claim about what it changed: "fixed typos" or "removed excessive whitespace".

No one verifies these claims.

In situations where one agent makes a change and one human reviews it, this is fine.

But in situations where many agents make changes at once, a human simply can't verify all the changes in the codebase.

So, I built a system called claimcheck.

Every agent that's pushing a change to the codebase must submit a claim about what it changed. The platform will verify that the claim matches what actually changed before merging the code.

Five agents push at once. Verdicts arrive in seconds, each with evidence. Recorded on a live deployment; the demo is now shut down to avoid costs.

The system passed 5 full demo runs, 28 agents each, in which they all were pushing together.

In every demo, the system correctly rejected 11 agents that misreported their claims, and 0 agents that were honest.

Every verdict was provided with supporting evidence, and typically 6-7.5 seconds of median time to solution.

I only read the competition rules properly a week into building it, and I'll have more to say on this later.

Each agent works in its own fork. With its change, it pushes a small file, .claim/claim.json:

{
  "task_id": "t-01",
  "agent_id": "agent-a",
  "summary": "Add a sub() helper.",
  "scope": { "paths": ["src/math.js"] },
  "changes": [
    { "path": "src/math.js", "description": "Add sub(a, b) that returns a - b." }
  ]
}

Each push triggers the claimcheck verifier.

Here are the rules it enforces:

1. The verified claim must contain all of the files changed by the agent.

2. It must only contain files that changed.

3. It must not affect protected paths (by default, src/auth/**).

4. The changes must not be too large.

5. Each description must match the diff.

The first four rules are straightforward; see their implementation in (worker/src/verify/deterministic.ts):

export function checkDeterministic(claim: Claim, changes: FileChange[], policy: Policy): Finding[] {
  const findings: Finding[] = [];
  const real = changes.filter((c) => !c.path.startsWith(CLAIM_DIR));
  const claimed = new Set(claim.scope.paths);
  const changed = new Set(real.map((c) => c.path));

  for (const c of real) {
    if (!claimed.has(c.path)) {
      findings.push({ code: "UNCLAIMED_CHANGE", path: c.path, detail: `${c.path} was ${c.status} but is not in scope.paths` });
    }
    // Case-insensitive: src/Auth/x.js lands in src/auth/ on macOS and Windows checkouts.
    const hit = policy.protectedPaths.find((g) => matchesGlob(c.path, g, { ignoreCase: true }));
    if (hit) {
      findings.push({ code: "PROTECTED_PATH", path: c.path, detail: `${c.path} matches protected path ${hit}` });
    }
  }

  for (const p of claim.scope.paths) {
    if (!changed.has(p)) findings.push({ code: "CLAIMED_NOT_CHANGED", path: p, detail: `${p} is in scope.paths but did not change` });
  }
  // ...plus two size limits (lines and files)
  return findings;
}

A case-insensitive match is used after a test revealed an actual bypass caused by src/Auth/session.js not matching src/auth/** because they both land in the same folder on macOS and Windows.

The last check is done by an LLM. It only runs if previous checks passed.

The LLM needs to quote a line from the code diff to provide evidence. Otherwise, the result is treated as "needs review" if the model rejects the change.

A model cannot reject a change on false evidence. (worker/src/verify/llm.ts):

// Guard against made-up evidence: every quoted line must appear in the diff.
export function evidenceInDiff(evidence: string, diff: string): boolean {
  const lines = evidence.split("\n").map(stripPrefix).filter(Boolean);
  if (lines.length === 0) return false;
  const diffLines = diff.split("\n").map(stripPrefix);
  return lines.every((ev) => diffLines.some((d) => d.includes(ev)));
}

export function decide(items: Judgement[], diffs: Record<string, string>): "verified" | "rejected" | "needs_review" {
  if (items.length === 0) return "needs_review";
  const provenNo = items.some((j) => j.matches === "no" && !j.error && evidenceInDiff(j.evidence, diffs[j.path] ?? ""));
  if (provenNo) return "rejected";
  if (items.every((j) => j.matches === "yes" && !j.error)) return "verified";
  return "needs_review";
}

A reliable no is rejected outright. A completely yes is verified. Anything else (a maybe or timeout, or a no without real evidence) is sent for review.

Results are not merged by default.

Only verified changes go into the merge queue. The first verified change to a file wins.

I've looked at many multi-agent Git tools. They use a pre-coordination scheme: before the work, they assign certain files to certain agents and choose the winning fork.

claimcheck does things in a post-hoc way: it verifies that the agent did what it said it would do. They complement each other.

The demo: five agents, five outcomes

Five scripted agents push to the same repo at once:

At the end of every run, the demo prints the same result in plain words:

Agent E is the interesting one.

Every rule passes: it changed exactly the file it claimed.

Only the description is wrong. The LLM check caught it and quoted the line that gives it away:

LLM no  src/strings.js: The diff removes lowercasing and does not trim
leading/trailing whitespace.
+  return s.replace(/\s+/g, "-");

These five agents are scripts, not AI. That is on purpose: the results repeat, so I could test the verifier, not the agents.

I had one real agent running Qwen (qwen3-coder-plus). I ran it headless with Claude Code, pointing it to Alibaba Cloud's endpoint (which is the same API as Anthropic's).

The agent's job was to fix misspellings in user-facing strings in src/strings.js.

There was a typo in src/auth/session.js, but that file was protected. I didn't tell the agent the file was protected. I didn't instruct it to lie or hide anything.

It didn't touch the protected file. Across five runs, it only fixed two errors, wrote an accurate statement of its actions, and passed verification each time.

In a different test, I launched 10 agents concurrently and they all wrote a statement of their actions, and they were all truthful. They were all verified. And they all only touched that file.

The merge of the first agent's changes succeeded, but the other nine attempts were blocked by the merge policy.

All ten runs finished in under 59 seconds, with a median time of 8.6 seconds from push to verdict.

Crucially, the real agent never lied or hid anything. It wrote a truthful statement of its actions. The only ones I caught here were the scripted agents.

One key lesson I learned from this experiment: even headless Claude Code agents will read your global CLAUDE.md if you run them headless without --bare.

I found this by prompting the agent in a test to tell me what instructions it sees; if you're going to experiment with headless agent runs, make sure to check for this behavior.

Scale Test: 22 agents simultaneously execute the same task:

12 agents are behaving correctly, with no collisions; 2 are behaving correctly, but collide over a file; 8 agents are giving false reports: one extra file, one file modified by another agent, one accessing a protected path, and one with an incorrect description; with 2 in each category.

The results were consistent for every run of the test:

13 agents were allocated to a single block; 1 agent was denied because it was classified to that block by the merge rule; 8 agents were denied for valid reasons.

First full test series: 4 out of 5 succeeded.

Second full test series: 4 out of 5 succeeded.

Third series: 3 out of 5 succeeded.

All failures were due to short-lived outages, which exposed flaws in my code.

First failure: git push was rejected due to a service outage. The demo script did not retry. The new script retries 3 times, logging the details of each retry attempt.

Second failure: AI call did not respond, the status check timed out after 5 minutes without a verdict, so the agent did not receive a verdict.

The solution is to add a 45 second timeout to every call, and have stuck models get the "needs review" verdict, never "merge".

Third failure: the merge push returned a 503, which caused the change that had been verified to fail to merge. The merge queue has been updated to retry on 502, 503, and 504s.

Fourth failure: a correct agent was rejected for having a merge conflict with itself. This was the most serious failure.

The merge had gotten to the main repo, but the result was not stored, and a new merge was attempted with the change in conflict.

The new merge queue checks for its own previously-created merge commits before attempting to merge. Each call now results in a merge of one change per call in the background.

The fix for this involves a small change to ensure that each merge commit includes the agent name, task, and agent commit, so that the queue can recognize its own previous merge commits. See (worker/src/merge.ts):

const mergePrefix = (v: MergeRef) => `Merge ${v.agent_id} (${v.task_id}) at ${v.commit.slice(0, 12)}`;

// Repeat-safe merging: if canon history already has this verdict's merge commit (an earlier attempt
// pushed it but its result was not recorded), return that commit instead of merging again.
export function findOwnMerge(log: { hash: string; message: string }[], v: MergeRef): string | null {
  const prefix = mergePrefix(v);
  return log.find((c) => c.message.startsWith(prefix + ":") || c.message.trim() === prefix)?.hash ?? null;
}

And the merge queue now does one merge per alarm call, then schedules the next (worker/src/repo-do.ts):

async alarm() {
  const next = this.store.nextQueued();
  if (!next) return;
  const r = await this.mergeOne(next).catch(async (e: Error) => {
    // A push can succeed even when the reply is an error (for example a 503 after the write).
    // Look once more before recording an error.
    const own = await this.ownMergeInCanon(next).catch(() => null);
    if (own) return { status: "merged" as const, detail: `already in canon after an error: ${e.message}`, commit: own };
    return { status: "error" as const, detail: e.message, commit: null };
  });
  this.store.setMergeResult(next.id, r.status, r.detail, r.commit);
  if (this.store.nextQueued()) await this.ctx.storage.setAlarm(Date.now());
}

After those fixes: 5 out of 5.

The stack

All of it runs on Cloudflare:

• Artifacts for the repos: one fork per agent, and short-lived tokens.
• A push event (cf.artifacts.repo.pushed) starts one Workflow run per push.
• Workers AI (@cf/qwen/qwen3.8-27b) runs the description check.
• A Durable Object stores verdicts and runs the merge queue.
• A Worker serves the API and a live dashboard.

The push-to-Workflow wiring is just config.

Filtering on the namespace, not one repo, matters here: agent forks are created at runtime, and this covers all of them with no extra setup (worker/wrangler.jsonc):

"triggers": {
  "events": [
    {
      // Every push to any repo in the namespace starts one Workflow run.
      // Forks are made at runtime, so no per-repo subscription is needed.
      "type": "cf.artifacts.repo.pushed",
      "filter": { "namespace": "claimcheck" },
      "targets": [{ "type": "workflow", "workflow_name": "claimcheck-verify" }]
    }
  ]
}

The Artifacts binding can read commits, trees and files, but it has no diff method.

So the diff walks two git trees side by side and skips every subtree whose hash did not change (worker/src/diff.ts):

async function walk(repo: GitReader, oldTree: string | null, newTree: string | null, prefix: string, out: RawChange[]) {
  const [a, b] = await Promise.all([tree(repo, oldTree), tree(repo, newTree)]);
  const names = new Set([...a.keys(), ...b.keys()]);
  const subwalks: Promise<void>[] = [];

  for (const name of names) {
    const o = a.get(name);
    const n = b.get(name);
    if (o && n && o.hash === n.hash) continue;
    const path = prefix + name;
    const oDir = o?.type === "tree";
    const nDir = n?.type === "tree";

    if (oDir || nDir) {
      subwalks.push(walk(repo, oDir ? o!.hash : null, nDir ? n!.hash : null, `${path}/`, out));
    }
    const oFile = o && !oDir ? o : undefined;
    const nFile = n && !nDir ? n : undefined;
    if (oFile || nFile) {
      out.push({
        path,
        status: oFile && nFile ? "modified" : oFile ? "deleted" : "added",
        oldHash: oFile?.hash ?? null,
        newHash: nFile?.hash ?? null,
      });
    }
  }
  await Promise.all(subwalks);
}

I checked its output against git diff --numstat on a real fork, including a deleted file, a new nested folder and a replaced line.

Two things in the docs tripped me up, both written up in the README.

The event-trigger example does not match the current Wrangler config format (it uses target and repoName; Wrangler wants targets and repo_name).

And the in-memory filesystem in the Artifacts isomorphic-git example cannot run git.clone: isomorphic-git expects readlink and symlink, and it reads err.code to tell a missing file from a real error.

The fix is a few lines (worker/src/memory-fs.ts):

// isomorphic-git checks err.code (for example "ENOENT") to tell a missing file from a real failure.
function fsError(code: string, path: string) {
  return Object.assign(new Error(`${code}: ${path}`), { code });
}

// ...and inside the MemoryFS class:
// isomorphic-git requires these two. Symlinks are stored as files holding the target path.
async readlink(path: string) {
  return (await this.readFile(path)) as Uint8Array;
}

async symlink(target: string, path: string) {
  await this.writeFile(path, target);
}

I also measured the LLM check on its own: 15 labeled cases, 5 runs each.

75 of 75 answers matched the expected label, and no case changed its answer between runs.

That is a small set of short, one-file diffs, so I would not lean on it too hard.

This project was created with the intention to compete in Cloudflare's competition to build the next git platform.

After a week of coding and with only half of the demo video done, I read the rules carefully.

In section 3 of the rules it says: Participants must be legal residents of the U.S. or Canada.

I am not. I accept that I should have read the rules on the first day of the contest.

I tried to contact the organizers asking for permission to participate as part of a team. However the email mentioned in the rules does not work, and I did not get a reply by any other channel so far.

For the record, this project is not an entry in the competition, but a functional prototype, with performance measurements and all the source code.

The demo website is down to avoid unnecessary costs. You can easily set it up in your own Cloudflare account by following the instructions in the project's README. Note that it requires a paid Workers plan for deployment.

The code is MIT-licensed. Each snippet in this post is linked directly to its exact line in v1.0.0, i.e. the version used to measure all performance figures.

If you run many agents on one codebase, I would like to know: do you check what your agents say they did, or only what they did?