DEV Community: xbill

Profile Sprawl: Solving a Problem You Didn't Know You Had

xbill — Sun, 01 Nov 2026 14:35:31 +0000

This article provides a step by step guide to keeping a public web presence consistent across many sites. An agent skill for Claude Code is built that treats a bio as source code: one file holds the facts, one table records where it has landed, and scripts read the live sites to confirm it.

https://github.com/xbill9/profile-sync

Every profile you own is a copy of your bio, and copies drift. The fix is the one software already uses: a single source, a status table and a check that reads production.

What Is Profile Sprawl?

A working developer collects profiles. LinkedIn, GitHub, Medium, dev.to, Substack, a certification wallet, a community portal, a program dashboard. Each one was accurate on the day you filled it in.

Profile sprawl is what happens next. A new role or credential goes on the site you happened to have open, and the rest keep the old text. No site reports an error, because each one is internally fine. The inconsistency exists only across sites, and nobody reads them side by side except a recruiter, a conference organizer or a reviewer.

How Bad Could It Be?

One audit of ten profiles belonging to one person, read on 2026-10-31, the day after a new certification was issued:

Site	Certifications	Years of experience
LinkedIn headline	10
LinkedIn About	9	over 30+
Medium	9	20+
Google Developers profile	9	20+
Credly	9	20+
Advocu	9

The same LinkedIn profile disagreed with itself. The headline said 10 and the About said 9, and each line looked fine alone.

The other four sites had a different problem. AWS Builder Center listed 0 credentials, dev.to held a tagline, GitHub held an unexplained string, and Substack was empty apart from the platform's default text.

At This Point You Should Have

A list of your public profiles and a browser where you are logged in to them
Python 3, with no extra packages
Claude Code with the Chrome extension, or the patience to run the same steps by hand
A git repository, which can be public as long as it holds public text only

Step 1 — Write One Source File

bio.md holds everything a site could need, in plain Markdown with no site-specific syntax. Facts are lists. Text comes in lengths, written from those facts.

---
version: 2026-10-31
---

# Bio

## Headline
## Tagline
## Short
## Summary
## Long
## Roles
## Community
## Certifications
## Education
## Links

Three lengths cover every form measured here: Short at 133 characters, Summary at 561 and Long at 735. A site gets its own text only when a limit forces it, and even then the shorter text becomes a section of the file. Per-site copies are how sprawl starts.

The rule that keeps this working: a change starts in bio.md, never on a site. If a site was edited directly, that change is copied back first.

Step 2 — Track Where It Has Landed

tracker.md is one table with one row per site. Status takes one of four values.

Status	Meaning
Current	Synced version matches `bio.md` and the live profile was checked
Behind	The site has an older version
Unverified	Nobody has compared this site to `bio.md` yet
Blocked	Something outside your control stops the update, explained in Notes

The Text used column records which sections a site takes. A change to Long reaches only the sites that use Long, and every other row moves to the new version with no edit.

Step 3 — Make the Computer Do the Counting

preflight.py compares the two files and exits 1 on any disagreement. It counts the Certifications list and compares that count with every claim in the text, so a number in a headline can never drift from the list it summarizes.

$ python3 .claude/skills/profile-sync/scripts/preflight.py
bio.md 2026-10-31, 10 sites: {'Behind': 0, 'Blocked': 1, 'Current': 9, 'Unverified': 0}
preflight OK

It also fails on a text longer than a site's form allows, a Current row with no check date, and a status outside the four values. The repository is public, so it scans both files for email addresses and phone numbers too.

Arithmetic belongs in code. A model or a person can miscount a list. A script compares 10 with 10.

Step 4 — Read the Live Sites

A tracker row is a claim. check-live.py fetches what a visitor gets from each site with a public source and compares it with bio.md.

$ python3 .claude/skills/profile-sync/scripts/check-live.py
  SKIP  LinkedIn: browser check
  OK    GitHub: live text matches Short
  SKIP  Medium: browser check
  OK    dev.to: live text matches Short
  OK    Substack: live text matches Short
  SKIP  AWS Builder Center: browser check
  SKIP  Google Developers profile: browser check
  SKIP  Google Cloud Skills Boost: browser check
  SKIP  Credly: browser check
  SKIP  Advocu: browser check
check-live OK

A site marked Current that differs fails the run. A check that can only pass proves nothing, so the same script was run against a stale value to confirm it exits 1.

Step 5 — Measure Every Form

Each site has its own field, limit and save button. These are read from the forms themselves, and they live in one reference file so the next run starts with them.

Site	Field	Limit
Medium	Short bio	160
dev.to	Bio	200
AWS Builder Center	About	255
Credly	Personal Summary	600
Google Developers profile	Bio	750
LinkedIn	About	2,600

A property of one site is evidence about that site only. The Long text at 735 characters fits 2 of these fields and fails the other 4, so the limits go in biolib.py and preflight.py enforces them before any browser opens.

🔎 Tip: Reading and Writing Are Different Routes

dev.to has a good API, and a key reads the profile with no trouble. Profile fields are read-only there: PUT and PATCH on /api/users/me return 404, so the bio is saved through the settings page. Test both directions before planning a rollout around an API.

Step 6 — Compare Before Saving

Browser automation can drop a character while reporting success, and a page can show a success message for the wrong text. So each save follows the same order:

Scroll to the field and confirm it has focus.
Select the old text and type the new text.
Compare the field with the exact payload from bio.md.
Read the position of the save button after scrolling.
Save, reload and compare the live text.

payload.py supplies step 3. It prints the text for a site, or a small JavaScript function that compares a field with it. The text travels base64-encoded, which survives quotes, emoji and line breaks.

$ python3 .claude/skills/profile-sync/scripts/payload.py Credly > /dev/null
Summary: 561 characters

🔎 Tip: Decline the Announcement

Adding a certification on LinkedIn opens a prompt to post about it to your network. Updating a profile and announcing to it are separate decisions. The skill clicks Skip unless an announcement was requested.

Step 7 — Package It as a Skill

The steps above become an agent skill, in the shape of publishing-kit:

.claude/skills/profile-sync/
  SKILL.md               the workflow
  references/sites.md    route, field, limit and save button per site
  scripts/biolib.py      reads bio.md and tracker.md
  scripts/preflight.py   the two files agree
  scripts/check-live.py  the live sites agree
  scripts/payload.py     the exact text for a field
tests/test_preflight.py  one case per failure, with a passing case beside it

The tests are the guard on the guards. Each failure gets a case that must fail and a case that must pass.

$ python3 -m unittest discover -s tests
Ran 18 tests in 0.004s

OK

The passing cases carry weight too. One holds a date, so a phone number pattern that matches 2026-10-31 fails the suite before it can fail every row in the tracker.

Step 8 — Roll Out One Change

The certification that started this run shows the whole loop.

# 1. Edit bio.md, set version to today
# 2. Confirm the files agree
python3 .claude/skills/profile-sync/scripts/preflight.py
# 3. In Claude Code, run /profile-sync to roll it out
# 4. Confirm the live sites agree
python3 .claude/skills/profile-sync/scripts/check-live.py

The skill works down the tracker, saves each field in Chrome in the logged-in session, reloads each page, records the check date, then commits and pushes. Credentials stay in the browser, so the repository holds public text only.

Before starting, confirm the fact at its source. Here that source was the Credly badge, issued 2026-10-30, which settled whether 10 or 9 was correct.

Compare and Contrast

	By hand	A notes file	profile-sync
Source of truth	Whichever site is newest	A notes file	`bio.md` in git
Knows which sites are behind	No	If you update it	🥇 Yes
Catches a wrong count	No	No	🥇 Yes
Reads the live site	When you look	When you look	🥇 3 sites by script, the rest by browser

So, Which One?

Start with Step 1 and Step 2, even with no automation. The first side-by-side read will show you what your profiles have been saying.

Add the skill once you hold more profiles than you can check in one sitting. That number is small, and a new credential is a good day to start.

Summary

The goal of this article was to keep one person's public profiles consistent across many sites. The key to the solution was treating a bio as source code, with one source file, a status table and a check that reads the live sites. The profile-sync results were:

🟢 9 of the 10 sites are Current, each confirmed on its live page
🟢 The certification count is computed from a list, so the headline and the About can no longer disagree
🟢 18 tests keep each failure beside a passing case
❌ 5 sites said 9 certifications the day after a tenth was issued, and 3 said 20+ years
⚠️ Google Skills stays Blocked: it has no bio field and its public profile page returns an error
⚠️ check-live.py reads 3 sites and only the Short text, so the other sites still need a browser check

Scope: one person's ten profiles on 2026-10-31, in one Chrome session. Form limits were read from each site's own form on that date and can change. Two edits on the same day share one version, so version equality alone does not show that a site has the latest text, and the live comparison covers that gap.

The strategy for keeping a public profile consistent was validated with an incremental step by step approach.

References

Profile Sprawl: Solving a Problem You Didn't Know You Had

xbill — Sun, 01 Nov 2026 14:35:26 +0000

https://github.com/xbill9/profile-sync

Every profile you own is a copy of your bio, and copies drift. The fix is the one software already uses: a single source, a status table and a check that reads production.

What Is Profile Sprawl?

A working developer collects profiles. LinkedIn, GitHub, Medium, dev.to, Substack, a certification wallet, a community portal, a program dashboard. Each one was accurate on the day you filled it in.

How Bad Could It Be?

One audit of ten profiles belonging to one person, read on 2026-10-31, the day after a new certification was issued:

Site	Certifications	Years of experience
LinkedIn headline	10
LinkedIn About	9	over 30+
Medium	9	20+
Google Developers profile	9	20+
Credly	9	20+
Advocu	9

The same LinkedIn profile disagreed with itself. The headline said 10 and the About said 9, and each line looked fine alone.

At This Point You Should Have

A list of your public profiles and a browser where you are logged in to them
Python 3, with no extra packages
Claude Code with the Chrome extension, or the patience to run the same steps by hand
A git repository, which can be public as long as it holds public text only

Step 1 — Write One Source File

bio.md holds everything a site could need, in plain Markdown with no site-specific syntax. Facts are lists. Text comes in lengths, written from those facts.

---
version: 2026-10-31
---

# Bio

## Headline
## Tagline
## Short
## Summary
## Long
## Roles
## Community
## Certifications
## Education
## Links

The rule that keeps this working: a change starts in bio.md, never on a site. If a site was edited directly, that change is copied back first.

Step 2 — Track Where It Has Landed

tracker.md is one table with one row per site. Status takes one of four values.

Status	Meaning
Current	Synced version matches `bio.md` and the live profile was checked
Behind	The site has an older version
Unverified	Nobody has compared this site to `bio.md` yet
Blocked	Something outside your control stops the update, explained in Notes

The Text used column records which sections a site takes. A change to Long reaches only the sites that use Long, and every other row moves to the new version with no edit.

Step 3 — Make the Computer Do the Counting

$ python3 .claude/skills/profile-sync/scripts/preflight.py
bio.md 2026-10-31, 10 sites: {'Behind': 0, 'Blocked': 1, 'Current': 9, 'Unverified': 0}
preflight OK

Arithmetic belongs in code. A model or a person can miscount a list. A script compares 10 with 10.

Step 4 — Read the Live Sites

A tracker row is a claim. check-live.py fetches what a visitor gets from each site with a public source and compares it with bio.md.

$ python3 .claude/skills/profile-sync/scripts/check-live.py
  SKIP  LinkedIn: browser check
  OK    GitHub: live text matches Short
  SKIP  Medium: browser check
  OK    dev.to: live text matches Short
  OK    Substack: live text matches Short
  SKIP  AWS Builder Center: browser check
  SKIP  Google Developers profile: browser check
  SKIP  Google Cloud Skills Boost: browser check
  SKIP  Credly: browser check
  SKIP  Advocu: browser check
check-live OK

A site marked Current that differs fails the run. A check that can only pass proves nothing, so the same script was run against a stale value to confirm it exits 1.

Step 5 — Measure Every Form

Each site has its own field, limit and save button. These are read from the forms themselves, and they live in one reference file so the next run starts with them.

Site	Field	Limit
Medium	Short bio	160
dev.to	Bio	200
AWS Builder Center	About	255
Credly	Personal Summary	600
Google Developers profile	Bio	750
LinkedIn	About	2,600

🔎 Tip: Reading and Writing Are Different Routes

Step 6 — Compare Before Saving

Browser automation can drop a character while reporting success, and a page can show a success message for the wrong text. So each save follows the same order:

Scroll to the field and confirm it has focus.
Select the old text and type the new text.
Compare the field with the exact payload from bio.md.
Read the position of the save button after scrolling.
Save, reload and compare the live text.

$ python3 .claude/skills/profile-sync/scripts/payload.py Credly > /dev/null
Summary: 561 characters

🔎 Tip: Decline the Announcement

Step 7 — Package It as a Skill

The steps above become an agent skill, in the shape of publishing-kit:

.claude/skills/profile-sync/
  SKILL.md               the workflow
  references/sites.md    route, field, limit and save button per site
  scripts/biolib.py      reads bio.md and tracker.md
  scripts/preflight.py   the two files agree
  scripts/check-live.py  the live sites agree
  scripts/payload.py     the exact text for a field
tests/test_preflight.py  one case per failure, with a passing case beside it

The tests are the guard on the guards. Each failure gets a case that must fail and a case that must pass.

$ python3 -m unittest discover -s tests
Ran 18 tests in 0.004s

OK

The passing cases carry weight too. One holds a date, so a phone number pattern that matches 2026-10-31 fails the suite before it can fail every row in the tracker.

Step 8 — Roll Out One Change

The certification that started this run shows the whole loop.

# 1. Edit bio.md, set version to today
# 2. Confirm the files agree
python3 .claude/skills/profile-sync/scripts/preflight.py
# 3. In Claude Code, run /profile-sync to roll it out
# 4. Confirm the live sites agree
python3 .claude/skills/profile-sync/scripts/check-live.py

Before starting, confirm the fact at its source. Here that source was the Credly badge, issued 2026-10-30, which settled whether 10 or 9 was correct.

Compare and Contrast

	By hand	A notes file	profile-sync
Source of truth	Whichever site is newest	A notes file	`bio.md` in git
Knows which sites are behind	No	If you update it	🥇 Yes
Catches a wrong count	No	No	🥇 Yes
Reads the live site	When you look	When you look	🥇 3 sites by script, the rest by browser

So, Which One?

Start with Step 1 and Step 2, even with no automation. The first side-by-side read will show you what your profiles have been saying.

Add the skill once you hold more profiles than you can check in one sitting. That number is small, and a new credential is a good day to start.

Summary

🟢 9 of the 10 sites are Current, each confirmed on its live page
🟢 The certification count is computed from a list, so the headline and the About can no longer disagree
🟢 18 tests keep each failure beside a passing case
❌ 5 sites said 9 certifications the day after a tenth was issued, and 3 said 20+ years
⚠️ Google Skills stays Blocked: it has no bio field and its public profile page returns an error
⚠️ check-live.py reads 3 sites and only the Short text, so the other sites still need a browser check

The strategy for keeping a public profile consistent was validated with an incremental step by step approach.

References

Agent Skills Drift: A Feedback Loop for Keeping One True

xbill — Sat, 31 Oct 2026 23:02:26 +0000

This article provides a step by step guide to finding drift in an agent skill and closing it with a repeatable feedback loop. The working example is publishing-kit, an agent skill for Claude Code, Codex and Antigravity that publishes one markdown article to five destinations.

https://github.com/xbill9/publishing-kit

A skill is a set of claims about a world that keeps moving. Every claim has a shelf life, so maintenance is a loop you run on a schedule.

What Is Skill Drift?

A skill is instructions plus scripts, and both describe something outside the skill: an editor, an API, a file format, or another script.

Drift is the distance between what the skill says and what is true today. It grows without any error message, because the skill keeps loading and its checks keep passing.

Why Does a Green Check Prove So Little?

The kit runs a pre-flight against its own article. On the day of this audit that run exited 0:

$ python3 skills/publishing/scripts/preflight.py articles/publishing-kit-skill/devto-publishing-kit.md --live
all checks passed

The same day, a code review of the scripts returned 20 findings and running every generator by hand added 2 more. One of those unpublished live articles.

The pre-flight walks a happy path. An article that never contains a ~~~ block can never show that 8 files mishandle one.

Step 1 — Measure Install Drift

Hosts install a skill as a snapshot, so the copy a session loads can lag the repository by weeks. Compare the two:

$ grep '"version"' .claude-plugin/plugin.json
  "version": "0.32.0",
$ grep -A4 'publishing@publishing-kit' ~/.claude/plugins/installed_plugins.json | grep version
        "version": "0.28.0",
$ git rev-list --count a49983f..dc4f31f
89

Every fix in those 89 commits existed in the repository and in no session. The whole Substack destination was among them.

Step 2 — Measure Version Drift

A version string identifies a release. It says nothing about whether the content changed, and an updater that compares strings inherits that blind spot.

$ git rev-list --count 631cf11..dc4f31f
29
$ git diff --stat 631cf11 dc4f31f -- skills .claude-plugin | tail -1
 16 files changed, 549 insertions(+), 26 deletions(-)

Those 29 commits changed behavior under an unchanged version. Bump the version in the same session as the change, so the update has something to see.

Step 3 — Find Warnings That Should Be Guards

A warning in a document depends on the next reader. The kit's notes recorded that --update pushes the source's published: false and takes a live article down. The script still did it and printed updated.

$ python3 -m unittest tests.test_regressions.DevtoUpdate
.....
----------------------------------------------------------------------
Ran 5 tests in 0.002s

OK

The script now reads the listing first, keeps a live article live, and exits non-zero if the article leaves the published listing.

Search the skill for words like "remember", "always" and "make sure". Each one is a candidate for a guard inside the script.

Step 4 — Test the Checks Themselves

A check drifts when its logic stops matching what its name promises. The fact checker joined every number in an evidence file into one string of digits:

"".join(["12", "3.4"])     # "123.4", so a claim of 123.4 ms traced

Comparing whole numbers fixed that and failed the kit's own article on the next run. Its 320px had been traced to a file with no 320 in it.

Give every check an input it must reject, and run that input. A check that has never failed tells you nothing about what it can catch.

Step 5 — Collapse Duplicated Logic

Copies drift apart. At audit time 9 files carried their own logic for a code fence. One handled ~~~, and the other 8 treated it as prose.

One script therefore saw a code block where the next saw paragraphs, and a fix in one file reached none of the others.

One function now classifies fences, and the repository's rules forbid a second.

Step 6 — Date Every Claim About a Destination

Destinations change without notice, so an undated claim about one has no expiry. The kit labels each behavioral claim with the day it was taken:

MEASURED 2026-09-01: PUT {"article": {"published": true}} returns 200 and changes nothing.

The skill carries 143 of those labels. A date tells a reader how far to trust the claim and tells a reviewer where to look first.

When a claim reverses, keep it visible and write a Correction beside it with the new measurement. A quiet edit removes the record that the old belief was ever held, and the next session can repeat it.

The Feedback Loop

Six kinds of drift, one loop. Each pass turns one surprise into something the next session cannot repeat.

  ┌─► 1 OBSERVE   a real run, a failed publish, a review, a user correction
  │       │
  │   2 MEASURE   reproduce it and archive the output, dated
  │       │
  │   3 ENCODE    guard in the script, test with the failing input,
  │       │       Correction in the reference, one source of truth
  │       │
  │   4 VERIFY    the test fails on the old code and passes on the new
  │       │
  │   5 RELEASE   bump the version, update the install, compare served bytes
  │       │
  └───────┘

Step 3 sets the order of preference. Code beats a test, a test beats a document, and a document beats memory, because each one depends less on someone paying attention.

Step 4 keeps the loop true. The regression tests here failed 39 of 44 against the scripts as they stood, and all 44 pass now. A test that passes on broken code proves nothing.

Step 5 is the step most often skipped, and skipping it produced the 89 commit gap from Step 1.

Compare and Contrast

Drift	Symptom	Detect with	Close with
Install	Sessions run old code	Installed version against the repo	Update, then compare content
Version	Updater sees no change	Commits since the last bump	Bump with the change
Document	A warning nobody enforces	Search for "remember" and "always"	A guard in the script
Check	Green over a real defect	An input the check must reject	A regression test
Duplication	Copies disagree	Search for the repeated test	One shared function
Destination	A claim goes stale	Oldest dated claims first	Re-measure, write a Correction

How Often Should the Loop Run?

Run the loop on three triggers:

🟢 Every failed or awkward run. A retry is the cheapest signal available, so write it down while it is fresh.
🟢 Every release. Run the tests, run the live pre-flight, bump the version and update the install.
🟢 On a calendar. A review reads the code cold and catches what a passing run cannot. It also finds claims that no recent run touched.

Give the calendar review to a reader with no stake in the code. Hand it the code and withhold your conclusions, so it can disagree.

So, Which One First?

Start with install drift. It takes one command, and until it is closed every other fix stays out of reach of your sessions.

Then add one rejecting input to the check you trust most. If that check accepts bad input, you have learned what your green runs are worth.

Summary

The goal of this article was to find and close drift in an agent skill. The key to the solution was treating every claim as perishable and every surprise as input to a loop. The publishing-kit results were:

🟢 22 defects surfaced on a day when the pre-flight passed
🟢 44 regression tests now keep the failing inputs, each beside a passing case
❌ The installed skill was 89 commits behind the repository
⚠️ The --update guard is tested against a fake API, and no live article has gone through it yet

Scope: one skill, one repository and one audit on 2026-10-31. Counts come from that repository's git history, and the defects come from one code review and one manual run, so they are a lower bound on what remains.

With this approach a skill stays true for as long as someone keeps running the loop. The full source and every measurement are in the repository.

References

Agent Skills Drift: A Feedback Loop for Keeping One True

xbill — Sat, 31 Oct 2026 23:02:02 +0000

https://github.com/xbill9/publishing-kit

A skill is a set of claims about a world that keeps moving. Every claim has a shelf life, so maintenance is a loop you run on a schedule.

What Is Skill Drift?

A skill is instructions plus scripts, and both describe something outside the skill: an editor, an API, a file format, or another script.

Drift is the distance between what the skill says and what is true today. It grows without any error message, because the skill keeps loading and its checks keep passing.

Why Does a Green Check Prove So Little?

The kit runs a pre-flight against its own article. On the day of this audit that run exited 0:

$ python3 skills/publishing/scripts/preflight.py articles/publishing-kit-skill/devto-publishing-kit.md --live
all checks passed

The same day, a code review of the scripts returned 20 findings and running every generator by hand added 2 more. One of those unpublished live articles.

The pre-flight walks a happy path. An article that never contains a ~~~ block can never show that 8 files mishandle one.

Step 1 — Measure Install Drift

Hosts install a skill as a snapshot, so the copy a session loads can lag the repository by weeks. Compare the two:

$ grep '"version"' .claude-plugin/plugin.json
  "version": "0.32.0",
$ grep -A4 'publishing@publishing-kit' ~/.claude/plugins/installed_plugins.json | grep version
        "version": "0.28.0",
$ git rev-list --count a49983f..dc4f31f
89

Every fix in those 89 commits existed in the repository and in no session. The whole Substack destination was among them.

Step 2 — Measure Version Drift

A version string identifies a release. It says nothing about whether the content changed, and an updater that compares strings inherits that blind spot.

$ git rev-list --count 631cf11..dc4f31f
29
$ git diff --stat 631cf11 dc4f31f -- skills .claude-plugin | tail -1
 16 files changed, 549 insertions(+), 26 deletions(-)

Those 29 commits changed behavior under an unchanged version. Bump the version in the same session as the change, so the update has something to see.

Step 3 — Find Warnings That Should Be Guards

$ python3 -m unittest tests.test_regressions.DevtoUpdate
.....
----------------------------------------------------------------------
Ran 5 tests in 0.002s

OK

The script now reads the listing first, keeps a live article live, and exits non-zero if the article leaves the published listing.

Search the skill for words like "remember", "always" and "make sure". Each one is a candidate for a guard inside the script.

Step 4 — Test the Checks Themselves

A check drifts when its logic stops matching what its name promises. The fact checker joined every number in an evidence file into one string of digits:

"".join(["12", "3.4"])     # "123.4", so a claim of 123.4 ms traced

Comparing whole numbers fixed that and failed the kit's own article on the next run. Its 320px had been traced to a file with no 320 in it.

Give every check an input it must reject, and run that input. A check that has never failed tells you nothing about what it can catch.

Step 5 — Collapse Duplicated Logic

Copies drift apart. At audit time 9 files carried their own logic for a code fence. One handled ~~~, and the other 8 treated it as prose.

One script therefore saw a code block where the next saw paragraphs, and a fix in one file reached none of the others.

One function now classifies fences, and the repository's rules forbid a second.

Step 6 — Date Every Claim About a Destination

Destinations change without notice, so an undated claim about one has no expiry. The kit labels each behavioral claim with the day it was taken:

MEASURED 2026-09-01: PUT {"article": {"published": true}} returns 200 and changes nothing.

The skill carries 143 of those labels. A date tells a reader how far to trust the claim and tells a reviewer where to look first.

The Feedback Loop

Six kinds of drift, one loop. Each pass turns one surprise into something the next session cannot repeat.

  ┌─► 1 OBSERVE   a real run, a failed publish, a review, a user correction
  │       │
  │   2 MEASURE   reproduce it and archive the output, dated
  │       │
  │   3 ENCODE    guard in the script, test with the failing input,
  │       │       Correction in the reference, one source of truth
  │       │
  │   4 VERIFY    the test fails on the old code and passes on the new
  │       │
  │   5 RELEASE   bump the version, update the install, compare served bytes
  │       │
  └───────┘

Step 3 sets the order of preference. Code beats a test, a test beats a document, and a document beats memory, because each one depends less on someone paying attention.

Step 4 keeps the loop true. The regression tests here failed 39 of 44 against the scripts as they stood, and all 44 pass now. A test that passes on broken code proves nothing.

Step 5 is the step most often skipped, and skipping it produced the 89 commit gap from Step 1.

Compare and Contrast

Drift	Symptom	Detect with	Close with
Install	Sessions run old code	Installed version against the repo	Update, then compare content
Version	Updater sees no change	Commits since the last bump	Bump with the change
Document	A warning nobody enforces	Search for "remember" and "always"	A guard in the script
Check	Green over a real defect	An input the check must reject	A regression test
Duplication	Copies disagree	Search for the repeated test	One shared function
Destination	A claim goes stale	Oldest dated claims first	Re-measure, write a Correction

How Often Should the Loop Run?

Run the loop on three triggers:

🟢 Every failed or awkward run. A retry is the cheapest signal available, so write it down while it is fresh.
🟢 Every release. Run the tests, run the live pre-flight, bump the version and update the install.
🟢 On a calendar. A review reads the code cold and catches what a passing run cannot. It also finds claims that no recent run touched.

Give the calendar review to a reader with no stake in the code. Hand it the code and withhold your conclusions, so it can disagree.

So, Which One First?

Start with install drift. It takes one command, and until it is closed every other fix stays out of reach of your sessions.

Then add one rejecting input to the check you trust most. If that check accepts bad input, you have learned what your green runs are worth.

Summary

🟢 22 defects surfaced on a day when the pre-flight passed
🟢 44 regression tests now keep the failing inputs, each beside a passing case
❌ The installed skill was 89 commits behind the repository
⚠️ The --update guard is tested against a fake API, and no live article has gone through it yet

With this approach a skill stays true for as long as someone keeps running the loop. The full source and every measurement are in the repository.

References

Desert Hours: Mapping When Transit Disappears With Amazon Location Service

xbill — Sat, 31 Oct 2026 19:05:23 +0000

This article provides a step by step guide to building an hourly transit reach map with Amazon Location Service. Walking areas come from Pedestrian isolines, journey times come from Transit routing, and the finished map runs as static files on Amazon S3 and Amazon CloudFront.

Repository: https://github.com/xbill9/desert-hours

Live map: https://d35zkoavyfqlat.cloudfront.net/

Demo video: https://youtu.be/jfC_d6nq2EM

What Is a Transit Desert?

A transit desert is a place where demand for transit outruns the service on offer. Published maps usually score that gap once per area. Service changes by the hour, so this project scores it for every hour of a weekday.

The study area is Mercer County, New Jersey. It holds Trenton, Princeton and a ring of suburbs, and 16,114 of its 144,114 households have no car.

What You Can Explore

Drag the clock from 5 AM to 4 AM and the blue walkable service area shrinks across the county. The running total shows how many car-free households are out of reach at that hour.

Click a census tract for its day: car-free households, the share of housing that can reach service, departures by hour, and transit times to three destinations at 8 AM, 5 PM, 11 PM and 2 AM beside the driving time.

At This Point You Should Have

An AWS account with access to Amazon Location Service
The AWS CLI, version 2
Python 3 with pyshp and shapely
A GTFS feed for your area, a car ownership table and Census block boundaries

Step 1 — Gather the Data

Source	What it holds	Use
NJ TRANSIT GTFS	Scheduled bus and rail departures	Departures per stop, per hour
Census ACS table B08201	Households by vehicles available	Car-free households per tract
2020 Census blocks	Housing units per block	Where households sit in a tract
Mercer County route layer	Route 130 Connection stops	A county service outside the feed

Every published figure is a county, municipal or tract total, so the map holds no personal data.

Step 2 — Count Departures by Hour

pipeline/build_data.py picks one Wednesday in the feed and counts departures per stop in each hour. It keeps stops inside the county and within 1.5 km of it, because people cross county lines to catch a bus. That leaves 1,012 stops.

Step 3 — Ask Amazon Location for a 10-Minute Walk

Each stop gets a Pedestrian isoline along the street network.

aws geo-routes calculate-isolines --region us-east-1 \
  --origin -74.7548 40.2186 --travel-mode Pedestrian \
  --thresholds '{"Time":[600]}' --isoline-geometry-format Simple

The shape depends on the streets around the stop. Both of these reach about 560 m at their farthest point:

Location	10-minute walk area	Farthest point
Trenton station	0.33 km²	562 m
Hamilton suburb	0.09 km²	560 m

The median across all 1,012 stops is 0.26 km², against 0.50 km² for a 400 m circle.

🔎 Tip: Place Households Where Housing Is

A tract includes parks, rail yards and farmland. Spreading its households evenly across that area counts people who live nowhere near those spaces.

Census blocks carry a housing unit count, so each tract's car-free households are shared across its blocks by housing units. A block counts as served in an hour when its interior point falls inside that hour's service area.

Step 4 — Build the Hourly Service Area

For each hour, the pipeline joins the walking areas of the stops with at least one departure in that hour. Testing every block against that shape gives the hourly total.

Code does the counting and the page quotes it:

Hour	Car-free households out of reach	Share
8 AM	6,388	39.6%
11 PM	8,129	50.4%
3 AM	14,095	87.5%

Step 5 — Ask for Real Journeys

CalculateIsolines covers walking and driving modes. CalculateRoutes adds a Transit mode with timetable departures.

aws geo-routes calculate-routes --region us-east-1 \
  --origin -74.7548 40.2186 --destination -74.6672 40.3495 \
  --travel-mode Transit --departure-time 2026-11-04T02:00:00-05:00

A request for 2 AM from Trenton to Princeton returns a timed journey:

Leg	Departs	Arrives
Walk to Trenton Transit Center	3:56 AM	4:11 AM
Train to Princeton Junction	4:11 AM	4:24 AM
Train to Princeton	4:40 AM	4:45 AM
Walk to the destination	4:45 AM	5:09 AM

The trip takes 3 hours 9 minutes from the moment you wanted to leave, and most of that is waiting for the first train. The pipeline stores that full wait.

Step 6 — Ship It as Static Files

The results are GeoJSON and HTML in a private S3 bucket behind CloudFront. Precomputing everything leaves the page with one Amazon Location key, limited to map actions and the site's domain.

aws cloudformation deploy --region us-east-1 \
  --stack-name desert-hours --template-file deploy/stack.yaml

The findings sit in the page's HTML, so they read without JavaScript. MapLibre draws the tracts and service areas over an Amazon Location base map.

One Tract, One Day

Trenton Tract 1 is a strip along the Delaware River with about 445 car-free households of 1,332, with a margin of ±227.

Measure	Result
Housing within a 10-minute walk of service, 8 AM	73%
Housing within a 10-minute walk of service, 11 PM	0%
Drive to Capital Health Regional Medical Center	11 minutes
Transit to the hospital, leaving at 8 AM	52 minutes
Transit to the hospital, leaving at 2 AM	3 hours 29 minutes

Compare and Contrast

Approach	Walking area	Time of day	Journey times
400 m circles	Same everywhere	Any	None
Static desert score	Varies by method	One score	None
Desert Hours	Street network	Every hour	Timetable by Amazon Location

What Did It Cost?

Item	Count	List price	Total
Isolines	1,012	$1.50 per thousand	$1.52
Journeys	1,260	$0.50 per thousand	$0.63

Totals are arithmetic from AWS Price List rates. Results are cached, so a rebuild makes no new calls.

So, Which One?

Use isolines when reach depends on the street network, which it does in any suburb. Use Transit routing for the stories a map shape cannot tell, such as how long a 2 AM trip takes. Keep both in a batch pipeline, and the live page never spends money on routing.

Summary

The goal of this article was to map how transit reach changes through a weekday. The key to the solution was Amazon Location Pedestrian isolines joined by hour and tested against Census blocks. The Mercer County results were:

🟢 6,388 car-free households are out of walking reach at 8 AM, 39.6% of the county's total
❌ 14,095 are out of reach at 3 AM, 87.5% of the total
🟢 Street-based walking areas separate a city grid from a suburb at the same distance
⚠️ Trenton Tract 1 falls from 73% of housing in reach at 8 AM to 0% at 11 PM

Scope: scheduled service for one weekday, which may be incomplete, inaccurate or out of date. TigerTransit, the Princeton municipal shuttle, the Z-Line and on-demand services are not included, so reach is understated where they run. Route 130 Connection stops use a September 2023 timetable and count in every hour with a departure. Car-free households are assumed to be spread like housing within a tract, and tract estimates carry wide margins. Journey times come from one point per tract. Schedule data comes from NJ TRANSIT's published GTFS through the Mobility Database archive, and this project is independent of NJ TRANSIT.

The strategy for using Amazon Location Service for an hourly transit map was validated with an incremental step by step approach.

References

Four GPUs, One Quantization Ladder: Gemma 4 E2B From a 4 GB Laptop to an NVIDIA L4

xbill — Fri, 30 Oct 2026 13:05:27 +0000

This article provides a step by step guide to running Gemma 4 E2B's quantization ladder, BF16 down to Q2_K, on four NVIDIA GPUs with the same nine GGUF files and the same llama.cpp build, and comparing the four cards question by question.

https://github.com/xbill9/gemma-graveyard

The first article asked every rung of the ladder the same 250 GSM8K questions on a Tesla T4. The second repeated it on an NVIDIA L4 in AWS. This one adds two consumer cards: a GTX 1650 Ti laptop GPU with 4 GB and an RTX 3050 desktop card with 8 GB. All four run llama.cpp b11699 on files with the same sha256, and each card gives identical text when BF16 is run twice.

The Results in Brief

The ladder keeps its shape on all four GPUs. Against each card's own BF16, F16 to Q5_K_M lose 1 to 8 answers, Q4_K_M and Q4_0 lose 9 to 19, Q3_K_M loses 42 to 49, and Q2_K gets 3 to 6 right out of 250.

The counts agree; the questions move. Any two GPUs grade 2 to 4 questions differently at BF16 and 12 to 22 at Q4_0. No question is lost at F16 or Q8_0 on all four cards, while 216 are lost at Q2_K on all four. 218 questions die at some rung on every GPU, and 147 of them die first at the same rung on every GPU.

The 4 GB laptop runs every quantized rung entirely on its GPU. BF16 and F16 do not fit, so llama.cpp splits them between GPU and CPU, and they decode at 9.2 and 9.6 tokens per second against 45.9 for Q8_0 on the same card.

The cover lays the four GPUs' grades over each other. One row per rung, BF16 at the top and Q2_K at the bottom, one cell per question: marigold where all four GPUs answer right, grey where all four answer wrong, violet where they disagree. docs/make_cover_gpu4.py draws it from the committed rows.

What Changes and What Stays

	Tesla T4	NVIDIA L4	GTX 1650 Ti	RTX 3050
Where	GCE `n1-standard-2`	AWS `g6.xlarge`	laptop	desktop
GPU memory	15,360 MiB	23,034 MiB	4,096 MiB	8,192 MiB
Compute capability	7.5	8.9	7.5	8.6
Host CPU	2 vCPU	AMD EPYC 7R13	Intel i7-10750H	Intel i7-5820K
Driver	615.71.10	595.112.07	615.71.10	615.78.08

The same on all four: llama.cpp b11699 with GGML_CUDA=ON, built for each card's own compute capability; the nine GGUFs and Google's QAT Q4_0 file, byte for byte; the 250 questions in data/heldout.jsonl; temperature 0, one request at a time, thinking off, 1,024 max tokens, llama-server -c 4096 -np 1 --seed 0. Every rung runs with -ngl 99 except BF16 and F16 on the GTX 1650 (Step 4).

At This Point You Should Have…

An NVIDIA GPU with at least 4 GB and a CUDA toolkit
About 58 GB of free disk for the checkpoint, the nine GGUFs and the QAT file
A clone: git clone https://github.com/xbill9/gemma-graveyard
The T4 run's committed results in runs/2026-10-27-ladder/

Step 1 — Build llama.cpp at the Same Tag

On the RTX 3050:

git -C ~/llama.cpp worktree add --detach /android/graveyard/llama.cpp b11699
cd /android/graveyard/llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j12 --target llama-server llama-quantize

The one build flag that changes per card is CMAKE_CUDA_ARCHITECTURES: 75 for the T4 and the GTX 1650, 89 for the L4, 86 for the RTX 3050.

nvidia-smi --query-gpu=name,memory.total,compute_cap,driver_version --format=csv
name, memory.total [MiB], compute_cap, driver_version
NVIDIA GeForce RTX 3050 OEM, 8192 MiB, 8.6, 615.78.08

🔎 Tip: keep the pinned build in its own worktree. The desktop already had a newer llama.cpp checkout (b11983). git worktree add --detach <dir> b11699 gives a second working copy at the tag from the same clone, so the measured build and the everyday build sit side by side.

Step 2 — Rebuild the Ladder and Check It Against the T4

ladder.sh converts the checkpoint to BF16 and F16 and quantizes seven rungs from BF16. Each run script then compares every file's size and sha256 with the T4's ladder.tsv:

diff <(cut -f1,3,4 runs/2026-10-27-ladder/ladder.tsv) <(cut -f1,3,4 runs/2026-10-29-3050/ladder.tsv) && echo "ladder matches T4 sha256"
ladder matches T4 sha256

The GTX 1650 and the RTX 3050 both print the same line, and Google's QAT Q4_0 file matches too (sha256 fa401b55…). Across the four machines, the GGUF bytes are identical; the card, its driver, its CUDA version and the kernels built for it are what differ.

🔎 Tip: the converter writes the checkpoint directory into the GGUF. convert_hf_to_gguf.py copies the model card and takes the model's name from the directory it reads. A checkpoint taken straight from the Hugging Face cache snapshot has neither the README.md nor the name gemma-4-E2B-it, and gives different bytes. Download with hf download google/gemma-4-E2B-it --local-dir <dir>/gemma-4-E2B-it.

Step 3 — Prove Each Card Repeats Itself

BF16 on 50 questions, twice, on every GPU:

./graveyard.py repeat runs/2026-10-29-3050/determinism-1 runs/2026-10-29-3050/determinism-2 BF16
{"rung": "BF16", "n": 50, "mismatched_extracted": 0, "mismatched_text": 0, "mismatched_right": 0, "right_a": 46, "right_b": 46}

The T4, the L4 and the GTX 1650 give the same result: 0 mismatched characters and 46 of 50 right both times. Each card on its own is deterministic, so a difference between two cards comes from the GPU and its CUDA stack.

Step 4 — Fit the 16-bit Rungs on 4 GB

BF16 and F16 need a 4,384.87 MiB GPU buffer for their 36 layers (from the T4's load log), and the GTX 1650 has 4,096 MiB. With -ngl 99 the load fails. graveyard.py run --ngl auto leaves -ngl off, and llama.cpp's fit step places what it can:

common_params_fit_impl:   - CUDA0 (NVIDIA GeForce GTX 1650 Ti with Max-Q Design): 14 layers,   2546 MiB used,   1125 MiB free
load_tensors: offloaded 14/36 layers to GPU
load_tensors:        CUDA0 model buffer size =  2407.92 MiB

Every other rung fits whole. Q8_0 on the same card:

load_tensors: offloaded 36/36 layers to GPU
load_tensors:        CUDA0 model buffer size =  2342.30 MiB

The split changes speed and leaves the grades in line with the other GPUs: the GTX 1650's BF16 scores 229, the same as the T4 and the L4, and grades 2 to 4 questions differently from each of them. The 8 GB RTX 3050 holds BF16 whole at 4,651 MiB.

Step 5 — Run the Ladder on Each Card and Score It

Each run script runs the nine rungs and QAT Q4_0, then scores every rung against that card's own BF16:

./graveyard.py bury runs/2026-10-29-3050 BF16 F16 Q8_0 Q6_K Q5_K_M Q4_K_M Q4_0 Q3_K_M Q2_K --json graveyard-ladder.json

Right answers, out of 250:

Rung	T4	L4	GTX 1650	RTX 3050
BF16	229	229	229	232
F16	229	229	228	232
Q8_0	227	228	229	226
Q6_K	229	227	226	227
Q5_K_M	231	228	226	226
Q4_K_M	224	221	222	221
Q4_0	222	220	216	218
Q3_K_M	191	186	183	191
Q2_K	3	6	5	3

Lost, against the same card's BF16:

Rung	T4	L4	GTX 1650	RTX 3050
F16	1	1	2	1
Q8_0	4	3	2	7
Q6_K	4	6	7	8
Q5_K_M	3	6	6	8
Q4_K_M	9	14	12	14
Q4_0	15	16	19	16
Q3_K_M	42	45	49	44
Q2_K	226	225	225	229

Lost: right at BF16, wrong at this rung, on the same GPU.

The failures fall into the same kinds on every card. Q3_K_M runs out of room: 33 to 36 of its losses on each GPU are answers cut off at 1,024 tokens. Q2_K loops: 208 to 228 of its 250 answers repeat a line five or more times, and 246 to 249 hit the token cap.

Step 6 — Line the Four GPUs Up, Question by Question

docs/gpu4_derived.py rebuilds the T4's grid from its committed results and reads the other three from their rows, then compares all six pairs of GPUs at every rung:

python3 docs/gpu4_derived.py > runs/2026-10-29-3050/four-gpus.txt

Rung	Graded differently, any two GPUs	Lost on all four
BF16	2 to 4	—
F16	3 to 5	0
Q8_0	3 to 9	0
Q6_K	4 to 10	2
Q5_K_M	7 to 10	1
Q4_K_M	9 to 15	5
Q4_0	12 to 22	5
Q3_K_M	26 to 34	20
Q2_K	6 to 9	216

Graded differently: right on one GPU and wrong on the other, at that rung, for the closest and the furthest of the six pairs.

The light rungs lose different questions on each card. F16 and Q8_0 lose 1 to 7 answers per GPU, and none of them is lost on all four.
The deep losses are shared. Q3_K_M loses 20 questions on all four GPUs and Q2_K 216.
Most of the graveyard is common ground. Each GPU has 226 questions that die at some rung, the RTX 3050 230. 218 die on all four, and 147 die first at the same rung on all four. One question dies only on the RTX 3050 (test-358) and one only on the GTX 1650 (test-147).
BF16 itself moves a few answers. 227 questions are right at BF16 on all four cards and 18 wrong on all four. Against the L4, the RTX 3050's BF16 changes the text of 31 answers out of 250, the extracted number of 4 and the grade of 3.

Step 7 — Google's QAT Q4_0 on Four Cards

./graveyard.py bury runs/2026-10-29-3050 BF16 QAT_Q4_0 Q4_0 --json graveyard-qat.json

GPU	QAT Q4_0 right	Q4_0 right	QAT Q4_0 lost	Q4_0 lost
T4	221	🥇 222	15	15
L4	219	🥇 220	15	16
GTX 1650	🥇 222	216	11	19
RTX 3050	🥇 221	218	15	16

QAT Q4_0 scores 219 to 222 on the four cards and plain Q4_0 216 to 222. On the T4 and the L4 the plain file is ahead by 1; on the GTX 1650 QAT is ahead by 6 and on the RTX 3050 by 3. The two files lose mostly different questions: on the RTX 3050, 5 questions are lost under both, 10 only under QAT and 11 only under the plain Q4_0. Across the four GPUs, 6 of QAT's losses and 5 of Q4_0's are lost on every card.

Step 8 — Speed on Each Card

Median decode speed in tokens per second. Each column is one machine; read down a column.

Rung	T4	L4	GTX 1650	RTX 3050
BF16	40.0	50.1	9.2	40.3
F16	39.6	50.0	9.6	40.3
Q8_0	64.3	83.0	45.9	67.8
Q6_K	70.8	98.1	57.2	75.3
Q5_K_M	78.7	107.7	61.9	82.2
Q4_K_M	84.1	118.7	66.5	91.5
Q4_0	🥇 93.6	125.6	🥇 70.8	🥇 99.7
Q3_K_M	74.2	111.1	55.9	64.2
Q2_K	74.0	🥇 139.5	62.2	95.0

Q4_0 is the fastest rung on three of the four cards; on the L4 it is Q2_K, which is wrong on 244 of 250 questions.
Q5_K_M decodes 1.21 to 1.35 times as fast as Q8_0 on each card (arithmetic, from the table). Q8_0 is the baseline here because it is the largest rung that fits whole on all four GPUs.
Q3_K_M is slower than Q4_0 on every card, and on the RTX 3050 it is slower than Q8_0.
BF16 and F16 run at the same speed on the T4, the L4 and the RTX 3050. On the GTX 1650, with 22 of 36 layers on the CPU, they run at a fifth of Q8_0's speed.

The full ladder with QAT took 5 hours 16 minutes on the RTX 3050 and 12 hours 25 minutes on the GTX 1650, from the first log line to the last (run-3050.log, run-1650.log).

Compare and Contrast

	T4	L4	GTX 1650	RTX 3050
BF16 right of 250	229	229	229	🥇 232
Q5_K_M lost	🥇 3	6	6	8
Q4_0 right, tok/s	222, 93.6	220, 125.6	216, 70.8	218, 99.7
Every rung whole on the GPU	🟢 yes	🟢 yes	❌ BF16, F16 split	🟢 yes
Same text twice at BF16	🟢 yes	🟢 yes	🟢 yes	🟢 yes

So, Which One?

On any of the four cards, take Q5_K_M: 39.0% of BF16's bytes, 226 to 231 right, 3 to 8 answers lost against the card's own BF16, and 1.21 to 1.35 times Q8_0's decode speed. For the fastest file that stays within 19 answers, take Q4_0, the fastest rung on three of the cards. On a 4 GB card, every quantized rung fits whole and the 16-bit rungs run at under 10 tokens per second. To compare two quantizations question by question, run both on one GPU: a different card moves 2 to 4 grades at BF16 and up to 34 at Q3_K_M.

Summary

The goal of this article was to run Gemma 4 E2B's quantization ladder on four GPUs, from a 4 GB laptop card to a data-centre L4, and see which grades hold. The key was holding everything else fixed: byte-identical GGUFs, the same llama.cpp tag and the same sampling, with each card checked to give identical text twice. The four-GPU results were:

🟢 All nine GGUFs and the QAT file match the T4's sha256 on every machine, and BF16 gives identical text twice on 50 questions on every card.
🟢 The ladder keeps its shape on all four GPUs: F16 to Q5_K_M lose 1 to 8, Q4_K_M and Q4_0 lose 9 to 19, Q3_K_M loses 42 to 49.
🟢 Q5_K_M is the best trade on every card: 226 to 231 right at 1.21 to 1.35 times Q8_0's speed.
🟢 A 4 GB GTX 1650 runs every quantized rung whole on the GPU; BF16 and F16 run split, 14 of 36 layers on the GPU, at 9.2 and 9.6 tokens per second.
⚠️ Any two GPUs grade 2 to 4 questions differently at BF16 and 12 to 22 at Q4_0; no question is lost at F16 or Q8_0 on all four.
⚠️ 218 questions die on all four GPUs, 147 of them first at the same rung.
⚠️ QAT Q4_0 and plain Q4_0 trade places by card: plain ahead by 1 on the T4 and L4, QAT ahead by 6 on the GTX 1650 and by 3 on the RTX 3050.
❌ Q3_K_M loses 42 to 49, mostly answers cut off at 1,024 tokens; Q2_K gets 3 to 6 right, with 208 to 228 looping answers per card.

Scope: one run of each rung per GPU on four NVIDIA cards (a Tesla T4 on GCE, an L4 on AWS, a GTX 1650 Ti laptop GPU and an RTX 3050 desktop card), 250 GSM8K test questions, greedy, thinking off, 1,024 max tokens, llama.cpp b11699 built for each card. Determinism checked at BF16 on each card. The T4 grid is rebuilt from its committed per-question results. The hosts also differ in CPU, RAM, driver and CUDA version, and on the GTX 1650 BF16 and F16 run split between GPU and CPU. Speed is compared within each card only.

The strategy for measuring Gemma 4's accuracy one quantization rung at a time on four GPUs was validated with an incremental step by step approach.

References

Performance vs Quality: Gemma 4 E2B Quantization on an NVIDIA L4 and a Tesla T4

xbill — Thu, 29 Oct 2026 14:39:09 +0000

This article provides a step by step guide to measuring Gemma 4 E2B's speed and accuracy at every quantization rung a second time, on an NVIDIA L4 in AWS, with the same nine GGUF files and the same llama.cpp build as the Tesla T4 run, and comparing the two GPUs question by question.

https://github.com/xbill9/gemma-graveyard

The first article quantized Gemma 4 E2B from BF16 to Q2_K on one Tesla T4, asked every rung the same 250 GSM8K questions, and recorded each answer's grade at every rung. This one moves the whole ladder to a g6.xlarge. The nine files have the same sha256 on both machines, both run llama.cpp b11699, and each machine gives identical text when a rung is run twice. The GPU changes, and with it the driver, the CUDA version and the kernels that do the arithmetic.

The Results in Brief

The L4 decodes every rung faster than the T4, from 1.25 times at BF16 to 1.89 times at Q2_K, and Q5_K_M stays the best trade between speed and accuracy: 228 right against BF16's 229, at 2.15 times BF16's decode speed on the L4.

The grades move with the GPU. With the same file and the same build, the L4 and the T4 grade 2 questions differently at BF16, 7 at Q5_K_M, 16 at Q4_0 and 33 at Q3_K_M. At Q5_K_M that is nearly the whole quantization effect on the T4, where Q5_K_M against BF16 was 8 questions: 3 lost and 5 gained. Of the 226 questions lost at some rung on each GPU, 180 are first lost at the same rung on both.

The cover lays the two GPUs' grades over each other. One row per rung, BF16 at the top and Q2_K at the bottom, one cell per question: marigold where both GPUs answer right, grey where both answer wrong, violet where the two GPUs disagree. docs/make_cover_gpu.py draws it from the committed rows.

What Changes and What Stays

	Tesla T4 (GCE)	NVIDIA L4 (AWS)
Instance	`n1-standard-2`, `us-west2-b`	`g6.xlarge`, `us-west-2c`
GPU memory	15,360 MiB	23,034 MiB
Compute capability	7.5 (Turing)	8.9
Host	2 vCPU, 7,436 MiB RAM	4 vCPU AMD EPYC 7R13, 15,363 MiB RAM
Driver, CUDA	615.71.10, 13.4	595.112.07, 13.2

The same on both: llama.cpp b11699 with GGML_CUDA=ON, built for each card's own compute capability; the nine GGUFs and Google's QAT Q4_0 file, byte for byte; the 250 questions in data/heldout.jsonl; temperature 0, one request at a time, no prompt cache, thinking off, 1,024 max tokens, llama-server -ngl 99 -c 4096 -np 1 --seed 0.

At This Point You Should Have…

An AWS account with quota for G instances (a g6.xlarge uses 4 vCPU of it)
The AWS CLI signed in, a key pair and a security group that allows SSH
A clone: git clone https://github.com/xbill9/gemma-graveyard
The T4 run's committed results in runs/2026-10-27-ladder/

Step 1 — Launch the L4

The AWS Deep Learning Base AMI ships with the NVIDIA driver and the CUDA toolkits, so the box needs no driver install. Its ID comes from a public SSM parameter:

aws ssm get-parameter --name /aws/service/deeplearning/ami/x86_64/base-oss-nvidia-driver-gpu-ubuntu-24.04/latest/ami-id \
  --query Parameter.Value --output text
ami-0fcc8fa7ee8d2dc4c

aws ec2 run-instances --image-id ami-0fcc8fa7ee8d2dc4c --instance-type g6.xlarge --key-name graveyard-l4 \
  --security-group-ids sg-0d721a15b91250482 --metadata-options HttpTokens=required \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=200,VolumeType=gp3,DeleteOnTermination=true}'

From nvidia-smi on the new box:

| NVIDIA-SMI 595.112.07             Driver Version: 595.112.07     CUDA Version: 13.2     |
|   0  NVIDIA L4                      On  |   00000000:31:00.0 Off |                    0 |
| N/A   55C    P0             28W /   72W |       0MiB /  23034MiB |      0%      Default |

A g6.xlarge in us-west-2 is $0.8048 an hour on demand, from the AWS Price List API.

🔎 Tip: work on the instance's own NVMe drive. A g6.xlarge carries a local NVMe disk, which the Deep Learning AMI mounts at /opt/dlami/nvme. The checkpoint, the nine GGUFs, the llama.cpp build and a 16 GB swapfile all go there. It is wiped when the instance stops, so copy the results off first.

Step 2 — Build llama.cpp at the Same Tag

git clone https://github.com/ggml-org/llama.cpp /opt/dlami/nvme/llama.cpp
cd /opt/dlami/nvme/llama.cpp && git checkout b11699
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j4 --target llama-server llama-quantize llama-cli
git describe --tags
b11699

The one build flag that differs from the T4 is CMAKE_CUDA_ARCHITECTURES: 75 for the T4, 89 for the L4. llama.cpp's log on the L4 confirms it:

ggml_cuda_init: found 1 CUDA devices (Total VRAM: 22565 MiB):
  Device 0: NVIDIA L4, compute capability 8.9, VMM: yes, VRAM: 22565 MiB

🔎 Tip: llama.cpp's pinned transformers cannot read Gemma 4's tokenizer. pip install -r requirements/requirements-convert_hf_to_gguf.txt installs transformers 4.57.6, and convert_hf_to_gguf.py then stops while loading the tokenizer:

AttributeError: 'list' object has no attribute 'keys'

pip install -U transformers (5.21.0) converts the checkpoint, and Step 3 shows the files match the T4's byte for byte.

Step 3 — Rebuild the Ladder and Check It Against the T4

ladder.sh converts the checkpoint to BF16 and F16 and quantizes seven rungs from BF16, as on the T4. runs/2026-10-28-l4/run-l4.sh then compares each file's size and sha256 with the T4's ladder.tsv:

./ladder.sh /opt/dlami/nvme/models/gemma-4-E2B-it /opt/dlami/nvme/graveyard-gguf /opt/dlami/nvme/llama.cpp
diff <(cut -f1,3,4 runs/2026-10-27-ladder/ladder.tsv) <(cut -f1,3,4 runs/2026-10-28-l4/ladder.tsv) && echo "ladder matches T4 sha256"
ladder matches T4 sha256

All nine files are identical, and so is Google's QAT Q4_0 file (sha256 fa401b55…). From here on, every answer that differs between the two machines comes from the GPU and its CUDA stack: the card, its driver, its CUDA version and the kernels built for it.

Step 4 — Prove the L4 Repeats Itself

The same check as on the T4: BF16 on 50 questions, twice.

./graveyard.py repeat runs/2026-10-28-l4/determinism-1 runs/2026-10-28-l4/determinism-2 BF16
{"rung": "BF16", "n": 50, "mismatched_extracted": 0, "mismatched_text": 0, "mismatched_right": 0, "right_a": 46, "right_b": 46}

Every character matches. Each GPU on its own is deterministic, so a difference between the two machines belongs to the GPU and its CUDA stack.

Step 5 — Run the Ladder on the L4 and Score It

run-l4.sh runs the nine rungs and QAT Q4_0, then scores each against BF16:

./graveyard.py bury runs/2026-10-28-l4 BF16 F16 Q8_0 Q6_K Q5_K_M Q4_K_M Q4_0 Q3_K_M Q2_K --json graveyard-ladder.json

Rung	Right, T4	Right, L4	Lost, L4	Gained, L4
BF16	229	229	0	0
F16	229	229	1	1
Q8_0	227	🥇 228	3	2
Q6_K	229	227	6	4
Q5_K_M	231	🥇 228	6	5
Q4_K_M	224	221	14	6
Q4_0	222	220	16	7
Q3_K_M	191	186	45	2
Q2_K	3	6	225	2

Lost: BF16 right, this rung wrong. Gained: BF16 wrong, this rung right. Both are counted against the same GPU's own BF16.

The shape of the ladder holds. Q8_0 to Q5_K_M stay within 2 of BF16; Q4_K_M and Q4_0 lose 8 and 9; Q3_K_M loses 43; Q2_K collapses. The failures fall into the same kinds as on the T4: Q3_K_M runs out of room (36 of its 45 losses cut off at 1,024 tokens), and Q2_K loops (218 of 250 answers repeat a line five or more times, 71 open a thinking block with thinking off, 63 leave content empty).

🔎 Tip: llama-server can refuse an answer it cannot parse. One Q2_K question, test-194, came back as HTTP 500 on the L4: The model produced output that does not match the expected peg-gemma4 format. graveyard.py records it as an error and grades it wrong, so Q2_K's 6 right are out of 250 with 1 error.

Step 6 — Line the Two GPUs Up, Question by Question

docs/gpu_derived.py rebuilds the T4's right/wrong grid for all 250 questions at all nine rungs from its committed results, and the rebuilt grid reproduces all 2,250 cells of the T4 article's cover and the T4's own right, lost and gained counts at every rung.

python3 docs/gpu_derived.py > docs/evidence-l4/derived-gpu.txt

Rung	Graded differently, T4 vs L4	Lost, T4	Lost, L4	Lost on both
BF16	2	—	—	—
F16	4	1	1	0
Q8_0	3	4	3	2
Q6_K	6	4	6	3
Q5_K_M	7	3	6	2
Q4_K_M	15	9	14	7
Q4_0	16	15	16	10
Q3_K_M	33	42	45	28
Q2_K	9	226	225	221

Graded differently: questions right on one GPU and wrong on the other, at that rung.

Three things show in the grid:

Above Q4_K_M, the GPU moves nearly as many grades as the quantization. At Q5_K_M the two GPUs disagree on 7 questions; on the T4, Q5_K_M and BF16 disagree on 8 (3 lost, 5 gained). Q5_K_M's 231 on the T4 and 228 on the L4 sit either side of BF16's 229 on both.
Most losses are shared, often at a different rung. 222 questions are lost at some rung on both GPUs, 4 only on the T4 and 4 only on the L4. 180 of them are first lost at the same rung on both.
The deep losses are the stable ones. Q2_K's losses are 221 of 226 shared and Q3_K_M's 28 of 42. Between F16 and Q5_K_M, 7 of 12 T4 losses are lost again on the L4 at the same rung.

The twins and triplets question shows it. The T4 article's one F16 loss, test-687 (answer 13), was right at BF16 on the T4 and answered 17 at F16. On the L4, BF16 itself answers 17, so the question counts as gained there: wrong at BF16, right again at Q8_0, Q6_K, Q5_K_M and Q4_0. test-93 (answer 36) runs the other way: gained on the T4, right at BF16 on the L4 and lost from Q8_0 on.

Step 7 — Google's QAT Q4_0 on the L4

./graveyard.py bury runs/2026-10-28-l4 BF16 QAT_Q4_0 Q4_0 --json graveyard-qat.json

File	Right	Lost	Cut off at 1,024	Median tok/s
QAT Q4_0	219	15	8	125.7
Q4_0	220	16	12	125.6

QAT Q4_0 gains 5 questions against BF16 and Q4_0 gains 7. The QAT file is 3,349,516,256 bytes and loads in 1,730 MiB; the plain Q4_0 is 3,360,161,216 bytes and loads in 1,740 MiB.

The same picture as the T4, one right answer apart again: 219 against the plain Q4_0's 220, where the T4 had 221 against 222. On the L4 the two decode at the same speed, 125.7 and 125.6 tok/s; on the T4 the plain Q4_0 was faster, 93.6 against 87.8. 10 of QAT Q4_0's 15 losses on the T4 are lost on the L4 too.

Step 8 — Speed and Memory on Each Card

Rung	Median tok/s, T4	Median tok/s, L4	L4 ÷ T4	× BF16, T4 / L4
BF16	40.0	50.1	1.25	1.00 / 1.00
F16	39.6	50.0	1.26	0.99 / 1.00
Q8_0	64.3	83.0	1.29	1.61 / 1.66
Q6_K	70.8	98.1	1.39	1.77 / 1.96
Q5_K_M	78.7	107.7	1.37	1.97 / 2.15
Q4_K_M	84.1	118.7	1.41	2.10 / 2.37
Q4_0	93.6	125.6	1.34	2.34 / 2.51
Q3_K_M	74.2	111.1	1.50	1.85 / 2.22
Q2_K	74.0	139.5	1.89	1.85 / 2.78

BF16 and F16 run at the same speed on both cards, within 0.4 tok/s of each other on each GPU.
The smaller the file, the bigger the L4's lead, up to 1.89 times at Q2_K. Q2_K is the fastest rung on the L4 and among the slowest on the T4; it is also wrong on 244 of 250 questions.
Q3_K_M is slower than Q4_K_M and Q4_0 on both cards.
The L4 uses 101 MiB more GPU memory at every rung (nvidia-smi after load: 4,772 against 4,671 MiB at BF16, 1,740 against 1,639 at Q4_0), the same fixed amount whatever the file size.

The L4 run took 4 hours 3 minutes from the start of the ladder build to the last scoring pass (run-l4.log), about $3.26 at the on-demand price (arithmetic).

Compare and Contrast

	Tesla T4	NVIDIA L4
BF16 right of 250	229	229
Q5_K_M right, × BF16 speed	🥇 231, 1.97	🥇 228, 2.15
Q4_0 right, median tok/s	222, 93.6	220, 125.6
Q3_K_M right	191	186
Same text twice on one GPU	🟢 yes	🟢 yes
GPU memory after load, Q4_0	1,639 MiB	1,740 MiB
Verdict	🟢 Q5_K_M, slower card	🟢 Q5_K_M, 1.37 times faster

So, Which One?

On either card, take Q5_K_M: 39.0% of BF16's bytes, within 1 of BF16's score on the L4 and 2 above it on the T4, and 1.97 times BF16's decode speed on the T4 or 2.15 times on the L4. For the fastest file that stays within 9 answers, take Q4_0. The L4 is the faster card at every rung, by 1.25 to 1.89 times. To compare two quantizations question by question, run both on one GPU: a different card moves a few grades at every rung, and up to 33 at Q3_K_M.

Summary

The goal of this article was to rerun Gemma 4 E2B's quantization ladder on a second GPU, measure speed against accuracy at every rung, and see which grades stay put. The key was holding everything else fixed: byte-identical GGUFs, the same llama.cpp tag and the same sampling, with each GPU checked to give identical text twice. The L4 results were:

🟢 All nine GGUFs and the QAT file match the T4's sha256; BF16 on the L4 gives identical text twice on 50 questions, the one rung checked for determinism.
🟢 BF16 scores 229 of 250 on both GPUs, and the ladder keeps its shape: Q8_0 to Q5_K_M within 2, Q4_K_M and Q4_0 within 9.
🟢 Q5_K_M is the best trade on both: 228 right at 2.15 times BF16's speed on the L4.
🟢 The L4 decodes 1.25 to 1.89 times faster than the T4, and BF16 runs at F16's speed on both cards.
⚠️ The two GPUs grade 2 questions differently at BF16, 7 at Q5_K_M and 16 at Q4_0; above Q4_K_M that is nearly the size of the quantization effect.
⚠️ 180 of 226 lost questions are first lost at the same rung on both GPUs; 222 questions are lost on both at some rung.
⚠️ QAT Q4_0 scores 219 against the plain Q4_0's 220 on the L4, one apart again, at the same speed.
❌ Q3_K_M loses 43, 36 of its 45 losses cut off at 1,024 tokens; Q2_K gets 6 right, with 218 looping answers and 1 answer the server refused to parse.

Scope: one Tesla T4 on GCE (n1-standard-2, us-west2-b) and one NVIDIA L4 on AWS (g6.xlarge, us-west-2c, $0.8048 per hour on demand), one run of each rung per GPU, 250 GSM8K test questions, greedy, thinking off, 1,024 max tokens, llama.cpp b11699 built for each card. Determinism checked on the L4 at BF16. The T4 grid is rebuilt from its committed per-question results and cover. The two hosts also differ in CPU, RAM, driver and CUDA version.

The strategy for measuring Gemma 4's speed and accuracy one quantization rung at a time on two GPUs was validated with an incremental step by step approach.

References

Performance vs Quality: Gemma 4 E2B Quantization on an NVIDIA L4 and a Tesla T4

xbill — Thu, 29 Oct 2026 14:38:56 +0000

https://github.com/xbill9/gemma-graveyard

The Results in Brief

What Changes and What Stays

	Tesla T4 (GCE)	NVIDIA L4 (AWS)
Instance	`n1-standard-2`, `us-west2-b`	`g6.xlarge`, `us-west-2c`
GPU memory	15,360 MiB	23,034 MiB
Compute capability	7.5 (Turing)	8.9
Host	2 vCPU, 7,436 MiB RAM	4 vCPU AMD EPYC 7R13, 15,363 MiB RAM
Driver, CUDA	615.71.10, 13.4	595.112.07, 13.2

At This Point You Should Have…

An AWS account with quota for G instances (a g6.xlarge uses 4 vCPU of it)
The AWS CLI signed in, a key pair and a security group that allows SSH
A clone: git clone https://github.com/xbill9/gemma-graveyard
The T4 run's committed results in runs/2026-10-27-ladder/

Step 1 — Launch the L4

The AWS Deep Learning Base AMI ships with the NVIDIA driver and the CUDA toolkits, so the box needs no driver install. Its ID comes from a public SSM parameter:

aws ssm get-parameter --name /aws/service/deeplearning/ami/x86_64/base-oss-nvidia-driver-gpu-ubuntu-24.04/latest/ami-id \
  --query Parameter.Value --output text
ami-0fcc8fa7ee8d2dc4c

aws ec2 run-instances --image-id ami-0fcc8fa7ee8d2dc4c --instance-type g6.xlarge --key-name graveyard-l4 \
  --security-group-ids sg-0d721a15b91250482 --metadata-options HttpTokens=required \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=200,VolumeType=gp3,DeleteOnTermination=true}'

From nvidia-smi on the new box:

| NVIDIA-SMI 595.112.07             Driver Version: 595.112.07     CUDA Version: 13.2     |
|   0  NVIDIA L4                      On  |   00000000:31:00.0 Off |                    0 |
| N/A   55C    P0             28W /   72W |       0MiB /  23034MiB |      0%      Default |

A g6.xlarge in us-west-2 is $0.8048 an hour on demand, from the AWS Price List API.

Step 2 — Build llama.cpp at the Same Tag

git clone https://github.com/ggml-org/llama.cpp /opt/dlami/nvme/llama.cpp
cd /opt/dlami/nvme/llama.cpp && git checkout b11699
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j4 --target llama-server llama-quantize llama-cli
git describe --tags
b11699

The one build flag that differs from the T4 is CMAKE_CUDA_ARCHITECTURES: 75 for the T4, 89 for the L4. llama.cpp's log on the L4 confirms it:

ggml_cuda_init: found 1 CUDA devices (Total VRAM: 22565 MiB):
  Device 0: NVIDIA L4, compute capability 8.9, VMM: yes, VRAM: 22565 MiB

AttributeError: 'list' object has no attribute 'keys'

pip install -U transformers (5.21.0) converts the checkpoint, and Step 3 shows the files match the T4's byte for byte.

Step 3 — Rebuild the Ladder and Check It Against the T4

./ladder.sh /opt/dlami/nvme/models/gemma-4-E2B-it /opt/dlami/nvme/graveyard-gguf /opt/dlami/nvme/llama.cpp
diff <(cut -f1,3,4 runs/2026-10-27-ladder/ladder.tsv) <(cut -f1,3,4 runs/2026-10-28-l4/ladder.tsv) && echo "ladder matches T4 sha256"
ladder matches T4 sha256

Step 4 — Prove the L4 Repeats Itself

The same check as on the T4: BF16 on 50 questions, twice.

./graveyard.py repeat runs/2026-10-28-l4/determinism-1 runs/2026-10-28-l4/determinism-2 BF16
{"rung": "BF16", "n": 50, "mismatched_extracted": 0, "mismatched_text": 0, "mismatched_right": 0, "right_a": 46, "right_b": 46}

Every character matches. Each GPU on its own is deterministic, so a difference between the two machines belongs to the GPU and its CUDA stack.

Step 5 — Run the Ladder on the L4 and Score It

run-l4.sh runs the nine rungs and QAT Q4_0, then scores each against BF16:

./graveyard.py bury runs/2026-10-28-l4 BF16 F16 Q8_0 Q6_K Q5_K_M Q4_K_M Q4_0 Q3_K_M Q2_K --json graveyard-ladder.json

Rung	Right, T4	Right, L4	Lost, L4	Gained, L4
BF16	229	229	0	0
F16	229	229	1	1
Q8_0	227	🥇 228	3	2
Q6_K	229	227	6	4
Q5_K_M	231	🥇 228	6	5
Q4_K_M	224	221	14	6
Q4_0	222	220	16	7
Q3_K_M	191	186	45	2
Q2_K	3	6	225	2

Lost: BF16 right, this rung wrong. Gained: BF16 wrong, this rung right. Both are counted against the same GPU's own BF16.

Step 6 — Line the Two GPUs Up, Question by Question

python3 docs/gpu_derived.py > docs/evidence-l4/derived-gpu.txt

Rung	Graded differently, T4 vs L4	Lost, T4	Lost, L4	Lost on both
BF16	2	—	—	—
F16	4	1	1	0
Q8_0	3	4	3	2
Q6_K	6	4	6	3
Q5_K_M	7	3	6	2
Q4_K_M	15	9	14	7
Q4_0	16	15	16	10
Q3_K_M	33	42	45	28
Q2_K	9	226	225	221

Graded differently: questions right on one GPU and wrong on the other, at that rung.

Three things show in the grid:

Above Q4_K_M, the GPU moves nearly as many grades as the quantization. At Q5_K_M the two GPUs disagree on 7 questions; on the T4, Q5_K_M and BF16 disagree on 8 (3 lost, 5 gained). Q5_K_M's 231 on the T4 and 228 on the L4 sit either side of BF16's 229 on both.
Most losses are shared, often at a different rung. 222 questions are lost at some rung on both GPUs, 4 only on the T4 and 4 only on the L4. 180 of them are first lost at the same rung on both.
The deep losses are the stable ones. Q2_K's losses are 221 of 226 shared and Q3_K_M's 28 of 42. Between F16 and Q5_K_M, 7 of 12 T4 losses are lost again on the L4 at the same rung.

Step 7 — Google's QAT Q4_0 on the L4

./graveyard.py bury runs/2026-10-28-l4 BF16 QAT_Q4_0 Q4_0 --json graveyard-qat.json

File	Right	Lost	Cut off at 1,024	Median tok/s
QAT Q4_0	219	15	8	125.7
Q4_0	220	16	12	125.6

QAT Q4_0 gains 5 questions against BF16 and Q4_0 gains 7. The QAT file is 3,349,516,256 bytes and loads in 1,730 MiB; the plain Q4_0 is 3,360,161,216 bytes and loads in 1,740 MiB.

Step 8 — Speed and Memory on Each Card

Rung	Median tok/s, T4	Median tok/s, L4	L4 ÷ T4	× BF16, T4 / L4
BF16	40.0	50.1	1.25	1.00 / 1.00
F16	39.6	50.0	1.26	0.99 / 1.00
Q8_0	64.3	83.0	1.29	1.61 / 1.66
Q6_K	70.8	98.1	1.39	1.77 / 1.96
Q5_K_M	78.7	107.7	1.37	1.97 / 2.15
Q4_K_M	84.1	118.7	1.41	2.10 / 2.37
Q4_0	93.6	125.6	1.34	2.34 / 2.51
Q3_K_M	74.2	111.1	1.50	1.85 / 2.22
Q2_K	74.0	139.5	1.89	1.85 / 2.78

BF16 and F16 run at the same speed on both cards, within 0.4 tok/s of each other on each GPU.
The smaller the file, the bigger the L4's lead, up to 1.89 times at Q2_K. Q2_K is the fastest rung on the L4 and among the slowest on the T4; it is also wrong on 244 of 250 questions.
Q3_K_M is slower than Q4_K_M and Q4_0 on both cards.
The L4 uses 101 MiB more GPU memory at every rung (nvidia-smi after load: 4,772 against 4,671 MiB at BF16, 1,740 against 1,639 at Q4_0), the same fixed amount whatever the file size.

The L4 run took 4 hours 3 minutes from the start of the ladder build to the last scoring pass (run-l4.log), about $3.26 at the on-demand price (arithmetic).

Compare and Contrast

	Tesla T4	NVIDIA L4
BF16 right of 250	229	229
Q5_K_M right, × BF16 speed	🥇 231, 1.97	🥇 228, 2.15
Q4_0 right, median tok/s	222, 93.6	220, 125.6
Q3_K_M right	191	186
Same text twice on one GPU	🟢 yes	🟢 yes
GPU memory after load, Q4_0	1,639 MiB	1,740 MiB
Verdict	🟢 Q5_K_M, slower card	🟢 Q5_K_M, 1.37 times faster

So, Which One?

Summary

🟢 All nine GGUFs and the QAT file match the T4's sha256; BF16 on the L4 gives identical text twice on 50 questions, the one rung checked for determinism.
🟢 BF16 scores 229 of 250 on both GPUs, and the ladder keeps its shape: Q8_0 to Q5_K_M within 2, Q4_K_M and Q4_0 within 9.
🟢 Q5_K_M is the best trade on both: 228 right at 2.15 times BF16's speed on the L4.
🟢 The L4 decodes 1.25 to 1.89 times faster than the T4, and BF16 runs at F16's speed on both cards.
⚠️ The two GPUs grade 2 questions differently at BF16, 7 at Q5_K_M and 16 at Q4_0; above Q4_K_M that is nearly the size of the quantization effect.
⚠️ 180 of 226 lost questions are first lost at the same rung on both GPUs; 222 questions are lost on both at some rung.
⚠️ QAT Q4_0 scores 219 against the plain Q4_0's 220 on the L4, one apart again, at the same speed.
❌ Q3_K_M loses 43, 36 of its 45 losses cut off at 1,024 tokens; Q2_K gets 6 right, with 218 looping answers and 1 answer the server refused to parse.

The strategy for measuring Gemma 4's speed and accuracy one quantization rung at a time on two GPUs was validated with an incremental step by step approach.

References

Accuracy vs Speed: Gemma 4 E2B Quantization Tradeoffs, BF16 to Q2_K

xbill — Wed, 28 Oct 2026 16:02:20 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

This article provides a step by step guide to quantizing Gemma 4 E2B down a ladder of nine GGUF files with llama.cpp, asking every rung the same 250 maths questions on one Tesla T4, and burying each answer at the rung where it first goes wrong. Every count is computed by code and every row is committed.

https://github.com/xbill9/gemma-graveyard

A score per quantization level says how many answers a smaller file loses; a graveyard says which ones. Against the 229 of 250 questions BF16 gets right, Q8_0, Q6_K and Q5_K_M stay within 2, Q4_K_M and Q4_0 lose 5 and 7, Q3_K_M loses 38 and Q2_K gets 3 right. Above Q3_K_M most of the dead come back: 16 of the 23 questions that die between F16 and Q4_0 are right again at a lower rung. With thinking on, Q3_K_M climbs from 191 to 208.

What I Built

Quantization is the most common thing done to an open-weight model before it ships to a small device. The usual report is one accuracy number per file size. Two files with the same score can still get different questions wrong, and a net loss hides its parts: Q4_0 here is 7 answers down on BF16, made of 15 questions lost and 8 gained.

The gemma-graveyard asks every rung of a quantization ladder the same 250 questions at temperature 0, so each answer is the model's single most likely one. BF16 is the reference, and every question a rung answers differently from BF16 lands in one of four records:

Dead: BF16 answers it right and the rung answers it wrong. A rung's dead count is every such question at that rung.
Tombstone: one per question, placed at the first rung, going down from BF16, where the question dies. A question dead at Q8_0 and again at Q4_0 has one tombstone, at Q8_0.
Rose again: a lower rung answers a tombstoned question right once more. test-687 dies at F16 and is right again at Q8_0, Q6_K, Q5_K_M and Q4_K_M.
Ghost: BF16 answers it wrong and the rung answers it right. test-1166 asks for an avocado tree's fruit over ten years, answer 2,350. BF16 says 1,000; Q8_0, Q6_K and Q4_0 get it right.

A rung's score is BF16's 229, minus its dead, plus its ghosts: Q4_0 is 229 − 15 + 8 = 222. Quantization moves answers in both directions, and a score that holds can hide questions lost and found, so the article reports both.

For Día de los Muertos: the dead are remembered one at a time, by name. Each tombstone here carries the question's id, the right answer, the wrong answer the rung gave, and every lower rung where it rose again.

What the project delivers:

ladder.sh: BF16, F16 and seven quantized GGUFs of Gemma 4 E2B, with bytes and sha256
graveyard.py run: one rung, its own llama-server, 250 questions, answers graded in code
graveyard.py bury: deaths, tombstones and ghosts per rung, computed in code
graveyard.py autopsy: how each rung's answers fail (cut off, looping, writing into the thinking block)
graveyard.py repeat: two runs of one rung compared question by question
one run of the whole ladder plus Google's QAT Q4_0 on a Tesla T4, committed with its report
a second pass with thinking on, over seven rungs from BF16 to Q3_K_M

Demo

The Results in Brief

Q5_K_M is the best trade between speed and accuracy on the T4. It keeps BF16's score, 231 right against 229, at 39.0% of the bytes and 1.97 times BF16's decode speed. The 2 extra right answers come from the 8 questions where the two disagree, 5 ghosts against 3 dead; two equally accurate models split 8 questions at least that unevenly about three times in four, so the gap is within the noise. Q4_0 is the fastest rung at 2.34 times BF16, and costs 9 right answers against Q5_K_M for 1.19 times its speed. Q4_K_M sits between the two.

Every other rung is beaten on both counts by one of those three: Q8_0 and Q6_K are slower than Q5_K_M with fewer or equal right answers, and Q3_K_M is slower than Q4_0 with 31 fewer. Below Q4 the accuracy collapses: Q3_K_M loses 38 and Q2_K gets 3 right. With thinking on, the order holds: Q5_K_M 230, Q4_K_M 226 and Q4_0 219, against BF16's 227.

The cover is the graveyard. One row per rung, BF16 at the top and Q2_K at the bottom, one cell per question: marigold where the rung answers right, a grey headstone where it answers wrong. docs/make_cover.py draws it from the committed rows.

The ladder, from graveyard.py bury, on 250 GSM8K test questions:

Rung	% of BF16 bytes	Right	Dead	Ghosts	Median tok/s
BF16	100.0	229	0	0	40.0
F16	100.0	229	1	1	39.6
Q8_0	53.3	227	4	2	64.3
Q6_K	41.3	229	4	4	70.8
Q5_K_M	39.0	🥇 231	3	5	78.7
Q4_K_M	36.8	224	9	4	84.1
Q4_0	36.1	222	15	8	🥇 93.6
Q3_K_M	34.4	191	42	4	74.2
Q2_K	32.1	3	226	0	74.0

Dead: BF16 right, this rung wrong. Ghosts: BF16 wrong, this rung right.

Code

https://github.com/xbill9/gemma-graveyard

Path	What
`ladder.sh`	Converts the checkpoint to BF16 and F16 GGUF, quantizes seven rungs from BF16, writes `ladder.tsv`
`f16_range.py`	Counts the weights that overflow or underflow when BF16 is cast to F16
`graveyard.py`	`run`, `bury`, `autopsy` and `repeat`
`docs/think_derived.py`	Thinking on against thinking off, question by question
`lazy_eval.py`	Prompt, answer extraction and grading, reused from Week 3
`run.md`	Every command for the T4, in order, and how to resume after the VM stops
`runs/2026-10-27-ladder/run-ladder.sh`	Runs the nine rungs in order, skips finished ones, then buries
`runs/2026-10-27-ladder/`	`REPORT.md`, the bury and autopsy tables, per-question tombstones

How I Built It

At This Point You Should Have…

A GCE n1-standard-2 in us-west2-b with one Tesla T4, 15,360 MiB, driver 615.71.10, 2 vCPU and 7,436 MiB of RAM
llama.cpp built with CUDA for compute capability 7.5, at b11699
google/gemma-4-E2B-it from Hugging Face, the BF16 checkpoint, text model only
google/gemma-4-E2B-it-qat-q4_0-gguf, Google's quantization-aware Q4_0 file
A clone: git clone https://github.com/xbill9/gemma-graveyard

Step 1 — Build the Ladder

ladder.sh converts the checkpoint twice, to BF16 and to F16, then quantizes seven rungs from the BF16 file with plain llama-quantize, with no importance matrix. That is the default a user gets.

TMPDIR=/opt1/tmp ./ladder.sh /opt1/models/gemma-4-E2B-it /opt1/graveyard-gguf /opt1/llama.cpp
rung    file    bytes   sha256
BF16    /opt1/graveyard-gguf/e2b-BF16.gguf  9311305152  9fe40af9a72302151a5abc28a5057c5fe335eb80145e795ba745085fc73a2ad6
F16 /opt1/graveyard-gguf/e2b-F16.gguf   9311305152  5ab927e4b24a248de319e70467fc74d4ab87347ee9d6fb3ee60b293b12198cf8
Q8_0    /opt1/graveyard-gguf/e2b-Q8_0.gguf  4967497152  6b0301104c89a43c3bec5bae4b14d744f6000bc21e00e46e887fbbfe024de38a
Q6_K    /opt1/graveyard-gguf/e2b-Q6_K.gguf  3845346752  e800e0389f48646994d8ff6d3409dbf0c13b3fc217d0647b481385da33bd3d99
Q5_K_M  /opt1/graveyard-gguf/e2b-Q5_K_M.gguf    3630288320  f6c3f4395a9d3839535d8c14ae48ab784096945cfc4485005853eede915cb4a9
Q4_K_M  /opt1/graveyard-gguf/e2b-Q4_K_M.gguf    3427880384  dd9dc465bb535989205f158d77f28f6f8348e47d8ac8870920c152b5996cbc16
Q4_0    /opt1/graveyard-gguf/e2b-Q4_0.gguf  3360161216  2d84758474be87d0a74edfa98efbeb602484e26f2e5fa8695fd9641ad323c7b2
Q3_K_M  /opt1/graveyard-gguf/e2b-Q3_K_M.gguf    3201351104  b10c1f38c93635ccee1c7f382e03f5d2f4e9516087d7d62a73ca7de0f52693ab
Q2_K    /opt1/graveyard-gguf/e2b-Q2_K.gguf  2989088192  65850f1a31efed2013ed148adc5dc937e973a8e79fd55cdba31edd53369fddb2

The reference is BF16 because that is the checkpoint's own dtype, so the BF16 GGUF holds the trained weights unchanged. F16 is a rung.

Step 2 — Check What F16 Costs Before Running It

F16 has more mantissa than BF16 and less exponent range, so the cast can overflow or round small weights to zero. f16_range.py counts both on the checkpoint, one line per tensor and a total:

python3 f16_range.py /opt1/models/gemma-4-E2B-it/model.safetensors > runs/2026-10-27-f16-range/out.txt
grep -E 'position_embedding_table|TOTAL' runs/2026-10-27-f16-range/out.txt
{"tensor": "model.vision_tower.patch_embedder.position_embedding_table", "n": 15728640, "overflow": 0, "underflow": 5507733, "subnormal": 834716, "absmax": 0.431640625}
{"tensor": "TOTAL", "tensors": 2011, "n": 5123178979, "overflow": 0, "underflow": 5523265, "subnormal": 7013885, "absmax": 900.0}

No weight overflows. Of the 5,523,265 that underflow, 5,507,733 are in the vision tower's position table, which the text GGUF does not carry.

Step 3 — Ask One Rung 250 Questions

graveyard.py run starts its own llama-server on one GGUF and asks each question once: temperature 0, one request at a time, no prompt cache, thinking off, 1,024 max tokens. Only the reply's content is graded, in code: the number after Answer:, compared with the reference.

./graveyard.py run BF16 /opt1/graveyard-gguf/e2b-BF16.gguf data/heldout.jsonl runs/2026-10-27-ladder --llama /opt1/llama.cpp
{"rung": "BF16", "n": 250, "right": 229, "errors": 0, "truncated": 4, "median_decode_tps": 40.0, "mean_tokens": 391.2}

The whole model fits on the T4 at BF16:

load_tensors: offloaded 36/36 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   768.00 MiB
load_tensors:        CUDA0 model buffer size =  4384.87 MiB

🔎 Tip: Gemma 4 E2B keeps its embedding tables in CPU memory. The per-layer embedding table holds 2,348,810,240 of the checkpoint's weights, and llama.cpp leaves it memory-mapped on the CPU with -ngl 99. So each rung's GPU memory is about half its file size: 4,671 MiB in use at BF16 for a 9.3 GB file, 1,639 MiB at Q4_0, 1,287 MiB at Q2_K.

Step 4 — Prove the Same Rung Gives the Same Answers

A tombstone means something only if the same model on the same question always gives the same answer. BF16 on 50 questions, twice:

for i in 1 2; do
  ./graveyard.py run BF16 /opt1/graveyard-gguf/e2b-BF16.gguf data/heldout.jsonl runs/2026-10-27-determinism-$i \
    --llama /opt1/llama.cpp --limit 50
done

The two runs compared in code, question by question:

0 mismatched extracted, 0 mismatched text, 46/50 both times

Every answer and every character of text matched. The same holds below BF16. graveyard.py repeat compares a second full run of four rungs with the ladder's:

./graveyard.py repeat runs/2026-10-27-ladder runs/2026-10-28-repeat Q4_0 Q3_K_M Q8_0 F16 --common
{"rung": "Q4_0", "n": 250, "mismatched_extracted": 0, "mismatched_text": 0, "mismatched_right": 0, "right_a": 222, "right_b": 222}
{"rung": "Q3_K_M", "n": 250, "mismatched_extracted": 0, "mismatched_text": 0, "mismatched_right": 0, "right_a": 191, "right_b": 191}
{"rung": "Q8_0", "n": 250, "mismatched_extracted": 0, "mismatched_text": 0, "mismatched_right": 0, "right_a": 227, "right_b": 227}
{"rung": "F16", "n": 250, "mismatched_extracted": 0, "mismatched_text": 0, "mismatched_right": 0, "right_a": 229, "right_b": 229}

A changed answer at a lower rung comes from the weights.

Step 5 — Run the Ladder and Bury It

runs/2026-10-27-ladder/run-ladder.sh runs the nine rungs in order and skips any rung whose results are complete, so it picks up again after the VM's evening stop. Then bury:

./graveyard.py bury runs/2026-10-27-ladder BF16 F16 Q8_0 Q6_K Q5_K_M Q4_K_M Q4_0 Q3_K_M Q2_K --json graveyard-ladder.json
{"first_deaths_by_rung": {"Q2_K": 177, "Q3_K_M": 26, "Q4_0": 9, "Q8_0": 4, "Q5_K_M": 2, "Q4_K_M": 6, "Q6_K": 1, "F16": 1}, "tombstones": 226, "ghosts": 11}

226 tombstones and 11 ghosts: 226 different questions die at some rung, and 11 questions BF16 gets wrong come back right at one rung or more. The first-death counts add up to the tombstones, one per question; the per-rung dead counts in the ladder table add up to more, because one question can be dead at several rungs. Per-question detail is in graveyard-ladder.json. The one question that dies at F16:

{'id': 'test-687', 'died_at': 'F16', 'answer': '13', 'said': 17.0, 'right_again_at': ['Q8_0', 'Q6_K', 'Q5_K_M', 'Q4_K_M']}

"One twin is 7 years older than one triplet. If their combined ages are 44, how old is one of the twins?" BF16 answers 13, from two twins and three triplets. F16 adds up two twins and one triplet, and answers 17. Four lower rungs answer 13 again.

Step 6 — Autopsy: How Each Rung Dies

graveyard.py autopsy counts, per rung, answers cut off at 1,024 tokens, answers that repeat one line five or more times, and answers that put text in the thinking block:

./graveyard.py autopsy runs/2026-10-27-ladder BF16 F16 Q8_0 Q6_K Q5_K_M Q4_K_M Q4_0 Q3_K_M Q2_K
{"rung": "Q3_K_M", "cut_off": 49, "looped": 0, "thought": 0, "empty_content": 0, "empty_right_in_reasoning": 0, "dead": 42, "dead_cut_off": 34, "dead_looped": 0, "dead_empty_content": 0, "first_deaths": 26, "first_deaths_right_again_lower": 0}
{"rung": "Q2_K", "cut_off": 246, "looped": 228, "thought": 75, "empty_content": 63, "empty_right_in_reasoning": 2, "dead": 226, "dead_cut_off": 222, "dead_looped": 207, "dead_empty_content": 60, "first_deaths": 177, "first_deaths_right_again_lower": 0}

Every rung:

Rung	Cut off	Looped	Thinking unprompted	Dead	Dead, cut off
F16	4	0	0	1	0
Q8_0	9	0	0	4	2
Q6_K	4	0	0	4	1
Q5_K_M	3	0	0	3	1
Q4_K_M	6	0	0	9	3
Q4_0	10	0	0	15	8
Q3_K_M	49	0	0	42	34
Q2_K	246	228	75	226	222

Three kinds of death:

F16 to Q4_0, a different road through the same problem. 23 first deaths; 16 of them are right again at some lower rung. Down to Q5_K_M most of the dead answers end normally with a wrong number: 4 of 12 were cut off. At Q4_K_M and Q4_0, 11 of 24 were. From F16 to Q5_K_M, ghosts (1 to 5 per rung) match deaths (1 to 4), so the score holds. From Q4_K_M down, deaths outnumber ghosts, 9 to 4 and 15 to 8.
Q3_K_M, out of room. 34 of its 42 deaths were cut off at 1,024 tokens, still working, with no repeated lines. BF16 used a median 466.5 tokens on the 44 questions Q3_K_M got wrong while cut off.
Q2_K, collapse. 246 of 250 answers run to the limit and 228 repeat a line five or more times. With thinking off, 75 answers open a thinking block anyway, and in 63 of them the whole answer lands there, leaving content empty. No other rung writes into the thinking block at all.

🔎 Tip: give two buries in one directory two names. bury writes its per-question tombstones to graveyard.json; --json picks the file, so the QAT comparison in Step 7 keeps the ladder's file intact.

Step 7 — Google's QAT Q4_0 Beside the Plain Q4_0

Google publishes a Q4_0 GGUF of Gemma 4 E2B trained for 4-bit (quantization-aware training). Its weights differ from the base checkpoint, so it sits beside the ladder, against the same BF16 reference:

./graveyard.py bury runs/2026-10-27-ladder BF16 QAT_Q4_0 Q4_0 --json graveyard-qat.json
{"rung": "QAT_Q4_0", "n": 250, "right": 221, "errors": 0, "truncated": 6, "median_decode_tps": 87.8, "mean_tokens": 391.9, "dead": 15, "ghosts": 7, "answer_changed": 27, "bytes": 3349516256, "gpu_mib_loaded": 1629, "gpu_mib_before": 0}
{"rung": "Q4_0", "n": 250, "right": 222, "errors": 0, "truncated": 10, "median_decode_tps": 93.6, "mean_tokens": 430.4, "dead": 15, "ghosts": 8, "answer_changed": 27, "bytes": 3360161216, "gpu_mib_loaded": 1639, "gpu_mib_before": 0}

Same number of deaths, same number of changed answers, one right answer apart. The graves differ: 6 of the 15 dead are the same question, 9 die only under QAT and 9 only under the plain Q4_0. QAT runs out of room less often: 2 of its deaths were cut off at 1,024 tokens, against 8 for the plain Q4_0.

Step 8 — Bury the Ladder Again With Thinking On

Gemma 4 can think before it answers. --think turns it on; the thinking goes to reasoning_content and only content is graded. Thinking needs room, so this pass allows 4,096 tokens in an 8,192-token context. It covers seven rungs, BF16 to Q3_K_M; F16, Q2_K and QAT Q4_0 were not run with thinking on. Each rung is buried against BF16 with thinking on, which gets 227 right. A second BF16 run of 50 questions matched the first exactly, 47 right both times.

./graveyard.py bury runs/2026-10-28-think BF16 Q8_0 Q6_K Q5_K_M Q4_K_M Q4_0 Q3_K_M --json graveyard.json
{"rung": "BF16", "n": 250, "right": 227, "errors": 0, "truncated": 0, "median_decode_tps": 34.5, "mean_tokens": 842.2, "dead": 0, "ghosts": 0, "answer_changed": 0, "bytes": 9311305152, "gpu_mib_loaded": 4699, "gpu_mib_before": 0}
{"rung": "Q3_K_M", "n": 250, "right": 208, "errors": 0, "truncated": 11, "median_decode_tps": 68.0, "mean_tokens": 1320.1, "dead": 26, "ghosts": 7, "answer_changed": 42, "bytes": 3201351104, "gpu_mib_loaded": 1517, "gpu_mib_before": 0}

docs/think_derived.py lines the two passes up question by question:

python3 docs/think_derived.py | tail -3
 "q3_dead_cut_off_with_thinking_off": 34,
 "of_them_right_with_thinking_on": 21
}

Rung	Right, thinking off	Right, thinking on	Dead, off	Dead, on	Dead in both
BF16	229	227	—	—	—
Q8_0	227	🥇 230	4	2	0
Q6_K	229	🥇 230	4	6	0
Q5_K_M	231	🥇 230	3	6	0
Q4_K_M	224	226	9	9	1
Q4_0	222	219	15	15	3
Q3_K_M	191	208	42	26	12

Three things change:

Q3_K_M recovers most of its loss. It gains 17 right answers. Of the 34 questions it lost by running out of room at 1,024 tokens, 21 are right with thinking on and 4,096 tokens. Both the thinking and the larger limit changed at once, so this run cannot say which one did it.
Above Q3_K_M the score holds and the graves move. Each rung stays within 3 answers of its thinking-off score, but the questions it loses are almost all new: 0 of the deaths at Q8_0, Q6_K and Q5_K_M are shared between the two passes, 1 of 9 at Q4_K_M and 3 of 15 at Q4_0. BF16 itself gets 218 questions right in both modes.
The cost is time. Answers use 2.08 to 2.18 times as many tokens, decoding runs at 0.79 to 0.93 of the thinking-off speed, and each rung takes 2.22 to 2.70 times as long on the T4.

Compare and Contrast

	Q5_K_M	Q4_K_M	Q4_0	QAT Q4_0	Q3_K_M
Right of 250 (BF16: 229)	🥇 231	🥈 224	🥉 222	221	191
Dead	🥇 3	🥈 9	🥉 15	🥉 15	42
GPU MiB	1,897	1,705	🥉 1,639	🥈 1,629	🥇 1,489
Median tok/s on the T4	78.7	🥉 84.1	🥇 93.6	🥈 87.8	74.2
Answers cut off	🥇 3	🥈 6	10	🥈 6	49
Verdict	🟢 same score as BF16	🟢 small loss	⚠️ 15 dead	⚠️ 15 dead, fewer cut off	❌ 38 fewer right

So, Which One?

To ship Gemma 4 E2B on a small GPU with BF16's score, take Q5_K_M: 39.0% of the bytes, 231 right against 229, and 1.97 times BF16's decode speed on the T4 (arithmetic from the medians). For the fastest file that stays within 7 answers, take Q4_0 or Q4_K_M. Below Q4, the losses stop being a few flipped answers: Q3_K_M saves 150 MiB of GPU memory over Q4_0 and costs 31 right answers, and Q2_K stops answering. With thinking on and 4,096 tokens, Q3_K_M gets 208 right, 11 short of Q4_0, at more than twice the time per question.

Prior Work

Prompt, answer extraction and grading (lazy_eval.py) and the 250-question GSM8K split (data/, seed 0): from my Week 3 entry, lazy-gemma, whose grader came from my earlier Gemma 4 TPU work.
Open source used: llama.cpp (MIT) for conversion, quantization and serving; Gemma 4 E2B and Google's QAT Q4_0 GGUF (Gemma terms); GSM8K.

Why Does Open Innovation Matter?

A graveyard needs the same model at nine sizes, run the same way.

The weights are on the disk, so the ladder is built here. ladder.sh produces every rung from one BF16 file with the open converter and quantizer, and ladder.tsv records each file's sha256. A hosted API serves whichever quantization its provider chose, and it can change without notice.
The same answer twice. Greedy decoding on a fixed llama.cpp build gave identical text on every repeated run, BF16 to Q3_K_M, so every tombstone counts. That check needs control of the server, the seed and the build.
The failure is visible. Q2_K's answers in the thinking block show up only because llama-server returns reasoning_content and content separately and every row is kept.
No per-token bill. Ten models times 250 questions ran on one rented T4.

The Spirit of It

Every row in the graveyard is a grade-school maths problem, and each tombstone keeps its own epitaph:

test-687, twins and triplets, answer 13. Died at F16, said 17. Rose again at Q8_0, Q6_K, Q5_K_M and Q4_K_M.
test-2, a house flipped for profit, answer 70,000. Died at Q4_0, said 120,000. Never rose.
test-564, Judy's two weeks of reading, answer 240. Died at Q8_0, said 3. Rose again at every rung from Q6_K to Q4_0.

Most graves above Q3_K_M have someone come back to them: 16 of 23.

Prize Categories

Gemma: Gemma 4 E2B, quantized locally with llama.cpp at nine levels and compared with Google's quantization-aware Q4_0 release, served on one Tesla T4.

Summary

The goal of this project was to find which answers Gemma 4 E2B loses at each quantization level, one question at a time. The key was a deterministic run: greedy decoding on one llama.cpp build, checked to give identical text twice at five rungs, so every changed answer comes from the weights. The results were:

🟢 Q8_0, Q6_K and Q5_K_M stay within 2 right answers of BF16's 229 of 250; Q5_K_M scores 231 at 39.0% of the bytes.
🟢 Every rung fits on the T4, BF16 included: all 36 layers on the GPU, 4,671 MiB in use, with the embedding tables in CPU memory.
🟢 Above Q3_K_M, 16 of the 23 questions that die are right again at a lower rung.
⚠️ Q4_K_M and Q4_0 lose 5 and 7, with 9 and 15 deaths.
⚠️ Google's QAT Q4_0 scores 221 against the plain Q4_0's 222; the two share 6 of their 15 deaths.
🟢 Q4_0, Q3_K_M, Q8_0 and F16 run a second time give the same text on all 250 questions.
⚠️ With thinking on, rungs above Q3_K_M keep their score but lose different questions; at Q4_0, 3 of 15 deaths are shared.
⚠️ Thinking on costs 2.1 times the tokens and 2.2 to 2.7 times the time per rung.
❌ Q3_K_M loses 38; 34 of its 42 deaths are cut off at 1,024 tokens. With thinking on and 4,096 tokens it loses 19.
❌ Q2_K gets 3 right: 246 answers hit the token limit, 228 loop, and 75 write into a thinking block with thinking off.

Scope: one Tesla T4 on GCE, llama.cpp b11699, 250 GSM8K test questions per rung, greedy. Thinking off with a 1,024-token limit on all ten models; thinking on with a 4,096-token limit on seven rungs, BF16 to Q3_K_M. Determinism checked at BF16, F16, Q8_0, Q4_0 and Q3_K_M. No importance matrix, none of the other 1,069 GSM8K test questions, and no other machine. Speeds compare rungs on this T4 only.

The strategy for burying Gemma 4's answers one quantization rung at a time was validated with an incremental step by step approach.

References

Size Your Model to Where You Run It: Gemma 4 E2B and E4B on an AWS L4 and Two Small GPUs

xbill — Tue, 27 Oct 2026 13:30:55 +0000

This article provides a step by step guide to serving Gemma 4 E2B and E4B, Google's quantization-aware trained (QAT) small models, on an NVIDIA L4 in an AWS EC2 G6 instance with llama.cpp, and compares the result with the same models on two small local GPUs: an 8 GB RTX 3050 desktop card and a 4 GB GTX 1650 Ti laptop card. Every card runs the same llama.cpp build, the same model files, the same drafter and the same tests. The measures are generation speed for one user and for many, what the drafter adds on each card and why, answer quality, and cost.

https://github.com/xbill9/gemma4-dev

For one user, the 24 GB L4 is 1.2-1.3x an 8 GB desktop card, with the same answers. The 4-bit small models use 13-29% of the L4's memory, and the rest of the card sits idle:

Gemma 4, 4-bit QAT, one user	GTX 1650 Ti, 4 GB laptop	RTX 3050 OEM, 8 GB desktop	NVIDIA L4, 24 GB, AWS
E2B, no drafter	76.5 tok/s	108.2 tok/s	133.4 tok/s
E2B, best drafter setting	90.3 tok/s	151.2 tok/s	217.6 tok/s
E4B, no drafter	38.0 tok/s	58.3 tok/s	74.0 tok/s
E4B, best drafter setting	56.5 tok/s	100.0 tok/s	129.7 tok/s
3050 as a share of the L4		69-81%

The desktop card delivers 79-81% of the L4 without the drafter and 69-77% with it. The L4 earns its memory with many users at once (1.6x the 3050 at 64) and with models too large for 8 GB. The rule this article arrives at: size the model to where it runs.

What Is Being Measured?

Gemma 4 E2B and E4B are the two small models of the family, served here as exact 4-bit (Q4_0) GGUF files whose every block sits on Google's QAT grid: xbill9/gemma-4-E2B-it-qat-q4_0-exact-gguf and xbill9/gemma-4-E4B-it-qat-q4_0-exact-gguf. Google also ships a QAT drafter for each, a small model that proposes the next few tokens for the main model to check in one pass; it is stored the same way, as xbill9/gemma-4-E{2,4}B-it-qat-assistant-q4_0-exact-gguf.

When one user is generating, every token reads every weight of the model once. The speed limit is then how fast the card reads its memory, and the size of the memory plays no part as long as the model fits. That is why a card with three times the memory can be only a little faster.

The three cards:

	GTX 1650 Ti Max-Q	RTX 3050 OEM	NVIDIA L4
Memory	4 GB	8 GB	24 GB
Host	laptop	i7-5820K desktop	EC2 `g6.2xlarge`, 8 vCPU
Where	local	local	AWS `us-east-1`, spot

All three run llama.cpp 77c4540ce with flash attention, a 16-bit context cache and the per-layer embedding table on the card. The L4 build uses CUDA 13.1 for sm_89.

Step 1 — Serve the Rigs

Each model has a rig directory with a tpu.env holding every setting and a Makefile to run on the instance:

git clone https://github.com/xbill9/gemma4-dev
cd gemma4-dev/gpu-llamacpp-g6-2b-q4_0      # or gpu-llamacpp-g6-4b-q4_0
make toolkit build download draft serve

The G6 instance comes from gpu-pytorch-g6-2b-q4w4a16's MCP tool create_g6_instance, on the PyTorch Deep Learning AMI. The AMI has an NVIDIA driver (595.112.07, CUDA 13.2) and no CUDA compiler, so make toolkit installs CUDA 13.1 from NVIDIA's apt repository, one minor version below what the driver supports. make build compiles llama.cpp for sm_89 in 633 s.

The serving line, E2B:

llama-server -m gemma-4-E2B-it-q4_0-exact.gguf -ngl 99 -c 131072 -ctk f16 -ctv f16 -fa 1 \
  -t 4 -tb 8 --parallel 1 --reasoning off --metrics -ot 'per_layer_token_embd\.weight=CUDA0' \
  -md gemma-4-E2B-it-qat-assistant-q4_0-exact.gguf --spec-type draft-mtp -ngld all --spec-draft-n-max 2

At the full 131,072-token context with the drafter, E2B peaks at 3,952 MiB and E4B at 6,774 MiB of the L4's 23,034.

Step 2 — Measure One User

speed.py sends six prompts (code, chat, story, explanation, facts, summary), 256 tokens each, three greedy repeats per prompt with the prompt cache off, and reports the geometric mean over prompts. Each arm gets its own server, started on a cooled card. On the L4:

drafted tokens	E2B tok/s	x none	E4B tok/s	x none	tokens kept
none	133.4	1.000x	74.0	1.000x	—
1	185.5	1.391x	106.8	1.443x	0.60-0.62
2	211.0	1.582x	121.8	1.646x	0.48-0.50
3	217.6	1.631x	129.7	1.754x	0.40
4	196.6	1.474x	129.7	1.754x	0.30-0.35

A no-drafter arm run last lands 0.7% (E2B) and 1.2% (E4B) under the first. Code gains most from the drafter (2.39x E2B, 2.55x E4B at 3 tokens) and stories least (1.31x, 1.38x).

With the drafter, E4B runs at 0.972x E2B without it: 129.7 against 133.4 tok/s.

The same prompts on the two local cards, best setting per card:

	1650 Ti	3050	L4
E2B best	1 token, 1.18x	2 tokens, 1.40x	3 tokens, 1.63x
E4B best	1 token, 1.49x	2 tokens, 1.72x	3 tokens, 1.75x

🔎 Tip: The Card Sets the Draft Length

The drafter's guesses are kept at the same rate on every card. What changes is what each guess costs. Splitting each reply into steps (one pass of the main model over the last token plus the drafted ones) shows it, here for E2B:

drafted tokens	tokens per step, all three cards	step time vs a plain step: 1650 Ti	3050	L4
1	1.60	1.35x	1.16x	1.15x
2	1.92-1.98	1.77x	1.39x	1.26x
3	2.12-2.20	2.12x	1.63x	1.36x

The speed-up is tokens per step divided by step time. On the L4 llama-bench times the main model over 1 to 6 tokens in one pass: 4 tokens take at most 1.04x the time of 1, at 0, 8,192 and 32,768 tokens of context. So checking 3 guesses is close to free there, and the added time is the drafter's own, 0.75-1.18 ms per guess. That is 33% of an E2B step at 3 guesses and 19% of E4B's longer step, which is why E4B gains more from its drafter. A slower card pays more per guess and peaks at fewer.

Step 3 — Hold a Long Conversation

longctx.py puts the first D tokens of a long document ahead of a story, a code and a summary request, at -c 131072. Mean speed-up over the requests at 2 drafted tokens:

tokens ahead	E2B, L4	E2B, 3050	E4B, L4	E4B, 3050
0	1.70x	1.51x	1.75x	1.86x
32,768	1.53x		1.69x	1.64x
120,000	1.49x	1.21x	1.57x	1.55x

Generation without the drafter slows with context (E2B story 134.1 tok/s empty, 66.7 at 120,000 ahead on the L4), and the drafter keeps paying at every depth on both cards.

Step 4 — Serve Many Users at Once

mixed.py sends c requests at once, each one of the six chat prompts, 256 tokens, five repeats, one shared context cache (-kvu) at -c 32768 --parallel 64. Total output tok/s:

requests at once	E2B, 3050	E2B, L4	E4B, 3050	E4B, L4
1	106.1	131.4	57.3	73.5
8	297.1	505.2	199.3	309.7
64	569.5	918.9	472.4	778.1

This is where the L4's size shows: 1.61-1.65x the 3050 at 64 requests. The drafter (2 tokens) helps on the L4 up to 16 requests (E2B 1.06-1.19x, E4B 1.15-1.34x) and costs output at 64 (0.87x for both).

Step 5 — Check the Answers

gen_eval.py runs GSM8K (1,319 grade-school math questions, up to 2,048 tokens per reply) and BFCL simple (400 single function calls) at -c 32768, 8 requests at once, with and without the drafter at 2 tokens:

L4	GSM8K, none / drafter	BFCL simple, none / drafter	GSM8K time, none / drafter
E2B	0.8992 / 0.8984	0.930 / 0.925	1,014 s / 713 s
E4B	0.9310 / 0.9287	0.910 / 0.915	1,416 s / 893 s

Paired question by question, every drafter-against-none difference has a 95% range that includes zero. The 3050 scored E2B 0.8984 / 0.9022 and E4B 0.9340 / 0.9325 on GSM8K under the same settings: the same model files give the same answers on every card.

E4B answers more GSM8K questions right than E2B, +0.0318 without the drafter (range +0.0167 to +0.0470) and +0.0303 with it, and BFCL is level between the two. With its drafter E4B finishes GSM8K in 893 s, sooner than E2B without one (1,014 s).

Step 6 — Price It

The L4 instance, g6.2xlarge in us-east-1, listed at $0.9776/h on demand and $0.8393/h spot on the day of the runs. Per million output tokens, one machine kept busy:

L4, spot	tok/s	$ per million tokens
E2B, one user, no drafter	133.4	$1.75
E2B, one user, drafter	217.6	$1.07
E4B, one user, drafter	129.7	$1.80
E2B, 64 users	918.9	$0.25
E4B, 64 users	778.1	$0.30

One user pays for the whole card while using 13-29% of its memory. Filled with 64 requests, the same card costs a sixth to a seventh as much per token. A local card's cost is the card and its electricity, neither measured here.

🔎 Tip: Where the Bigger Card Earns Its Keep

The 8 GB 3050 runs every larger Gemma 4 too, at a price paid in speed once the model outgrows the card. From the same series on the same card:

on the RTX 3050, 8 GB	where the model lives	generation, with drafter
12B, 6.7 GB	all on the card	55.1 tok/s
26B A4B, 14.4 GB	experts of 21 of 30 layers in system RAM	33.7 tok/s
31B, 17.7 GB	40 of 60 layers in system RAM	4.14 tok/s

All three fit a 24 GB card whole; this article measures them on the 3050 only.

Compare and Contrast

Gemma 4 E2B / E4B, 4-bit QAT	🥇 RTX 3050, 8 GB, local	🥈 NVIDIA L4, 24 GB, AWS	🥉 GTX 1650 Ti, 4 GB, local
One user, E2B / E4B, best	151.2 / 100.0 tok/s	217.6 / 129.7 tok/s	90.3 / 56.5 tok/s
64 users, E2B / E4B	569.5 / 472.4 tok/s	918.9 / 778.1 tok/s
Best draft length	2 tokens	3 tokens	1 token
GSM8K, E4B with drafter	0.9325	0.9287
Cost	owned	$0.84/h spot	owned

So, Which One?

For one user on E2B or E4B, a local 8 GB card. It delivers 69-81% of the L4's speed with the same answers, and the model uses a fraction of either card.

For many users at once, the L4. At 64 requests it serves 1.6x the 3050's output, and at about $0.25-0.30 per million tokens on spot.

For quality at small-model speed, E4B with its drafter. On the L4 it generates at 0.97x E2B without a drafter and scores 3 points higher on GSM8K.

For the 26B and 31B, a card that holds them. On 8 GB they spill into system RAM and slow to 34 and 4 tok/s.

Summary

The goal of this article was to find what a 24 GB cloud GPU buys over small local GPUs when serving Gemma 4's small models. The key to the answer was running the same llama.cpp build, model files, drafter and tests on all three cards. The results were:

🟢 One user: the L4 is 1.23-1.27x the 3050 without the drafter and 1.30-1.44x with it, at 13-29% of its memory
🟢 The same model files give the same answers on every card; the drafter costs no GSM8K or BFCL answers
🟢 The drafter's guesses are kept at the same rate on every card; the card's cost per guess sets the best draft length, 1 / 2 / 3 tokens on the 1650 Ti / 3050 / L4
🟢 E4B with its drafter generates at 0.97x E2B without one, 3 GSM8K points higher
🟢 64 users: the L4 serves 1.6x the 3050, $0.25-0.30 per million tokens on spot
⚠️ The drafter helps on the L4 up to 16 users at once and costs output at 64
❌ One user on the L4 pays for a whole card and uses under a third of its memory

Scope: Gemma 4 E2B-it and E4B-it QAT only, exact Q4_0 GGUF files and exact Q4_0 QAT drafters, text only, llama.cpp 77c4540ce on all three cards, 16-bit context cache, thinking off. One L4 instance on one day (spot g6.2xlarge, us-east-1d); the 3050 (i7-5820K desktop) and 1650 Ti (laptop) runs are from 2026-10-14 and 2026-10-15 on different hosts with the same prompts and scripts, so cross-card ratios are indicative. The many-users test uses short chat prompts and a shared context cache. Larger cloud GPUs and the 12B, 26B and 31B on the L4 were not run.

The strategy for using MCP for local accelerator comparison was validated with an incremental step by step approach.

References

Part 2, Do Drafters Work on Small Models?: https://dev.to/gde/do-drafters-work-on-small-models-gemma-4-e2b-and-e4b-on-two-small-gpus-1b4g
Part 6, A Dense 31B on an 8 GB GPU: https://dev.to/gde/a-dense-31b-on-an-8-gb-gpu-gemma-4-31b-on-an-rtx-3050-26p9
Repository: https://github.com/xbill9/gemma4-dev
E2B on the L4: https://github.com/xbill9/gemma4-dev/blob/main/gpu-llamacpp-g6-2b-q4_0/benchmarks/runs/2026-10-26-drafter-g6/REPORT.md
E4B on the L4: https://github.com/xbill9/gemma4-dev/blob/main/gpu-llamacpp-g6-4b-q4_0/benchmarks/runs/2026-10-26-drafter-g6/REPORT.md
Drafter cost and many users on the L4: https://github.com/xbill9/gemma4-dev/blob/main/gpu-llamacpp-g6-2b-q4_0/benchmarks/runs/2026-10-26-drafter-cost-g6/REPORT.md
E2B GGUF: https://huggingface.co/xbill9/gemma-4-E2B-it-qat-q4_0-exact-gguf
E4B GGUF: https://huggingface.co/xbill9/gemma-4-E4B-it-qat-q4_0-exact-gguf
E2B drafter: https://huggingface.co/xbill9/gemma-4-E2B-it-qat-assistant-q4_0-exact-gguf
E4B drafter: https://huggingface.co/xbill9/gemma-4-E4B-it-qat-assistant-q4_0-exact-gguf
llama.cpp: https://github.com/ggml-org/llama.cpp
Amazon EC2 G6 instances: https://aws.amazon.com/ec2/instance-types/g6/

Size Your Model to Where You Run It: Gemma 4 E2B and E4B on an AWS L4 and Two Small GPUs

xbill — Tue, 27 Oct 2026 13:30:52 +0000

https://github.com/xbill9/gemma4-dev

For one user, the 24 GB L4 is 1.2-1.3x an 8 GB desktop card, with the same answers. The 4-bit small models use 13-29% of the L4's memory, and the rest of the card sits idle:

Gemma 4, 4-bit QAT, one user	GTX 1650 Ti, 4 GB laptop	RTX 3050 OEM, 8 GB desktop	NVIDIA L4, 24 GB, AWS
E2B, no drafter	76.5 tok/s	108.2 tok/s	133.4 tok/s
E2B, best drafter setting	90.3 tok/s	151.2 tok/s	217.6 tok/s
E4B, no drafter	38.0 tok/s	58.3 tok/s	74.0 tok/s
E4B, best drafter setting	56.5 tok/s	100.0 tok/s	129.7 tok/s
3050 as a share of the L4		69-81%

What Is Being Measured?

The three cards:

	GTX 1650 Ti Max-Q	RTX 3050 OEM	NVIDIA L4
Memory	4 GB	8 GB	24 GB
Host	laptop	i7-5820K desktop	EC2 `g6.2xlarge`, 8 vCPU
Where	local	local	AWS `us-east-1`, spot

All three run llama.cpp 77c4540ce with flash attention, a 16-bit context cache and the per-layer embedding table on the card. The L4 build uses CUDA 13.1 for sm_89.

Step 1 — Serve the Rigs

Each model has a rig directory with a tpu.env holding every setting and a Makefile to run on the instance:

git clone https://github.com/xbill9/gemma4-dev
cd gemma4-dev/gpu-llamacpp-g6-2b-q4_0      # or gpu-llamacpp-g6-4b-q4_0
make toolkit build download draft serve

The serving line, E2B:

llama-server -m gemma-4-E2B-it-q4_0-exact.gguf -ngl 99 -c 131072 -ctk f16 -ctv f16 -fa 1 \
  -t 4 -tb 8 --parallel 1 --reasoning off --metrics -ot 'per_layer_token_embd\.weight=CUDA0' \
  -md gemma-4-E2B-it-qat-assistant-q4_0-exact.gguf --spec-type draft-mtp -ngld all --spec-draft-n-max 2

At the full 131,072-token context with the drafter, E2B peaks at 3,952 MiB and E4B at 6,774 MiB of the L4's 23,034.

Step 2 — Measure One User

drafted tokens	E2B tok/s	x none	E4B tok/s	x none	tokens kept
none	133.4	1.000x	74.0	1.000x	—
1	185.5	1.391x	106.8	1.443x	0.60-0.62
2	211.0	1.582x	121.8	1.646x	0.48-0.50
3	217.6	1.631x	129.7	1.754x	0.40
4	196.6	1.474x	129.7	1.754x	0.30-0.35

A no-drafter arm run last lands 0.7% (E2B) and 1.2% (E4B) under the first. Code gains most from the drafter (2.39x E2B, 2.55x E4B at 3 tokens) and stories least (1.31x, 1.38x).

With the drafter, E4B runs at 0.972x E2B without it: 129.7 against 133.4 tok/s.

The same prompts on the two local cards, best setting per card:

	1650 Ti	3050	L4
E2B best	1 token, 1.18x	2 tokens, 1.40x	3 tokens, 1.63x
E4B best	1 token, 1.49x	2 tokens, 1.72x	3 tokens, 1.75x

🔎 Tip: The Card Sets the Draft Length

drafted tokens	tokens per step, all three cards	step time vs a plain step: 1650 Ti	3050	L4
1	1.60	1.35x	1.16x	1.15x
2	1.92-1.98	1.77x	1.39x	1.26x
3	2.12-2.20	2.12x	1.63x	1.36x

Step 3 — Hold a Long Conversation

longctx.py puts the first D tokens of a long document ahead of a story, a code and a summary request, at -c 131072. Mean speed-up over the requests at 2 drafted tokens:

tokens ahead	E2B, L4	E2B, 3050	E4B, L4	E4B, 3050
0	1.70x	1.51x	1.75x	1.86x
32,768	1.53x		1.69x	1.64x
120,000	1.49x	1.21x	1.57x	1.55x

Generation without the drafter slows with context (E2B story 134.1 tok/s empty, 66.7 at 120,000 ahead on the L4), and the drafter keeps paying at every depth on both cards.

Step 4 — Serve Many Users at Once

mixed.py sends c requests at once, each one of the six chat prompts, 256 tokens, five repeats, one shared context cache (-kvu) at -c 32768 --parallel 64. Total output tok/s:

requests at once	E2B, 3050	E2B, L4	E4B, 3050	E4B, L4
1	106.1	131.4	57.3	73.5
8	297.1	505.2	199.3	309.7
64	569.5	918.9	472.4	778.1

Step 5 — Check the Answers

L4	GSM8K, none / drafter	BFCL simple, none / drafter	GSM8K time, none / drafter
E2B	0.8992 / 0.8984	0.930 / 0.925	1,014 s / 713 s
E4B	0.9310 / 0.9287	0.910 / 0.915	1,416 s / 893 s

Step 6 — Price It

The L4 instance, g6.2xlarge in us-east-1, listed at $0.9776/h on demand and $0.8393/h spot on the day of the runs. Per million output tokens, one machine kept busy:

L4, spot	tok/s	$ per million tokens
E2B, one user, no drafter	133.4	$1.75
E2B, one user, drafter	217.6	$1.07
E4B, one user, drafter	129.7	$1.80
E2B, 64 users	918.9	$0.25
E4B, 64 users	778.1	$0.30

🔎 Tip: Where the Bigger Card Earns Its Keep

The 8 GB 3050 runs every larger Gemma 4 too, at a price paid in speed once the model outgrows the card. From the same series on the same card:

on the RTX 3050, 8 GB	where the model lives	generation, with drafter
12B, 6.7 GB	all on the card	55.1 tok/s
26B A4B, 14.4 GB	experts of 21 of 30 layers in system RAM	33.7 tok/s
31B, 17.7 GB	40 of 60 layers in system RAM	4.14 tok/s

All three fit a 24 GB card whole; this article measures them on the 3050 only.

Compare and Contrast

Gemma 4 E2B / E4B, 4-bit QAT	🥇 RTX 3050, 8 GB, local	🥈 NVIDIA L4, 24 GB, AWS	🥉 GTX 1650 Ti, 4 GB, local
One user, E2B / E4B, best	151.2 / 100.0 tok/s	217.6 / 129.7 tok/s	90.3 / 56.5 tok/s
64 users, E2B / E4B	569.5 / 472.4 tok/s	918.9 / 778.1 tok/s
Best draft length	2 tokens	3 tokens	1 token
GSM8K, E4B with drafter	0.9325	0.9287
Cost	owned	$0.84/h spot	owned

So, Which One?

For one user on E2B or E4B, a local 8 GB card. It delivers 69-81% of the L4's speed with the same answers, and the model uses a fraction of either card.

For many users at once, the L4. At 64 requests it serves 1.6x the 3050's output, and at about $0.25-0.30 per million tokens on spot.

For quality at small-model speed, E4B with its drafter. On the L4 it generates at 0.97x E2B without a drafter and scores 3 points higher on GSM8K.

For the 26B and 31B, a card that holds them. On 8 GB they spill into system RAM and slow to 34 and 4 tok/s.

Summary

🟢 One user: the L4 is 1.23-1.27x the 3050 without the drafter and 1.30-1.44x with it, at 13-29% of its memory
🟢 The same model files give the same answers on every card; the drafter costs no GSM8K or BFCL answers
🟢 The drafter's guesses are kept at the same rate on every card; the card's cost per guess sets the best draft length, 1 / 2 / 3 tokens on the 1650 Ti / 3050 / L4
🟢 E4B with its drafter generates at 0.97x E2B without one, 3 GSM8K points higher
🟢 64 users: the L4 serves 1.6x the 3050, $0.25-0.30 per million tokens on spot
⚠️ The drafter helps on the L4 up to 16 users at once and costs output at 64
❌ One user on the L4 pays for a whole card and uses under a third of its memory

The strategy for using MCP for local accelerator comparison was validated with an incremental step by step approach.

References

Part 2, Do Drafters Work on Small Models?: https://dev.to/gde/do-drafters-work-on-small-models-gemma-4-e2b-and-e4b-on-two-small-gpus-1b4g
Part 6, A Dense 31B on an 8 GB GPU: https://dev.to/gde/a-dense-31b-on-an-8-gb-gpu-gemma-4-31b-on-an-rtx-3050-26p9
Repository: https://github.com/xbill9/gemma4-dev
E2B on the L4: https://github.com/xbill9/gemma4-dev/blob/main/gpu-llamacpp-g6-2b-q4_0/benchmarks/runs/2026-10-26-drafter-g6/REPORT.md
E4B on the L4: https://github.com/xbill9/gemma4-dev/blob/main/gpu-llamacpp-g6-4b-q4_0/benchmarks/runs/2026-10-26-drafter-g6/REPORT.md
Drafter cost and many users on the L4: https://github.com/xbill9/gemma4-dev/blob/main/gpu-llamacpp-g6-2b-q4_0/benchmarks/runs/2026-10-26-drafter-cost-g6/REPORT.md
E2B GGUF: https://huggingface.co/xbill9/gemma-4-E2B-it-qat-q4_0-exact-gguf
E4B GGUF: https://huggingface.co/xbill9/gemma-4-E4B-it-qat-q4_0-exact-gguf
E2B drafter: https://huggingface.co/xbill9/gemma-4-E2B-it-qat-assistant-q4_0-exact-gguf
E4B drafter: https://huggingface.co/xbill9/gemma-4-E4B-it-qat-assistant-q4_0-exact-gguf
llama.cpp: https://github.com/ggml-org/llama.cpp
Amazon EC2 G6 instances: https://aws.amazon.com/ec2/instance-types/g6/

Inference on Google Cloud: Precision, Memory and Capacity on TPU, from v5e to Ironwood

xbill — Sun, 25 Oct 2026 23:02:38 +0000

This article is the written companion to the talk "Inference on Google Cloud". It covers serving Gemma 4 on Cloud TPU v5e and v6e through vLLM, measured on the gemma4-dev Cloud TPU test setups between July and October 2026, and compares them with the NVIDIA L4 on Google Cloud, Blackwell list prices, an AMD MI300X and AWS Inferentia. Four ideas explain most of the results: the number format the chip computes natively, the memory bandwidth that feeds it, the slice shape, and the rules that decide whether a TPU is there to rent at all.

https://github.com/xbill9/gemma4-dev

The hands-on steps run from a second repository, which starts one TPU and serves Gemma 4 on it with a single command:

https://github.com/xbill9/gcp-inference

The slides: https://docs.google.com/presentation/d/1hT8UXOEBueZfoFdIH0ZyhYhQ3WZJ_0xbCsvNDDs8H4A/edit

What Is the Problem?

This article compares four TPU generations, the L4 and Blackwell GPUs beside them, and every way Google Cloud prices a TPU chip-hour. The same Gemma 4 E2B serves at 144 tokens a second on a v5e in bf16 and at 243 in int8 with 4-bit embedding tables, on the same chip. A v6e-1 serves 3.3 times the L4's tokens on the same 26B checkpoint. And a zone that reports quota for 1,536 chips can still return a stockout.

Which Generations?

Chip	HBM	Peak bf16	Native low precision	Max pod	On-demand
v5e	16 GB	197 TFLOPs	int8, 393 TOPs	256	$1.20/hr
v6e Trillium	32 GB	918 TFLOPs	int8, 1,836 TOPs	256	$2.70/hr
v5p	95 GiB	459 TFLOPs	int8; fp8 at bf16 rate	8,960	$4.20/hr
TPU7x Ironwood	192 GiB	2,307 TFLOPs	fp8, 4,614 TFLOPs	9,216	$12.00/hr

Google's first TPU, in its datacenters since 2015, was an 8-bit inference chip with a 92 TOPS peak, and Google announced Ironwood on 2025-04-09 as "the first Google TPU for the age of inference". The gemma4-dev test setups measured v5e and v6e. v5p and TPU7x rows come from Google's documentation only.

vLLM reaches the TPU through tpu-inference, a JAX backend. XLA compiles one program per input shape, which makes the compile the largest part of a cold start.

At This Point You Should Have…

The gcloud CLI with its alpha and beta components, signed in with gcloud auth login and gcloud auth application-default login
A project with Cloud TPU quota for one v5e chip in us-west4-a
python3 with pip install 'google-adk[extensions]' google-genai mcp
A clone of xbill9/gcp-inference

Step 1 — One TPU, One Command

demo.sh asks for a TPU v5e-1 as a flex-start request in us-west4-a, waits for the node, opens an SSH tunnel to vLLM on the TPU's port 8000, waits for the model to load and compile, then runs a Google ADK agent against it. Flex-start stops the node by itself after 4 hours. On a TPU that is already up, ./demo.sh again skips the first two steps:

./demo.sh again

== [0m01s] 3/5  Connecting to it over SSH (the demo reaches vLLM on the TPU's port 8000 through a tunnel)
   [0m05s] SSH works
   [0m11s] tunnel open: localhost:8005 -> TPU port 8000
== [0m11s] 4/5  Starting Gemma on vLLM (first start downloads and compiles: several minutes)
   [0m30s] vLLM is answering
  20:21:25Z  PASS  tpu        chat 0.198s; tool run filters=['id >= 10'] quoted=8 3.71s
READY
== [0m37s] 5/5  The demo: Gemma on the TPU counts ids with the count_ids tool from the MCP server
Question: How many of the ids are 10 or more? Answer with the number.
Ids: [0, 2, 3, 20, 21, 22, 23, 10, 11, 12, 13]
True answer: 8  tools: mcp
  tpu        run 1/3  ok   quoted=8  filters=['id >= 10']  3.87s
  tpu        run 2/3  ok   quoted=8  filters=['id >= 10']  0.51s
  tpu        run 3/3  ok   quoted=8  filters=['id >= 10']  0.49s
backend     runs  correct  right-filter  errors  median s  filters sent
tpu            3    3/3             3/3       0      0.51  id >= 10

The agent is an ADK LlmAgent whose model is LiteLlm pointed at vLLM's OpenAI-compatible API, and an ADK McpToolset starts mcp_server.py over stdio and offers it one tool. The model never counts. It calls count_ids on that MCP server, which returns the exact count, and each run is graded on the filter it sent as well as the number it quoted. Gemma 4 E2B on the v5e sent id >= 10 in all three runs.

The node serves google/gemma-4-E2B-it in bf16 with vllm/vllm-tpu:v0.30.0, one chip, no sharding.

Step 2 — What the 16 GB Chip Holds

vLLM logs the chip's memory as it starts. In order:

Init worker | ... | hbm=[(0.0, 15.75)]GiB
Init model | hbm=[(8.97, 15.75)]GiB
Memory statistics | total_hbm_limit_gb=15.75GiB | total_hbm_limit_cap_gb=14.49GiB | total_hbm_used_gb=8.97GiB | total_hbm_avail_gb=5.52GiB
TPU KV cache size: 321,280 tokens, Maximum concurrency for 8,192 tokens per request: 39.22x
Init kv-cache | num_total_layers=15 | num_blocks=[1255, ...] | regular_attn_shape=(num_blocks, (256, 1, 2, 256)) | regular_attn_dtype=bfloat16 | hbm=[(14.51, 15.75)]Gb

HBM on one v5e chip	GiB
Weights, bf16	8.97
vLLM's ceiling (92% of 15.75)	14.49
Left for the KV cache	5.52
In use after the KV cache is allocated	14.51
Never allocated	1.24

Each of the 15 cached layers holds 1,255 blocks of 256 tokens, one KV head of dimension 256, in bf16. That is 1,024 bytes per token per layer, 15,360 bytes per token across all 15, and 321,280 token slots in all. The model's config lists 35 layers with num_kv_shared_layers: 20: 20 layers reuse another layer's keys and values, so the cache holds 15.

Step 3 — The Full 131,072-Token Context

Gemma 4 E2B's config allows 131,072 positions. vLLM sizes its KV cache from the memory left after the weights, so raising --max-model-len from 8,192 to 131,072 changes the concurrency and leaves the cache the same size. To change a flag on a running TPU, replace the container:

gcloud compute tpus tpu-vm ssh gcp-inference-tpu-node --project <tpu-project> --zone us-west4-a --command \
  'sudo docker rm -f vllm && sudo docker run -d --name vllm --restart unless-stopped --privileged --net=host \
   -v /dev/shm:/dev/shm --shm-size 10gb -e HF_HOME=/dev/shm vllm/vllm-tpu:v0.30.0 \
   vllm serve google/gemma-4-E2B-it --served-model-name gemma --max-model-len 131072 --port 8000 \
   --max-num-batched-tokens 4096 --enable-auto-tool-choice --tool-call-parser gemma4'

Memory statistics | total_hbm_limit_gb=15.75GiB | total_hbm_limit_cap_gb=14.49GiB | total_hbm_used_gb=8.97GiB | total_hbm_avail_gb=5.52GiB
TPU KV cache size: 321,280 tokens, Maximum concurrency for 131,072 tokens per request: 2.45x
init engine (profile, create kv cache, warmup model) took 503.54 s (compilation: 413.41 s)
Application startup complete.

`--max-model-len`	8,192	131,072
KV cache, tokens	321,280	321,280
Full-length requests at once	39.22	2.45
Engine start	502.84 s	503.54 s
of which XLA compile	408.59 s	413.41 s

One full-length request needs 1.875 GiB of KV cache. scripts/longctx.py in tpu-survey buries a passphrase at a chosen depth in up to 124,949 tokens of filler, starts each prompt with a unique line so vLLM's prefix cache cannot reuse an earlier run, and asks for the passphrase:

$ python3 scripts/longctx.py $TPU_URL 8000 0.5
{"prompt_tokens": 7949, "completion_tokens": 10, "depth": 0.5, "seconds": 0.61, "prompt_tokens_per_second": 13110.4, "answer": "violet-harbor-7341", "found": true}
$ python3 scripts/longctx.py $TPU_URL 120000 0.1
{"prompt_tokens": 124949, "completion_tokens": 10, "depth": 0.1, "seconds": 9.53, "prompt_tokens_per_second": 13116.9, "answer": "violet-harbor-7341", "found": true}
$ python3 scripts/longctx.py $TPU_URL 120000 0.5
{"prompt_tokens": 124949, "completion_tokens": 10, "depth": 0.5, "seconds": 9.54, "prompt_tokens_per_second": 13102.3, "answer": "violet-harbor-7341", "found": true}
$ python3 scripts/longctx.py $TPU_URL 120000 0.9
{"prompt_tokens": 124949, "completion_tokens": 10, "depth": 0.9, "seconds": 9.41, "prompt_tokens_per_second": 13274.6, "answer": "violet-harbor-7341", "found": true}
$ python3 scripts/longctx.py $TPU_URL 140000 0.5
HTTP 400: {"error":{"message":"This model's maximum context length is 131072 tokens. However, you requested 16 output tokens and your prompt contains at least 131057 input tokens, for a total of at least 131073 tokens. ..."}}

The passphrase comes back at 10%, 50% and 90% depth of a 124,949-token prompt, in 9.4 to 9.5 seconds each, about 13,100 prompt tokens a second as timed from the client. vLLM's own log reads 12,495 tokens a second, a 0.0% prefix cache hit rate, and 38 to 39% of the KV cache in use while the request runs. A prompt past the limit is refused before it reaches the chip.

🔎 Tip: Raise the Context on the Chip You Have

Flex-start capacity is granted per request. Replacing the container keeps the node, and its 4-hour clock, and costs one engine start, 503.54 s on this v5e. Deleting the node to relaunch it means a new flex-start request, and a new wait for capacity.

Step 4 — Precision: What the Chip Computes

A smaller format saves memory and bandwidth on every chip. It saves compute only where the matrix unit multiplies that format natively.

Format	v5e	v6e	v5p	TPU7x
bf16	native	native	native	native
int8	2x bf16	2x bf16	native	native
fp8	storage	storage	bf16 rate	2x bf16
int4	storage	storage	storage	storage

"Storage" means the weights widen to bf16 before every matrix multiply. On v5e that ordering shows up directly in E2B throughput at one stream:

E2B format on v5e-1	tok/s
bf16	144
4-bit QAT + 4-bit embeddings	150
fp8	182
fp8 + 4-bit embeddings	203
int8 W8A8	220
int8 + 4-bit embeddings	243

Accuracy holds: 0.683 for bf16 against 0.677 for int8 with 4-bit embeddings, on a 3,880-record suite. At 12B, fp8 serves 518 tok/s at 16 streams against 624 for int8 at the same file size.

On v6e the 4-bit path pays off through a fused kernel. Gemma 4 26B-A4B in 4-bit QAT W4A16 serves 1,283 tok/s at 16 streams against 668 for fp8, in 17.43 GiB of weights against 27.99 GiB, and holds 53,888 KV-cache tokens against 3,456. Accuracy is 75.3% against 76.0%. W4A16 needs tpu-inference PRs #3299, #3653 and #3660 on top of the stock image.

Step 5 — Memory and Bandwidth

Each generated token reads every weight once, so bandwidth sets the ceiling for one request.

Chip	Memory	Bandwidth	E2B bf16 tok/s, 1 stream	tok/s per TB/s
TPU v5e	16 GB	859 GB/s	120	140
TPU v6e	32 GB	1,638 GB/s	199	122
TPU v5p	95 GiB	2,765 GB/s	not measured	—
TPU7x	192 GiB	7,380 GB/s	not measured	—
NVIDIA L4	24 GB	300 GB/s	46	154

The three measured chips land between 122 and 154 tokens a second per TB/s. Google quotes v5e bandwidth as 800 GiBps, which is 859 GB/s. The L4 figure was measured on an AWS g6.2xlarge, the same chip as Google Cloud's g2.

Step 6 — Topologies and Slices

Weight size picks the slice. Every Gemma 4 size serves on one host:

Model	v5e-1	v6e-1	v6e-4, TP=4
E2B	bf16 or int8	bf16	—
E4B	int8 or 4-bit; bf16 does not fit	bf16	—
12B	int8 or 4-bit	bf16, 22.18 GiB	—
26B-A4B	4-bit, 201 tok/s	4-bit, 1,283 tok/s	bf16, 1,982 tok/s
31B	does not fit	4-bit, 500 tok/s	bf16, 2,031 tok/s

Small models scale out. On a v5e-4, four one-chip E2B engines serve 1.39x the output of one TP=4 engine at 16 streams, 2.52x at 64 and 3.36x at 256. E2B has one KV head, so tensor parallelism keeps four copies of the KV cache. 12B gains from TP=4: 2.25x the v5e-1 output at one stream, and 122,624 KV tokens against 9,728.

🔎 Tip: Quota Lets You Ask, Availability Decides

Quota is permission to request a TPU. Availability decides whether the request gets hardware, through three checks: hardware in the zone, the provisioning model offered there, and capacity at that moment. 37 zones report v6e quota; 18 offer v6e-1 hardware. Three of five zones holding 1,536 chips of quota returned stockout. us-central1-b has hardware, quota and a flex-start price, and rejects flex-start. Check capacity with a spot request: it fails fast with "reason: stockout", where a flex-start request with no quota queues as PENDING exactly like a stockout.

Ask for fewer chips. In europe-west4-a a v6e-8 flex-start request expired after 26 minutes, and a v6e-4 was granted at once less than an hour later.

Step 7 — Against Other Accelerators

On the same 26B 4-bit checkpoint, with the same flags and client:

	NVIDIA L4	TPU v6e-1
Output tok/s, 16 x 256 tokens	393.7	1,283.3
KV cache tokens	17,990	53,888
3,880-record suite, wall time	139 s	58 s
4-bit kernel	Marlin	gmm_v2
Software	stock vLLM 0.30.0	three tpu-inference patches
List price per hour	$0.71 (g2-standard-4)	$2.70

Cloud Run serves the L4 as a managed service: one L4 per instance, weights mounted from Cloud Storage at /mnt/models, a startup check that waits 180 s before its first check, and a default that scales between zero and one instance, so an idle service means a cold GPU start.

Blackwell on Google Cloud lists above v6e per accelerator-hour: an RTX PRO 6000 on g4-standard-48 at $4.50 on-demand and $2.25 flex-start, and a B200 at $8.06 per GPU flex-start ($64.44 for the 8-GPU a4-highgpu-8g, divided by 8). Both are list prices only, with no measurements in this article.

Per dollar, on E2B bf16 with vllm bench serve at 1,024 prompt and 128 output tokens, v5e at its $0.58 spot price leads to 16 streams and the MI300X at $1.99 leads at 64. The v6e-1 is priced at the europe-west4 on-demand rate of $2.97:

tok/s per $/hr	MI300X ($1.99)	v5e-1 spot ($0.58)	v6e-1 europe-west4 ($2.97)
1 stream	153	208	67
4 streams	487	711	233
16 streams	1,333	1,572	587
64 streams	2,713	1,972	719

Each software stack fails in its own way:

	TPU, vLLM + JAX	CUDA, L4	ROCm, MI300X	Neuron, inf2 / trn1
Start to healthy, E2B	510 s v6e, 986 s v5e	334 s	about 160 s	171 s on trn1
Gemma 4 support	4-bit needed patches	26B 4-bit loads stock	fp8 only as fnuz	vLLM-Neuron cannot load it
Hazard	OOM after a long compile	sm_89 missing; runs sm_86 code	AITER flag 0.97 to 0.99x	large gather returns zeros, HTTP 200

🔎 Tip: Cache the Compile

XLA compile is 81 to 91% of vLLM's engine start for E2B: 329 of 404 seconds on v6e-1 and 738 of 814 on v5e-1, inside 510 and 986 seconds from container start to healthy. On today's v5e-1 with vllm/vllm-tpu:v0.30.0 the engine start was 503.54 seconds, 413.41 of them compile. A mounted compile cache cut a v5e restart from 857 to 497 seconds, 42%. 31B weights load in 5.6 minutes from in-region Cloud Storage against 60 to 90 from Hugging Face.

Serving flags that work on GPUs fail late on TPU, after the compile. --gpu-memory-utilization 0.95 failed on v5e after a 691-second compile, twice; 0.92 serves. --kv-cache-dtype fp8 logs fp8 on v6e and allocates bf16. The vllm/vllm-tpu:nightly tag broke two 4-bit builds on 2026-10-02, so pin an image digest.

Step 8 — What Each Chip Costs

$ per chip-hour	On-demand	Flex-start	Calendar	1-yr CUD	3-yr CUD	Spot read
v5e	$1.20	$0.60	$0.84	$0.84	$0.54	$0.58
v6e Trillium	$2.70	$1.35	$1.89	$1.89	$1.22	$1.40
v5p	$4.20	$2.10	$2.94	$2.94	$1.89	—
TPU7x Ironwood	$12.00	$6.00	$8.40	$8.40	$5.40	—

Flex-start halves on-demand on every generation and can undercut spot: v6e in us-east5 lists $1.35 flex-start against a $1.40 spot price. Flex-start runs 10 minutes to 7 days and bills until deleted.

Utilization sets the price per token. A v6e-1 at the europe-west4 on-demand rate of $2.97 an hour serves E2B at $4.14 per million output tokens with one stream, $0.70 with eight and $0.55 with 32, at 1,024-token prompts. A v5e at the $0.60 flex-start rate goes from $1.31 per million at one stream to $0.09 at 64, at 128-token prompts.

Compare and Contrast

Option	Precision served	Bandwidth	Measured result	List $/hr	Software
🥇 TPU v6e-1	4-bit QAT W4A16	1,638 GB/s	26B at 1,283 tok/s	$2.70	tpu-inference, patched
🥈 TPU v5e-1	int8 W8A8 + 4-bit embeddings	859 GB/s	E2B at 243 tok/s, one stream	$1.20	tpu-inference
🥉 NVIDIA L4	4-bit QAT	300 GB/s	26B at 394 tok/s	$0.71	stock vLLM
TPU7x Ironwood	fp8 native	7,380 GB/s	not measured	$12.00	allowlisted
Blackwell B200	fp4 native	—	not measured	$8.06 flex, per GPU	not measured

So, Which One?

For E2B or E4B at the lowest cost, run v5e-1 in int8 with one engine per chip, on spot or flex-start. For 12B to 31B on one chip, run v6e-1 in 4-bit QAT. For bf16 26B or 31B with long context, run v6e-4 with TP=4. For stock software and the fastest boot, run the L4, on Compute Engine or on Cloud Run. TPU7x brings fp8 math and 192 GiB per chip, behind an allowlist. Whichever you pick: check capacity with a spot request, ask for the smallest slice that fits, pin the image digest and mount the compile cache.

Summary

The goal of this article was to serve Gemma 4 on a Cloud TPU in one command, then explain the results across TPU generations through native precision, memory bandwidth, slice shape and capacity. The key to the solution was measuring each claim on real hardware, with every ratio and price computed in code. The results were:

🟢 One v5e chip holds Gemma 4 E2B's full 131,072-token context: a 321,280-token KV cache beside 8.97 GiB of bf16 weights, and a passphrase found at 10%, 50% and 90% depth of a 124,949-token prompt in 9.4 to 9.5 s.
🟢 The agent sends the right filter: 3 of 3 runs on the v5e called count_ids with id >= 10 and quoted 8.
🟢 int8 is the fast format on v5e: with 4-bit embedding tables it serves E2B at 1.7x bf16, with accuracy within a point.
🟢 v6e-1 serves 3.3x the L4's tokens on the same 26B 4-bit checkpoint, with 3x the KV cache.
🟢 Flex-start halves on-demand on every generation, and can undercut spot.
⚠️ Cold start takes 8.5 to 16.4 minutes, most of it XLA compile; cache it.
🟢 One-stream decode follows bandwidth: 122 to 154 tok/s per TB/s on v5e, v6e and the L4.
❌ Quota is permission to request; availability decides the grant: 37 zones report v6e quota, 18 have v6e-1 hardware, and zones holding full quota still return stockout.

Scope: the hands-on steps ran on one flex-start v5e-1 in us-west4-a on 2026-10-25, google/gemma-4-E2B-it in bf16 on vllm/vllm-tpu:v0.30.0, three demo runs and one long-context request per depth. The rest covers Gemma 4 E2B to 31B on Cloud TPU v5e and v6e, measured on the gemma4-dev test setups between July and October 2026 with vLLM and tpu-inference; L4 on Google Cloud g2 and AWS g6; MI300X on DigitalOcean; Inferentia2 and Trainium on AWS. The cross-vendor tokens-per-dollar table mixes vLLM builds, hosts and repeat counts, so it shows direction. v5p, TPU7x and Blackwell figures are Google's published specifications and list prices, read on 2026-10-12 and re-read on 2026-10-25 with the same values; spot prices come from the Cloud Billing Catalog on recorded dates.

The strategy for serving Gemma 4 on Google Cloud from v5e to Ironwood was validated with an incremental step by step approach.