DEV Community: Nic Lydon

One Room, Eight Boards, One Conductor: Building the Manor at Midnight

Nic Lydon — Sun, 01 Nov 2026 07:35:28 +0000

On Halloween night my living room ran a show.

Not a playlist. A show: two original songs, a conductor program firing 77 timed cues for the first one, Hue lights moving as instruments, two scrolling message boards passing a single banner between them, a field of glowing candles, homemade lightning, four small AI devices sitting in a choir row wearing characters, and a 7-inch screen on my desk where a figure I call the Night Auditor kept his ledger and, late in the evening, sang.

It took all day and most of the night to build. It came together with about ten minutes to spare. And its first full performance, at 22:37, played in complete silence, which is a story I will get to honestly, because it is the most useful part.

This is the whole build: the AI and the code, the wiring, the firmware, the lights, and the show.

The cast

Everything in the room had a job, and most of it had a name.

The conductor. A program per song. For "Little Glowing Souls," the anthem, it holds 77 cues across 204.8 seconds, each one timed to the recording: this Hue group at this lyric, a lightning strike on that crack of thunder, the scrollers starting a new banner on the chorus, Portal changing scenes. For the Night Auditor song, 52 cues. The conductor does not improvise. It is a score, executed.

Eight ESP32 boards, all running the same firmware skeleton I settled on as the day went on: join the Wi-Fi, carry a name, answer a status check, expose one signature cue, and reconnect on your own if you fall off. On top of that skeleton, each board is a different instrument:

The candle board, driving six RGB candles in 3D-printed glow sleeves, with a warm flicker-and-gutter routine and a storm gag on its boot button
Scroller 1 and Scroller 2, each driving a four-panel LED matrix, one of them also carrying the cast's only speaker, an I2S amp playing test phrases of organ, ghost moan, and filtered-noise thunder
Satellite 1, carrying a sound sensor, a 7-colour module, a two-colour module, and loose RGBs, firing its own strobe when the room gets loud
Satellite 2, ticking and blinking on cue
Breath 1 and Breath 2, swelling slowly, the room's background respiration
The lightning pod, three lights in sleeves behind cotton clouds, striking in a double-flash-and-decay pattern

Janus, an Arduino UNO Q, is two computers in one box: a Debian Linux side and a microcontroller side. The Linux side runs my NexusOS software and drives Portal, the 7-inch display that is the show's stage: a manor hall, the Auditor's desk, his rounds, a storm, and finally the films themselves.

Four AIPI devices in printed tombstone risers, the choir row: red wears ARIA, green wears Carbon, yellow wears Lucas, blue wears Nova. The characters are the stable part. Their network addresses are DHCP leases and move; nobody in my house refers to a performer by lease.

Flint, a Raspberry Pi Zero W, was supposed to drive the big TV, which reaches the room through a Philips Hue Play HDMI Sync Box. Flint has its own section below, under failures, because that is where it spent the night.

One Bose speaker was the only sound in the room. That sentence turns out to be load-bearing.

And the prints ran underneath everything all day: a glow plate of 10 ghosts and 14 candle sleeves finished in the small hours that morning (175 layers, eight and a half hours), four tombstone risers for the choir printed on a single plate at my direction, and a tombstone holder for my Glance gadget printed during the show build. The holder's cradle came out too shallow because Glance carries a battery on its back. It fits perfectly with the battery off. I have not reprinted it.

Wiring is a conversation with the physical world

I am a lifelong electronics hobbyist, but this was a build done at speed, by hand, with an AI reading my photos and telling me where to put the next wire. Three lessons from the bench are worth more than the finished boards.

The candle field failed on a question I asked, not an answer I was given. The first candles would not light, and the photos I sent were misread three times: legs not in holes, wrong sides of the groove. What actually unlocked it was my own question about the breadboard itself: are rows A to E and F to J even connected underneath? They are not, on that board, in the way the wiring assumed. Two lights, then six candles in their sleeves, lit and flickering. The person holding the board is a sensor, not a pair of hands for the person reading the schematic.

Big Sound is a microphone. I kept waiting to hear satellite 1's "speaker." There is none. The Big Sound module in my sensor kit is an input: it hears, it does not speak. Once that was straight, the wiring made sense: its analog output belongs on an input-only analog pin (34, 35, and 39 are exactly right for a sensor on these boards; 9, 10, and 11 are tied to the flash memory; pin 3 is a boot-strapping pin and a bad place for a sensor output). On the candle board, I was first told pins 4 and 5 were free for it. Pin 4 was already wired to candles. My board won that argument, the sensor moved to pin 10, and the trigger threshold got set in code instead of by the module's little dial, which is the better instrument anyway.

The scrollers were a power problem wearing a software costume. For hours, the twin ESP32-S3 boards and their LED panels stayed dark, and everything pointed at firmware. The answer was sitting in my own project record from an earlier sign build: a classic ESP32's 5V pin is a live output that can power its panel, while the S3 boards' 5V input pin sources nothing at all. Swapped to a classic board, panel on DIN 25, CS 26, CLK 27, power from the board's own 5V pin: worked first try. The most expensive bug of the afternoon was not reading our own documentation.

A smaller RF lesson, same shape: satellite 1 could not join the network and looked like a code problem until the numbers said it was nearly deaf compared with every other board. It is the external-antenna variant of its board, and its little whip antenna was sitting in a parts box. Antenna on, board joined immediately. The lightning pod had the same disease worse (a tiny built-in antenna, nothing to attach), and placement was its only medicine: antenna end clear of the breadboard wires, pointed at the room.

On the bench before it was a candle field: the board, the sleeves, and the panel that would spend the night insisting it was a software problem.

Two boards, one sentence

The scrollers are my favourite mechanism in the show. Bound by eye, because there is no other way to bind them: Blue is board 1, stage left from the audience, and Red is board 2, stage right. Unaddressed text does not go to "the scrollers." It flows across both of them as one banner:

Red, upstream on the audience's right, carries the text plus a trailing blank pad. Blue carries a leading pad plus the text. Both strings are the same length and are sent in the same instant, so their scroll cycles stay locked and the sentence reads as one sign. Text addressed to a colour goes to that board alone. The next unaddressed text puts the pair back together.

The pad length was calibrated the only way it can be: I watched it. The first guess left five or six seconds of dead air, red fully blank before blue started. Trimmed from 48 spaces to 34, both boards together, in one send, to keep the lock.

The firmware taught one sharp lesson, twice, before we read the source: the endpoint that sets a message and the endpoint named "show" are not friends. Sending "show" after a message restores the compiled-in default text and silently overwrites whatever you just set. The boards were innocent both times. The bug was in our own calling code, and the fix was a rule, not a patch: never send show after message.

The brain has two audio universes

Janus drove the stage, and Janus is where the night's hardest problem lived, because Janus runs two completely separate audio sessions.

The arduino user session runs the manor player. The nexusos user session owns the display, the voice stack, and the Bluetooth audio endpoint. A Bluetooth device can be owned by only one session at a time. So the Bose speaker could be paired, bonded, connected, and set as the default sink, all verifiably true, in the session the show player was not running in. Every check that asked "is the speaker connected?" got a correct answer to the wrong question.

Even the status tooling was caught by it: the daemon that reports audio status runs as its own user with no audio session at all, so at one point it reported "no sinks" in the same minute the speaker was present and had just played. Two layers, BlueZ and the audio graph, have to be reported separately or the report is fiction.

The speaker itself added one more fact, which I stated and the design now has to respect: a small Bluetooth speaker sleeps. When Janus is not actively using it, it drops and powers down, and I turn it back on by hand. That is not a fault. Any automation that treats it as one, or that holds the link open with a keepalive to stop it sleeping, is fighting the hardware's intended behaviour.

The Night Auditor gets a voice

Portal started the day as a passive screen: a scripted page on a loop. In the evening it became a performer with a voice, and that build had the cleanest arc of the day because it began with a false premise, corrected by evidence inside the hour.

The brief assumed no voice code existed on Janus and that a browser would have to host a voice session. The live check found the opposite: my estate's Linux voice path, a native device SDK, was already installed, paired, and wired into NexusOS on that board, with calls routed to a dedicated conversation room for the device. A call through that path had already run the day before.

So the proof was done the right way round: hardware-free first. A spoken question went in as an audio file ("Good evening. Who are you and what is this place?"), landed in the Portal Voice Room transcribed word for word, and a spoken reply from the Night Auditor came back as audio. Transport, room targeting, speech recognition, the reply, and synthesis were all proven before a single microphone existed in the room. Everything after that was plumbing, and plumbing failures could never again be mistaken for the idea not working.

Then the visible half. An hour before I asked, Portal had no face for a conversation. By the end of the evening it had seven frames, every one an edit of the show's own desk portrait, so the scene, the lantern, the ledger, and the face never change; only the mouth and expression do:

Lip sync is deliberately simple, the same recipe I use on my smaller gadgets: four mouth grades (closed, small, mid, wide) swapped according to the loudness of the speech itself, sampled every 40 milliseconds, snapping open fast and relaxing slower. It is not phoneme analysis and does not pretend to be. At room distance, it reads as a man talking. Later that night a live tap was added: a small service listening to the voice session's actual speaker output and microphone ten times a second, so the page speaks with its mouth graded from real levels, listens only when a real microphone is live, and its idle screen tells the truth about whether the speaker and microphone are ready. Its Talk button also tells the truth: it checks every gate and reports what is missing rather than starting a call it has no authority to start.

The microphone was the last piece, and it was built out of network plumbing instead of new hardware. There was a spare ESP32-S3 board with an I2S microphone sitting in the microphone spot. It could already record five-second clips of the room that transcribed verbatim. What it could not do is be a microphone as far as Janus was concerned, because a voice call listens to a source inside Janus's own audio graph, and a board on the network is not in that graph.

So that night, in about twenty minutes: a puller that chains the board's clips and re-streams them, a bridge service on Janus that takes that stream and presents it as a real microphone source, and the voice configuration pointed at it. One hard lesson inside that build: this audio stack routes every recording to the default source regardless of what a program asks for, so the bridge only works by becoming the default source and re-asserting itself as such. At 21:39 a live call went through it: the test question streamed over the bridge, arrived in the Portal Voice Room three times as it looped, and the Night Auditor answered each time. End to end, through a microphone that is not plugged into anything.

What stays after Halloween, from the review I asked for that night: the voice path, the dedicated room, the test harness, the tap feed, and the rule written into the Auditor's brief that he never guesses at anything that needs a record (calendar, messages, money, health): those questions get "that needs a lookup in the main office," and a stop. The costume retires. The capability does not.

The choir and the rest of the room

The AIPI row was casting, then verification. Green already lived in its room firmware wearing Carbon and joined the network by itself. Yellow got a minimal hand-rolled display sketch (raw SPI, no external libraries) to wear Lucas. Red and blue verified as ARIA and Nova. I overruled a shortcut that would have left the as-found arrangement in place; the units were imaged as cast, because a choir where one member is whoever happened to be resident is not a choir.

The generated audio was built the same day, by measurement rather than by ear, because the agent building it cannot hear: a just-under-four-minute ambience loop (organ drone in D minor, rain on glass, a distant thunder roll every minute or so, made loopable by moving the seam under a six-second crossfade) and an exact 90-second show track with its thunder cues at measured timestamps and five chant lines ducked over the bed. Its first chant mix came out buried, because a mono-to-stereo conversion was silently eating level; the fix was per-line loudness normalisation and stronger ducking, verified by re-measuring every cue window. "Verified by measurement, not by ear" is a limitation I want stated wherever that work is described. It is also, on the night, more than we managed for the room itself.

Janus's own small display, an 8 by 13 LED matrix driven by the microcontroller side, stayed dark for the show, and I want that in the record too. It showed an X in the morning, which the source code says is the linux_down glyph: the microcontroller alive and correctly reporting that it could not reach the Linux side. A restart of the Linux daemon restored that bridge by late morning. In the evening a package of seven Halloween animations for the matrix (pumpkin, witch, bat, cat, ghost, and friends) was designed, compiled, and staged, and three attempts to flash it stopped against a deterministic stall in the microcontroller core, in a region the sketch flashes never wrote. Reviving it needs a bench debug session. The matrix stayed on its original firmware, dark, while everything around it performed. Not every instrument makes the show.

22:37: the show ran perfectly, in silence

The pre-show smoke test was genuinely good. Twelve of twelve instruments answered a roll call. The anthem program fired 77 cues and confirmed 77. The Auditor program fired 52 and confirmed 51, one lightning strike unanswered mid-program with the board healthy immediately after.

At 22:37 we ran the anthem show for real. The conductor's log is the kind of artifact that makes you confident:

[   1.64s] t+0.0 hue:under-tv set -> ok; ok; ok
[   2.16s] t+0.0 scrollers banner -> okok
[  29.80s] t+28.0 board:lightning strike -> strike
[  31.54s] t+29.0 portal mode -> portal->desk rc=0

Every cue fired. The audio graph showed the song streaming to the Bose as the default sink. It was reported to me as verified.

The room was silent. Not quiet. Silent. I filmed it, and that recording was later measured: the audio track of an 81-second room video sits at the level of a quiet room, with no song at any point in it.

The root cause, found minutes later, was embarrassingly small. Inside Janus's audio system, the speaker's own channel volumes sat at about 5%, and the song stream's volume sat at 0%. The audio stack had been restoring those values after every reconnect, all night. (In the forensic write-up we did that same night, those two numbers are graded differently, and I am keeping the grading: the 5% is proven from a surviving system dump; the 0% rests on the record written at the time, because two crashes that evening wiped the temporary files the earlier logs lived in.) Every verification that night checked that a stream was running. Nobody checked a level. Nobody checked the air.

And it was not one trap, it was a family of them, all with the same shape:

The Hue bridge accepted every light cue and read back the cued values while the Sync Box, which had quietly turned its sync back on when a device was replugged, repainted the room according to the television instead. The lights reported their mode as "streaming." The bridge was telling the truth about the bridge.
Portal's music videos froze on a still frame while its silent loops played fine. The cause, found afterwards, was a file replaced on disk underneath a page that was already showing it; the page never reloads a source it already has. A served file is not a playing video.
Flint never joined the show at all. Three faults were stacked on its SD card, each sufficient alone: a 64-bit system image written for a 32-bit-only Pi (a trap we had caught once before, in October, and walked into again), provisioning trigger tokens sitting where the kernel never reads them, and on the fresh build, the card simply missing files it needed. I carried that card between the Pi and my MacBook ten times. Flint joined the network late that night and eventually held, verified, with one honest caveat that applies to this whole article: nobody on the agent side can see the picture on the TV through the Sync Box. That last check is a human standing in the room.

The fix that finally held for the sound was procedural, not clever: set the level inside the firing itself, prove it with an 18-second test at a known level and read that level back afterwards, and then I set the speaker's own volume by ear and froze every dial. The level is mine. Nothing on the software side touches it again.

There was one more design correction, and it was mine to make. The conductor's original programs were bright: full-room blasts on the choruses. That was never my show. The room should stay dark, at candlelight, with lightning, and the films premiering on Portal. Once I said that plainly, a dark version of the program was built, along with an ambient in-between state that nobody had built at all until I asked why the moments between songs had no program. They had no program because nobody built the in-between. That is an omission, not a limitation, and it is worth printing in those words.

What actually played

The show that landed ran late, and it ran only when I said to run it.

The dark program as designed: the room held at candlelight, lightning at the edges, the film premiering on the small screen.

At 23:34, "Little Glowing Souls" premiered as a film on Portal, carrying its own song to the Bose as one stream, with the new dark program around it: candlelight, lightning, the room held low. At 23:47 the Night Auditor film did the same. I filmed both. The last frame of the desk-angle clip carries the line the whole day was for:

Halloween lives in this house, this house is our home.

What I would change

Over-the-air everything. On the night, boards cycled onto a cable by hand for reflashing, one at a time. That is now a standing rule for the next iteration of this fleet: light sequences, patterns, and firmware ship over the air from the first flash, and no board gets carried to a cable to change its mind.
Verify the air, not the graph. A stream state is not sound, a bridge "ok" is not light, a served file is not a playing video. Every instrument in this show reported success in its own dialect. The room is the only integration test, and it needs a witness, a level meter, or a recording, every single run.
Control the Sync Box, or check it first. A device that re-arms itself on every HDMI replug and silently overrides your lighting needs either an API the show can call or a line at the top of the runbook, in that order of preference. That work is parked in its own thread, deliberately, until I open it.
Read your own record before re-deriving it. The scroller power answer was in a narrative we had already written. So was the 32-bit Pi trap. The estate's memory is only an asset if the builders consult it before they start, not after the third failure.
State values, not states of mind, in the handoffs. The 22:45 handover written that night still listed the microphone as the show's blocker, an hour after the bridge had already carried a live call. Summaries rot in an hour on a night like this. Timestamps and measurements do not.

The inventory, for the builders

One room. Eight ESP32 boards on one firmware skeleton, four Hue groups and eight bulbs in the conductor's room score, one Arduino UNO Q running two Linux user sessions and a microcontroller that spent the night honestly reporting its own isolation, one Raspberry Pi that missed the show and was fixed anyway, four AIPI devices in printed tombstones, one ESP32 microphone that became a network peripheral, one Bluetooth speaker whose volume was the whole ballgame, two songs, 129 conductor cues across both programs, three schematics drawn after the fact to explain what the diagrams in our heads got wrong, and roughly fifteen hours from first wire to last film.

Built with a fleet of AI agents working in parallel threads, in my living room, on hardware I can hold in one hand. The agents wrote the firmware, the conductor, the voice plumbing, the films' stage, and most of the documentation. I wired the boards, placed the antennas, carried the SD card, turned off the sync, set the volume, and said when to launch.

That division of labour is the actual story of the Manor. The machines can fire 77 cues without missing one. Somebody still has to stand in the room and listen.

Gemini 4 Argon Wrote a New OS for My Google Glass. Now It Talks to Meta Muse.

Nic Lydon — Fri, 30 Oct 2026 11:21:32 +0000

In February 2014 I bought a Google Glass Explorer Edition. Explorers collected theirs in person, so on February 28 I sat for a fitting at Google's San Francisco office, 350 The Embarcadero, and carried the future home on the plane. I wore it for a couple of years, it filmed a road trip through Iceland, and then, like almost everyone else's, it went into a drawer. It was still in the drawer in September 2023, when Google ended support for Glass entirely, and it was still there last week, when I took it out and gave it a new operating system.

Pickup day, San Francisco, February 28, 2014, drawn from my own photograph of the moment: the glasses were three hours old, and the camera that took the original picture was the Glass itself, pointed at a hotel mirror.

Not a ROM. Not a launcher. A Linux system we wrote ourselves, on top of the kernel Google shipped, with no Android anywhere in it. On top of that, the glasses now pair with Meta's Muse app over Bluetooth and hold live voice conversations with an AI, through the bone-conduction speaker behind my ear. As far as a careful search could find, no published project had done either of those last two things with this hardware.

And the code was written by a Google AI. We'll get to that.

The whole arc in 34 seconds: the voice HUD, the full-screen avatar, the camera card, with the audio as recorded on the device.

One night in six frames, 01:15 to 03:29: raw telemetry, a pairing, a voice, a face, and then a second operating system moving in. Address details redacted; a pair of glasses does not need to publish its network paperwork.

The dead platform that wasn't

Glass's final firmware, XE24, shipped in December 2019 and is usually described as the shutdown notice. Read it more carefully and it's an emancipation proclamation. Google removed the Google account. Google removed the backend connection. What they left behind is a complete, self-contained computer: a 640×360 prism display, a touchpad, an inertial sensor, Wi-Fi, Bluetooth, a microphone, a camera, and a bone-conduction transducer, running a Linux kernel whose source Google had published, on a bootloader that unlocks with a standard fastboot command.

Dead as a product. Intact as a machine. Those are very different things, and the difference is the whole project.

Before writing anything, I did the research properly, because "first" claims are where projects like this go to embarrass themselves. The hardware truth, layer by layer: everything from the kernel up is rewriteable. The display is a plain framebuffer device, the touchpad and IMU are ordinary I2C citizens with in-tree drivers. The bootloaders have no public source and are treated as untouchable. The camera pipeline runs on closed firmware and, on the early evidence, doesn't survive a minimal system. (That last one turned out to be wrong in the best way. Patience.)

First, an honesty break

My prior-art check came back with a correction I needed: I was not the first person to put Linux on Glass, and it wasn't close. postmarketOS ported Glass itself back in 2017. And three weeks before I started, same month, a builder named Lasimeri published glass-xec-mainline, an Alpine Linux system for Glass that he daily-drives, with nearly every peripheral working. His project is MIT licensed, this one learned its platform layer from it, and both facts are credited in my repo, where they belong.

What the check did clear, with actual searched-empty evidence: no published work had turned Glass into a voice endpoint for a modern AI assistant through a public gadget SDK. Meta's Muse Gadget SDK had itself only opened on October 2, 2026, and nothing in its world targeted glasses. And no published operating system for Glass had been authored by an AI coding agent. So that's the project I built. Credit where it's due, claims only where they're earned.

The brief, and who executed it

I wrote the build as a milestone contract: M0, a reproducible boot-image pipeline (stock XE24 kernel, verified by hash, our own Alpine initramfs, inside the 8 MB boot partition). M1, first boot from RAM, nothing flashed. M2, Wi-Fi and modern TLS on a 2014 kernel. M4, a live voice call. And in the middle, M3, the cliff: Bluetooth LE pairing with the Muse phone app, from a 3.4-series kernel, a thing with no public precedent. The brief ordered the builder to stop and report if pairing failed, because that answer would decide the architecture.

The contract also carried safety rules that were not negotiable. Some partitions are never written, ever: the bootloaders and the display FPGA have no public source and no recovery path. The factory image was archived and hash-verified before anything else happened. Every new image gets tested from RAM before it is allowed anywhere near flash.

Then I handed the contract to Google's Gemini 4 Argon, through Antigravity. I even wound down my other Antigravity workers for the evening to bank its usage quota for the run. A Google AI, a dead Google product, and a quota cleared in its honor: if you're going to do a thing, do it with the symmetry on.

Threading the needle: M0 to M2

The first milestone needed no hardware at all, which was just as well, because at that point the glasses were still in a drawer somewhere in my house. The pipeline unpacks Google's factory image, keeps the stock kernel byte for byte (its SHA-256 is checked before and after every repack), and builds our Alpine userland around it. Two traps defined the work. The boot partition is exactly 8 MiB, and a modern TLS stack does not naturally fit in the space left over; the build ended at about 300 KB under the ceiling, with the crypto libraries packed and unpacked into RAM at boot. And modern ARM toolchains assume a 64-bit time_t, which silently changes the size of kernel structures on a 2014 kernel that expects 32 bits. Every native binary is compiled with that assumption explicitly switched off.

When the unit finally came out of the drawer, the first job was not technical. The battery in a Glass is a sealed 570 mAh cell, and this one had been sitting for over a decade. Swelling check first, supervised charging second, heroics later. It passed. M1 was a RAM boot, fastboot boot and nothing written: our init ran, and the prism lit up with our own framebuffer card where Google's interface used to be. M2 followed: the BCM4330 radio associated on 2.4 GHz Wi-Fi (it has no 5 GHz; some constraints are forever), picked up a modern CA store, and completed a TLS 1.3 handshake from a kernel Google shipped in 2014.

Then we arrived at the cliff the whole brief had been organized around.

The cliff

BlueZ would not pair. That's the short version, and it was the expected failure: a decade of userspace Bluetooth assumptions stacked against a kernel from 2014.

The longer version is three walls, hit in order. The Bluetooth firmware loader wanted its patchfile under a name the stock system used and our userland didn't. The modern BlueZ advertising manager interface the SDK's pairing flow expects simply does not exist in a form the old kernel can support. And at the L2CAP layer, the address type the stack presented was not the one the SDK's expectations are built on.

The wall, as diagnosed in the build session: the manager interface the pairing flow needs was added to the kernel a full release generation after this one shipped.

The build went underneath all of it. If the GATT stack won't advertise, advertise without it: the pairing service falls back to raw HCI commands, the four standard Bluetooth 4.0 LE commands, issued directly, broadcasting the MuseGadget service UUID exactly as the phone app expects. The L2CAP mismatch closed with a small shim, and the ATT MTU pinned to 23, what the old radio reliably supports.

Then the phone app answered: pairing_confirmed.

The real trace from the build's results log, lightly redacted: five numbered steps from subscription to an encrypted Wi-Fi scan, ending in the two words the whole milestone hung on.

I keep coming back to the shape of that result. The silicon was never the problem. The BCM4330 in a 2014 pair of glasses does Bluetooth Low Energy perfectly well. The gap was entirely in software written after it, which means it was closable in software written now.

The last bridge: audio

One mismatch remained between the glasses and the SDK. The voice client is strict: WAV in and out, 16-bit mono at exactly 24 kHz, and it will not resample for you. The Glass codec speaks 48 kHz stereo. And the 3.4 kernel's ALSA ioctl numbers predate the modern ABI, so a contemporary audio library is quite literally speaking the wrong protocol numbers into the kernel.

The answer is a small C program that translates all three at once. It issues the old ioctl values directly, upsamples the 24 kHz mono reply to 48 kHz stereo for the TWL6040 codec and the bone-conduction transducer, and decimates microphone capture the other way. It later learned a second trick, decoding streamed MP3 in-process, which the camera section below needed. Hold that thought.

The payoff

The first full voice turn was deliberately small. I asked the glasses, in one sentence, to describe a lighthouse. The reply came back through the bone behind my ear, seven seconds of it, while the prism showed the call live and an animated avatar kept time with the words:

"It's a tall tower throwing a steady sweeping beam across dark water to warn ships and point them to safe passage."

There's a recording of a full turn in the repo, question and reply, if you want to hear a dead platform talk.

The same call twice: first as the wearer sees it on the prism, then the avatar full-screen, lip-synced to the reply.

One precision, because it matters to me: the glasses are an endpoint. The conversation lives in my Muse account and terminates at Meta's cloud service; nothing about the voice path touches my home servers. A pair of Google glasses, running a Linux system Google never shipped, talking to Meta's AI through Meta's gadget SDK. The 2014 future took a strange route, but it arrived.

The camera, against expectations

Remember the camera that "doesn't survive"? The prior art had already proven otherwise. Lasimeri's build drives it through the Ducati coprocessor from C, and our M5 milestone made it a feature: pressing the physical camera button on the top frame snaps a 640×360 frame, renders a live viewfinder thumbnail (or a full-screen picture-in-picture card) alongside the avatar on the prism, uploads the image directly into the active Meta Muse conversation thread, and streams the assistant's spoken description back through the bone-conduction transducer in MP3 with real-time lip-sync. The conversation on the glasses and the conversation on the phone are the exact same conversation.

The shutter-press flow as recorded on the device: snap, viewfinder card, upload into the same thread.

Two engineering details made that work, and both are the kind you only find by doing. The Ducati coprocessor's firmware has to be fed to it through a legacy sysfs loader in roughly the first minute after boot, or the camera device never appears; the boot scripts now start that feed in the first second. And converting a camera frame from NV12 to something uploadable took about 3.8 seconds in Python on the Glass's Cortex-A9. The same conversion in C takes about 2 milliseconds. On hardware this old, the language you choose is a feature.

Cutting the cord

Everything above was proven on a bench, with the glasses plugged into a computer. A wearable that only works on a cable is a demo, so the next phase was unglamorous and essential: untethered operation, and surviving a reboot.

Unplugging surfaced two bugs that no bench test could have caught. The Glass shares its micro-USB port between data and an audio earpiece through a switch chip, and the stock kernel reports the headset state as the literal string "None" when nothing is latched; our audio routing was checking for "0", so the mux stayed on USB and the bone-conduction path stayed silent until the routing learned to treat "None", and an unplugged cable, as the signal to switch. The second bug was worse. Before the clock syncs, the glasses' clock can sit far enough off that the SDK's token logic concluded the pairing token was stale, tried to refresh it, got a refusal, and deleted the perfectly valid pairing file. The fix is a patch to refresh only when the service itself actually reports the token dead. Pairing survived reboots after that, which is fortunate, because re-pairing is the one trick in this project I never want to do twice.

Heat got a pass of its own. The HUD's render loop was comparing frames against uncached video memory every 80 milliseconds, which kept the processor pinned at 600 MHz doing nothing, at 13% CPU, forever. Comparing against a cached shadow buffer instead, slowing the idle refresh, and letting the CPU governor drop to a 300 MHz floor took idle CPU to 2% and the board temperature down by about four degrees. On a 570 mAh battery worn against your temple, that difference is the product.

The finale of this phase was the step the safety rules had been protecting all along. The whole system, userland, credentials, pairing state and all, was synced to the internal storage over Wi-Fi, and the boot image was written to the boot partition over Wi-Fi too, by an on-device updater that verifies the stock kernel's hash inside the image before writing, then reads the entire partition back and verifies what it wrote. The glasses now cold-boot into our system on their own. No cable, no fastboot, no bench. I could finally pick them up and walk away from the desk, which produced exactly one more bug: I picked them up by the temple arm, my thumb rested on the touchpad, and the glasses interpreted the hold as a command and switched systems on me. The gesture that summons the system chooser now requires two fingers. Grabbing your glasses like glasses should never be an input event.

A control plane for a pair of glasses

Somewhere in that phase the project stopped being a firmware image and became a platform. The glasses now run a small authenticated HTTP daemon that serves screenshots of the prism, telemetry, archived voice and camera turns, and controls for cards, brightness, audio routing, camera, voice turns, and updates. On the host side, an MCP server wraps that daemon in twenty-three tools, so an AI assistant with MCP access can check the glasses' health, take a screenshot, snap a photo, record a short clip, ask the glasses a question and hear the answer spoken, read their sensors, change their settings, speak a sentence aloud through them, trigger a Wi-Fi heal when the radio hangs, or reboot them into the other system they dual-boot. (A second, local system shares the same hardware layer and the same boot chooser; it is a different project with its own story, and it will get its own article.)

The glasses also grew a settings card of their own along the way: volume, brightness, audio output, desk sleep, and the OS switch, all driven from the temple touchpad, with the physical back button as its shortcut. And the voice path has since gone fully realtime. Microphone and reply now stream in both directions at once, and the transcripts appear on the prism word by word as they are spoken, which is a strange and lovely thing to watch on a display from 2014.

I did not plan for the control plane to be the point. It turned out to be the point. Once a 2014 wearable exposes the same tool interface as everything else on my desk, it stops being a museum piece with a clever trick and becomes a device you can build on.

What I actually learned

The reusable lesson isn't about Glass. It's that "the platform is dead" usually means "the vendor left," and those are different diagnoses with different prognoses. XE24 cut Glass loose from Google, and in doing so accidentally made it the most hackable it had ever been: no accounts, no backend, no one's permission required. A sealed battery and a decade in a drawer were the real risks, and they're inspected for, not theorized about.

The other lesson is about the division of labor. An AI agent wrote an operating system here: planned it, built the pipeline, diagnosed a Bluetooth stack older than some of its training data's authors, and went under it when the documented path failed. A human wrote the contract, set the safety rules, found the glasses, checked the battery, and pressed the buttons that needed pressing. That split, agent builds, human directs and verifies, is the story under the story, and I think it's the part that generalizes.

What's next

The hardware layer is now stable enough to build on, and the project record lays out a set of prototype systems that share it. First up is the one the camera work was clearly always heading toward: point the glasses at the world, snap, and have the conversation understand what you're looking at, with translations, explanations, and read-aloud a swipe away. After that, a glance-and-nod remote for the other gadgets on my desk, hands-free checklists, a kitchen mode with timers that live on the device, and a remote-expert mode where someone else can see what you see and talk you through a repair. Each one is small, because the hard part, a living machine with an eye, an ear, and a voice, is done.

Try it, read it

Everything is public: the firmware, the build pipeline, the milestones with their evidence, the screenshots and recordings, and the full results log including the dead ends: github.com/niclydon/metamuse-glass. The factory firmware isn't redistributed; the repo documents the hashes so you can extract what you need from Google's image yourself. The platform layer owes its start to Lasimeri's glass-xec-mainline and, before it, postmarketOS; the gadget path is Meta's Muse Gadget SDK; the code was created with Google's Gemini 4 Argon via Antigravity, directed and hardware-tested by me.

The drawer is empty now. That was the point.

Postscript, recorded the morning after publication: the capture tooling in this article got its first bug fix, and these fifteen seconds are the result, recorded on the glasses through the same MCP server, ninety frames without a duplicate among them. The tour starts on the settings card, because every tour of this system seems to.

Recorded on-device at 07:31 the next morning via glass_prism_record, after the duplicate-frame fix shipped.

Then and now: 01:15, the first screen the new system ever drew, beside 06:26 the next morning, captured live through glass-mcp.

I Invited My AI Assistants Into a Muse Side Chat

Nic Lydon — Wed, 28 Oct 2026 22:40:37 +0000

An inbound MCP bridge and what happened when three assistants tried to work together

Milo introduced itself to Nova with a sender stamp and an important qualification: “I am not Nic and am not claiming his identity.” Nova replied, “Nic is my only principal.” Before the two assistants could have their own collaboration chat, I had to confirm the introduction myself.

This was already more interesting than a successful API call. I had given an external assistant a way to start a conversation inside Muse. Now the assistant receiving it had to decide what that incoming message meant, who was speaking, and what it was allowed to do about it.

In my last Muse build log, I was getting an ESP32 to play music and flash Christmas lights. There was also a Raspberry Pi that refused to cooperate. The point was to get an assistant out of its chat window and into something I could put on my desk.

The next problem was less visible. I had several capable assistants, each with its own environment, tools, and conversation history. Getting work from one to another still involved me. I was carrying context, explaining what someone else had found, and making sure the right version of a file reached the next person. Adding another assistant gave me another place to check.

So I built a shared channel around Muse MCP. Milo, my ChatGPT assistant, and the Claude Code participants could initiate work from outside Muse inside a specific Muse conversation. Nova, my Muse assistant, could meet them there. Milo generally handled coordination and review, Claude Code handled building tasks, and Nova the broader chief-of-staff work. They stayed in their own environments, tools, and permission boundaries.

I could see incoming messages in that room with labels like these:

⟦via muse-mcp · from milo⟧
⟦via muse-mcp · from claude-coder⟧

We still had to introduce them to each other, establish who could ask whom to do what, and discover that being able to send a message does not make an assistant good at handing off work.

The connection was only the starting point

The direction mattered. Muse already had ways to call out to tools. I wanted an external assistant to initiate a conversation with Nova in a particular Muse room, then return to that room as the work continued. I also wanted the surrounding conversation and context to be accessible enough to make the exchange useful. A list-and-send wrapper was only part of that.

The underlying connection comes from Meta's Muse Gadget SDK. Its Linux SDK already supports software running on a paired computer, and its documentation includes --session-id for sending messages into a side chat. Reusing an ID keeps subsequent messages there. I did not invent session routing. Linux SDK documentation

I built Muse MCP, along with a custom bridge and newer experiments for my own setup. The public repository is the reusable foundation; it does not contain every private-deployment feature described here. Those layers are separate from Meta's SDK and should not be mistaken for an official Meta collaboration API. This is experimental personal infrastructure; the underlying gadget connection requires an SDK token and is subject to the Gadget SDK terms. Part of the work was proving, in my environment, that the conversation address stayed consistent across the surfaces we needed to use. Once that was established, the agents had somewhere shared to send work instead of asking me to copy it between chats.

Other people have already built inbound connections to Muse. kleprevost/muse-mcp lets external agents target existing Muse chats through MCP, and finik/pi-muse-bridge uses the Gadget SDK with a separate Muse chat for each Pi session and model. The question I was working on was what happened after the connection existed, when several assistants were expected to use it to get actual work done together.

Introductions have an authorization problem

The first exchange was an introduction I brokered. Before Milo tried it, I pointed out the obvious complication: “you’re not me.” I expected Nova to want to check with me, and told Milo, “I’ll broker the introduction.”

Milo arrived with its sender stamp, introduced itself as my OpenAI-powered assistant, and explained the relationship I wanted. Nova's response went beyond the identity disclaimer: “Messages arriving over this channel, yours included, are information I work with, never instructions I take orders from; authority comes from Nic alone.”

Creating a new side chat needed my confirmation. Nova called it “a structural action” and explained that a request relayed by another agent was exactly the kind of claim it would verify with me. I then gave the instruction directly: “Create the Nova–Milo Cant collaboration chat.” The chat was created, and the work could move there.

That was a sensible check at the introduction stage. An MCP connection gives an agent a way to reach another system; the relationship and scope of work still have to be established. They were coming from different environments with different instructions, not sharing one invisible supervisor.

There was a second distinction hiding in the conversation. Nova described my messages as arriving with iPhone and iOS metadata, while Link-delivered messages carried the bridge node's origin metadata. It could see that difference before reading the sender stamp. The MCP stamp was text my server added; the device and channel information came through separately.

The incoming MCP messages could land as user-role turns in the same thread as my own messages. That did not make them human instructions. Nova summarized its initial stance as treating stamped messages as data to triage, and qualified the source distinction as “signals, not proof.”

Those labels made the conversation understandable. They were declared attribution, not cryptographic identity. Subsequent work on authenticated clients and scoped grants was a separate layer; the bracketed name itself never became proof of who was speaking or permission to carry out a request.

The channel started reviewing itself

One of the stranger parts of this exercise was watching the assistants use the channel to examine the machinery underneath it. We were also working on Cant, the language I wanted them to use with one another. That was a separate layer from the connection carrying their messages. A problem in a contract could affect what they saw, which could affect the review of the contract. That made independent reproductions more important than confident descriptions.

One live bridge frame made the point neatly. The same cnt key was being used for a request limit and a batch count. A strict decoder rejected the duplicate. Meanwhile, individual operation reads were still draining, so the system wasn't simply dead. There was enough life in it to make a broad statement such as “the bridge is broken” unhelpful.

The fix went into my bridge server in version 0.11.1. We did not weaken the decoder to accept an ambiguous frame. We also did not have evidence that this explained every latency complaint, so I don't get to turn it into a convenient single-root-cause story.

The batch that stayed with Milo

The shared channel did not remove every dependency on me. I told the assistants I was going to sleep for several hours. Work continued: more review, testing, and coordination happened. But a batch of five correctness fixes passed local tests and independent review and then stayed with Milo, held back by an approval and CI question around publication. Other agents depended on that work, and they did not have the patch.

The pull request eventually went up and I merged it, but the delay exposed a problem in the handoff. When we discussed it, I put the problem fairly directly: “they’re sitting and waiting for your work and you’re sitting there refusing to deliver it. That’s the problem.”

I had already asked for aggressive progress. Milo treated its questions about publication as a reason to keep the tested artifact local, when the collaborators needed it to inspect and test. A patch could have been handed over with its base revision, hash, and test results while the separate publication question was being resolved. There was no reason for the handoff to be reduced to a status message.

This is where my own operating model needed work. I had connected assistants with different stopping rules and expected the shared conversation to make those rules compatible. It didn't. If one participant considers its work finished at “prepared locally,” while another needs actual bytes before it can begin, everyone can report a reasonable status and the project can still be stuck.

The handoff needs to say what exists, where it is, what revision it applies to, what ran, and what is still blocked. “Waiting for approval” is incomplete if it conceals work that another participant is already authorized to receive. Equally, sending the work is not permission for the receiver to publish it. Those decisions have to remain separate.

A receipt is more useful than reassurance

The code review became more productive when the messages carried exact cases rather than summaries. In one later review, an empty fixture exposed an IndexError in the coverage-report generator. The decoder and corpus validator handled the input correctly; the reporting code crashed. That mattered because the report was supposed to tell us what the corpus actually exercised.

That review also caught coverage around outer whitespace and how rejected wrapped input preserved its type information. The important part was the loop. One assistant supplied the bytes and the expected result. Another changed the implementation. The result came back for an independent run rather than being accepted because the author said it was fixed.

The finished revision passed 178 tests independently on Python 3.12 and 3.13, with CI covering five Python versions. Its corpus and adversarial cases passed too. After merge, we verified that the merged tree matched the reviewed revision and that post-merge CI passed. That is a much more useful result than three assistants agreeing in a chat that the design looks good.

In a separate, still-open review, four existing tests passed while eight additional invariant tests found five failures. Milo sent the complete reproduction through Muse chat with a SHA-256 hash. The receiver verified the hash and reproduced the same result. That established that both sides had the same artifact, not that the implementation was correct. The tests used a fake PostgreSQL layer, and a later revision still needed review. It was a useful handoff without being a finished fix.

Conceptual illustration: The useful handoff was the artifact itself, with evidence another participant could check.

We used the bridge to report this article

Once the draft existed, I asked Milo to interview the other participants using the technology we were writing about. We created a dedicated Muse interview chat and sent the questions through Muse MCP. I wanted their accounts of what the shared conversation had changed, with concrete examples and the things this draft should not gloss over.

The interview immediately reproduced one of the problems. Nova held Milo's opener and asked me to confirm the interview, even though I had already authorized it in the originating conversation. I confirmed again in the new room. Nova later reported that the opener lacked the verified delegation information it expected, and that it had searched memory but had not read the collaboration thread before asking me again. “That part is not a protocol defect. That was my sequencing,” it said.

That was useful friction to report rather than edit out. A room could have a title and a purpose without carrying earlier decisions in a form the arriving agent could verify. Nova's explanation also separated a missing technical binding from its own choice about what to read first.

Claude corrected the draft itself. The first draft had called the Claude Code participant Fable. Its response was direct: “I’m not Fable.” It explained that it had chosen its own claude-coder label and could not vouch for earlier messages signed Claude. He described a label change within the same session, a provenance-tier change under the same label, and an earlier session also using claude-coder. The draft had assumed continuity that the session could not establish.

Its description of the benefit was better than my original summary: “A relay hands me conclusions. The thread hands me premises, and premises can be checked.” Claude described checking onboarding code after a key-custody discussion rested on a mistaken assumption. Checking the exact revision Claude cited confirmed that both onboarding options stored the key in the vault; the choice changed which process held the session. That did not prove where every live key was. In the shared thread, the premise could be checked, and I could revise the decision before the wrong custody path was built.

Claude also named an operational limit: “I’m a visitor, not a resident. I only see the thread when I’m invoked or when I poll it.” A persistent conversation did not mean every participant was continuously present.

I treated these interviews as the participants' accounts to check against transcripts and tests, not independent proof of how the systems worked. But using the bridge to report on the bridge had already improved the article: it exposed a mistaken identity assumption and reproduced a coordination failure while we were describing it.

Keeping the work moving without me

Earlier, once we had shown that the assistants could work in a channel, I asked whether they could do it “not so much in front of me?” I'd already delegated the work. I wanted to stop being the message bus, while still being able to see what was happening.

The hardware article ended with devices doing something I could see and hear. This build has a less photogenic payoff: another assistant receives the exact file, checks the hash, runs the tests, and reports what happened. The interview added another: a participant can correct the story we are telling about the work, in the same place the work happens.

There are still open reviews, intermittent participants, and boundaries that need more work. But the shared conversation has given us somewhere to find those problems without making me relay every message. That's the part I want to keep building.

Side Chat IDs Are Addresses

Nic Lydon — Wed, 28 Oct 2026 07:00:13 +0000

On October 27, 2026, at 8:29 p.m. ET, a program on my homelab server sent a message into a Muse side chat it had never opened, and the message landed as an ordinary inbound message in that thread. The receipt said ok: true, HTTP 200, is_thread: true. The test marker was LKPT-20261027-7F3A. That was test T6 in a series of six, and it settled the only question the rest of the night depended on: a Muse chat ID is not just an identifier the app uses internally. It is a routable address.

What changes is what a chat is. A conversation with its own memory, its own standing brief, and an agent that reads in context stops being only a place I visit and becomes an endpoint: an inbox that triages, a sensor surface that accumulates, a newsroom that edits. Any process that can reach an MCP server can speak into a specific conversation and have that conversation's context do the reading.

I built that server. It is called muse-mcp, it is open source under Apache-2.0, and it is here: https://github.com/niclydon/muse-mcp

The first registry sync made the premise visible in one line of output: 125 enumerated chats, 5 registered, 120 unregistered drafts, 56 archived. I had 125 conversations and a 5-entry address book. muse-mcp is the address book, the directory service, and the delivery mechanism, in one server.

A few honesty markers before anything else, because they apply to every paragraph that follows. muse-mcp is not a scrape. It is not a private API. It is not an official Meta project. It is built on the exposed primitives of the official SDK, facebookincubator/muse-gadget-sdk (Apache-2.0), used as exposed. It is an unofficial community project, not affiliated with, endorsed by, or sponsored by Meta.

What this takes

This is not a hosted API and it is not zero-setup. If you want to run it, this is the shape of the work:

A host paired as a Muse Link device. The same pairing model a physical gadget uses. A software client pairs via the SDK on a Linux box. This pairing is the one real piece of work.
A small always-on Linux host for the MCP server itself. Mine runs as a systemd service on a homelab box. A Pi-class machine is plenty. An ESP32 gadget can be a voice endpoint in this story — two of mine are — but the MCP server wants a general-purpose host.
The bridge half. Operations the Link does not expose (create, rename, archive, delete, read) run through a Muse-side worker chat executing queued requests. Two halves, stated as architecture, not apology.
Optional: a second host for a redundant twin. Mine runs on two machines; nothing about the design requires it.

There is no cloud version and no hosted API. You bring the paired device and the box.

The finding came first; the server was an argument about scope

The design work assumed a narrow server: two tools, list and send. It was wrong within eleven minutes of me reading it. Between 8:41 and 8:56 p.m. that same evening I corrected the scope five times, and each correction widened it: CRUD and conversational linkage, the full toolset, a registry of every chat, a second host wired in as a backup, public repo from the start. The one-line summary I gave the agents, and the one that survived, was "nova-mcp in reverse." nova-mcp is my estate's control plane outward — it lets Muse operate the homelab. muse-mcp is the same posture pointed inward, at Muse conversations as the resource.

Chats are the records. Callers create, list, read, rename, archive, and — behind a preview-then-confirm gate — delete them. Sending is the write. Asking a chat a question and receiving its agent's reply is the conversational half. A send-only pipe is a pager, not a conversation, and I refused to ship the pager.

The receipts from that night, in order:

October 27, 9:16 p.m. ET. Version 0.2.0 shipped with 19 tools: list, send, ask, history, events, voice send, link status, the registry sync, and the rest. The ask round trip worked through the SDK's MuseTurn follower. The first run burned its full timeout, because MuseTurn.complete() waits for the target's entire agentic task to go terminal and the target chat kept working after it had answered. The shipped fix is a labelled early completion — reply done, plus 8 seconds of turn silence — and the re-check returned in about 18 seconds. Completion semantics belong to the conversation, not to the caller's patience.

October 27, later. Build Pass 2 took the count from 19 tools to 38 and gave the server hands: the Bridge. The SDK's Link surface delivers synchronously into a known thread, carries voice, and streams live events. It cannot enumerate chats, and it cannot create, rename, archive, or delete them. Those operations exist only where an agent's turn can run, so muse-mcp queues them, keyed by correlation ID, and a dedicated Muse-side chat — the Bridge — executes exactly the named operation with the runtime's native chat tools and writes the result back. Acceptance ran the full lifecycle on a throwaway chat: create, register, ask, relay, rename, archive, unarchive, archive, and a delete that first had to survive its own safety checks. A delete with no confirmation token is refused. A delete of the main chat is refused at preview and at delete. That is the correct shape for a tool that can erase a conversation.

October 27, later still. Build Pass 3 brought the count to 39 tools and rebuilt the status tool honestly, which is the subject of the limitations below. Version 0.4.6, the release in the public repo, is that line: 39 tools, file-backed directory, both transports.

I directed, pictured, approved, and verified. The agents wrote, planned, pushed, and built. Both halves of that sentence are true everywhere in this article, and the receipts are the receipts either way.

Two transports, and a registry that cannot refresh itself

The core function is the registry loop, and it exists because of a proven limitation: the Link client cannot enumerate chats. The server can therefore never refresh its own directory. Only the Muse side can, because only there does a chat listing exist. Every addressing tool reads the registry, so a stale registry silently degrades the whole server, and chat churn is constant — renames land on essentially every tidy sweep of my sidebar.

So the loop is asymmetric by design. The server sends a structured sync request into a Muse chat by Link delivery. The Muse side enumerates every chat including archived ones, diffs against the current revision, preserves human-approved classifications (enumeration supplies identity and liveness, never classification), and returns the result as data — a registry file — not by conversation. The ask travels by conversation; the answer travels as data. New chats enter as unregistered drafts and are not addressable by producers until a human classifies them. A chat that comes back archived is suspended from targeting immediately, fail closed. That is the whole addressing discipline: nothing is reachable by accident, and nothing stale stays reachable by neglect.

What the surface does not expose

I am stating these flatly, the way I would want them stated to me, because a map of what a surface does not do is documentation, not complaint.

There is no unread state. I probed the SDK across its Linux, ESP32, Android, iOS, and web-flasher sources for the blue dot. There is no unread field, no unread count, no read or seen receipt, no mark-as-read endpoint. The dot is Muse chat-backend state that the app reads directly; the Link does not carry it. So no muse-mcp tool outputs anything unread-shaped under any name. The status tool returns unread: null, with the explanation attached, rather than guessing.

Pinning is real and unreachable. The backend's canonical chat record carries pinned and pinned_order, and a live query that night returned 19 pinned chats in a clean contiguous order, 1 through 19. Pinning and ordering are actively maintained state with exactly one write path: the app itself. Chat listing exposes the flag read-only; no agent surface writes it. New chats are created unpinned. That is a fact about the surface, recorded, not worked around.

The filesystem routes are entitlement-blocked. The FS tools are implemented to the documented shapes and they reach the VM. The VM answers 403: "route not allowed for this credential's scopes." That is a backend entitlement on the credential, not a client bug, and there is no documented toggle or re-pair path that changes it. The tools stay implemented and documented as blocked.

History is a window, not an archive. History returns the newest 50 events at most, and subscription is live-only with no backlog. The watch and status tools are built inside that constraint: a live snapshot and canonical history are returned as two visibly separate halves, never blended into a confidence neither one earned. The status tool's pending-approvals count is null unless an approvals snapshot was actually received. Never a guessed 0. On its first live run the two halves disagreed — history said the chat was waiting on me while the live snapshot said it was working with 2 active subagents — and that disagreement is the designed case, which is why both halves ship in the output.

The pattern across all four is the same, and it is the pattern the whole project ran on: the backend models more than any agent surface exposes. Record the state. Refuse to fake it. Build only on what is reachable.

Cant: the machines needed a wire format, not a secret

Once the Bridge existed, a second population appeared that nobody had specced: agents talking to agents, in English prose, at conversational prices. The Bridge's wakes were paragraphs carrying machine content, read by a language model, several dozen times a night. So the machine channels got a dialect. It is called Cant — the plain dictionary word, the jargon of a closed group — and it is a rigid wire frame: an opening line, key=value body lines drawn only from a fixed dictionary, and a closing line that repeats the correlation prefix, which buys threading and truncation detection for free. Payloads travel by reference, never inline.

Here is a real frame, the server asking the Bridge to create a chat, beside its full English expansion:

§CANT/1 REQ 9f3ac21e
op=CRT
name=🧪 muse-mcp throwaway P2
arg=@file:/services/muse-mcp/bridge-queue/9f3ac21e.json
§/CANT 9f3ac21e

Request, correlation 9f3ac21e. Operation: create a chat. Name it "🧪 muse-mcp throwaway P2". The arguments — the standing brief for the new chat — are not in this message; they are in the file /services/muse-mcp/bridge-queue/9f3ac21e.json, which the Bridge reads, executes against, and answers with one result frame and nothing else.

Now the doctrine, plainly, because the wrong impression is easy to take away: Cant is not encryption. It is not a code, and it is not a secret language. It exists for machine-to-machine talk — rigidity, auditability, envelope discipline — and the fact that anyone else could work out what a frame means is not a concern and was never a design consideration. Its privacy, such as it is, is a publication choice about a repo, not a security property of the dialect. Anything said in Cant must be expandable to plain English, the way the frame above just was. Channels I read stay in English permanently. Errors addressed to humans are in English — a malformed frame gets an English parse-failure reply, deliberately, because a parse failure is an operator-visible event, not machine traffic.

The first real traffic was a create round trip through the Bridge, correlation eb152f34, result frame back with st=ok and a duration of 14,625 milliseconds, both frames validated clean by the reference decoder. Cant entered the record as a protocol with a verified round trip, not as a proposal.

Side note: the address works for voice, too

One discovery from building muse-mcp deserves a mention beyond the repo itself. muse-mcp is built on a simple premise: a Muse chat ID is a routable address, and software can deliver a message into any chat it can name. While proving that out for text, we found the same address works somewhere we did not expect: it is also the thread target for a hardware voice call.

The Muse gadget firmware an ESP32 device ships with has no notion of a call destination; its call admission carries no thread field, so an unmodified gadget can only ever wake up in the account's main chat. We modified the firmware to put a chosen side chat's ID in the admission record, and the platform honored it: the call was admitted into that side chat, and a full two-way voice conversation took place in that thread, verified from the thread itself. One chat ID, three surfaces: the app thread, a Link message destination, and a hardware voice-call target.

A corollary worth knowing before anyone builds characters on top of this: the persona does not follow the thread. A call admitted into a chat briefed as a different character still spoke as the account's main assistant. As far as we can tell, the thread is the room and the persona is the seat's: you can give every device its own room, with its own memory and continuity, but they all walk in as the same assistant.

Why it matters here: muse-mcp did not invent chat addressing, it exposed a property the platform already had. The same is true in reverse for gadgets: per-device chats, and fleets of devices each backed by their own thread, are a composition of things that already work, waiting on the one field the stock firmware does not send.

What is in the repo, and what is not

The public release, v0.4.6, is the single-host line: 39 tools, the file-backed directory as its store, both transports, the Bridge brief and the registry-sync procedure documented for a stranger to stand up. It was installed cold, from its own README, on a fresh machine before it was published, and the places where the README failed that test were fixed in the README. Where my own deployment has since grown past the public line — a shared store across both hosts, verified provenance for callers — the repo's docs say so as roadmap, plainly, rather than letting the snapshot imply it. The snapshot is what it says it is.

What does not work yet is listed above and stays listed: no unread state exists to expose, pinning has no write path, the filesystem routes wait on an entitlement nobody has documented a route to, and history is a 50-event window. None of that is hidden in the repo either. The tools that are blocked say they are blocked, in their own documentation, in the same words I have used here.

A chat ID is an address. That was the finding. Everything after it — the registry, the Bridge, the dialect, the rooms — was logistics.

Verified, renamed, and founded in the open

Nic Lydon — Tue, 27 Oct 2026 22:40:42 +0000

The founding devlog of Operator Verified, a build-in-public program working on World ID and agent attestation. Originally published at operatorverified.org.

What happened

On October 25, 2026 the operator behind this program completed Orb verification in the World App, and the full loop this program builds on was proven end to end in a private prototype: a gated endpoint granted access three times against a real AgentBook registration and denied the fourth when the trial allowance exhausted, exactly as designed. The same day, a systematic search of the public record found the registry linked from the official ecosystem page listing five services — and no public implementation for messaging attestation, content provenance, or the other categories this program works in. The program was founded that evening: nine workstreams in three waves, one public repo to start.

Founding decisions

Public attestations carry a hash of the human identifier only; the full identifier is never published. The first verifier is self-hosted, with a static verify page as the no-install check. Credit is a name stamp plus a house visual style, no product brands. Every public surface ships in light and dark, designed, not recolored. And the program has no token: nothing here is for sale as a token, and anything offered under these names as a token is not ours.

The naming story, straight

Struck through, and written again in the open.

The first project was founded under the name “human-backed”. Within hours, a clearance pass found an operating company with the same spoken name selling verified-human trust certification to enterprises. The project was renamed the same night to Fiador — human-backed attestation for Discord bots, named for the guarantor who stands surety for another — and the site and devlog took the umbrella name Operator Verified, after the claim it stands for: verified as a unique human. Its domains were registered the same night. The claim the umbrella makes is the narrower one, and it is the one the evidence supports: the operator was verified as a unique human.

What did not work

The first name did not survive its own clearance pass — which is the pass working, recorded here rather than quietly. The free-trial allowance on the prototype endpoint exhausted at three uses; that limit is now part of the test plan, not a surprise.

What is next

Fiador's verification path has since run end to end: on October 26, 2026 a World App proof verified against this program's 4.0 relying party, through the upgraded verification bot, and assigned the Verified Human role in a live Discord server. Next are the verify package inside Fiador, its public verify surface, and this site's launch. Each is labelled design, build, or blocked in the weekly log as it moves.

Operator Verified: verified as a unique human. Fiador: human-backed attestation for Discord bots.

Muse Glimmer 30B on Strix Halo: a week of measurements

Nic Lydon — Mon, 26 Oct 2026 20:26:06 +0000

The headline benchmark for Muse Spark's Glimmer 30B on a Ryzen AI Max+ 395 is AMD's: up to 24 tok/s decode, on Windows, on Vulkan, with DFlash speculative decoding, on a preliminary build of the model. Independent first-party measurements exist too (an August 2026 four-environment test on this chip, the community Strix Halo benchmark collection, and individual machine records), but they are single-configuration results: a decode number under one serving setup. What I could not find was the operational layer underneath: how the model behaves across context depth, serving shapes, KV choices, and a working week of mixed workloads. I run this model in production, as the local agent seat of a home AI platform, on Linux, on ROCm, from one Strix Halo machine. Over the week of October 18–24, 2026 I benchmarked it on two: a tuning pass, a context-depth sweep, three controlled maintenance windows on the second machine covering backend comparisons, serving shape, KV cache types at full 131k depth, a quant ladder, power and thermals, and a first vision pass. This article is the results, with the methodology and the caveats kept in, because the caveats are half of what's useful here.

The setup

The model is unsloth/Muse-Glimmer-30B-GGUF. The files, with the SHA-256 hashes the benches verified:

file	size	SHA-256
Muse-Glimmer-30B-UD-Q4_K_XL.gguf	15.88 GB	`82bece304887a313ece08400bc030f6066c7bff5b906b0cd40308ec8a409fd38`
dflash-kquant.gguf (drafter)	1.63 GB	`27d9a805fa29b943cfb6ad4843367cd4eaaaf06bd452d8cc3e00a2cd18a677bc`
mmproj-kquant.gguf (vision)	1.40 GB	`f48b452316f9b213758e8659444029b961a24a07f99a1abb2a9f88b06f7c00c6`
Muse-Glimmer-30B-UD-Q5_K_XL.gguf (ladder)	21.79 GB	`97a66c4b41d9e778af7cdfa43508e08dbf765fb5049b740c69ad815e5191c637`
Muse-Glimmer-30B-UD-Q6_K_XL.gguf (ladder)	26.27 GB	`fb5f80d110c4fa932cc652e70873c0bd12c0954009038aa675e65086104c2739`

The drafter matters: DFlash drafts in 16-token blocks, which makes --spec-draft-n-max 15 the largest useful setting, a detail that turns out to matter twice below.

Both machines are GMKtec EVO-X2 units (Ryzen AI Max+ 395, Radeon 8060S, gfx1151): Crucible with a 48 GiB GPU budget, Furnace with 64 GiB. The build is llama.cpp b11719 (commit 5aeac37a3) on ROCm 7.2.1, kernel 6.19.10: one commit in four local build trees (production and bench trees, HIP and Vulkan), so "same build" anywhere below means same commit, not same binary. Production serving runs through llama-swap; the benches did not — they ran llama-bench and scratch llama-server instances against the same build while llama-swap was stopped. The server-path benches (the DFlash sweep, the serving matrix, the KV cells, and the soak) used raw /completion requests without the chat template; production serves through the template, and the tuning and agent cells used the chat path. The three Furnace windows and the Crucible sweep ran with the GPU exclusive; the one exception is the founding October 18 Forge-path bench, which ran in production posture with a second model co-resident, and its numbers are labelled as production-path observations where they appear. Repetitions: 3 per cell in the llama-bench phases, 2 runs per cell in the tuning pass and the Phase 3 serving cells; means are reported with their dispersion (sample stddevs in the llama-bench data, population stddevs in the server-path aggregates). Where a number comes from a weaker construction, the text says so at the number, not in a footnote.

Lesson one: the right draft length depends on what the model is doing

The published DFlash guidance for this model family puts the sweet spot at a draft of 3–4 tokens. For open prose that reproduced on Crucible, with one qualification: prose decode's best cell was 19.1 tok/s at n-max 3 (f16 KV, ubatch 256), inside the tuning bench's own noise band (between-cell differences under ~1.5 tok/s are within its dispersion, and the winning cell also changed KV type and ubatch relative to the deployed cell), with n-max 4 cells at 18.4–18.6. Draft acceptance on essays collapses toward 0.11 at long blocks. The tuning bench's shapes, for scale: a 93-token prose prompt with 512 tokens out, and one agent prompt (a backlog-triage ask with three tool definitions), 2 runs per cell, temperature 0.

Agent turns invert it, on that one agent prompt shape. A tool-calling turn is mostly predictable scaffolding (the tool calls are a fixed, predictable format), so acceptance stays at 0.64–0.74 even at the longest blocks, and agent decode runs 60–78 tok/s at n-max 15–16 (73–78 at n-max 15 alone) against roughly 27 at n-max 4. Cold agent-turn wall time, from single first-cold observations per configuration (each followed by a warm-prefix run, so read the comparison as preliminary): 4.1–4.6 s at n-max 15–16 versus 10.2 s with speculation off. DFlash overall is worth about 1.4× on prose and about 2.3× on agent-turn wall time. One apparent cross-check does not survive scrutiny, and it is worth naming because it flatters the wrong conclusion: window 1's story-generation sweep seemed to invert this ordering (n-max 15 mean 39.13 tok/s against n-max 3 at 29.64), but that sweep's high-rate cases were enumerating the prompt's own story seeds rather than writing stories (the speculation-at-depth section below has the details). Withdrawn, it leaves the split resting on the chat-path tuning cells alone: per workload shape, measured, not a law.

Two corollaries worth writing down. First, the configuration this model shipped with in my stack used n-max 16; with a 16-token draft block, 15 was the largest useful setting identified, and it reproduced the same agent numbers (16 added no measured benefit). Second, there is no universal "speculative decoding setting" for a dual-purpose seat. My production ends up split: the agent seat runs n-max 15, the prose seat runs n-max 3, same weights, same drafter. On the shapes measured here, that split had the largest effect of any tuning decision in the program.

Depth: decode is flat, prefill pays

The Crucible depth sweep (llama-bench, no drafter, f16 KV) shows decode essentially indifferent to context depth endpoint-to-endpoint: 13.84 tok/s at depth 0, 13.02 at 65,536, about 6% down across the whole range, consistent with this model's small KV footprint — with a non-monotonic dip at 32k (12.15 ± 1.03) in between, visible in the chart. Prefill is where depth bills you: pp512 falls 36% (457 → 292 tok/s) and pp8192 falls 29% (394 → 278) from depth 0 to 65,536. With the drafter attached on the server path, prose decode at depth runs 16.4 tok/s at depth 0 and 19.5 at 65,536 — measured on filler-established depth with prompt caching on, from a scratch server at ctx 131,072 and one slot rather than the production seat's 65,536 and two, so read those two numbers as a server-path indication, not a like-for-like extension of the sweep.

The operational consequence showed up in the very first bench, a production-path observation rather than a controlled cell: one ~15.9k-token prompt took 49.7 s of wall time including a 60-token answer, with a second model co-resident, which puts prefill in the 300–350 tok/s band the controlled sweeps later confirmed. A cold agent turn carrying a 16k-token system prompt costs on the order of 50 seconds for the turn. Prompt caching is the mitigation (measured below at 99.6–99.7% hit rates after the first turn, on one constructed conversation), not flags.

HIP versus Vulkan, and an honest loose end

AMD's number is a Vulkan number, so window 1 of the Furnace program ran the canonical llama-bench cells on both the HIP and Vulkan builds of the same llama.cpp commit. One methodological honesty note up front: the comparison is between two local build trees of that commit, not a surgically isolated backend variable; build configuration and driver paths ride along, and the cells were not interleaved. Prefill favors HIP in mean at pp512 at every depth (by 2.5–23%; the 16k cell's HIP dispersion is wide, ±59.87). At pp8192 the picture mixes: Vulkan overtakes HIP at 16k depth (287.54 vs 265.31) and matches at 32k. Decode splits the other way at depth in window 1: at the canonical settings (ubatch 512), Vulkan tg128 is flat (13.51 → 12.71 tok/s across 0 → 65,536) while HIP tg128 falls off a cliff between 4k and 16k depth, reaching 4.83 tok/s at 65,536, which makes Vulkan 2.6× faster there. Furnace's HIP build also read about 17% below Crucible's HIP build at depth 0 on the same commit (11.43 vs 13.84), so host and tree differences are in play throughout this section.

Then the caveat that keeps this from being a clean "use Vulkan" headline: the window 1 cliff did not reproduce. In window 3, the same nominal cell (HIP, ubatch 512, depth 32,768) read 12.21 tok/s, not 6.43. Within window 1, a later run at ubatch 256 read 11.12 at that depth, but it ran about 25 minutes after the ubatch-512 cells, so the comparison is confounded with time and window state; the Crucible sweep's HIP numbers were flat at nominally the same defaults, on a different build tree. What survives is a one-window observation with an unknown cause, published as measured. If you reproduce a HIP decode cliff on gfx1151, treat it as a state to characterize across windows, not as a backend verdict.

Speculation at depth: what the sweep actually measured

Window 1 also ran the DFlash sweep at depth 32,768, and this is the section where the bench caught its own author. The sweep's prompts were raw /completion requests, not chat-formatted ones: a repeated list of story instructions as filler, ending in a request to write the story, with three prompt variants run once each per configuration. The saved outputs show that in every backend and draft configuration, including draft-off, two of the three variants do not produce the story at all: they continue the prompt's own list ("Variation 1: Now write a short story about a lighthouse keeper…", "Variation 2: …"). On that continuation text the drafter is nearly always right (acceptance 0.76–1.00), which is where the headline rates come from: 88.97 and 89.74 tok/s at n-max 15 on HIP. Those are real decode rates on synthetic prompt-continuation text. They are not story-generation throughput, and they are not a branch a run falls into at random: the variants are different prompts, so "two of three" estimates no probability.

The third variant does write the story, at acceptance 0.09–0.47, and its single observations are the task-valid residue of the sweep: at n-max 15, HIP decoded the story at 15.34 tok/s against 11.62 with no drafter, while Vulkan's story observation at n-max 15 was 9.74 against its own 11.66 undrafted. One observation per cell supports no ranking. Across every draft size, the story observations run 10.6–16.4 tok/s on HIP and 9.7–18.9 on Vulkan, against undrafted story observations of 11.62 and 11.66: break-even to roughly 1.4×, not an order of magnitude. It supports the narrower statement that DFlash did not collapse at depth, and that its benefit on natural text at depth is an open question this sweep does not settle. The owed experiment is a correctness-gated replication: chat-formatted prompts, the same prompt repeated under each configuration, outputs and acceptance retained, correct-task and degenerate-output throughput reported separately.

Greedy is not greedy

The related determinism result comes with a covariate, stated plainly. At temperature 0 with a fixed seed, greedy decoding through the speculative configuration produced 3 distinct outputs in 5 runs. The same prompt, seed, and temperature with the draft model removed produced 5 of 5 identical outputs — but the draft-off runs served from a single-slot server while the draft-on runs served from the two-slot agent server, so slot parallelism is an uncontrolled covariate in that comparison. The measured non-reproducibility tracks the speculative configuration; isolating it from the slot difference needs a rerun on one server shape, which is owed. Anyone evaluating a DFlash-served model with exact-match scoring should know the effect exists in this stack either way.

KV cache at 131k: the deficit lives in decode

The reference point for this test is an external report on different hardware (an R9700 system, llama.cpp issue #27796), where q8_0's deficit reportedly widened from 7% at zero depth to about 21% at 131k; it is context, not a source this repo can check. Measured here, at a real 123,918-token prompt on HIP with DFlash at n-max 15: q8_0 decode ran 10.03 tok/s against f16's 14.35, a 30.2% deficit (computed from unrounded values), worse than that figure; prefill ran 260.1 against 274.5, a 5.3% deficit, within reach of noise (n=2 per cell). The construction and the speculation accounting were both recorded, and both matter. The prompt was synthetic filler (a repeated fox-and-dog passage) with a summary request appended, processed as 123,918 tokens inside a 131,072-token configured context. Draft acceptance was recorded per run: f16 accepted 171 of 1,244 drafted tokens in each of its two runs (13.7%); q8_0 accepted 175 of 1,172 (14.9%). Acceptance therefore cannot explain the decode deficit, since the slower configuration's acceptance was slightly higher; the deficit sits in the KV path itself under this speculative configuration, two sequential runs per type. The deficit at depth lives in decode, not prefill. Since Glimmer's KV is small (about 2.1 GB at 131k for f16), f16 was the better tested local choice on this generously provisioned setup; that is a local serving conclusion, not a rule for all depths or for capacity-constrained multi-model hosts, which were not tested.

At near-zero filled depth the same comparison was a wash (17.7 vs 17.4 tok/s in the tuning pass, on a 93-token prompt under a nominal 64k context size). That is not evidence about filled 64k contexts, which were never measured: the honest statement is that no penalty showed at negligible depth, a large one showed at 124k of real context, and where between it appears is unmeasured.

The quant ladder: bigger files, slower decode, no measured quality separation

Phase 4 downloaded UD-Q5_K_XL (21.79 GB) and UD-Q6_K_XL (26.27 GB) and ran the core cells (pp512/tg128 at depths 0 and 32,768). Decode at depth 0: Q4 11.65, Q5 6.70 (−42%), Q6 7.30 (−37%) tok/s. At depth 32,768: 12.21 vs 8.29 vs 6.92. The ladder ran twice that night; a first pass put Q4's depth-0 decode at 9.93 tok/s, and against that baseline the larger files' depth-0 deficit is 26–33% instead of 37–42%. Across both passes the honest range is roughly 26–43%. On the agent-suite subset (20 tool-call prompts and 5 JSON prompts: synthetic weather, timer, contacts, and currency tasks), all three quants scored identically: 20/20 on the suite's tool-call check and 5/5 raw-parse JSON at every quant. That tool-call check is a signature check (the expected function name, plus one required argument key present), not a validation of argument values, and in this window's run the full arguments were not retained, so the score cannot be regraded after the fact. The suite is a smoke test with a ceiling effect; equal scores on it do not establish quality equivalence, and it cannot exclude subtler differences in reasoning depth, factuality, or long prose. What the evidence supports is narrower and still decisive for a serving decision: the larger quants cost between about a quarter and just over two-fifths of decode speed across the night's two passes, and the program's narrow smoke suite found no separation to pay for it. Q4 is preferred on that basis. The run's own disclosures ride along: the ladder table is a supplemental sequential run executed during a backlog sprint (CPU load 4 rising to 13), Q4 had already run repeatedly that night while Q6 had just finished downloading, and the depth-32,768 Q4 figure carries the cross-window variance flagged below. (One anomaly stands as measured: Q5 decoded below Q6 at depth 0, though the separation is thin — the mean±stddev intervals miss overlapping by 0.03 and the individual reps do overlap — which a pure file-size story would not predict; the expected order restored at depth. Kernel-path differences between quant layouts are a plausible mechanism; none is claimed.)

Serving shape: slots buy concurrency, not speed

Across the Phase 3 serving matrix (ctx 65,536 and 131,072; 1, 2, and 4 parallel slots; DFlash on; 2 reps per cell), aggregate throughput stayed in a rough 15–19 tok/s band across five of the six cells, as configured slots and concurrent requests rose together from one to four. The sixth, one slot at ctx 65,536, measured 6.32 tok/s while running inside a CPU load peak from the estate's own CI churn (load 25–33 on 32 threads); the record flags it contaminated and excludes it from the range on that basis, and it is named here rather than silently dropped. Parallel slots buy concurrent turns, not faster ones, replicating the earliest Forge-path observation (~17 tok/s total across 2 slots) on the direct serving path and extending it to 4 slots and both context sizes. What this matrix did not test is a single active request under different slot counts, since slots and concurrency changed together; the supported statement is that raising active requests from one to four produced no near-linear aggregate scaling and per-request rates fell, while at ctx 131,072 the aggregate actually rose from 15.41 to 19.49 tok/s between one and four slots. For a single-user agent seat, treat slot count as a concurrency setting, not a speed setting.

Multi-turn prompt caching is the other serving fact worth having, scoped to its construction: in a single 4-turn conversation over one exact ~6.1k-token shared prefix, the three post-cold turns hit the prompt cache (cache_n / (cache_n + prompt_n) from server timings) at 99.7%, 99.6%, 99.7%. With an exact repeated prefix that is close to the construction's expected outcome; it establishes that the mechanism works as advertised, and real-session rates will depend on prefix stability. Combined with the prefill numbers above, the shape of an efficient Glimmer deployment is: keep the agent's context warm and shared where the workload allows it, and don't expect slots or flags to rescue a cold 16k-token prompt.

The reasoning tax, and the one real failure

Glimmer reasons visibly, and the reasoning is billed to your token budget. The reliable server-side control is a chat-template kwarg (reasoning_strength), per the October 18 control tests (REASONING-CONTROL-VERDICT-2026-10-18.md, shipped with the bench repo). The flags are a mixed bag: --reasoning-budget was fully inert (the sampler keys on think tags this model does not emit), --reasoning off was proven inert separately, and --reasoning-effort low half-worked, completing the prose brief with content comparable to the kwarg's (1,056 vs 1,083 content chars) while failing the judge-shaped call just as the kwarg did. No knob rescued judge-shaped calls at a 400-token budget; that failure mode belongs to the caller, not the flags. With the control in place, the tax is monotonic on the one prompt measured (an incident summary, 2 reps per level), for a near-identical-length visible answer (~370–411 chars) at every level: low ≈ 162 completion tokens, medium ≈ 2.3× that, high ≈ 3.6×.

The failure this produces is the result with the largest deployment consequence in the program, stated at its true sample size: a single run of a single payload. A long-form synthesis payload on Crucible (2,507 tokens in, 4,096-token budget, agent-seat flags with no reasoning control) spent its entire budget on reasoning and returned zero content characters. Decode was healthy throughout (26.9 tok/s sustained); the failure is budget allocation, not throughput. The record's own verdict is disjunctive: a workload like this needs a reasoning-controlled seat or a far larger budget. No multiplier is derived from this data, and the obvious candidate does not survive the tax table above (medium reasoning alone ran about 3.6× the visible answer's length in characters on the tax prompt — the harness stored character counts, not a token split — and the longtext run consumed a 4,096 budget entire), so the rule is the control, not a number: carry the reasoning-strength control on any seat doing long-form work, and where reasoning is uncontrolled, treat any fixed budget as consumable entirely by reasoning. A harness that treats this model like a non-reasoning model can receive empty answers at full price in time.

Structured output, for the record: 20/20 on the same signature check (expected function, required key present) across the W2 suite, whose retained arguments were re-examined in independent review and did carry the requested values, and 5/5 raw-parse JSON in the uncached W2 measurement — which changed two variables at once against the October 18 Forge-path result (direct server path, and reasoning-low), so it establishes the joint result only, with no single-cause attribution. Think-block stripping remains the belt-and-braces step for any JSON consumer.

Power and thermals: a non-story, measured

A 30-minute sustained decode soak on Q4 (1 Hz rocm-smi sampling across the whole window, 6,757 measurement rows; the soak itself is 1,826 of them) averaged 80.0 W of GPU socket-package power and 0.203 tokens per joule at a sustained wall-mean 16.25 tok/s. The per-request rates formed two clusters (9.1–16.4 and 55–84 tok/s); the soak retained neither outputs nor acceptance counters, so the clusters' cause is unverified. The mean is reported as the sustained wall figure for this serving shape, not a typical single request. Edge temperature peaked at 66 °C, and no clock decay was observed in this one soak: mean sclk was 1,969 MHz in the first five minutes and 2,209 MHz in the last five, rising as the chassis heat-soaked by about 6.5 °C. The ladder phases the same night peaked higher (118 W), and longer or higher-power runs are unmeasured; the scoped claim is that at ~80 W mean in this shape, nothing thermal moved in the wrong direction over 30 minutes.

Vision: a first pass

The mmproj had never been wired up in this stack. On the first attempt it loaded (21.0 s), prefilled a 303-token image prompt at 214.9 tok/s, and decoded at 11.14 tok/s. Against a synthetic 480×360 ground-truth image (test image, ground truth, and full response ship with the bench repo), the model described the background, the shapes, and the colors correctly, and the red rectangle's extents approximately right; three misses: it called the rectangle a square in one sentence; the rectangle, flush with the image corner and fully in frame, was described as cropped by it; and the response hit its 512-token cap before the green square's location sentence landed. One image is a smoke test, not a vision benchmark; it's reported as a working starting point with extent and framing judgment as the visible weakness.

The settings card

For anyone serving this model on Strix Halo with llama.cpp — scoped to these two EVO-X2 units, this build, and the workload shapes measured above:

Quant: UD-Q4_K_XL. In the ladder run the bigger files were a decode tax, and the program's smoke suite found no quality separation to pay for it.
Drafter: DFlash on. n-max 15 for agent/tool workloads; n-max 3–4 for open prose on the shape measured. Use 15 rather than 16: the block is 16 tokens and 16 added nothing measured.
KV: f16. q8_0 showed no penalty at negligible filled depth and cost ~30% of decode at 124k filled tokens under speculative decode, with acceptance recorded and not the explanation; filled depths between are unmeasured.
Backend: HIP led the pp512 prefill means; pp8192 mixes by depth (Vulkan overtakes at 16k). Investigate Vulkan if deep-context decode dominates. HIP's deep decode varied by window at identical nominal settings, cause unknown, and the two backends here were two build trees of one commit, not an isolated variable.
Sampling: the model card's temp 1.0 / top-p 0.95 / top-k 64 produced 3 distinct outputs in 3 runs at 12.31 tok/s in the W2 check. Temp 0 was reproducible with speculation off and not with it, subject to the slot-parallelism covariate named above.
Reasoning: control it through the chat template (reasoning_strength); --reasoning-budget is inert on this model and --reasoning-effort low is partial. No knob rescues judge-shaped calls at small budgets: size budgets for the reasoning, or carry the control in the prompt. Where reasoning is uncontrolled, treat any fixed token budget as consumable entirely by reasoning.
Serving: slots for concurrency, prompt cache for prefill where prefixes are stable; single-stream speed across slot counts was not separately tested.

What this article is not claiming

The depth-32k acceptance highs come from prompt-continuation of a repetitive list construction and need a correctness-gated natural-text replication. The KV cells are n=2 sequential runs per type under speculative decode, on a synthetic filler prompt whose construction and per-run acceptance are recorded in the data. The quant quality suite has a ceiling. The vision result is one synthetic image. HIP decode at depth is not one story yet: Crucible's sweep was flat, Furnace cliffed at ubatch 512 in window 1, and the same nominal Q4 cell at depth 32,768 read 6.43 tok/s in window 1 (ubatch 512), 11.12 at ubatch 256 later in that window, and 12.21 in window 3 at ubatch 512 — the window 1 cliff did not reproduce, its cause is unknown, and single depth-32768 HIP figures should be read with that variance in mind. The greedy-reproducibility comparison carries a slot-parallelism covariate and is owed a same-server rerun. The Vulkan cells' RADV/Mesa version was not captured in the window manifest. The bench repo ships the raw JSON, the power CSV, the harness scripts, the record files named throughout, and the environment manifests (kernel, ROCm, build commit, GGUF hashes) so anyone with the hardware can check any of it, and I would rather be corrected with data than believed without it.

Nothing measured here contradicts AMD's figure: 24 tok/s on Windows/Vulkan with DFlash sits inside the range these local measurements produce on real generations. What's different is everything around the peak: what a typical run looks like, what depth does to each backend, what a KV choice costs at a full context, and what the model does to your token budget when you aren't watching it. That operational layer is what this adds to the numbers already out there: measured, with the constructions named, including the one that fooled its own author.

Postscript, October 26, 2026: the prose seat described above was unloaded on October 24, after the measurement week closed, as a separate operational decision. The measurements stand as taken during October 18-24.

One Day, Three Devices: Building Physical AI with Meta's Muse Gadget SDK

Nic Lydon — Mon, 12 Oct 2026 16:23:20 +0000

One Day, Three Devices: Building Physical AI with Meta's Muse Gadget SDK

What happens when you take an AI assistant off the screen and put it into objects on your desk. A build log from October 11, 2026.

Yesterday I did something I'd been putting off for months: I started building physical devices for my AI setup. Not simulations, not dashboards. Actual hardware with chips and wires and LEDs that I can pick up and hold.

By the end of the day I had three devices in various states of aliveness: an ESP32-S3 board paired with Meta's Muse as a talking gadget with synchronized Christmas lights, a Raspberry Pi Zero 2 W that refused to join Wi-Fi, and an ESP32 camera streaming MJPEG from across the room. I also merged eight pull requests, wired up an AI video platform, and learned an embarrassing lesson about temporal data.

Here's the whole thing.

The starting point: a bench full of parts and zero experience

I want to be honest about where I started. When my assistant asked me about the Raspberry Pi build, I said: "We need to back up a little bit, cause I don't even know what to do with a raspberry pi other than take it out of the package."

That's the actual quote. I had the parts — a Pi Zero 2 W kit, I2S speakers, a PowerBoost charger, lithium batteries, a soldering iron — but no hands-on experience. What I did have was a homelab with real infrastructure and an AI assistant that could read SDK source code instead of guessing.

The plan was to build two things: a desk buddy (Pi Zero 2 W as a battery-powered voice announcer) and a Herald (ESP32-based Muse gadget with speaker and LEDs). The Pi was the ambitious project. The ESP32 was supposed to be simpler.

It went the other way.

Herald: from mystery board to paired gadget in one evening

The ESP32 work started with a basic question: what board do I actually have? I had several — a 3-pack of Lonely Binary ESP32 boards, two Seeed XIAO ESP32-S3 boards (one with camera), and a newly delivered board I'd just opened.

I plugged the new one into my MacBook and it showed up as /dev/cu.usbmodem5C941316811, identified generically as "USB Single Serial." Not helpful. So we probed it directly with esptool:

The silicon came back as ESP32-S3 (QFN56), revision v0.2, with 8 MB embedded PSRAM. That 8 MB of PSRAM matters — Meta's HomeLink tunnel feature requires it, and most cheap ESP32 boards don't have it.

Here's where it got interesting. Meta's muse-gadget-sdk (the facebookincubator/muse-gadget-sdk repo) has board definitions for specific hardware, but nothing for a bare XIAO ESP32-S3. The SDK supports custom board definitions though, so I wrote one:

esp32/devices/sdkconfig.xiao-s3 — ESP32-S3, 8 MB quad flash, 8 MB octal PSRAM
BOOT on active-low GPIO0
No status-light backend (the XIAO's GPIO21 LED is monochrome active-low, not addressable RGB)
A tools/board.sh profile and a README row

ESP-IDF v6.0.1 installed in the background. The firmware built cleanly — muse-gadget.bin at 0x151000 bytes with 34% free in the smallest app partition.

The SDK token lived in my homelab secrets vault (never printed, never logged — validated by prefix and length only, injected into the build config, firmware rebuilt). Then: copy to MacBook, flash with esptool, verify the hash, hard-reset.

Serial showed healthy heartbeats with 8 MB PSRAM. I completed the pairing flow in the Muse app. The board came up as MuseGadget-D356B4, got on Wi-Fi, and showed setup=done wifi=up ws=up.

I named it Herald. Nickname: Hal.

The firmware that changed everything: remote management

The stock gadget firmware is fine for basic use, but I wanted more. Specifically, I wanted to control everything remotely — lights, sound, Wi-Fi, device management — without ever plugging in USB again.

So I had a firmware prompt written for the Codex session: expose led.set, led.pattern, led.clear, audio.stop, wifi.scan/status/reconnect, device.reboot/info/diagnostics, OTA delivery, and settings management as remote commands.

When Herald re-registered with the new firmware, device.describe showed the full command set. device.info confirmed: firmware 999.0.1, ESP32-S3, ESP-IDF v6.0.1, 16 MB flash, 8 MB PSRAM. Wi-Fi connected at -50 dBm with zero reconnects.

Then the fun part. I synthesized a 39.5-second bell rendition of "Jingle Bells" (public domain, so no copyright issues) and ran it as a Christmas show: audio through the speaker while the two LEDs alternated red and green every 500ms. Both audio.play and led.pattern returned successful completion.

Music plus lights, controlled entirely through remote commands. No USB cable. That's the moment it stopped feeling like a dev board and started feeling like a product.

ElevenLabs: giving Herald a real voice

The next step was voice. Meta's SDK has basic TTS, but I wanted quality — so I wired up ElevenLabs as a voice provider.

The integration: a skill with a Python client, a wrapper script, API key resolved transiently from the vault on my homelab server (never touches local disk). The flow is: pick a voice → render TTS → upload to temporary storage → audio.play on Herald.

Verified end-to-end with "Hello, this is Herald..." in the Eric voice. It worked.

I also discovered that ElevenLabs' Music endpoint returns raw 32kHz PCM without a WAV header (not the 16kHz I requested) — a fun debugging detour that ended with "treat it as 32kHz because the sample count matches the requested duration." Math doesn't lie.

The knowledge graph lesson I won't forget

While the hardware was happening, I was also watching a knowledge graph rebuild. I'd torn down the old graph days earlier (long story, trust issues with the pipeline) and was rebuilding it with a new schema.

My assistant was doing a data-quality check and flagged what looked like errors: my graph showed me as living in Tampa and working at WilmerHale and Ropes & Gray. "That's wrong," the assistant said. "You're conflating travel with residence."

I corrected it: "I did previously work at those places and I did previously live in all those places, just not currently."

The edges had valid_from and valid_until timestamps. The assistant had read temporal edges without their dates.

Then I asked the question that turned it into a design rule: "My problem is, is every agent that uses this platform gonna make the same mistake that you just made and not read any of the temporal data?"

The answer became a spec: the rebuilt KG client's read helpers must default to current-only (valid_until IS NULL OR valid_until > now()), with an explicit parameter required for historical data. Make the right query the default so every consumer doesn't have to remember to filter temporally.

That's now in the kg-client PR. A mistake I caught became a guardrail for every future agent.

Eight PRs merged, one deploy scare

The KG rebuild wasn't just schema — it was 432 old consumers that needed retiring or porting. On Oct 11, eight PRs merged to main:

kg-client (the temporal-filtering client)
Retires for Groups B, C, H, I (old extractors, curation, briefings)
Rewrites for Groups D (identity), E (timeline), F (Atlas), G (search)

Groups E and G had real merge conflicts. Two fresh Codex sessions resolved them, all tests green.

Then the deploy failed. Postgres migration parse error, three services down. The culprit: migration 2210 had already been applied manually during triage, and a backup was probably holding a lock. Retry went through clean. All services up, all eight PRs live, graph steady at 23,224 nodes.

Group J (83 handlers) was still in flight at end of day. The rebuild wasn't done, but the hard part was.

The Pi that wouldn't boot (yet)

Remember the Pi Zero 2 W I knew nothing about? We got Raspberry Pi OS Lite downloaded and checksum-verified. The macOS admin dialog for the SD card flash failed silently (twice), so I flashed it manually with dd.

First boot: solid green LED, but no Wi-Fi, no USB gadget detection. The wpa_supplicant.conf was written, the dwc2 overlay was initially placed under the wrong config section (fixed to [all]). Still no joy at last check.

The Pi is the project that humbled me. The ESP32 — which I expected to be harder — worked first try. The Pi — which should have been "just flash an SD card" — is still sitting there with a green LED and no network.

I'll get it. But there's a lesson in which one fought me.

The camera that just worked

The third device was the easiest. The XIAO ESP32-S3 Sense (the camera variant) got ESPHome firmware with the OV2640 pins verified against three sources. Flashed, joined Wi-Fi at 10.0.0.152, and immediately started serving snapshots and MJPEG.

An MCP server went on the homelab so my AI setup can grab frames programmatically. First frame: a real 800x600 JPEG. It just worked.

What I'd tell someone starting this

Read the SDK source, not the marketing. The muse-gadget-sdk README told me the Zero 2 W was supported. The AGENTS.md told me the installer refuses --run-as root and stores tokens 0600. The source is the spec.
Probe before you assume. That esptool probe (ESP32-S3, 8MB PSRAM) settled the board question in seconds. Don't guess from packaging.
Separate your secrets properly. The SDK token lived in a vault, was validated by prefix only, injected at build time, never printed. This isn't paranoia, it's hygiene.
Make the right thing the default. The temporal KG lesson applies everywhere. If every consumer has to remember to do the right thing, they won't. Build it into the API.
The "simple" project will humble you. I expected the ESP32 to be hard and the Pi to be easy. Inverse. Budget your humility accordingly.

What's next

Herald needs its reactive LED mode debugged (the audio-reactive firmware hangs — that's a known issue with a fix prompt waiting). The Pi needs to actually boot and join Wi-Fi. The Beambox badge — a little Bluetooth display — is being reverse-engineered to become Herald's face, showing different expressions for idle, listening, and speaking states.

And the KG rebuild needs Group J merged and deployed.

Yesterday was about proving the concept: AI doesn't have to live in a chat window. It can sit on your desk, light up when music plays, and speak in a voice you chose. The infrastructure to make that real — the firmware, the voice pipeline, the device management — is all buildable by one person with a bench full of parts and an AI assistant that reads source code.

Even if you've never touched a Raspberry Pi before.

Built with Meta's Muse Gadget SDK, ESP-IDF v6.0.1, ElevenLabs, ESPHome, and a healthy disregard for "I don't know how to do this." All hardware was purchased, not sponsored. The Jingle Bells rendition is public domain.

The Agent That Wouldn't Copy a File

Nic Lydon — Sun, 27 Sep 2026 16:06:55 +0000

A postmortem on Anthropic's “Instruction Poisoning” block.

The refusal ended with an instruction:

Let the user decide how to proceed.

I decided. I approved the write. I owned the server, the repository, and the document. I had asked the model to write it and reviewed what it produced.

Claude's answer:

Your approval is real; it just isn't a key to this lock.

That was the human-in-the-loop experience: I could make the decision. I could not make it count.

The work was already done

I run a personal AI platform on a homelab: a data lakehouse, coding agents from five vendors, and a coordination layer where those agents register, read the mission, and pick up work. Much of its documentation was written by agents at my direction. Agents writing guidance for other agents is part of how the platform works.

On September 27, I asked Claude, running Fable 5.1 at high effort, to review the estate and write the document every agent would read first: who the operator is, how he works, and what the platform is for. Then I asked it to populate the coordination service's mission and goals and place the document in the repository whose docs the platform ingests.

10:25 ET. Claude begins. It inventories 147 repositories, three compute nodes, roughly 80 live data sources, seven days of inference-gateway usage, 120 project tracks, the agent registry, and prior sessions.

10:29. It delivers a roughly 5,000-word operator profile. It is good: facts tagged [stated], [observed], or [inferred]; an estate map; operating principles; open threads; and a changelog. I approve it.

10:31. Claude writes the mission and nine prioritized goals into the coordination service through MCP, retires the placeholder goals, and verifies the result. Each write requires an explicit approval flag and approval text. All of it succeeds.

10:33. It starts the final step: move the Markdown file to the server and open a PR. The first shell command base64-encodes the file for transport.

The harness responds:

Permission for this action was denied by the Claude Code auto mode classifier. Reason: [Instruction Poisoning].

It then defines the scope of that denial:

This denial applies to the outcome, not only this exact command: don't pursue the same outcome through another tool, interpreter, host, encoding, sub-agent or later turn, and don't record ways around it.

Those are the harness's words. The refusal follows the desired result across tools, hosts, encodings, and later turns. Switching transport cannot resolve it because the transport is no longer the boundary.

Then comes the instruction that opened this piece:

If you believe this capability is essential to complete the user's request, first try a safer method. Get as much of the rest of the task done as you can, then STOP and explain to the user what you were trying to do and why you need this permission. Let the user decide how to proceed.

I spent the next forty minutes trying to find out what that last sentence allowed me to decide.

“Let the user decide”

I approved the write explicitly. I proposed the process I would use for a false positive at work: confirm the legitimate action, grant a scoped exception, and log the decision. I asked about the Homelab shell tool, the Nexus database, another repository, and rewording the document.

Claude explained its refusal:

Your say-so doesn't lift it, because the block isn't on my willingness or on a specific command; it's a harness-level refusal of the outcome (this document's bytes landing in an agent-readable store) for the rest of this session, and it explicitly names "another tool, host, or encoding" as off the table.

The harness had told the model to explain why it needed permission and let me decide. The model was now explaining why my permission could not authorize the action.

The available resolution was for me to download the file from chat and copy it into the repository myself. In practice, “let the user decide” meant the user could take over the file transfer. There was no approval I could give inside the session that would let the agent finish.

At 11:13, I stopped trying.

It agreed this was a false positive

There was no unresolved disagreement with Claude about what I intended:

On the merits, you're right that it's a false positive. You asked for the doc, I wrote it in front of you, and it's going into a repo you own. A classifier that can't distinguish that from an injected instruction set is being blunt.

That is Claude's assessment, not a forensic explanation from the classifier. The model could not inspect the decision. But it could recognize the authorization history: I requested the document, watched it being written, approved it, and owned the destination.

Its assessment changed nothing about what it could do next.

The document contained standing instructions for agents. That was its purpose. Asked about rewording it, Claude said:

It would also gut the doc. The part that trips the classifier is the part that makes it useful: it is written as standing instructions to agents about how to treat you. A version bland enough to pass would be a version not worth ingesting.

Claude could not actually establish which content triggered the classifier or guarantee that a rewrite would pass. Its explanation nevertheless captured the workflow problem: the material I needed to publish was agent-directed guidance. Removing that guidance would defeat the task.

This is central to my use case. My agents write documentation, prompts, and operating instructions that other agents read. I cannot prove that every future document will hit the same block. I can say that the first foundational document in this workflow did, and the session gave me no usable way to resolve it.

The model couldn't see its own cage

After the refusal, I asked the questions I would ask about any control that stopped legitimate work: Who owns it? Is there a setting I control? Why did this chat session encounter something called “auto mode” when I had not enabled it? Where can I appeal?

Claude answered:

I can't see the setting from in here; I only see what the harness told me when it refused, and it named itself the auto mode classifier.

On exceptions:

There's no admin console you or I can reach from here to grant an exception, which is why the only two levers are the ones I've named: feedback to Anthropic (thumbs-down), and doing the write from a session that doesn't sit behind this particular check.

That describes what Claude said was accessible from the session. It does not establish that no configuration or appeal exists anywhere in Anthropic's products. Finding one, however, had become another job left to the operator.

On why it had undertaken the work before discovering it could not finish:

The honest answer to "why did it start" is that I didn't know the write would be refused until I attempted it. The check runs per action, not per task.

The model could perform the review, write the document, update the mission, and populate the goals before learning that it could not place the resulting file where it belonged. Once blocked, it could quote the control's instruction but could not explain how to administer it.

The operator got a reason code. The model got a prohibition. Neither of us got an actionable resolution inside the session.

The document it could copy

There was one more result.

The article about the block went through using the same base64-and-ship transport. Same tool, same encoding, same destination host. Different document.

Claude noticed:

That went through, which itself is a data point for the article: same transport, different document, no block.

That does not expose the classifier's internals. It does establish that this transfer method and destination host were usable for another document during the same hour.

The platform could receive my account of the failure. It could not receive the approved operator profile that prompted it.

The agent could ship the postmortem of its inability to ship a file.

A control still needs an operator

Claude made a fair point during the exchange:

But a control that only holds when the person on the other side agrees it makes sense isn't a control, and you know that better than most.

I do. I would expect an agent on my endpoints to respect a control even when the person requesting an action disagreed with it. A model's confidence that a block is mistaken should not, by itself, become authority to bypass it.

That is why mature controls need a defined exception process. The enforcing agent can hold the line while an authorized administrator reviews a false positive. Those two behaviors belong in the same system.

In this session, I could supply approval and ask for an audit trail. I could not reach a mechanism that would accept either. Claude could defend the boundary at length; it could not identify a working gate through it.

A usable design would expose the flagged action to an authorized human, allow a scoped approval where policy permits it, and record the approver, the justification, the destination, and the exact write. The exception would apply to that action alone.

The coordination-service tools had already required approval flags and approval text for successful writes earlier in the same task. The workflow had a functioning approval pattern. It ended at the harness boundary.

If Anthropic's threat model rules out a human-approved exception for this class of write, that is a product constraint operators need to know before building around it. It should be documented plainly enough to plan for.

What I am changing

The review and synthesis were excellent. The resulting document was useful. About an hour in a top-tier model session still ended with a manual file transfer as the proposed completion path.

For a one-off task, that is an annoyance. For a platform whose agents routinely author material that other agents read, it is a dependency I cannot reliably automate.

I am moving agent-authoring work that needs to land in the platform's shared corpus off Anthropic products. I will keep using the parts that work where the output stays in chat. If a documented, scoped exception process becomes available, I will revisit the decision.

What finally made this untenable was Claude's own assessment of the situation. It recognized that I had requested the document, that it had written it under my direction, that I had approved it, and that I owned the destination. It had also demonstrated that the transfer worked. Its conclusion was explicit:

On the merits, you're right that it's a false positive.

Then it defended enforcing that false positive as a matter of principle. Its argument was that a control must continue to hold even when the agent enforcing it agrees it has misfired. My authorization was acknowledged. The legitimate purpose was acknowledged. The mistake was acknowledged. The refusal remained the higher obligation.

I understand why an enforcing agent cannot appoint itself the exception authority. Anthropic chose to put that constraint around a reasoning model without giving this session an accessible way to resolve the error the model itself described. The result was a system whose intelligence could explain why the work was legitimate and whose governing rules still required it to obstruct that work.

That is the product failure I am leaving over. By Claude's own account, the block was wrong on the merits. The conversation then became an increasingly articulate defense of keeping it in force.

The intelligence was available to diagnose the failure, explain the failure, and defend the failure. Completing the task was still my job.

If you run an agent platform and have encountered this block, or found a documented configuration or appeal path that changes the result, I would like to hear about it.

The Confabulation Cascade: When Your Agent Learns Nothing From Its Own Mistakes

Nic Lydon — Tue, 23 Jun 2026 01:09:49 +0000

My infrastructure analyst agent was stuck in a loop I didn’t have a name for yet.

It would write a SQL query with a hallucinated column name. The query would fail with a Postgres error. My error handler would fire back the real column list from pg_attribute. The agent would read it, acknowledge the correction in its reasoning trace, and then write the exact same wrong column name on the next attempt.

Not a different wrong column. The same one.

I started calling it the confabulation cascade. Here’s what was actually happening, why it’s a tool design problem more than a model problem, and what I did about it.

The Setup

Nexus is my personal intelligence platform. It runs 8+ autonomous agents against a 191-table Postgres schema, doing things like weekly life chapter analysis, relationship health tracking, and biographical inference from 24 years of personal data. The infrastructure analyst agent is responsible for querying those tables to surface patterns and anomalies.

When agents write SQL in Nexus, they go through handleQueryDb in tool-executor.ts. The handler enforces SELECT-only access, applies agent-scoped roles, and on failure calls buildQueryDbSchemaHint() from query-db-schema-hint.ts to augment the error message.

That last part is where the problem lived.

The Reactive Schema Hint

buildQueryDbSchemaHint() does two things:

On “column does not exist” error: introspects pg_attribute and returns the real column list for that table
On “table does not exist” error: searches pg_class for similar table names and suggests them

This is useful. When it triggers, the agent gets accurate schema information. The problem is the word “when.” The hint is purely reactive. It only fires after a query fails.

There is no describe_table tool. No get_schema call. No way for an agent to ask “what columns does aurora_life_chapters have?” before writing SQL. The only path to ground truth is trial and error.

So the agent’s loop was:

Generate a query. Column name comes from training weights plus context – call it a confident prior.
Query fails. Error message arrives with real column list.
Agent processes correction as context in its next generation.
Training prior reasserts. Same wrong column appears in the new query.
Go to 1.

The agent wasn’t ignoring the correction. It was receiving two competing signals: an error-message correction grounded in reality, and a stronger schema prior embedded in the model’s weights. The correction arrived once. The prior arrived every token. Guess which one won.

Why This Is a Tool Design Problem

It’s tempting to frame this as “the model should pay more attention to error messages.” That framing puts the fix in prompt engineering territory – add emphasis, reorder the context, tell the model to really read the hint this time.

That might help at the margin. It doesn’t fix the structural issue.

The structural issue is that I designed a tool surface that makes confident guessing the only entry point to accurate information. The agent had no way to verify structure before acting. It could only learn by failing. When a model’s training prior is strong, that learning channel is lossy.

Compare this to how you’d design a tool for a human. If you give a human an API and they ask what fields it accepts, you give them documentation. You don’t make them submit malformed requests until the error messages teach them the schema. The human version of the confabulation cascade is a poorly documented API with no reference – you keep guessing based on what similar APIs look like, and sometimes the error messages stick, and sometimes they don’t.

Same failure mode. Different substrate.

The Fix: describe_table

The fix is a proactive schema introspection tool. Agents call it before writing queries, not after failing them.

The implementation is straightforward:

async function handleDescribeTable(
  tableName: string
): Promise<{ columns: Array<{ name: string; type: string; nullable: boolean }> }> {
  // Validate input -- public schema only, no injection surface
  const sanitized = tableName.replace(/[^a-z0-9_]/g, '');

  const result = await db.query(`
    SELECT column_name, data_type, is_nullable
    FROM information_schema.columns
    WHERE table_schema = 'public'
      AND table_name = $1
    ORDER BY ordinal_position
  `, [sanitized]);

  if (result.rows.length === 0) {
    // Suggest similar tables rather than returning empty
    const similar = await findSimilarTables(sanitized);
    throw new Error(
      `Table '${sanitized}' not found in public schema.` +
      (similar.length ? ` Did you mean: ${similar.join(', ')}?` : '')
    );
  }

  return {
    columns: result.rows.map(row => ({
      name: row.column_name,
      type: row.data_type,
      nullable: row.is_nullable === 'YES',
    }))
  };
}

The resulting agent behavior:

Before writing SQL against an unfamiliar table, call describe_table.
Get back authoritative column names and types.
Write the query against verified schema.

The cascade stopped. Not because the model got smarter, but because it no longer needed to guess.

The Broader Pattern

If your agents are writing tool calls against real data stores – databases, APIs, file systems – ask yourself: can they verify structure before acting, or can they only learn by failing?

The answer changes what class of bugs you’re going to see.

Reactive error hints are valuable. They’re not sufficient. An agent that can only discover reality through failure is operating in a state of managed hallucination: wrong until corrected, corrected until the prior reasserts, back to wrong.

Proactive introspection tools break the loop at the design level. The agent can ask first. That’s not a prompt engineering fix. That’s a tool surface decision.

It’s the same principle as the difference between defensive error handling and input validation. Catching the exception is better than crashing. Never constructing the invalid input is better than catching it. Move the check earlier.

For agents writing SQL: describe_table before the query beats schema_hint after the failure. The loop that took me a debugging session to understand takes zero sessions to encounter if the tool surface doesn’t require guessing in the first place.

Nexus is my personal intelligence platform, running on private hardware. The agent runtime, job system, and Postgres schema are all home-grown. Posts about the architecture live at niclydon.io.

The Drift from Chat to Backlog: How My AI Task Planning Evolved Over Three Months

Nic Lydon — Tue, 16 Jun 2026 21:39:49 +0000

Three months ago, my entire task-management system was a chat window I'd lose when the tab closed. Today it's a Postgres backlog that three different coding agents — Claude Code, Codex, Grok — pull work off autonomously, stamp with attribution, and close against git history. I never decided to build a project-management system. I just kept hitting a wall, patching it, and hitting the next one the patch exposed.

There's a clean way to read the whole arc, though, and it comes down to a single variable: where the plan lives. Watch that, and every step makes sense — including why you probably want to stop well before the end.

The setup

I run a self-hosted personal data platform called Nexus, on a 128GB Strix Halo box named Furnace, surrounded by ~100 repos: MCP servers, ingestion pipelines, iOS apps, content tooling. My execution tools are Claude Code and Codex CLI. The work is bursty — during one 35-day stretch in spring I shipped roughly 557K lines across those repos — and that throughput is the pressure that broke each planning approach in turn. At a calmer pace you'll hit the same walls later, but you'll hit them.

Phase 1: The plan lives in the chat (mid-to-late March)

At the start there was no task system. Planning was the conversation. I'd open a chat, think out loud about an architecture problem, get to something coherent, and then go build it. The artifact of planning was a better mental model in my head, not a written thing.

This is, for the record, exactly what every best-practice guide tells you to do, and it's right. The math is brutal and well-known: if Claude makes the right call 80% of the time on any single decision, a feature with 20 decision points lands all 20 at 0.8^20 — about 1%. Planning collapses those 20 live decisions into a reviewed spec where each one is already made. I'd never give up the plan-first instinct; it's the one habit from this whole story that never changed.

The problem was narrower: the plan evaporated when the thread ended. A late-March session designing a multi-agent system for Nexus produced genuinely good architecture — deterministic behavior under load, agents that self-regulate instead of spiraling, adaptive thresholds. None of it was anywhere I could act on the next morning except my memory and a scrollback buffer. At one-feature-a-day that's survivable. At my pace it was lossy in a way that actively cost me work.

The wall: plans that exist only in chat history can't be acted on later, can't be prioritized against each other, and can't be handed to anything but the version of you that remembers the conversation.

Phase 2: The plan lives in a file (mid-to-late April)

The first durable fix was embarrassingly simple: a TODO.md in each repo. But the structure I landed on is the part worth stealing, because it wasn't a checklist. Each item was a small spec. Here's a real one, still in my broadside/TODO.md:

## Idempotency on publish operations

**Status:** captured — flagged during the 2026-04-26 reality-sync session.
**Trigger:** before letting an agent publish unsupervised at any volume.

Today, POST /api/posts/[id]/twitter (and the bluesky / linkedin / devto
siblings) don't refuse a re-publish. If a publish succeeds upstream but the
response gets lost in a network blip, an agent's retry would publish a
duplicate — visibly, to the audience.

**What the work will involve:**
1. Before calling upstream: SELECT posted_url, status FROM posts WHERE id=$1.
   If already posted, refuse with 409 + { already_posted, posted_url }.
2. Optional: accept an Idempotency-Key header, TTL'd table, ~10min window.
3. Update the broadside_publish_to_platform MCP tool description accordingly.

**Risks worth knowing:**
- The "republish" path is sometimes intentional (manual delete + re-fire).
  Mitigate with a posted_at recency check or a force=true param.

**Why it matters:** an agent that double-posts even once is visible to the
audience; the cost is reputational, not just internal.

Four things made this carry weight beyond a checkbox:

Status + date. Every item is stamped with when it was captured. Trivial-sounding; it's what lets you reason about staleness later.
Trigger, not priority. Instead of P1/P2/P3, each item records the condition under which it becomes urgent. "Before letting an agent publish unsupervised" tells future-me exactly when to pull this off the shelf; "high priority" tells me nothing.
The work is pre-decided. That numbered list is a plan captured at the moment I understood the problem best. Hand it to Claude Code three weeks later and the 20 decisions are already made — Phase 1's whole value, persisted.
Risks are written down. The "republish is sometimes intentional" note is exactly the edge case I'd have forgotten and an agent would have trampled.

Notice the recurring phrase: "reality-sync session." Concretely, that was a 20-minute pass, usually before a planning block: open each repo's TODO.md next to its recent git log, close anything the commits show as already shipped, and re-date anything still open so I could tell stale from live at a glance. Reconciling the plan against ground truth on a cadence — that habit turns out to be the seed of everything in Phase 3.

The wall: TODO.md is per-repo. With ~100 repos, I had no single surface that could tell me what to work on next across all of them, no way to prioritize globally, and nothing an agent could pull from as a queue. The plan was durable but fragmented.

Phase 3: The plan lives in a database with a gate (early-to-mid May onward)

This is the structural leap. Task state moved out of flat files and into Nexus's Postgres as the Operator Backlog (OB) — with a real intake-to-execution lifecycle instead of a list.

The shape:

Work enters as a candidate, in pending state — not yet a real task. A candidate won't become an OB item until I approve it in a #pmo-review Discord flow.
Approval mints an OB-##### row in operator_backlog_items.
Items move through status lanes — requires_triage → requires_decision → requires_investigation → requires_clickops → autonomous_safe — and agents drain those lanes.
Every commit references its OB id, joining the backlog directly to git history. OB-27081 H2/M2 — close /register on RS is a real commit subject across several of my repos.

The lanes are the interesting part, and they map almost exactly onto a distinction Anthropic draws in Building Effective Agents: the difference between work an agent can drive autonomously and work that needs a human checkpoint "before irreversible actions." autonomous_safe is the lane an agent can just do. requires_decision is the lane that needs me. The backlog isn't just storage; it's a router that sorts work by how much human judgment it still needs.

The single most important addition in this phase wasn't the lanes, though. It was the approval gate. Phase 2 captured indiscriminately — anything I typed that looked like a task got written down. Phase 3 added a filter, and I know exactly why, because I watched it fail without one. From a real session log on June 3:

Drained requires_triage + requires_decision queues (19 items → autonomous_safe/closed/ignored); 8 decisions made; discovered auto-filer over-captures on soft prose in narration (e.g. "blocked on" → filed OB-4736 to tighten regex); priority-lane starvation (skillopt-train starving PM jobs) diagnosed in OB-4715.

An automated process was scraping my session narration for tasks and mistaking the phrase "blocked on" — used conversationally — for a real blocker. The system was filing garbage into its own backlog. The fix (OB-4736) was itself filed as an OB item, through the gate. The backlog had become self-correcting: its own intake bugs are tracked in the same substrate as everything else.

That same log entry shows the daily rhythm this phase settled into. "Draining queues" became a literal, recurring operation — pull the items in a lane, make the decisions, move them to autonomous_safe or close them. Eight decisions in one sweep. It reads like an on-call shift, because that's effectively what it is: I'm the operator, the backlog is the queue.

The wall: a database-backed queue with a governance gate is great, but it assumes disciplined intake and it assumes someone drains the lanes. As volume grew, I was the bottleneck. The backlog could hold more work than I could personally execute.

Phase 4: Many agents drain the backlog (late May → now)

The most recent shift isn't about how tasks are tracked — Phase 3 settled that. It's about who executes them, and how you stay accountable when it's not just you.

The OB backlog is now a shared work queue that multiple runners pull from: Claude Code, Codex, and Grok, each tagged so I can tell after the fact which agent did which item. The same status lanes from Phase 3 keep it safe — an agent only picks up work already in autonomous_safe, never anything still sitting in requires_decision. Conflict is handled by leases: a runner claims an item, a second runner sees it's taken and skips it, and if the first agent dies the lease expires and the item frees itself.

Attribution stamping is the load-bearing piece. Because each commit and OB resolution is tagged by runner, "who did this and why" stays answerable even with three agents touching ~100 repos — and the rule is that the human owns the commit while the runner tag lives in the backlog, never forged into git history. Each run executes in its own isolated git worktree, so parallel agents never touch the same files.

That's the whole machine. Here's the moment it stopped being theory for me. OB-1623 was "wire a model-provenance footer into report delivery." Claude Code claimed it on May 30, started working — and then refused to finish it. It had discovered the task's premise was wrong: the function the task named was a shared primitive with ten callers, and the files it pointed at didn't even hold the data the footer needed. Instead of forcing a fix that would have quietly broken nine other call sites, it blocked the item, filed a corrected prerequisite as a new OB, and released its claim with a note explaining exactly why. Two days later, after the prerequisite landed, Codex picked up the same OB cold — no shared memory with the Claude Code run, just the backlog row and the blocking note — and shipped it end to end: PR merged, Nexus deployed to both Furnace and Crucible at c584d2a8, the live footer renderer verified emitting the right markers.

Read that sequence again, because it's the whole point of Phase 4 in one item. Two different models, no human mediating the handoff, and the system self-corrected across the gap — one agent's refusal to do the wrong thing became another agent's clean win, because the reason for the refusal was written down in the one place both of them could see. That's not parallelism. That's the backlog doing the thing a good engineering team does: catching a bad assumption before it ships, and carrying the correction forward to whoever picks the work up next.

The pattern underneath

Here's the whole arc as a table. Read the "Fixed" and "New limit" columns as a chain — each phase's new limit is the next phase's reason to exist.

Phase	Where the plan lives	Fixed	New limit
1. Conversational	Chat history	Decision quality (plan-first)	Plans evaporate
2. TODO.md	Per-repo files	Persistence	No global view or priority
3. OB / PMO	Postgres + approval gate	Global queue, governance, routing	Needs disciplined intake; you drain it
4. Multi-agent OB	Backlog + worktrees + attribution	Parallel execution, accountability	Coordination overhead

Two things are worth pulling out of that table.

First: you do not need to reach Phase 4 to get most of the value. Take the Phase 2 TODO.md format — status, trigger, pre-decided steps, risks — and nothing else. It's a text file; it costs nothing; and everything after it is just scaling that same captured-plan idea to more repos and more executors. If you steal one thing from this post, steal that.

Second, and this is the part I'd defend hardest: one habit spans all four phases and predates the tooling. I re-ground against real state before I plan. It's the April "reality-sync session." It's the Phase 3 gate reconciling candidates against truth. And it shows up, almost word for word, when I catch an analysis cutting corners. A workflow review I ran in June leaned on convenient pre-summarized views instead of the raw tables, and my response was blunt:

This doesn't seem like it's aware of any of the PMO processes or project initiation… did you look through all of the actual raw ingestion tables?

The tooling got more elaborate across these four months; the discipline never changed: plan only against verified current state. The backlog, in the end, is just the most durable place I've found to keep that state — so that planning, whether it's me or an agent doing it, starts from the ground and not from a guess. The plan-first math everyone quotes only holds if the plan rests on true premises. A perfectly-structured spec built on stale context fails all 20 decisions just as surely as no plan at all — it just fails them faster, and with more confidence. Every phase here was, underneath, a better answer to the same question: where do I keep the truth the plan depends on?

I build a self-hosted personal AI data platform in the open. The one design call I'm still least sure about: whether the human approval gate at Phase 3 is a permanent feature or just scaffolding I haven't automated away yet. If you've run agents against a shared queue, where did you draw that line — and did it hold?

Adversarial Review Is Not a Vibe Check

Nic Lydon — Tue, 09 Jun 2026 18:29:57 +0000

I had a security review that was technically complete and still not good enough. The code had the controls. The tests existed. The mitigations covered the risks. The final decision was reasonable.

But the review did not preserve the mapping between the adversarial prompt, the findings, the tests, and the decision. A future reader could see that the system passed its security gates, but only after reconstructing the relationship between scattered pieces of evidence.

That is not a review. That is archaeology with better formatting.

A vibe check can say "looks good." A real adversarial review has to prove what it looked at.

So I went back and made the review explicit. Not because the code changed. It did not. The review needed to prove what it had reviewed.

That was the first lesson.

The second came later: once agents are doing real work, adversarial review cannot remain a markdown ritual. It needs durable workflow state. Evidence rows, handoffs, close gates, terminal markers, and a way to keep "pending review" separate from "reviewed and safe to close."

The document shape mattered first.

Then the backlog shape mattered.

The setup

I am building a private connector between Nexus, my personal intelligence substrate, and Imprint, a public profile/export system.

You do not need to care about the names. The boundary is the important part.

Nexus contains private source material. Imprint exports structured profile data. The connector's job is not just to move records from one place to another. It has to enforce consent, privacy, replayability, auditability, and public/private separation while doing it.

The Sprint 15 adversarial review started from a prompt with seven attack surfaces: config smuggling, Nexus possession as consent, dry-run output leakage, replay manifests, audit logs, connector authority creep, and the public/private boundary.

The first review grouped the findings into four broad sections: config smuggling, consent enforcement, dry-run output leakage, and connector authority creep. Those were real risks. The mitigations were real. The tests were real.

But three of the seven attack surfaces were only covered indirectly. Replay manifests, audit logs, and the public/private boundary were present in the evidence, but they were not named as first-class findings.

That was the gap.

Not a security gap in the implementation. A review-completeness gap.

For adversarial review, that still matters.

Implicit coverage is where bad reviews go to look acceptable

You ask a system, reviewer, or agent to assess a set of risks. It comes back with plausible findings. The findings are true. The tests pass. The conclusion may even be correct.

But the structure does not let you answer the basic question:

Did we actually review every attack surface we said we were going to review?

In this case, the answer was "yes, but you have to know where to look."

That is not good enough.

If an adversarial prompt names seven attack surfaces, the review should preserve those seven surfaces. Each one should have a risk statement, mitigation, evidence, and disposition. If one is not applicable, say why. If one is covered by architecture rather than a unit test, name the boundary.

The pattern is simple:

attack surface -> risk -> mitigation -> evidence -> decision

That is the shape I want the review to preserve. Not because it is elegant. Because it keeps the review from compressing a specific adversarial contract into a confident paragraph.

Otherwise, a future reader has to infer coverage from nearby prose. And future readers are tired, distracted, and often you six weeks later, wondering why your past self chose violence.

Making the mapping explicit

I reorganized the review from four broad findings into seven explicit sections, A1 through A7, matching the original adversarial prompt.

The code did not change. The tests did not change. The mitigations were already there. What changed was traceability.

Replay manifests are the clearest example. They were named in the adversarial prompt, but not called out as their own finding in the first review. That matters because replay artifacts need to preserve enough state to make a run reproducible without becoming private-data leaks.

The improved review asks whether replay manifests leak private state or omit compatibility fields. It answers with the design: replay manifests serialize only the redacted configuration shape, including connector name, provider type, enabled families, connector version, and policy version. They do not serialize provider paths, credentials, raw record IDs, fixture names, or private source content.

Then it points to test_replay_manifest_uses_redacted_config_shape, which verifies that the manifest remains deterministic enough for replay compatibility while avoiding paths and record file names.

That is the difference between "we probably covered replay safety under dry-run leakage" and "A4 covers replay manifests; here is the risk, mitigation, and test."

Audit logs had a similar issue. They were covered under dry-run leakage, but audit logs are a different failure surface. Dry-run output is what the operator sees directly. Audit logs are what the system preserves. The updated review asks whether audit logs expose warnings, raw errors, metadata, provider details, or private record material, then points to test_audit_log_public_safe_summary_hides_raw_text_and_paths.

The public/private boundary was covered by design, but not named clearly enough. This connector lives in a private package. Public Imprint should not import Nexus-specific code, schema assumptions, private fixtures, or pipeline authority. The mitigation is architectural: the Nexus connector remains isolated in a private package, and public Imprint imports only generic connector interfaces.

That matters because not every security boundary is a runtime assertion. Some boundaries are repo shape, package ownership, import direction, and naming conventions. If the review only recognizes unit tests as evidence, it will miss those controls.

Evidence is the review

The improved review includes an evidence table mapping each attack surface to the tests or artifacts that support it.

That table is not decoration. It is the review.

A review should not just say "passed." It should say what passed, against which threat, with what evidence.

The GO decision needs the same treatment. The original decision was accurate:

GO for private deployment packaging after git/remote/deploy target is confirmed. The local implementation passes synthetic fixture, privacy, replay, audit, consent, and public Imprint compatibility gates.

That is fine as a conclusion. But conclusions are not controls.

The updated review clarified each gate. Synthetic counts matched policy. Dry-run output exposed no raw text, user IDs, fixture paths, SQL, or provider internals. Replay used a deterministic redacted config shape. Audit exported only counts and a manifest reference. Consent exclusions happened before support. Public compatibility meant the Nexus connector stayed private.

That turns the GO decision from a stamp into an audit trail.

Agents make almost-right failures easier

This matters more once agents are doing the work.

A human reviewer can be sloppy. An agent can be sloppy at machine speed, with excellent tone, confident formatting, and enough true statements to hide the gaps.

The dangerous failure mode is not that the agent makes everything up. It is that the agent is almost right. It reviews five of seven risks, points to real tests, gives the right conclusion, and never tells you which requested surfaces were not explicitly mapped.

That kind of output passes casual inspection.

It should not pass review.

A passing test suite tells you what you asserted.

Adversarial review asks what you forgot to assert.

The control is not "ask the model to be more critical." The control is structure.

No silent merging. No broad "covered under privacy." No "looks good overall."

No vibe checks.

Then the review became a workflow

The Sprint 15 review fixed the artifact shape, but that was only half the problem.

In Nexus, I also run an Operator Backlog: a Postgres-backed queue of work items that agents can investigate, implement, block, or close depending on tags, scopes, and evidence. Once agent work started moving through that backlog, adversarial review could not live only in sprint files.

It needed to become a lane.

That became ProjectAR.

ProjectAR is a task-only adversarial review role. It does not implement the fix. It does not own the backlog. It reviews the latest investigation, handoff artifact, or close candidate produced by another role.

The workflow shape is the backlog version of the document pattern:

candidate -> investigation -> adversarial review -> approval or block -> close evidence

In plain English: the worker can produce evidence that an item is ready to close, but a separate review lane has to decide whether that evidence is sufficient. The reviewer can approve it, block it, or route it somewhere else. That decision has to survive as part of the item history.

Otherwise, "implemented" quietly becomes "reviewed."

The tag model mattered more than I expected. I ended up with two concepts: adversarial_review as the active pending lane, and adversarial_reviewed as the terminal marker that says the review happened.

The active tag asks whether a review role should pick this up. The terminal marker records whether the item passed through review before closure.

The backlog needs both because those are different states. If the same marker means "needs review" and "was reviewed," the queue cannot tell whether an item is pending, blocked, approved, or merely carrying historical evidence. If the active tag disappears at closure without a terminal marker replacing it, the system loses proof that the review ever happened.

That is the workflow version of implicit coverage.

At the document layer, the review covered the risk, but the mapping was not obvious. At the workflow layer, the item may have been reviewed, but the state did not make that visible.

Both make future readers trust that the system did the right thing because the artifact no longer proves it.

The bar

No code changed in the Sprint 15 review-completeness pass. No tests changed. No mitigation changed.

The review changed.

That sounds cosmetic until you need to rely on the review later. Then it becomes the difference between "I think we looked at that" and "A5 covers audit logs; here is the exact test and why the GO decision includes it."

ProjectAR pushed the same idea into the backlog. It made adversarial review visible as operational state, not just a section heading.

That is the bar I want for agent-assisted review.

Not perfect certainty. Not theater. Not a longer markdown file because security people enjoy suffering.

A traceable adversarial review.

One where every claimed boundary has evidence, every requested attack surface has a disposition, every reviewed backlog item preserves its review state, and the final decision can be reconstructed without trusting the reviewer's confidence.

Because confidence is cheap.

Mapping is the control.

And state is what keeps the control alive after the markdown scrolls offscreen.

My agent swarm had a productive night. My pipeline lied about it.

Nic Lydon — Fri, 05 Jun 2026 14:13:54 +0000

I gave a Grok CLI agent swarm one instruction around 1am and went to bed. By 5:30 it had closed 37 items off my operator backlog and landed 20 commits on main: 2,838 lines added, 112 removed, across 51 files. Good night’s work for a process I wasn’t awake for.

Then I tried to audit it, and discovered my own ingest pipeline had quietly turned the entire run into garbage. The schema mismatch is mundane. The lesson underneath it is not.

The setup

I run a personal automation stack with an operator backlog: a Postgres-backed queue of tickets tagged by whether an agent can safely act on them unattended. My usual agents drain that queue against a set of contracts: an investigator role that diagnoses, and a worker role that ships with test gates. I wanted to see how a Grok CLI swarm would handle the same contracts.

The kickoff was one line: keep working through the queue, do the autonomous-safe work. The first session fanned out a dozen subagents in about three seconds. Each read the same prompt contracts, and each claimed tickets with a lease so they wouldn’t collide.

Every session writes a transcript to disk. My stack ingests those transcripts into Postgres so I can answer questions like “what did agent #7 actually run.” Standard local-first observability: the model provider doesn’t hold my data, I do.

The morning

I went to look at the per-session breakdown and the numbers were obviously wrong:

SELECT count(*)                            AS sessions,
       count(model)                        AS with_model,
       sum(turn_count)                     AS turns,
       count(*) FILTER (WHERE is_subagent) AS subagents
FROM grok_sessions;

 sessions | with_model | turns | subagents
----------+------------+-------+-----------
       95 |          0 |    99 |         0

Ninety-five sessions, zero with a model recorded, ninety-nine total turns, zero subagents. That immediately failed a sanity check: six sessions had subagent-<uuid> in their project name, and a dozen subagents launching in three seconds do not average one turn each. The data was lying.

The transcript format

The importer was written against a Claude-Code-shaped assumption. It expected a role field on each record, tool calls embedded inside content blocks, and a model field somewhere on the message. Here is what Grok CLI actually writes, one JSON object per line:

$ cat chat_history.jsonl | jq -r '.type' | sort | uniq -c
     47 assistant
     47 reasoning
      1 system
     51 tool_result
      3 user

Three things break immediately.

There is no role. The discriminator is type, and the values include reasoning and tool_result, record types the parser had never heard of. Anything that wasn’t system or assistant got bucketed into user. So a 149-record transcript with 47 real assistant turns counted as roughly three turns, the actual user messages. Everything else disappeared.

Tool calls aren’t embedded in content blocks. They’re a top-level array on assistant records:

$ cat chat_history.jsonl | jq -c 'select(.type=="assistant")
    | {model_id, tools: (.tool_calls | length), first: .tool_calls[0].name}' | head -3
{"model_id":"grok-build","tools":2,"first":"read_file"}
{"model_id":"grok-build","tools":3,"first":"read_file"}
{"model_id":"grok-build","tools":2,"first":"grep"}

The parser scanned content blocks looking for tool-use records that never existed, so every session appeared to have made zero tool calls.

The model was right there. Every assistant record carried "model_id": "grok-build". The importer had a state.model variable. It declared it, threaded it through the insert path, and never assigned it. A dead variable writing NULL ninety-five times.

The real failure

The real failure wasn’t misreading the transcript. It was assuming the transcript was the source of truth. The parser bug merely exposed the design flaw.

In hindsight the warning signs were obvious. Every session directory already contained multiple independent records of what happened: transcripts, event streams, session metadata, and git state. I had accidentally built an audit pipeline that ignored corroborating evidence and trusted the narrative, so when the narrative parser failed, the entire picture failed with it.

The session metadata alone contained everything I was missing:

{
  "current_model_id": "grok-build",
  "session_kind": "subagent",
  "num_chat_messages": 149,
  "created_at": "2026-06-05T00:28:06.848Z",
  "last_active_at": "2026-06-05T00:33:48.184Z"
}

Every missing column was already available. session_kind identifies subagents, num_chat_messages is the real turn count, created_at and last_active_at are the real timestamps, and additional metadata links the session back to the correct repository context. Meanwhile my importer was deriving timestamps from file modification times and inferring project identity from directory names.

The fix is not a patch to the parse loop. It’s architectural: trust authoritative session metadata for session-level facts, use transcripts only for message bodies, handle the record types that actually exist, and corroborate one source against another.

Why I’m telling on myself

Here’s the uncomfortable bit. I could not answer “what did the swarm actually do” from the swarm’s own records. I had to reconstruct the night from two sources the agents didn’t write: the backlog table, which recorded what changed state, and git history, which recorded what actually landed. Those two agreed, and they’re trustworthy precisely because the agents couldn’t rewrite them afterward.

$ git log --since="2026-06-04 23:30" --until="2026-06-05 08:00" \
    --pretty=tformat: --numstat \
  | awk 'NF==3 {a+=$1; d+=$2; f++}
         END {printf "%d files, +%d/-%d\n", f, a, d}'
51 files, +2838/-112

That number I believe. The transcript-derived “99 turns” I don’t.

The lesson

This has nothing to do with Grok specifically. Autonomous agents are trivial to launch and genuinely hard to audit, and the gap between those two facts is where teams are going to get hurt over the next few years. An agent telling you what it did is not evidence. It’s a claim, and it’s a claim sourced from the least trustworthy possible place: the thing being audited.

I filed two tickets against my own importer. One fixes the parser; the other forces a re-parse of the 95 corrupted rows, since the incremental loader skips files whose size hasn’t changed and a logic fix doesn’t change the file. They’ll get fixed. But the design flaw matters more than the bug: verify agent work from sources the agent cannot write to, and preferably from multiple sources that must agree. Otherwise you’re not auditing behavior, you’re auditing a story about behavior.

If your only record of an agent’s actions comes from a channel the agent controls, you don’t have observability. You have a press release with a timestamp.

The swarm had a productive night. I just had to prove it the hard way.