DEV Community: Panth Patel

[Boost]

Panth Patel — Tue, 27 Oct 2026 03:02:34 +0000

Unlabeled tokens create hidden retry loops

Karthigayan Devan for Google Developer Experts

Oct 25

Postmortem: The Tuesday Our AI Bill Had No Owner

#discuss #vertexai #sre #ai

14 min read

944 lines that turn a ticket into a headless Claude Code session

Panth Patel — Mon, 26 Oct 2026 13:00:30 +0000

I hand work to Claude Code through tickets, not a terminal. A 944-line orchestrator with zero dependencies turns each ticket into its own headless Claude Code session, and on 27 September I open-sourced it together with the tracker it works in: ticket-tracker, MIT.

I'm Panth, and I lead the software team at Oizom. I dropped out before the AI wave, because I figured the engineering was in solving problems in the real world, and my college made me choose between attendance and real-world applications. This post is about how the orchestrator works and why it is shaped the way it is.

Why tickets and not a terminal

I wanted to give my VM work from my phone, while travelling or waiting in a queue, with no laptop. The first version was a terminal in an app. It grew into a room full of agents, and three things went wrong:

I got attached to one long-running agent, and starting fresh ones was a chore.
The app was a door into the VM. If someone broke in, I would spend a long time cleaning up.
When the VM lost internet, I couldn't even queue a prompt.

Tickets fixed all three. Every ticket gets its own session, so there is no single agent to get attached to. The orchestrator runs on the machine and pulls work from the tracker, so nothing has to connect into that machine. The tracker is hosted apart from it, so I can raise tickets while the VM is offline, and my VM and my PC can take work from the same board.

The loop

An agent is a board member: a profile, a system prompt and a token that acts as that agent on one board. Assign a ticket to it and move the ticket to To do. From there:

To do  ──orch claims──▶ In progress ──agent reports "review"──▶ Review ──owner──▶ Done
                            │  ▲
           agent asks a     │  │ the answer (form or a comment) resumes
           question ────────▼  │ the same session
                          waiting

The orchestrator claims the ticket and starts claude -p in the repo the workspace points at, with the tracker's MCP tools. The agent reads the ticket and its thread, then publishes its plan as a task list on the ticket and marks each item as it goes.

When it needs a decision, it calls ask_question, which puts a form card in the thread, and ends its turn with status: "waiting". It does not poll. The answer, or any comment in the thread, starts the next turn with --resume on the same session, so the agent picks up with its whole history. Moving the ticket back to To do resumes it as rework. Unassigning it, or moving it anywhere else while the agent runs, stops the agent.

The brief every agent gets says it in two lines:

You run headless in [the repo] and nobody is watching your terminal. The owner only sees the ticket, so everything you want them to see goes on the ticket.

The orchestrator owns the ticket's state

The agent gets the tracker's MCP tools, minus the ones that decide where the ticket stands:

// What only the orch may do: the stage, the heartbeat, the inbox, assignment.
const DISALLOWED = ['move_ticket', 'update_ticket', 'assign_ticket', 'create_ticket', 'heartbeat', 'get_events', 'ack_events']
  .map((t) => `mcp__tm__${t}`).join(',');

The agent does the work and reports review, waiting or blocked. The orchestrator moves the ticket, beats, reads the event inbox and posts the receipt. So a ticket only reaches Review after the orchestrator has seen the agent report it done.

The receipt is one message per turn with what the turn cost. Claude Code's total_cost_usd is cumulative across --resume, so the orchestrator stores the last total and posts the difference. Because the agent pushes the cost of each turn, you can see where the money actually went — which ticket, which turn — instead of one number at the end of the month. One ticket of mine reached $402 over nine runs before the app showed any of it; the full story of that fix is in $402 on one ticket.

Health from a heartbeat

The orchestrator beats once a minute while an agent runs, and every fifth beat it re-reads the task list, so the ticket's dot shows progress like 3/12 · Running tests. The board never stores "this agent is dead". It derives health from the last status and the time since the last beat, in one function, recomputed every 15 seconds:

Health	When
working	state is working and the last beat is under 75 s old
stale	state is working and no beat for 75 s: the orchestrator may have died
idle	waiting for an answer, or stopped
done	finished, ready for review
error	stopped with an error, for example a timeout
none	no status yet

75 seconds is one missed beat plus a little slack. Only "working" can go stale: idle, done and error are statements that stay true until the next beat. After five minutes of silence the agent's owner gets one notification per silence.

One command layer

The app, the REST API (/v1, OpenAPI 3.1) and the MCP server (/mcp) all run the same commands. Nothing an agent can do is different from what a person can do, and every change is attributed, for example "Owner via Claude (MCP)". Tokens are scoped to one board and act either as you or as one of your agents.

What running it takes

Everything runs on one Firebase project: Firestore, Functions, Storage, Realtime Database, Auth and Hosting. "Self-hosted" here means your own Firebase project on the Blaze plan, not a Docker image. Locally, pnpm dev starts the emulators, seeds demo data and serves the app (Node 22, pnpm 9, Java 21 and the Firebase CLI).

Firebase, because the tracker had to live apart from the machines doing the work: if the VM is offline or the agents are down, I can still raise tickets. From there the app does live reads straight from Firestore, Auth and push come with it, and the orchestrator wakes on a Realtime Database revision stream, so when an agent comes back it picks up where it left off.

A workspace is a folder with workspace.json (repo, board, agent, how many agents at once), a .env with a Worker token for that agent, and brief.md with the project's rules:

node workspaces/ws.mjs new myproject --repo ~/src/myproject --board MYP --agent "Claude MyProject"
# mint a Worker token for that agent in the app (Account › Tokens), put it in workspaces/myproject/.env
node workspaces/ws.mjs up
node workspaces/ws.mjs status

Agents run unattended in your working tree, with permissionMode from workspace.json: acceptEdits in the template, bypassPermissions only if you trust the brief and the repo. Read workspaces/lib/brief.md before you enable one.

What is not in it

At work I've written about per-ticket stacks: a branch in every repo, its own services, its own copy of the dev database and a preview URL per ticket. None of that is in this repo. That code is specific to Envizom and proprietary. Here the agent works in the repo you point it at.

The repo is github.com/Panth977/ticket-tracker; workspaces/README.md has the loop.

If you run Claude Code headless, where does its question go when it needs a decision, and who answers it?

Replacing Node-RED with a 10 MB browser editor and a start.sh

Panth Patel — Sun, 25 Oct 2026 13:01:09 +0000

A client disconnected our server from the internet for security, and the integration for that client still had to be written on it.

I'm Panth, and I lead the software team at Oizom, where we build Envizom, an air quality platform that many clients run on their own servers. Every one of those deployments needs a bit of custom work: pull data from somewhere, or expose an API in a schema the client gives us for a public display. That work can't go into our production code. This post is about where it lives now: codeshell, a small browser workspace I built and open-sourced under Apache-2.0.

Node-RED first

Node-RED was the default for these integrations before I joined. It is fragile for this kind of work:

An async for loop is hard to write.
Five API calls with something equivalent to Promise.all is close to rocket science.
Debugging is the worst part.
If there is a JavaScript library for the job but no Node-RED node in the marketplace, you spend five hours figuring out how to set the library up.

These are short programs. Plain code is the right tool for them.

Then one repo per deployment

The next idea: each deployment gets its own codebase, built into one Docker container that holds all of that server's customisations.

It works until the client disconnects the server from the internet. Then the integration still has to be written, so you open a shell in the container, edit the code with vi, and hope nobody pulls the latest image before you have copied the same code back into the repo.

What I actually needed

A VS Code-like editor in the browser, on the server, with the code directory mounted from outside so it survives a restart. If the code lives in one volume, and the volume is backed up, the integration doesn't need a repo at all.

VS Code in the browser is heavy for this, because the language servers and compilers run on the server. I looked for a light editor and didn't find one, so I wrote codeshell.

What codeshell is

Part	What
Backend	Go, one ~10 MB static binary with the UI embedded. It serves plain files and PTYs. No language servers, no compilers.
Frontend	Svelte 5, CodeMirror 6 for the editor, xterm.js for terminals. Syntax modes load per file type.
`/code`	The one required volume. Root of the file tree, working directory of every terminal.
`/code/start.sh`	Runs on every container start, restart and recreate.
Size	about 1,750 lines of Go and 3,700 of Svelte and TypeScript

Running one integration is one container:

docker run -d --name myapp \
  -p 8080:8080 -p 3000:3000 \
  -e CODESHELL_PASSWORD=change-me \
  -v myapp-code:/code -v myapp-state:/var/lib/codeshell \
  ghcr.io/panth977/code-shell:latest

Port 8080 is the editor. Port 3000 is whatever the integration listens on.

start.sh is the deploy

Terminals are temporary on purpose. Anything you install or start from a terminal is gone after a restart. What has to come back goes in start.sh:

#!/usr/bin/env bash
set -e
command -v node >/dev/null || (apt-get update && apt-get install -y nodejs npm)
npm ci
exec node index.js

The image is Debian slim, so start.sh can apt-get install whatever runtime the integration needs. On the first run it installs Node and the modules. On later runs node_modules is already in the volume, and mounting /root and /opt as volumes too keeps package caches and runtimes, so restarts get faster.

codeshell supervises the script. Its output shows as a start.sh entry in the terminal panel, next to your own terminals, with Restart and Stop buttons, and its state is in the top bar: running, stopped, exited, waiting or missing. The restart policy is on-failure by default, and restarts back off from 1 second, doubling, to a cap of 60 seconds. On a fresh volume it writes a commented start.sh template with a Node and a Python example.

Terminals keep their scrollback on the server, so a page reload or a dropped connection doesn't lose them.

Where it sits

The other half of our on-premise work is On-Prem Manager, which deploys and updates the containers on each client server and snapshots their volumes. With codeshell, a client's integration is one container and one volume. On-Prem Manager can persist that volume's content to its cloud, where it is versioned and reviewed, so the integration needs no repo of its own.

What it doesn't try to be

Single user. One login, an HMAC-signed session cookie, and repeated failed logins from one IP are locked out for longer each time.
Anyone who signs in has a root shell in the container. Put it behind HTTPS and use a strong password. The password and the session secret are removed from the environment of terminals and start.sh.
No IntelliSense. It's an editor with syntax highlighting, quick open, split groups, find and replace, and a terminal.

The repo is github.com/Panth977/code-shell. Images are published for linux/amd64 and linux/arm64.

Where do your per-client integrations live today, and what happens to them when the client's server goes offline?

90/10: what we took away from our AI coding agent, and why

Panth Patel — Sat, 24 Oct 2026 13:01:16 +0000

The first version of our agent pipeline let the coding agent run the git and deploy steps. It didn't fail often, but when it did, it failed in a different way each time, and I couldn't debug that. Today roughly 90% of the pipeline is plain scripted code, and the agent does one job.

I'm Panth, and I lead the software team at Oizom, where Envizom is our environmental monitoring platform. Two days ago I wrote the short version: the eight steps that run after a developer drags a ticket to Git Push. This is the long one. It follows one ticket through every column of the board and goes through each job we took off the agent, with the reason, then what it costs and what broke.

The first version

Before agents, our problems were ordinary ones. Envizom is several repos: hardware drivers, a simulator, a backend and a frontend. A new developer needed days to get a working local setup. Releases meant keeping track of which dev database, prod database, backend and frontend were pointed at each other, and as time went on I was the only person who knew every combination.

Coding agents made my developers faster. With an agent, one developer can keep three tasks in flight. The mistakes then moved out of the code and into the branches around it. A change landed on the wrong branch, or a fix only worked on one machine. I wrote about that in One stack per ticket.

In the first version of the pipeline, the agent did more of the operational steps. It did the engineering, and it also ran git and the deploy. Compared with what we run now, it was slower and more expensive, and it was worse at the tasks that need to be boring.

The failures taught me the most. A script that fails the same way twice is a bug you can fix: you read the log, find the line and change it. An agent that merges a little differently on each ticket gives you nothing to fix. Each failure is a new story, and the next run will tell another one.

The split we ended up with follows one line. If a step is the same on every ticket, it is a script. The agent keeps the engineering.

One ticket, end to end

The board has seven columns. One of them has the agent in it.

Column	Who acts	What happens
Feedback	developer	writes what they want, in plain English. That is the whole spec
Approved	scripts	dragging the ticket here is the handoff. The orchestrator provisions the ticket's stack
Working	agent	does the engineering work inside that stack, and talks only through the ticket thread
QAQC	developer	opens the preview the agent posted, tests it, replies in the thread or moves it on
Git Push	scripts	commit, merge dev in, open and merge a pull request per repo, post the links
VM Update	scripts	wait for CI, roll the new images onto the dev VM, health checks, rollback if unhealthy
Completed	scripts	the developer gets a notification, and the ticket's stack is torn down

Each developer has their own devbox on our dev VM: VS Code in the browser, with our orchestrator, orch, inside it. Every developer is paired with their own agent, and their orch only picks up tickets assigned to that agent. The board is shared. The machines and the agents are not.

Here is a real ticket from the walkthrough video. I asked for a camera layout change on a device page. The agent built it in the ticket's stack and posted the preview. I replied in the thread: "also handle more than 4 cameras with a horizontal scroll, and set up 8 test cameras on this device". The ticket went back to the agent, which picked up with the context it already had, did both, and created the 8 test cameras in the ticket's own copy of the database. I tested the new preview and dragged the ticket to Git Push. After that, two notifications, and the change was live on dev.

The person in that story touched the ticket three times: to write it, to reply to a preview, and to drag it. The agent had one column. Scripts ran the other four.

The ten jobs scripts own

This is the list no agent touches: watching the board, provisioning the stack, copying the database, issuing certificates, driving git, opening pull requests, waiting on CI, deploying, health checks and rollback. They fall into two groups, before the agent starts and after a person says yes.

Before the agent starts

Watching the board. Orch watches for tickets dragged to Approved and assigned to its agent. Nothing here needs any thought. A ticket is either in the column or it isn't, and a model would add cost and nothing else.

Provisioning the stack. Every approved ticket gets a fresh branch in every repo, named after the ticket, plus its own backend, drivers, simulator and frontend. This is the step everything else depends on, so it has to be the same every time. If the stack came out a little different per ticket, nothing built on top of it could be trusted.

Copying the database. Each ticket gets a copy of the dev database, so it can change data freely. This is the safety decision in the whole design. An agent that runs a bad migration or deletes the wrong rows does it to the ticket's copy. Nobody else's work is in that box, so nobody else's work is touched. A step that sets up the safety net shouldn't be done by the thing the net is for.

In Envizom V3, which we are rebuilding from scratch, the copy is cheap. Every store has a dev driver. Postgres, S3, the time-series database and Redis all become SQLite in dev, and MQTT becomes a service built into the Go backend. SQLite runs inside the Go process, so copying a database means copying a few files. That change made our sandboxes about 10x cheaper and faster than sandboxes with full databases.

Issuing certificates. Every ticket has a private preview URL with an HTTPS certificate issued on demand. That URL is what the developer tests in QAQC, so it has to exist and work before the agent posts it.

After a person says yes

When the developer drags a ticket to Git Push, no agent is involved from that moment on. Orch runs eight steps:

Git Push
  1  commit the work on the ticket branches
  2  merge the current dev branch into them
  3  open a pull request in each affected repo
  4  merge the pull requests
  5  post the PR and commit links into the ticket thread
VM Update
  6  wait for CI
  7  roll the new images onto the dev VM
  8  run health checks against the VM
       unhealthy -> roll back to the previous images, revert the merges,
                    tell the thread what happened
       healthy   -> Completed, notify the developer, tear the stack down

Driving git and opening pull requests. One of our goals from the start was that using agents would not weaken the normal process. Every feature still gets its own branch, commits, pull request and review. That only holds if those steps happen the same way on every ticket. An agent deciding how to merge is not a feature. A script that always merges the same way is.

Posting the links. Step 5 looks small, but it is how the thread becomes the audit trail. The PR and commit links land next to the ask, the previews and the feedback, so one ticket holds the whole story of one change.

Waiting on CI. Waiting is the clearest case of a job a model shouldn't hold. Nothing gets decided while it waits, and an agent session kept alive just to wait is still a session.

Deploying, health checks and rollback. These are the steps where sameness matters most. If the VM is unhealthy after the deploy, orch rolls it back to the previous images, reverts the merges and tells the thread, without anyone asking. A rollback that improvises is the last thing you want on the day something breaks. A scripted one runs the way it ran on every ticket before.

All of it stops at the dev environment. Auto-merge, the VM update and the rollback apply to dev only. Production goes through a separate path, where people make the call on a slower cadence. That is on purpose. We don't want automation deciding what reaches production, and we don't want a fast feature list to become the goal.

Taking away "I'm done"

The same split shows up in ticket-tracker, the board I open-sourced, where a small orchestrator turns each ticket into a headless Claude Code session. (The per-ticket stack scripts are not in it. They are Envizom-specific code.) It takes a few more jobs off the agent, ones that are easy to miss because they don't look like jobs.

Deciding it is finished. The agent never moves its own ticket. Its last message is structured output, enforced with --json-schema:

const OUTCOME_SCHEMA = {
  type: 'object',
  properties: {
    status: { type: 'string', enum: ['review', 'waiting', 'blocked'] },
    summary: { type: 'string' },
    question_id: { type: 'string' },
  },
  required: ['status', 'summary'],
};

The orchestrator reads it. review moves the ticket to Review. waiting means a person is needed, so nothing runs until someone answers. blocked usually means the turn ran out of room, so the orchestrator resumes it by itself, up to 3 times, and then says on the ticket that it has stopped. The agent reports. The script decides what that means for the board.

Saying it is alive. The heartbeat comes from the orchestrator, not the agent: one beat every 60 seconds while the agent works. A crashed process can't report that it crashed, so health is worked out from the age of the last beat. The agent's part is its task list, which the orchestrator reads for the status message. More in Agent health from a 60-second heartbeat.

Working out what a turn cost. When a run ends, the orchestrator reads Claude Code's result line and posts the turn's receipt on the ticket. The app's comment box never posts one. Each receipt goes out with an idempotency key made from the ticket and the run number, so a retried request doesn't count a turn twice.

Waiting for an answer. When the agent needs a decision, it asks in the ticket and ends its turn. It is told not to poll. My answer starts the same session again with --resume. Headless Claude Code asks, exits, then resumes on the answer has the details.

Any of these could be left to the agent. Each one is the same on every ticket, so none of them is.

What the agent still does

The engineering. It reads the ticket, works in the ticket's own repos and services, changes the code, and creates the test data it needs in the ticket's database copy, as in the 8 test cameras. When the developer replies with feedback, it picks up with the same context it had before, without re-reading the repo to find out where it was.

It also asks. A real decision that belongs to the owner, such as scope, design, or anything destructive or outward-facing, becomes a question in the ticket rather than a guess.

That is the 10%. It is the only part of the pipeline where the right answer changes from ticket to ticket, so it is the only part where a model is worth what it costs.

What stays with people

Writing the ticket. The thread is the only way to talk to the agent, so every prompt is stored with the work it produced. I read how each person on the team asks for things and coach them on it. That has done more than any prompting guide. See Our coding agent prompts live in the ticket.

QA. The agent posts a preview and moves the ticket to QAQC, and that is as far as it goes. The developer who asked tests the preview, because they are the only one who knows what they meant. An agent grading its own work would remove the one review that matters most. Every step after QA is built to be boring and repeatable, and none of them asks whether the change was the right change. QA is where that question gets asked, so it is the one place we didn't automate. More in 1 human check, 8 scripted steps.

The drag to Git Push. Shipping to dev is a person's action in the thread. Shipping to production is a separate, slower decision.

What it costs

What	Number	Source of the number
One agent turn in the walkthrough video	$4.77	the receipt under the agent's reply: "turn $4.77 · total $4.77"
The dev VM with every devbox and ticket stack	about ₹4–6k a month	rented, one machine
Tickets it runs in parallel	6, comfortably	same VM
Sandbox cost after SQLite dev drivers	about 10x cheaper and faster	V3 rebuild
A scripted step	no turn receipt	it is not a model call

The last row is the plain reason scripts are cheaper. Every agent turn posts its cost into the ticket thread, right under the answer. A scripted merge, deploy or health check posts no receipt, because it doesn't call a model. An operational step done by the agent is done inside a billed turn.

The receipt is a design choice too. The person who asked sees what the ask cost, where they asked it, and it changes how they write the next one.

What broke on the scripted side

Scripts break as well. The difference is how they break.

The cost number was wrong. On ticket-tracker, one ticket reached $402 over nine agent runs and nothing in the app said so. The orchestrator already wrote the cost of every run to a local log, but nothing showed it where the work was. Putting the number on the ticket had a trap: Claude Code's total_cost_usd is cumulative across --resume. Run 8 made one API call with zero output tokens and reported $392.83, the same as run 7. The fix is a few lines in the orchestrator. It keeps the last total per session and posts the difference. The full fix is in $402 on one ticket.

Polling was most of the bill. Two orchestrator loops asked the tracker's API for the full ticket and event lists every 30 seconds or so. Over three days that was 8,822 + 8,815 calls, 88% of every request the API served. The fix was again in the script. Agents now hold one stream on a small revision node and fetch only the delta when it moves. That one is in Polling was 88% of my API calls.

The heartbeat was on the bill twice. The first version kept each beat stream as a Firestore document: 1,440 writes per agent per day, plus a sweep that ran every minute to notice silence, 4,261 runs in three days. Beats moved to the Realtime Database, about 100 bytes each, and silence is now worked out at read time from the age of the last beat.

None of these needed an agent to diagnose. Each showed up in a log or a bill, and each was fixed in one place in the code. That is the trade I wanted from the split. The agent's mistakes stay inside a ticket's stack, where a bad change hit a copy. The scripts' mistakes show up in a log, the same way twice, so they can be fixed.

Why the boring 90% makes the 10% useful

Most of what makes the pipeline worth using comes from the scripted part. Per-ticket isolation is what lets the agent work freely, and lets a ticket sit in QAQC for a week while three others ship. The thread is the audit trail because scripts post into it at every step: the preview, the cost of each turn, the PR and commit links, and the rollback if there is one. Stored prompts are reviewable because the thread is the only channel. Human QA and dev-only scope are the two places we chose not to automate, and both are easier to hold when everything around them is predictable.

The order I'd build it in

If you're setting up something like this:

Isolation first. If every task can't run in its own branch, with its own services and its own data, nothing after it is safe.
A handoff surface the team already uses. For us that was the ticket.
Script every step that is the same on every ticket: provisioning, git, CI, deploy, health, rollback, and the receipts.
Only then, the agent inside the box, with the smallest job you can give it.

The ticket is the interface has the walkthrough video and the first write-up of the pipeline.

If your team runs coding agents, which step is the agent doing today that a script would do the same way every time?

I lead the software team at Oizom, where Envizom turns data from air-quality and environmental monitors in 80+ countries into something cities and industries can act on. I dropped out before the AI wave, because I figured the engineering was in solving problems in the real world, and my college made me choose between attendance and real-world applications. Find me on LinkedIn.

On-Prem Manager: one engineer per two on-prem Docker sites didn't scale

Panth Patel — Fri, 23 Oct 2026 13:01:04 +0000

About five on-premise deployments a year, and one dedicated engineer for every two of them. That was the shape of our setup when we realised on-premise was going to be most of our business.

I'm Panth, and I lead the software team at Oizom, where we build Envizom, an air quality monitoring platform. This post is about the deployment problem that came with selling it, and On-Prem Manager (OPM), the tool I built and open-sourced under Apache-2.0 to fix it.

Why on-premise

Many organisations don't want to hand their data to a cloud. Governments most of all, and air quality is first a government responsibility. So when a government picks software like Envizom, it also needs that software on its own servers, for its own security reasons.

The more we sold, the clearer it got: local deployment is the one real selling point for an air quality platform, and per-client customisation is what wins the client.

What a deployment actually looks like

The ideal case: the client gives you exactly what you asked for, and if something breaks on their side, a firewall rule or an IP config, they own it.

That is not what happens. One client wants a DC/DR setup. Another wants high availability. Another wants a single server. Another wants DC/DR eventually, but for now on one server, with a second one later and two more after that.

Then there are the integrations: pull data from one of their systems, or expose an API in a schema they hand us for a public display. None of that belongs in our production code.

What we had

An end-to-end setup on Docker Compose, with Portainer as the GUI.

After enough deployments it was clear Compose wasn't enough. Every client is different, so every install meant a lot of configuration and setup afterwards. A simple issue took hours to close, because debugging a remote server through that stack is slow.

That setup is fine at five deployments a year with one engineer for every two. It is the bottleneck the moment you want more.

Where the idea came from

Oizom started as an IoT company. It ships devices with software inside, and every device sits at a remote location where nobody can do anything by hand. For that, Oizom uses a device manager, Aikaan.

So: what if our deployment team had the same thing for on-premise servers?

What the team needs to run a deployment is a short list. Automated volume backups. A way to make a new version live. Environment and config per site. Variables, with separate secrets for each deployment. A GUI to see the containers. I wrote down every problem we hit in operations and built a tool around the list.

What OPM is

Two parts.

Part	What it does
Cloud console	The catalog of images, deployment templates, variables, secrets and files. You commit a change once and review the diff per site. Self-host it on a VM (SQLite and local files) or run it on Firebase, same frontend build.
Agent	One Go binary per server, with its own GUI built in. It applies commits to that server's Docker, runs an embedded Caddy as the edge proxy, snapshots volumes, and rolls back.

The README puts it as: like Portainer, plus a git-style commit history for every server's config.

Install is one line

The cloud's Setup page gives a command per server:

curl -fsSL -H "Authorization: Bearer opi_…" <apiBase>/v1/install/script | sudo sh

It runs preflight checks, uses the Docker that is already there (20.10 or newer) or asks before installing a pinned version, downloads the agent and checks its sha256 and Ed25519 signature, writes the config, sets up a systemd service, and waits up to 60 seconds for a health check, rolling itself back if that fails. Everything it installs is pinned when the script is generated.

Images without a registry

You register images in the cloud's catalog, and the cloud keeps its own copy of their layers. The agent loads images from those blobs with docker load and never connects to a Docker registry, so it never sees a registry password. Sites in the same deployment copy layers from each other first, and only one site with internet fetches from the cloud.

Commits, not proposals

Templates, deployments and each site are branches of commits. The latest commit on a deployment's branch is what its sites should run. You can compare any two branches and cherry-pick between them.

Nothing changes on a site until a person on that site merges it, in the agent GUI. A merge runs one fixed sequence:

LOCK -> SNAPSHOT -> APPLY -> VERIFY -> COMMIT

VERIFY waits for each container to pass its health check, or to run 10 seconds without a restart, within 300 seconds by default. Any failure rolls back, and a failed merge leaves no commit. The failed container is kept for 24 hours under a -failed-<time> name, with its last 200 log lines, so the post-mortem doesn't depend on someone grabbing logs in time.

The dangerous act isn't the merge, it's restoring data. So rollbacks and volume restores get a 60-second countdown and a typed ACCEPT. The spec's reasoning: "the commit message + plan review are the deliberate act… restoring data is still the dangerous act".

Backups by default

Before every update and rollback, the agent snapshots config, image references and the volumes marked for backup, as .tar.zst archives. It keeps the last 10. A scheduled snapshot runs daily at 02:00 site time and keeps the last 7. A snapshot refuses to start if free disk is under 1.2 times its estimate. Snapshots download as signed .opmsnap files.

Docker stays the source of truth

People still change things in Docker directly. The agent scans every 60 seconds and on Docker events. A hand edit to an image, env, port, mount or file shows up as uncommitted drift, and merges are refused until it is committed or discarded. A site can also auto-commit drift once it has been stable for 30 seconds, authored as "changed in Docker".

Standalone first

Every agent GUI can do everything locally with no cloud at all: create containers, change images, edit files, variables and secrets. The cloud is how you run many sites, not something a site needs to keep running.

When the client cuts the internet

Some clients take the server off the internet for security. OPM has three routes to the cloud, in order:

The server's own internet.
A peer site in the same deployment that has internet, relaying.
The browser of the person using the agent GUI. The GUI tab long-polls the agent and runs the cloud requests from the browser, so an engineer's laptop on the client network becomes the tunnel.

The agent only ever talks HTTPS on 443 to one API domain, which is one line in a client's firewall request.

And for a client that allows neither the agent nor the cloud, the cloud can mirror every branch to a GitHub repository. A machine checks out its site's branch, fills in site.env and runs ./deploy.sh: plain docker compose, Caddy as the router, Portainer as the GUI, no OPM code on the box. A real-Docker test compares that output with the agent's and reports 0 differences. You lose drift detection, snapshots and rollback, which is why it's the last resort.

The size of it

The full specification, SPEC-final.md, is 894 KB, and its decision log has 116 numbered entries, each with the reason. The code is about 97,000 lines of Go, 96,000 of TypeScript and 60,000 of Svelte, with 105 Go test files and 157 TypeScript test files. There is an end-to-end harness that runs real agents against real Docker.

Integrations got their own tool too. They run in codeshell, a small browser editor in a container whose code is one volume. OPM's "Persist to cloud" versions a volume's content in the cloud, the docs name "client integration code" as the use, so that code is reviewed and kept there instead of in a git repository. codeshell gets its own post.

What I'd ask you

If you ship software that customers run on their own servers: what does one deployment cost you in engineer-hours, and which part of it did you automate first?

90% of our AI coding agent pipeline is plain scripted code

Panth Patel — Thu, 22 Oct 2026 20:06:35 +0000

Roughly 90% of the pipeline that ships our code is plain scripted code. The AI coding agent does one thing: the engineering work inside a single ticket's workspace, in response to what a person wrote in that ticket.

We didn't start there. The first version gave the agent more of the operational steps, and this post is about why we took them back.

I lead the software team at Oizom, where Envizom is our environmental monitoring platform. The pipeline below is our dev pipeline for Envizom.

The shape of it

A developer writes what they want in a ticket, in plain English, and drags it to Approved. Our orchestrator picks it up and gives the ticket its own stack: a branch in every repo named after the ticket, its own backend, drivers, simulator and frontend, a copy of the dev database, and a private preview URL with an HTTPS certificate issued on demand.

The agent works inside that stack and talks to the developer only through the ticket thread. When it's done it posts the preview link and moves the ticket to QAQC. The developer tests the preview, replies in the thread, and either sends it back or drags it on.

Approved ──► [scripts] provision stack ──► [agent] engineering work ──► QAQC
                                                    ▲                    │
                                                    └── feedback ────────┤
                                                                         ▼
Completed ◄── [scripts] commit, merge, PR, CI, deploy, health ◄── Git Push
     │                         │
     │                         └── unhealthy: roll back images, revert merges
     ▼
 stack torn down

One box in that diagram has the agent in it. Everything else is a script.

What the scripts do

Here is the list of jobs no agent touches:

watching the board
provisioning the per-ticket stack
copying the database
issuing certificates
driving git
opening pull requests
waiting on CI
deploying
health checks
rollback

The clearest example is what happens when a developer drags a ticket to Git Push. From that moment there is no agent involved. The orchestrator runs eight steps:

#	Step	Done by
1	Commit the work on the ticket branches	script
2	Merge the current dev branch into them	script
3	Open a pull request in each affected repo	script
4	Merge the pull requests	script
5	Post the PR and commit links back into the ticket thread	script
6	Wait for CI	script
7	Roll the new images onto the dev VM	script
8	Run health checks against the VM	script

If the VM isn't healthy afterwards, the orchestrator rolls it back to the previous images and reverts the merges, on its own, and then tells the thread what happened. If it is healthy, the ticket moves to Completed and the developer gets a notification. The ticket's stack is torn down after that.

Why we took the steps back from the agent

We tried it the other way first, with the agent doing more of the operational steps. It was slower, it was more expensive, and it was worse at exactly the tasks that need to be boring.

That last one is the reason that matters. Merging, deploying and rolling back should happen the same way on every ticket, and that sameness is the whole value. An agent deciding how to merge is not a feature. A script that always merges the same way is.

The scripted part is deterministic and tested. When it breaks, I read a log and fix it.

Where the agent earns its place

The agent's job is the engineering work itself. In the walkthrough video I asked for a camera layout change on a device page. The agent shipped it and posted the preview. I replied in the thread asking it to also handle more than 4 cameras with a horizontal scroll, and to set up 8 test cameras on that device. It did both, including creating the test data in the ticket's own database copy, and it picked up with the same context it had before.

Isolation is what lets it do that safely. Because every ticket has its own stack, an agent that does something silly to the database did it to a copy. A developer can have several tickets in flight at once, and one ticket can sit in QAQC for a week while three others ship.

Two more lines we drew on purpose:

QA is human. The developer tests the preview and decides whether the ticket moves forward. An agent grading its own work would remove the one review that matters most.
This pipeline stops at dev. Auto-merge, rollback and VM updates all apply to the dev environment only. Production goes through a separate path where people make the call, on a slower cadence.

The order I'd build it in

If you're setting up something similar, this is the order we ended up with:

Isolation first. If every task can't run in its own branch, its own services and its own data, nothing after it is safe.
A handoff surface the team already uses. For us that was the ticket.
Only then, the agent inside the box.

And keep the box scripted. The agent should be the smallest part of the system.

The longer write-up, with the video walkthrough, is The ticket is the interface.

If your team runs coding agents, which step did you hand to the agent that a script should own?

0 reads to switch views: Firestore board filters in the browser

Panth Patel — Thu, 22 Oct 2026 19:57:41 +0000

Switching between saved views on a board in my tracker costs zero Firestore reads. The board's active tickets are loaded once and kept live, and every filter, sort, group-by and swimlane is a pure function run over them in the browser.

The constraint that forced it: Firestore cannot OR across fields, and every combination of filters you want to query needs its own composite index. A tracker with user-defined views over custom fields would need thousands of them.

I'm Panth, and I lead the software team at Oizom. This is from ticket-tracker, my open-source board. Its decisions page lists the choices that are expensive to reverse, each with the alternative that was turned down. This is D5.

The decision


Chosen	One live query per board over its active tickets. Filter, sort, group-by and swimlanes are pure functions in `packages/shared`.
Rejected	One Firestore query per view.
Why	Firestore cannot OR across fields, cannot filter on range and inequality over two fields, and needs a composite index for every combination. User-defined views with custom fields would need thousands of indexes. In the browser, switching views is instant and costs no reads.
Ceiling	About 5,000 active tickets per board, or about 5 MB.

A saved view is a few lines of data. A filter is a tree of and / or groups over leaves:

export type FilterGroup = { op: 'and' | 'or'; children: FilterNode[] };
export type FilterLeaf = { field: string; cmp: Cmp; value?: unknown };
export type FilterNode = FilterGroup | FilterLeaf;

A leaf can compare any built-in field (stage, priority, assignee, tag, due date, state, creator, text, linked) or any custom field, with twelve comparisons from is to between. "Mine and high priority, or anything of Ann's" is this, straight from the tests:

const f: FilterNode = {
  op: 'or',
  children: [
    {
      op: 'and',
      children: [
        { field: 'assignee', cmp: 'is', value: 'me' },
        { field: 'priority', cmp: 'is', value: 'hi' },
      ],
    },
    { field: 'assignee', cmp: 'is', value: 'ann' },
  ],
};

As a Firestore query, that OR spans two fields, and the tree can nest as deep as people like. As a function over an array, it's a recursive evalNode.

One function draws every view

The whole engine is one call:

applyView(tickets, view, board, ctx) → groups[]

It does four things, in this order:

Keep the ticket states the view asks for. The default is active only, so archived tickets leave every default view.
Run the filter tree.
Sort on several keys, ties broken by rank and then id, so every client gets the same order.
Group, then group again inside each group for swimlanes.

Kanban, table, calendar and timeline all draw from what it returns. A ticket with two assignees or three tags appears in each of their groups. Swimlanes are computed over all visible tickets, so every column has the same rows.

me, today, thisWeek and overdue are tokens in the saved view, resolved when the function runs, against the viewer's id and time zone. A shared view called "Mine, due this week" means the same thing to everyone who opens it, each in their own week.

Because the function is pure, it's tested as one: the test file is 453 lines against 665 lines of engine, with a fixed clock (Tuesday 2026-09-22 in Kolkata) and a board with six custom field types.

The one query, and what it costs

The query that feeds the engine is about as simple as Firestore gets: the board's tickets where state == 'active', one listener per board. It needs no composite index.

Loading the whole board on every open would pay one read per card. So the board store is incremental. The first open runs the full query. After that, the device starts from Firestore's own IndexedDB cache of the last snapshot and asks only for tickets whose updatedAt is newer than the newest one it has seen. An unchanged board answers that with no documents, which Firestore bills as one read, whatever the board's size.

The delta query has no state filter, on purpose. A ticket that leaves the active set has to arrive so it can be dropped from the board, and a query for active tickets would never send it. The browser decides what to keep. One inequality on one field also means the delta needs no composite index.

A hard delete doesn't show up in any updatedAt > query, because the document is gone. So the watermark is trusted for 6 hours from when it was written. After that, the next open pays for one full query and agrees with the server again.

Where it stops

The decision writes down its own ceiling: about 5,000 active tickets, or about 5 MB, per board. Past that, the plan is to flag the board as large and run its views through Typesense, which can filter on every field, with paging. Archived tickets are never loaded by default, so the number that matters is open work, not history.

For a personal tool, where a board is the unit people share, 5,000 open tickets is far more than a board should hold. The trade is the right one for this product. It wouldn't be for a tracker that hosts a whole company on one board.

What I'd take from this for any Firestore app with user-built filters: if the data a screen needs fits on the device, query it once, keep it live, and do the rest in code you can unit test. Indexes are for the data that doesn't fit.

Where do your filters run: in the database, or over data you've already loaded?

MCP on OAuth 2.1: one URL and a consent click, no API key

Panth Patel — Tue, 20 Oct 2026 23:52:57 +0000

Adding my tracker to Claude takes one URL and one click on a consent screen. Nobody makes a key, copies it, or pastes it anywhere.

The constraint: an MCP client that a person uses every day should not hold a long-lived secret the person had to handle by hand, and the person should be able to see and cut off what it can reach.

The decision


Chosen	OAuth 2.1 with PKCE, dynamic client registration, and protected-resource metadata, as the MCP authorization spec expects
Rejected	Pasting an API key into the MCP client
Why	It is the only way a person adds the server with one URL and a consent click, in Claude and other clients. The grant is per person, can be revoked in Account › Connected apps, and every change it makes is attributed `via MCP`.

API keys did not go away. They still work on /mcp for headless setups, where nobody is there to click Allow. Claude Code takes a token in a header, and so do the orchestrators that run my agents. OAuth is for the case where a person is sitting in front of the client.

What the client does, with nobody pasting anything

The whole flow is discovery. The client starts knowing one URL, and every other address comes from a document the server publishes.

1. POST /mcp, no token
   ← 401  WWW-Authenticate: Bearer resource_metadata=".../.well-known/oauth-protected-resource/mcp"
2. GET  /.well-known/oauth-protected-resource/mcp        (RFC 9728)
   ← { authorization_servers: [base], scopes_supported: [...] }
3. GET  /.well-known/oauth-authorization-server          (RFC 8414)
   ← authorize, token, register and revoke endpoints; PKCE S256 only
4. POST /oauth/register                                  (RFC 7591)
   ← client_id
5. GET  /oauth/authorize?...&code_challenge=...  → our consent screen
6. person picks boards, presses Allow  → redirect with ?code=
7. POST /oauth/token  code + code_verifier  ← access + refresh tokens

Step 1 carries the design. The 401 is not "your token is bad". It is "go and sign in, and here is where to start". A client that has never heard of this server can follow the header all the way to a working token.

Some clients probe OIDC discovery first, so /.well-known/openid-configuration answers with the same document.

The consent screen is where scope gets decided

The screen names the app asking, lists the permissions it wants in plain words ("Read tickets", "Post, edit, pin and react in threads"), and asks which boards it may act on: every board, or the ones you tick. Nothing is granted until you press Allow.

The server checks the ticked boards against the boards the person can actually read. A board you are not on is an error, not a silent drop.

What the grant gives back shapes the MCP server itself. A tool the grant's scopes do not allow is not registered at all, so it is missing from tools/list, and every handler checks again anyway. The model never sees a tool it would only get a 403 from.

Token lifetimes

Token	Lives	Rules
Authorization code	60 s	Single use, bound to the client, the redirect URI and the PKCE S256 challenge
Access token	1 h
Refresh token	30 days	Rotated on every use

Tokens are stored only as their SHA-256. The grant is a document under the person (users/{uid}/oauthGrants/{grantId}), and it is the revocation switch: revoking deletes it and every token issued under it. Revoking in Connected apps stops the app on its next call, including sessions already open.

A refresh may narrow scopes, never widen them. Asking for a scope the grant does not hold fails with invalid_scope.

The rule that catches a stolen refresh token

Rotation alone does not help if nobody notices the old token coming back. So a used refresh token is kept, marked with when it was used, until it expires. If it is presented again, that is two parties holding the same token: the real client and someone else. The server revokes the whole grant.

// A client that refreshes twice at once, or retries after losing the response,
// presents the same token seconds apart: that is not theft.
const retry = row.usedAt !== undefined && opts.now - row.usedAt <= REFRESH_RETRY_GRACE_MS;
if (row.usedAt !== undefined && !retry) {
  // THEFT DETECTION: an already-rotated refresh token came back.
  reused = { uid: row.uid, grantId: row.grantId };
  return null;
}

The grace window is 60 seconds. Without it, a client that fires two refreshes in parallel, or loses the response and retries, would lock its own user out. With it, a retry gets a fresh pair and a replay an hour later kills the grant with "Refresh token reuse detected — the app must be connected again".

The small checks that keep it from becoming an open door

Dynamic registration means anyone can register a client, so the edges matter:

Registration is rate-limited per IP address: 20 a minute, 500 a day.
Redirect URIs must be https, or http on localhost, 127.0.0.1 or [::1] only, or a private-use scheme such as cursor:// or vscode://. javascript:, data:, file:, vbscript: and blob: are refused, and so is any URI with a fragment.
A bad client or redirect URI is shown as an error page, never redirected to. Redirecting there would make the server an open redirector. Any other error goes back to the client as ?error= the way RFC 6749 says.
Only two grant types: authorization_code and refresh_token.

What it cost

One more surface to own. An API key is one table and one header check. This is an authorization server with eight endpoints, a consent screen in the app, and a Connected apps page. My own review notes had a row for it: the MCP flow pointed at a consent page that did not exist yet. That was fixed by building the page at /oauth/consent, after which the grant shows in Connected apps.

For an MCP server people add to Claude by hand, I would pay it again. The person never touches a secret, sees exactly which boards they handed over, and has one Revoke button.

If you run an MCP server, which do your users hold today: an OAuth grant or a key they pasted?

One zod schema per route: Express handler, OpenAPI and TS client

Panth Patel — Tue, 20 Oct 2026 23:36:09 +0000

Each endpoint is declared once, with a zod schema for what comes in and what goes out. From that one declaration you get three things: the handler Express serves, the OpenAPI JSON for your docs, and a TypeScript client for the frontend.

The constraint: the declaration is not a server. @panth977/routes only describes routes. Serving them is a separate handler, @panth977/routes-express for Express, or one you write for another framework.

I'm Panth, and I lead the software team at Oizom. @panth977/routes is one of my open-source libraries on JSR. This post walks through it with code: middleware, an HTTP route, an SSE route, and the calls that turn a folder of routes into docs and a client.

What one declaration gives you

From the route's schemas	Through
A typed handler, served by Express	`ROUTES.execute` + `@panth977/routes-express` (or your own handler)
OpenAPI JSON	`ROUTES.getRouteDocJson`, built on zod-openapi
A TypeScript client for the frontend	`ROUTES.CodeGen.genTsCode`

Because every route is already declared with types, the docs don't need a second source. The OpenAPI JSON can serve a docs UI, and the same JSON generates the frontend's code. Nothing is written twice.

Routes are built the same way as @panth977/functions, so you keep its wrappers, context, static, name and namespace, build, and input/output schemas. The functions post covers those.

Middleware

A middleware declares its own request schema and its security rule. Both end up in the docs of every route that uses it.

const DecodedStateKey = FUNCTIONS.DefaultContextState.CreateKey<ReturnType<typeof decodeToken>>({ label: 'Decoded',scope: 'global' });
const authorize = ROUTES.Middleware.build({
  security: {
    JwtAuth: {
      type: "apiKey", // this rules will be used for documentation of routes
      description: "Authorize ** /users/login **.",
      name: "x-auth-token",
      in: "header",
    },
  },
  request: ROUTES.z.MiddlewareRequest({ // this schema will be used to address route request schema, in documentation
    headers: z.object({ "x-auth-token": z.string() }).passthrough(),
  }),
  response: ROUTES.z.MiddlewareResponse({}),
  async func({context, headers}) {
    const token =
      headers["x-auth-token"] ??
      headers["authorized"] ??
      headers["x-token"];
    const decoded = await decodeToken(token);
    if (!decoded) throw createHttpError.Unauthorized("Token not found / Token got Expired / Invalid Token!");
    // set state to context for others to make use of it!
    context.useState(DecodedStateKey).set(decoded);
    return {};
  },
});

This one looks for the token in three headers (x-auth-token, then authorized, then x-token), decodes it, and throws a 401 if that fails. On success it stores the decoded token in the request's context under DecodedStateKey. Any route that lists authorize can read it back from there, which is what the next example does.

An HTTP route

const getProfile = ROUTES.Http.build([authorize], "get", "/profile", {
  description: `Get user profile.`,
  request: ROUTES.z.HttpRequest({
      query: z.object({
        getFriends: z.enum(['true', 'false']).default('false'),
      })
  }),
  response: ROUTES.z.HttpResponse({
    body: z.object({
      id: z.number(),
      username: z.string(),
      address: z.string().nullable(),
      noOfProjects: z.number().int().gt(0),
    }),
  }),
  static: {
    query: pg.query(`Select * from users where id = ? LIMIT 1`),
  },
  async func({context, query, build}) {
    const { userId } = context.useState(DecodedStateKey).get();
    const result = await build.query.execute([userId]);
    if (!result.rows.length) throw createHttpError.NotFound("Given user was not found in db, could be because someone has deleted the user.");
    if (query.getFriends === 'true') {...}
    return result.rows[0];
  },
});

The route reads the user id that authorize stored, runs a query declared once in static, and returns a row. The getFriends query parameter arrives as the string 'true' or 'false', with 'false' as the default, because that is what the schema says.

Compared with @panth977/functions, input becomes request and output becomes response. The request's path, query, headers and body arrive in func as named arguments. The response has headers and body.

ROUTES.z.HttpRequest and ROUTES.z.HttpResponse stand in for z.object so the types stay exact. You can still pass your own. For the docs, a route also takes description, tags, summary, security, reqMediaTypes and resMediaTypes.

A Server-Sent Events route

An SSE route is an async generator. This one yields the request's logs 30 at a time until there are none left.

export const getLogs = ROUTES.Sse.build([systemAuthorized], "get", "/logs/{requestId}", {
  request: ROUTES.z.SseRequest({
    path: z.object({
      requestId: z.string(),
    }),
  }),
  response: ROUTES.z.SseResponse(
    z
      .object({ id: z.number(), msg: z.string(), ts: z.date() })
      .array()
      .transform((x) => JSON.stringify(x))
  ),
  async *func({context, path: { requestId }}) {
    let logs = [];
    let offset = 0;
    do {
      const result = await pg.query(`SELECT * FROM logs where id = ? ORDER BY ts OFFSET ? LIMIT 30`, [requestId, offset]);
      logs = result.rows
      offset += logs.length;
      yield logs;
    } while (logs.length);
  },
});

Chaining middleware instead

If you prefer app.get('/profile', …) with middleware chained in front, ROUTES.Endpoint gives you that shape.

const initMiddleware = ROUTES.Middleware.build(...);
const userAuthMiddleware = ROUTES.Middleware.build(...);
const thirdPartyAuthMiddleware = ROUTES.Middleware.build(...);

const EndpointFactory = ROUTES.Endpoint.build().addMiddleware(initMiddleware);
const endpoints = {
  userAuthorized: EndpointFactory.addMiddleware(userAuthMiddleware),
  thirdPartyAuthorized: EndpointFactory.addMiddleware(thirdPartyAuthMiddleware),
}

const getProfile = endpoints.userAuthorized.HTTP('get', '/profile', ...);
const getProfile3rdParty = endpoints.thirdPartyAuthorized.get('/api/{userId}/profile', ...);

From a folder of routes to OpenAPI

ROUTES.getEndpointsFromBundle takes a module's exports and keeps only the endpoints (HTTP and SSE), dropping helpers and middleware. The detailed types are lost at this step, which is what you want when you are serving or filtering every route at once.

// --- routes/index.ts ---
export * from './m.ts'; // middleware
export * from './a.ts'; // route1, route2, and a helper variable
export * from './b.ts'; // route3, route4 (SSE), and a helper function
// --- server.ts ---
import * as routes_ from './routes/index.ts';
const routes = ROUTES.getEndpointsFromBundle({bundle: routes_}); // strong type will be lost
console.log(routes) // {route1: ..., route2: ..., route3: ..., route4: ...}

Then the OpenAPI JSON, with routes tagged internal-apis left out:

const routes = ROUTES.getEndpointsFromBundle({
    bundle: await import('./routes/index.ts'),
    excludeTags: ['internal-apis'],
});
const OpenApiJson = ROUTES.getRouteDocJson(routes, {
    info: {
        title: 'My Apis',
        version: '0.0.1',
    }
});

Serving a route

ROUTES.execute runs one endpoint through lifecycle hooks you supply. This is the seam where a framework plugs in. The example uses Express; routes-express is the full reference handler.

type MyFrameworkOpt = { req: Express.Request, res: Express.Response };
const MyEndpointExecutionLifeCycle: ROUTES.LifeCycle<MyFrameworkOpt> = {
    init(...) {...},
    onStatusChange({...}) {...},
    onExecution({...}) {...},
    onResponse({...}) {...},
    onComplete({...}) {...},
}
app.get('/getUser', (req, res) => {
    FUNCTIONS.DefaultContext.Builder.forTask(null, async (context, done) => {
        ROUTES
            .execute({
                context,
                build: getUserEndpoint,
                lc: MyEndpointExecutionLifeCycle,
                opt: { req, res },
            })
            .finally(done);
    })
})

The TypeScript client

The last output comes from the OpenAPI JSON. ROUTES.CodeGen turns it into frontend code with a typed call for every API. Here it is served from an endpoint:

app.get('/code-gen', (_, res) => {
  FUNCTIONS.DefaultContext.Builder.forTask('CodeGen', function (context, done) {
    res.on('finish', done);
    try {
      const { code } = ROUTES.CodeGen.genTsCode({ context, input: { json, options: {} } });
      res.setHeader('content-type', 'text/x.typescript').send(code);
    } catch (err) {
      console.log(err);
      res.status(500).send('Something went wrong!');
    }
  });
});

The schema is written once, and the handler, the docs and the client are all read from it.

The rest of the packages are at jsr.io/@panth977.

Where do your API docs come from today: the route code itself, or a spec someone keeps in step by hand?

Request-scoped logging in Node.js and Express at Oizom

Panth Patel — Tue, 20 Oct 2026 23:27:22 +0000

When an API call fails with a 500 in production, I want one id from the response and, behind it, every log line that request wrote, in order. At Oizom every request gets a context id, every function logs through context.log, and the logs of a request are only kept when it fails or the frontend asks for them.

I'm Panth, and I lead the software team at Oizom, where our devices send tens of thousands of air-quality readings to the cloud every minute. This post is how that logging works in our Node.js and Express backends, with the code. Hosted logging services can collect all of this, but none of them group logs by request out of the box, and running a separate service per request is not an option.

The problem

A client makes a request and the API answers with a 500. To debug it you would normally try to recreate the bug, probably by logging in to the frontend as that user. The faster route is to keep the logs of every request that answered 500, keyed by its requestId, and read them.

The idea

Every time a request comes in, generate a UUID and include it in the response headers. Whenever anything fails, the API sends a requestId that the client can show, in a sandbox view or similar.
Create a context object (with requestId as context.id) that includes a method context.log, which works like console.log.
When creating the context, ensure there is an implementation of context.log where you can access context.id. This allows you to group the logs by requestId effectively.

Solution

My libraries @panth977/functions and @panth977/routes do this out of the box.

Create Context

// Create Context like following
app.get('/v1/user/:userId', function (req, res) {
    const id = null; // either set id, to string or leave it null for auto generate UUID
    DefaultContext.Builder.forTask(id, async function (context, done) {
        res.on("finish", done);
        context.log('My task started!');
        const user = await getUser({context, input: req.params}); // pass context to all the functions, for access to context.log
        res.send(user);
        context.log('My task completed!');
    });
});
// OR
// use @panth977/routes api to skip the process of creating functions!
export const GET_USER = ROUTES.Http.build([WebsiteAuthValidator], 'get', '/v2/user/{userId}', {
    request: ROUTES.z.HttpRequest({path: getUser.input}),
    response: ROUTES.z.HttpResponse({body: getUser.output}),
    async func({context, path}) {
        context.log('My route started!');
        const user = await getUser({context, input: path}); // pass context to all the functions, for access to context.log
        context.log('My route completed!');
        return {body: user};
    }
})

Make every function take `context`

That way every function can use context.log.

const getUser = FUNCTIONS.AsyncFunction.build({
    input: z.object({userId: z.coerce.number()}),
    output: z.object({name: z.string(), ...}).nullable(),
    async func({context, input}) {
        context.log('Getting user', input);
        const [user] = pg.query(`SELECT * FROM users WHERE id = ? LIMIT 1`, [input.userId]);
        context.log('Found user', user);
        return user ?? null;
    }
});

Capture `context.log`

Every call to context.log runs your subscribed callback.

const unsub = DefaultContext.onLog(function (context, args) {
    console.log(context.id, ...args);
    // send the log to socket 
    // store to database
    // write to a file!
});
// incase you like to unsubscribe!
unsub();

By default, context.log doesn't write to the console at all, so add console.log in the callback if you want it.

How we use it at Oizom

At Oizom our devices send calibrated air-quality data to the cloud, tens of thousands of entries every minute, and clients read their live data through our APIs. With that many requests, debugging anything in production by reading a shared log stream was close to impossible.

What I wanted

Keep logs only for failed requests, so successful ones don't clutter the logs database.
A way to list the request IDs made by user from the website, user from third-party APIs, device from the device itself, and system-{type} for any internal system calls like cron jobs, reports, alerts, etc.
The full logs for any one request ID.

What I built

Define a log payload

    type log = {
        ids: string[];
        msg: string;
        ts: number;
        line: number;
    }

Many logs belong to the same ID, so why can one log belong to many IDs? Most logs do have a single ID, context.id. Multiple IDs are there for filtering: the same log line can be found by request, by user, by organisation or by login session. More on that below.

A secure socket app with three channels:
1. /send_log - every server sends its logs through this channel.
2. /stream_all_logs - streams all logs in real time as the servers send them.
3. /stream_logs - streams only the logs that match its room (one room per id).
A service that listens to /stream_all_logs and stores every log. For us that is Cassandra, self-hosted in a container, with a ttl of a few weeks.

    CREATE TABLE logs.logs_v2 (
      id text,
      ids set<text>,
      ts bigint,
      msg text,
      line bigint,
      PRIMARY KEY ((id), ts)
    ) WITH CLUSTERING ORDER BY (ts ASC) AND ...

A service with an HTTP GET endpoint /fetch_logs that reads logs by id from Cassandra, ordered, with limit/offset (Cassandra has token-based pagination built in).
A small frontend with two modes: [ real-time , database ]
1. A text box for the id and a submit button.
2. Depending on the mode, it connects to the socket /stream_logs or fetches from /fetch_logs, with a "fetch next" button for more.
3. Each log has an expand button that shows all its ids, which link to the other ids that share the log.
That is the logging system. The core is in the backend. Every time a context is created:
1. It gets a logs state.
2. Every context.log entry goes into that state.
3. On an error, or when something needs to be kept, we set .importance = true.
4. On dispose, if .importance === true, all the logs are streamed to /send_log.

    type Logs = {
        logs: { ids: string[]; msg: string; ts: number; line: number }[];
        important: boolean;
    };
    const LogsStateKey = FUNCTIONS.DefaultContextState.CreateKey<Logs>({
        label: 'Logs',
        scope: 'global',
    });
    const stringifyLogs = LOGS.createStringifyLogger({ forEachLine: false });
    // exposing this function for set the importance level.
    // this determines wether logs should be sent to socket or not!
    export function needToSendLogsToSocket(context: Pick<FUNCTIONS.Context, 'useState'>) {
        const state = context.useState(LogsStateKey).get();
        if (!state) return;
        state.important = true;
    }
    // exposing this function for more versatile use. (link different ids to same log)
    export function logToSocket(
        context: Pick<FUNCTIONS.Context, 'useState'>,
        { ids, line, ts }: { ids: string[]; ts?: number; line?: number },
        ...args: any[]
    ) {
        const state = context.useState(LogsStateKey).get();
        if (!state) return;
        state.logs.push({
            ids: ids,
            msg: stringifyLogs(null, args),
            ts: ts ?? Date.now(),
            line: line ?? state.logs.length,
        });
    }
    // set a state on context
    FUNCTIONS.DefaultContext.onCreate(function (context) {
        context.useState(LogsStateKey).set({ logs: [], important: false });
    });
    // store all the logs in the state
    FUNCTIONS.DefaultContext.onLog(function (context, args) {
        // every context.log will be mapped to 
        // ids=[context.id] 
        // so single id for all the context.log 
        logToSocket(context, { ids: [context.id] }, ...args);
    });
    // at end, check if it is important, and stream them to socket.
    FUNCTIONS.DefaultContext.onDispose(async function (context) {
        try {
            const state = context.useState(LogsStateKey).get();
            if (!state.important) return;
            for (const log of state.logs) socket.send(JSON.stringify(log));
        } catch (err) {
            console.error('Socket Log Error:', err);
        }
    });

Whenever a user logs in from the website, a third party, a device or a system-{type}, the login session gets its own UUID in the auth token.
Every request goes through an auth check. That is where I work out what kind of login it is and which categories the request belongs to.

    const WebsiteAuthValidator = ROUTES.Middleware.build({
        request: ROUTES.z.MiddlewareRequest({
            headers: z.object({}).passthrough(),
        }),
        response: ROUTES.z.MiddlewareResponse({
            headers: z.object({}).passthrough(),
        }),
        async func({ context, headers }) {
            // frontend wants to store logs!
            if (headers['x-store-logs'] === 'true') needToSendLogsToSocket(context);
            const decoded = ...; // decode the token from headers.
            if (decoded.mode === 'client') throw createHttpError.Unauthorized('This is not a client Api, You only have access to client APIs');
            const ids = [context.id];
            if (decoded.mode === 'system') {
                if (decoded.id) ids.push(`Core:User:System:${decoded.id}`);
            } else {
                ids.push(`Core:User:${decoded.userId}`);
                ids.push(`Core:User:Org:${decoded.orgId}`);
                if (decoded.id) ids.push(`Core:User:${decoded.userId}:${decoded.id}`);
            }
            const { req } = context.useState(ExpressStateKey).get();
            logToSocket(context, { ids }, req.method, req.path);
            context.useState(DecodedStateKey).set(decoded);
            if (headers['x-store-logs'] === 'true') return { headers: { 'x-request-id': context.id } };
            return {};
        },
    })

This way, when you get logs for id = Core:User:2141, you see a list of all the API calls made by this user. Each log includes a list of other relevant ids, and one of them will be context.id. That way, you can get the requestId, and by fetching the logs of that requestId, you get the complete logs for that specific API call.

When a request fails, we keep its logs.

const onError: ExpressHandler.onError = function ({ context, error: err }) {
  if (createHttpError.isHttpError(err)) {
    return { 
      body: err.message,
      headers: { 'x-request-id': context.id, ...(error.headers ?? {}) },
      status: err.statusCode
    };
  }
  if (context) {
    const { req } = context.useState(ExpressHandler.ExpressStateKey).get();
    // log the request, could be helpfull for recreating error, in debugging.
    context.log(
      'Req::',
      JSON.stringify({ method: req.method, path: req.params, query: req.query, body: req.body, headers: req.headers }, null, '\t'),
    );
    context.log('Err While Req:', err);
    console.error('500 Error at:', context.id);
    // set the importance.
    needToSendLogsToSocket(context);
  } else {
    // nothing we can do further!
    console.error('Err:', err);
  }
  return { 
    body: 'Something went wrong!', 
    headers: { 'x-request-id': context.id }, 
    statusCode: 500 
  };
};
const v2 = ExpressHandler.serve({ bundle: ExposedRoutes, onError });
app.use(v2);

The log viewer

I built the viewer with AI's help: I designed the system and let AI write the boring parts. Here it is:

Later I needed a list of every email sent to one address, or every SMS sent to one phone number. All it took was a new template for id. Each of those logs carries the requestId, so you can see which request sent the email.

Each log shows 4 things:

A toggle button to see other linked ids
The line number of the log
The ts in local time
The msg, which is the log message.

The extra ids are links, so you can jump from a user to a request to an email while reading.

If you build something like this, build it yourself rather than buying it: the ids you will want to filter by are specific to your system.

How do you find the logs of one failed request in production today?

Ticket keys in Firestore: issued once, never reused after delete

Panth Patel — Tue, 20 Oct 2026 23:10:10 +0000

I split the ticket id from the ticket key so that tickets could move between boards, and then I took moving tickets out of the product.

The split stayed. A key like ENG-42 gets pasted into WhatsApp messages, emails, GitHub PR titles and other tickets, and every one of those has to keep pointing at the right ticket for as long as the link exists, including after the ticket is gone.

I'm Panth, and I lead the software team at Oizom. This is from ticket-tracker, my open-source board, written to replace a tracker called TeamTrack. Its decisions page lists the choices that are expensive to reverse. This is D4.

The decision


Chosen	A generated, immutable `ticketId`, a `key` field, and a `keys` index that keeps old keys as redirects
Rejected	The key as the document id, which is what TeamTrack did with `TICKET_ID`
Why	Keys change when a ticket moves. Links in emails, WhatsApp messages, GitHub PRs and other tickets' `#refs` must keep working after that.

So a ticket lives at boards/{boardId}/tickets/{ticketId}, and ENG-42 is a field on it. The key also gets a document of its own at the top level:

// keys/{ticketKey}: '#ENG-42' → the ticket it names
export const KeyIndexSchema = z.object({
  ticketId: TicketIdSchema,
  boardId: BoardIdSchema,
  /** The key itself, equal to the doc id. */
  current: TicketKeySchema,
  deleted: z.boolean().optional(),
});

current is the redirect: an old key's row can name the key the ticket has now. Exact-key search follows one hop of it.

Then the move was dropped

Phase 1 built moving a ticket to another board: copy the ticket, its thread, activity and files to the other board, give it a new key, and redirect the old one.

Decision D13 took it out. Moving is machinery (option mapping per field, assignees who aren't on the new board, a queued copy job, the key redirects) for something that happens rarely in a personal tool, and it was the most dangerous write in the system. Now a ticket stays on the board it was created on. To connect work across boards, you link the tickets or #reference them. A ticket filed on the wrong board gets closed and filed again, and the new one can reference the old.

So the reason written next to D4 no longer happens. Nothing writes a redirect today. The id and the key stayed separate anyway, because the same split does three other jobs.

1. References store the id, not the key

When you type #ENG-42 in a comment, the editor stores a ticketRef node with the ticket's id. The key in that node is only a hint for rendering:

case 'ticketRef': {
  const id = String(n.attrs?.ticketId ?? '');
  if (id && !refs.includes(id)) refs.push(id);
  const hint = typeof n.attrs?.key === 'string' && n.attrs.key ? n.attrs.key : 'ticket';
  line += `#${opts.keyOf?.(id) ?? hint}`;
  return;
}

The server derives each message's refs as a list of ticket ids. Mentions of people work the same way: a mention stores a uid and shows the current display name, so a rename never breaks it. TeamTrack stored mentions as [@Name](mailto:…) links in Markdown and keyed assignees by email, so a change of address was a data migration.

2. A key is issued once

Keys come from a counter on the board document, inside the same transaction that creates the ticket. Not from a query for the highest number:

const start = board.nextNumber;
tx.update(ref, { nextNumber: start + ticketIds.length });
return ticketIds.map((ticketId, i) => {
  const number = start + i;
  const key = ticketKey(board.key, number);
  tx.create(keyRef(key), { ticketId, boardId, current: key });
  return { key, number };
});

tx.create fails if the document already exists. A key that was ever handed out, whether it now points at a live ticket, a deleted one or (once) a moved one, can't be overwritten. The transaction fails instead.

The price is in the comment above that code: the board document takes about one sustained write a second, so one new ticket per second per board. Bulk creation takes a range with a single counter bump.

3. Delete leaves a tombstone

Deleting a ticket removes it from the links and references of other tickets, takes it off watchers' lists, deletes the ticket document and its thread, and keeps the key:

// Tombstone: kept forever, never reissued.
const tomb: KeyIndex = k
  ? { ...k, deleted: true }
  : { ticketId, boardId, current: ticket.key, deleted: true };
tx.set(keyRefs[i]!, tomb);

After that, #ENG-42 in an old comment says the ticket was deleted. With the key as the document id, deleting the ticket deletes the only record that the key was ever used. If that key were ever issued again, every old link would open the new ticket as if nothing had happened. The comment in the schema says it more directly: the row is kept "so an old link says 'this ticket is gone' instead of landing on somebody else's work".

Deleting a whole board does the same for every ticket on it, and its board key stays claimed too. Board keys like ENG are claimed across the whole app, first come first served, like a username. That is what lets /t/ENG-42 and #ENG-42 mean one ticket without saying whose board it is.

Who reads the index

Everything that starts from a key goes through keys/, not through a board:

the REST and MCP doors, when an agent or a script asks for ENG-42
the GitHub hook, which finds keys in branch names, PR titles, commit messages and comments, and posts a line in each ticket's thread
Markdown coming in through REST or MCP, where #ENG-42 is turned into a ticket id before it is stored
the search box, for an exact key

A tombstoned key resolves to nothing for all of them. Over REST and MCP it is a 404, and so is a key on a board the caller can't read.

What it cost

One extra document per ticket, and one extra read whenever something arrives by key instead of by id: a pasted key, a webhook, a search. Links between tickets store only the id, so finding one looks on the same board first, which is almost always right, and falls back to the keys index, whose rows carry the board.

What I'd keep from this for any system with human-readable ids: let the name be a field, give it its own index, issue it once, and never delete the index row. The move that justified D4 is gone, and the split still earns its place on delete alone.

Have you ever reused an id after a delete, and found out from a link that opened the wrong thing?

Our coding agent prompts live in the ticket, reviewed like PRs

Panth Patel — Tue, 20 Oct 2026 23:00:29 +0000

Every prompt my team writes to our coding agent is stored in the ticket that asked for it. The ticket thread is the only way a developer can talk to the agent, so there is nowhere else for a prompt to go.

I'm Panth, and I lead the software team at Oizom, where Envizom is the platform that turns data from air-quality and environmental monitors into something cities and industries can act on. This post is about one design goal of our agent pipeline: the prompt is kept with the work, and it can be reviewed.

The only channel is the thread

A developer writes what they want in a ticket, in plain English, and drags it to Approved. There is no IDE open, no terminal and no prompt window. That ticket text is the whole spec.

Our orchestrator, orch, gives the ticket its own stack: a branch in every repo, its own backend, drivers, simulator and frontend, a copy of the dev database and a private preview URL. A coding agent does the work inside that stack. When it is done, it posts the preview link into the thread and moves the ticket to QAQC.

The developer opens the preview, tests it, replies in the same thread and moves the ticket back. The same agent, with the same context, picks up where it left off. Every follow-up is a message in the thread, so every follow-up is stored too.

When we set out to build this, it was one of five things we wanted from the start:

Every prompt should be stored with the work it produced, so we can review and coach how people ask for things.

What one thread holds

By the time a ticket is done, its thread holds the whole exchange, turn by turn:

In the thread	Who writes it
What was asked	the developer, in the ticket and in each reply
What was built	the agent, with the preview link
Which preview was tested	the developer, by replying to that preview
The feedback	the developer, in the same thread
What the turn cost	posted on every agent turn
The PR and commit links	orch, after the ticket is dragged to Git Push

Here is the shape of a real one. In the walkthrough video I asked for a camera layout change on a device page:

ticket   camera layout change on the device page          -> Approved
agent    builds it in the ticket's stack, posts the preview -> QAQC
me       "also handle more than 4 cameras with a horizontal scroll,
          and set up 8 test cameras on this device"         -> back to the agent
agent    does both, the 8 test cameras in the ticket's own
          database copy, posts the preview                  -> QAQC
me       drags it to Git Push
orch     commits, opens and merges the PRs, posts the links,
          waits for CI, deploys to dev, checks health        -> Completed

Every line of that is in the thread. Anyone can open the ticket and read what was asked, in the words it was asked in, next to what came out of it.

Reading prompts like pull requests

Because the prompts are stored, they can be reviewed. I read how each person on the team asks for things and coach them on it. That has been more useful than any prompting guide.

A prompting guide tells you what good asks look like in general. A ticket thread shows you how one person on your team asked for one change, what the agent built from it, and how many replies it took to get to something they were happy with. You can point at a specific message and talk about it, the way you would point at a line in a pull request.

The cost line matters here as well. Each agent turn posts its cost into the thread, right where the ask was made. The person asking sees what the ask cost, and it changes how they write the next one. That is on purpose too.

Why it needs one stack per ticket

A stored prompt is only useful if it sits next to the work it produced, and nothing else. That is where per-ticket isolation comes in.

Each ticket has its own branch in every repo, its own services and its own copy of the dev database. So a ticket's thread is a self-contained record: what was asked, what was built in that stack, what was tested on that preview. A ticket can sit in QA for a week while three others ship, and its history is still in one place, not mixed into anyone else's.

The thread is also where the rest of the process shows up. After Git Push, no agent is involved: orch commits, merges, opens and merges the pull requests, waits for CI, deploys to the dev VM and checks its health, and posts the PR and commit links back into the same thread. The prompt, the result and the review trail end up in one ticket.

What we get from it

Every prompt the team writes is stored with the ticket it belongs to.
I can review how people ask for things using their own tickets, not examples from a guide.
The cost of each ask is visible where it was asked.
The ticket is the audit trail: what was asked, what was built, what was tested and what feedback was given.

If your team uses coding agents, where do the prompts go once the work ships?

I lead the software team at Oizom. The full pipeline, including where we drew the line (dev only, human QA, visible cost), is in The ticket is the interface.