DEV Community: Rohan Sen Sharma

I Tried 28 Claude Code Skills. Here Are the 5 I Kept

Rohan Sen Sharma — Thu, 15 Oct 2026 03:00:00 +0000

The best Claude Code skills, for me, are ponytail, caveman, graphify, grill-me and ui-ux-pro-max: five out of 28 I tried. Each one does a narrow job the model does not already do well. The other 23 went, mostly because the models have absorbed what they used to add, and every installed skill costs context.

That is the thesis of this post. Six months ago, the big skill frameworks were the fastest way to make a coding agent plan, test and review properly. Today a lot of that behaviour is built into the models. So I now keep the smallest set that still changes the result, and I judge every skill by one question: does it earn the tokens it loads?

The 5 best Claude Code skills at a glance

Skill	What it is for	Stars (5 Oct 2026)	Works in Codex	Install in Claude Code
ponytail	Making the agent write the least code that works	155k	Yes, as a Codex plugin	`/plugin marketplace add DietrichGebert/ponytail`
caveman	Terser agent output; optional proxy that compresses what it reads	110k	Yes, via `npx skills add`	`claude plugin install caveman@caveman`
graphify	A queryable knowledge graph of your code and docs	124k	Yes	`pip install graphifyy && graphify install`
grill-me	A relentless interview to sharpen a plan	276k (repo)	Yes, via `npx skills add`	`/plugin install mattpocock-skills`
ui-ux-pro-max	Design intelligence for building interfaces	133k	Yes, via its CLI	`/plugin marketplace add nextlevelbuilder/ui-ux-pro-max-skill`

Stars are context, not a ranking.

1. ponytail: the laziest senior dev in the room

ponytail (MIT) pushes the agent to write the least code that does the job. Since version 5 (8 October 2026), its README reports about 53% less code, 26% lower cost and 41% faster runs; those are the author's numbers, from their own benchmark, not mine.

Install: in Claude Code, /plugin marketplace add DietrichGebert/ponytail, then /plugin install ponytail@ponytail. In Codex, codex plugin marketplace add DietrichGebert/ponytail, then codex plugin add ponytail@ponytail.

Gotcha: it works through hooks that run on their own: three in Claude Code (SessionStart, SubagentStart and UserPromptSubmit), and in Codex you have to trust its two lifecycle hooks before it works.

2. caveman: why many token when few do trick

caveman (Apache-2.0) makes the agent say less in controlled, terse English while keeping code exact. An optional local proxy also compresses what the agent reads, such as logs, JSON and test output; the README reports 33.2% fewer input tokens across 54 Claude Code runs.

Install: claude plugin marketplace add JuliusBrussee/caveman && claude plugin install caveman@caveman, or npx skills add JuliusBrussee/caveman -g for Codex and other agents.

Gotcha: two facts worth knowing. The proxy sits between your agent and the model API, on your machine, with your keys. And the separate caveman CLI sends usage statistics by default: commands, token counts, a random install ID, OS and IP, never prompts or code. Turn that off with caveman telemetry off or DO_NOT_TRACK=1. The skill on its own sends nothing.

3. graphify: your codebase as a graph

graphify (Apache-2.0) turns a codebase, its docs, SQL schemas and PDFs into a queryable knowledge graph, written out as graph.html, GRAPH_REPORT.md and graph.json, and queried with /graphify. Code is parsed locally with tree-sitter and does not leave the machine; docs and media go through your assistant's model or an API key you configure.

Install: pip install graphifyy && graphify install (two y's; uv tool install graphifyy works too); graphify install --project --platform codex sets it up for Codex.

Gotcha: it is a busy project, with around 1,500 open issues, and the README points to a paid graphify.com platform.

4. grill-me: a relentless interview for your plan

grill-me, part of Matt Pocock's skills collection (MIT), runs "a relentless interview to sharpen a plan or design". It hands off to a grilling skill that asks the hard questions in rounds, each with a recommended answer you can accept or override. It does not load itself: the skill sets disable-model-invocation: true, so it only runs when you call it.

Install: /plugin install mattpocock-skills (it is in Claude Code's official marketplace), or npx skills@latest add mattpocock/skills for Codex and other agents.

5. ui-ux-pro-max: design help that knows patterns

ui-ux-pro-max (MIT) adds design intelligence for building interfaces across platforms. Version 2.0 adds a design-system generator that proposes patterns, palettes and placement of calls to action for a project.

Install: /plugin marketplace add nextlevelbuilder/ui-ux-pro-max-skill, then /plugin install ui-ux-pro-max@ui-ux-pro-max-skill. For Codex, install its CLI with npm install -g ui-ux-pro-max-cli and run uipro init --ai codex in the project.

What I removed (23 of 28)

The two biggest names I removed were Superpowers and Get Shit Done, now continued as GSD Core. Both are good. Superpowers is a complete development methodology built from composable skills; GSD is a meta-prompting and spec-driven development system. Six months ago they made my agent plan, test and review in a way it would not on its own, and I used both.

What changed is the models. A lot of what these frameworks did is now built in. Next to that, a large framework felt bloated for my use. I also try to keep context tokens for the task, which means loading the fewest skills that still change the outcome, and a full methodology loaded into every session is the opposite of that.

Most of the rest fell into the same bucket: things the model now does by default, or things I could describe in two lines of project instructions.

Do these skills work in Codex?

Mostly. Skills follow the open Agent Skills standard: a folder with a SKILL.md file and optional scripts. Codex reads skills from .agents/skills in a repository and from ~/.agents/skills for your user, and you call one with $skill-name. Plugins are tool-specific, because they can bundle hooks and commands, so use each project's Codex install where it has one. All five above have a Codex route today.

Are skills safe to install?

A skill is instructions plus optional scripts that run with your permissions. Before you install one:

Read the SKILL.md. It is short, and it tells you exactly what the skill asks the model to do.
Check for hooks. Hooks run automatically on events, without a prompt. ponytail's Codex plugin asks you to trust two.
Check for telemetry and network calls. caveman's CLI is a good example of a tool that tells you plainly what it sends and how to stop it.
Prefer a clear licence and an active maintainer. Every skill here is MIT or Apache-2.0 and was actively maintained when I checked in October 2026.

If you are deciding whether a capability belongs in a skill at all, or in an MCP server, Claude Skills vs MCP servers covers the difference. If you want to write your own, start with how to create and install a skill.

Best Claude Code skills: common questions

What are the best Claude Code skills? After trying 28, the five I kept are ponytail for writing less code, caveman for terser output, graphify for a knowledge graph of the codebase, grill-me for stress-testing a plan, and ui-ux-pro-max for interface design. The best set is a small one, because every installed skill competes for context.

Do Claude Code skills work in Codex? Mostly, yes. Skills follow the open Agent Skills standard, a folder with a SKILL.md file, and Codex reads them from .agents/skills in a repository or in your home directory. Plugins that bundle hooks or commands are tool-specific, so check each skill's README for a Codex install.

How do I install a skill in Claude Code? Most published skills install as plugins: run /plugin marketplace add with the repository, then /plugin install. You can also copy a skill folder into .claude/skills in a project, or into ~/.claude/skills for every project. Writing your own takes one SKILL.md file with a name and a description.

Are Claude Code skills safe? A skill is instructions plus optional scripts that run with your permissions, so treat it like any code you install. Read the SKILL.md, look for hooks that run automatically, check whether any bundled tool sends telemetry, and prefer skills with a clear licence and an active maintainer.

What is the difference between Claude Code skills and MCP servers? A skill teaches Claude a procedure, such as how to review code or plan a feature. An MCP server gives Claude access to another system, such as GitHub or a database. Skills change what Claude knows; MCP servers change what it can reach, and many workflows use both.

Fewer skills, better results

The models will keep absorbing what today's skills add, so this list will shrink again. The habit that lasts is the question behind it: does this skill change the result enough to earn its place in every session?

Which skill would you keep if you could only have one?

Originally published on nulltensor.com.

How to Build an MCP Server: Python, Remote and Ready for Production

Rohan Sen Sharma — Wed, 14 Oct 2026 03:00:00 +0000

To build an MCP server, wrap the capabilities you already have in a few well-described functions, using the official SDK, then serve them over HTTP with a token check in front. The protocol work is done for you. What decides whether the server is any good is which tools you expose and how you describe them.

This guide builds a small remote server in Python: one tool, the HTTP transport, authentication, a Cloud Run deploy and the monthly cost, then the lessons that only show up once a real agent uses it. The example is generic, the patterns are the ones we use.

Start with the decision, not the code

An MCP server is worth building when several AI clients need the same capability, or a person's permissions must follow them into the agent. If one script calls your API on a schedule, just call the API, MCP vs API covers that decision.

A minimal Python MCP server, with the official SDK

Install the official Python SDK. Version 2 calls the server class MCPServer; on version 1 it is FastMCP, in mcp.server.fastmcp.

uv add "mcp[cli]" httpx uvicorn

Here is one tool over an existing orders API:

import os

import httpx
from mcp.server import MCPServer
from mcp.types import ToolAnnotations

API_BASE = os.environ.get("API_BASE", "https://api.example.com/v1")

mcp = MCPServer("orders")


@mcp.tool(annotations=ToolAnnotations(read_only_hint=True))
async def order_status(order_id: str) -> dict:
    """Return the status, items and expected delivery date of one order.

    Use this when someone asks where an order is or when it will arrive.
    """
    async with httpx.AsyncClient(base_url=API_BASE, timeout=10) as client:
        resp = await client.get(f"/orders/{order_id}")
        resp.raise_for_status()
        return resp.json()

The type hints become the input schema the model fills in. The docstring is what the model reads to decide when to call the tool, so write it for someone who has never seen your system. The read-only annotation tells the client the tool changes nothing.

Run it locally with mcp.run() and connect it from Claude Code over stdio before you think about hosting. If you want to know how to add an MCP server to your client, be it Claude Code, Claude Desktop or claude.ai, What is Claude MCP walks through the commands and config files.

Building a remote MCP server: streamable HTTP

A local server talks over standard input and output (stdio). A remote server answers HTTP on one path, so any client that can reach the URL can use it, including claude.ai in a browser. To put the same server on a port, name the transport in run():

if __name__ == "__main__":
    mcp.run(transport="streamable-http", port=3001)

That one line builds a Starlette app and serves it with uvicorn, and clients connect to http://127.0.0.1:3001/mcp. Transport options such as host, port and streamable_http_path go to run(), not to MCPServer(...).

run() starts one process on localhost. To deploy, build the ASGI app yourself and allowlist the hostname you serve:

from mcp.server.transport_security import TransportSecuritySettings

security = TransportSecuritySettings(
    allowed_hosts=["mcp.example.com", "mcp.example.com:*"],
    allowed_origins=["https://app.example.com"],
)
app = mcp.streamable_http_app(transport_security=security)

Then serve it with uvicorn:

uvicorn server:app --workers 4

allowed_hosts is the go-live gate. By default the app only accepts requests addressed to localhost, a guard against DNS rebinding, so a deployed server answers every request with 421 until you list its real hostname; list both the bare host and the :* form. allowed_origins only matters for browser clients.

If TLS ends at a proxy in front of uvicorn, add --proxy-headers and set --forwarded-allow-ips to the proxy's address, or to '*' when nothing but the proxy can reach uvicorn; otherwise the app's redirects point at http:// and clients refuse them. Clients on the current protocol revision (2026-07-28) send self-contained requests, so any worker can answer any of them. Current Claude Code speaks that revision. Older clients (2025-11-25 and earlier) keep a session in one worker's memory, so they need sticky sessions, or stateless_http=True on streamable_http_app(), which drops sessions and server-to-client features such as sampling.

Authentication: check a token before any tool runs

A remote server without auth is a public API to your data. The SDK's integration point is a token verifier:

import secrets

from pydantic import AnyHttpUrl
from mcp.server.auth.provider import AccessToken, TokenVerifier
from mcp.server.auth.settings import AuthSettings


class EnvTokenVerifier(TokenVerifier):
    async def verify_token(self, token: str) -> AccessToken | None:
        expected = os.environ.get("MCP_TOKEN", "")
        if expected and secrets.compare_digest(token, expected):
            return AccessToken(token=token, client_id="internal", scopes=["orders:read"])
        return None


mcp = MCPServer(
    "orders",
    token_verifier=EnvTokenVerifier(),
    auth=AuthSettings(
        issuer_url=AnyHttpUrl("https://auth.example.com"),
        resource_server_url=AnyHttpUrl("https://mcp.example.com/mcp"),
        required_scopes=["orders:read"],
    ),
)

With that in place, a request without a valid token gets a 401 whose WWW-Authenticate header points to the server's protected-resource metadata, which the SDK serves for you at /.well-known/oauth-protected-resource/mcp. Clients that support OAuth follow it to your identity provider.

A shared token is a sensible first step for an internal tool, not the last. The next step is verifying tokens your identity provider issues to each user, so that every tool call runs with that person's permissions. The specification also forbids passing the client's token straight through to your backend; the server should hold its own credentials for the API, scoped to what the tools need.

I checked this exact code with version 2.3.0 of the SDK: without a token the server answered 401 with the metadata link, a wrong token was refused, and with the right token a client listed the tool and called it.

Deploying to Cloud Run

Any platform that runs a Python web app works. Cloud Run suits a small team server because it scales to zero when idle. A Dockerfile:

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY server.py .
CMD exec uvicorn server:app --host 0.0.0.0 --port ${PORT:-8080} --proxy-headers --forwarded-allow-ips='*'

Store the token in Secret Manager, then deploy from the source directory:

gcloud run deploy orders-mcp --source . --region us-central1 \
  --allow-unauthenticated \
  --set-secrets MCP_TOKEN=orders-mcp-token:latest \
  --set-env-vars API_BASE=https://api.example.com/v1

Only Cloud Run's front end can reach the container, so the CMD trusts its forwarded headers. --allow-unauthenticated looks alarming, but it only switches off Cloud Run's own Google-identity check; your token verifier is the gate. Put the service URL that Cloud Run prints into allowed_hosts and resource_server_url, or map your own domain first and use that.

What it costs

For a low-traffic internal server, usually nothing. Cloud Run's free tier with request-based billing includes 2 million requests, 180,000 vCPU-seconds and 360,000 GiB-seconds a month, per billing account. A team tool serving a few thousand calls a day stays well inside it.

The surprise is cold starts: a server scaled to zero takes a few seconds to wake. Keeping one instance warm fixes that, but idle time is then billed. The model's tokens for tool definitions and results usually cost more than the server, one more reason to keep the tool list short.

PostgreSQL MCP servers: start read-only

The tempting design is one run_sql tool. Start instead with a few named, read-only tools over queries you wrote, a database role that can only read, and a row limit. The read-only role is the one guard that doesn't rely on the model behaving.

What we learnt building MCP servers

We put one in front of our own APIs; MCP vs API has the longer story:

Fewer, task-shaped tools. Our first version mapped endpoints to tools one to one and exposed around 150 functions. The agent was poor at choosing among them. A handful of tools shaped like the questions people actually ask worked far better.
Read before write. Start with read-only tools, and add writes one at a time, each narrowly scoped and confirmed by the user.
The user's permissions, not the server's. A tool call made for a person should see only what that person can see in your app. Design for this from the start; retrofitting it is painful.
Descriptions are the interface. The model only knows a tool through its name, schema and docstring. When an agent picks the wrong tool, improve the description before you add code.

How to build an MCP server: common questions

How do I build an MCP server? Install the official SDK, create an MCPServer, and turn each capability into a decorated Python function with type hints and a clear docstring. Run it over stdio for local use, or as a streamable HTTP app for remote use, then connect it from Claude Code with claude mcp add and call one tool to check it works.

How can I host my own MCP server? Any platform that runs a Python web app will do, because a remote MCP server is an ASGI app on one URL path. A container on Cloud Run, Fly or a small VM all work. Set the allowed hostnames, put a token check in front of the tools, and keep secrets in the platform's secret store.

How do I build a remote MCP server? Use the streamable HTTP transport instead of stdio. With the Python SDK, call streamable_http_app() and serve the result with uvicorn. The server then answers at one path, /mcp by default, so any MCP client that can reach the URL can connect. Add a token verifier before you expose it.

How much does an MCP server cost to run? For a low-traffic internal server, often nothing. Google Cloud Run's free tier includes 2 million requests and 180,000 vCPU-seconds a month with request-based billing, which covers most team tools. Cost appears when you keep instances warm to avoid cold starts, because idle time is then billed too.

Should a PostgreSQL MCP server run raw SQL? Not at first. Give the model a few named, read-only tools, such as find_customer or recent_orders, that run queries you wrote, and connect with a database role that can only read. Raw SQL is tempting for analysis, but it hands the model your whole schema and every mistake it can make.

Where to start

Build one read-only tool for the question your team asks most, run it locally for a week, and only then make it remote. Once it is connected, a Skill can teach Claude how your team uses it; Claude Skills vs MCP servers explains how the two fit together.

What is the first tool you would expose?

Originally published on nulltensor.com.

Claude MCP: What It Is and How to Add a Server

Rohan Sen Sharma — Sun, 11 Oct 2026 03:00:00 +0000

Claude MCP is Claude connected to other systems through the Model Context Protocol, Anthropic's open standard for giving AI assistants tools and data. You add an MCP server, the server exposes what one system can do, and Claude can then read from it and act in it. In Claude Code, adding one is a single claude mcp add command.

That is the short version. The rest of this post covers what an MCP server actually gives Claude, how to add one in each Claude app, where the configuration lives, and the few security habits worth having from the first day.

What a Claude MCP server gives Claude

An MCP server is a small program that speaks the protocol and offers three kinds of things:

Tools: actions Claude can call, such as "create an issue" or "run this query".
Resources: data Claude can read, such as a file, a table or a page.
Prompts: reusable templates the server suggests for common tasks.

Servers run in one of two ways. A local server is a process on your own machine that Claude starts and talks to over standard input and output (stdio). A remote server runs somewhere else and Claude reaches it over HTTP. Local is simpler for your own files and tools; remote suits services you sign in to, and it works from claude.ai in a browser, where nothing can run on your machine.

Anthropic released MCP as an open standard in November 2024, so the same server works with Claude Code, Claude Desktop, claude.ai and other MCP clients. Whether a system needs its own MCP server, or whether the API it already has is enough, is a separate decision; I worked through it in MCP vs API.

How to add an MCP server in Claude Code

The command is claude mcp add. For a local server, everything after the two dashes is the command Claude Code runs to start it:

claude mcp add filesystem -- npx -y @modelcontextprotocol/server-filesystem ~/projects

Pass secrets as environment variables, not as arguments, with --env:

claude mcp add --env AIRTABLE_API_KEY=your-key airtable -- npx -y airtable-mcp-server

For a remote server, give the transport and the URL:

claude mcp add --transport http notion https://mcp.notion.com/mcp

Remote servers usually sign you in with OAuth. Open a Claude Code session, type /mcp, pick the server and finish the login in the browser. A server that takes a fixed token instead accepts a header: --header "Authorization: Bearer your-token". The older SSE transport still works but is deprecated; use HTTP where a server offers it.

Scope: who gets the server

Every server is added at one of three scopes, set with --scope:

Scope	Flag	Stored in	Who sees it
Local (default)	`--scope local`	`~/.claude.json`, under this project	Only you, only in this project
Project	`--scope project`	`.mcp.json` in the project root	Everyone who clones the repository
User	`--scope user`	`~/.claude.json`, top level	Only you, in every project

Use project scope for servers the whole team needs, such as the issue tracker for that repository, and commit the file. Use user scope for personal tools you want everywhere. Leave one-off experiments at the default.

The .mcp.json file (Claude MCP JSON config)

Project-scoped servers live in .mcp.json, and you can write it by hand:

{
  "mcpServers": {
    "issues": {
      "type": "http",
      "url": "https://example.com/mcp"
    },
    "database": {
      "type": "stdio",
      "command": "/path/to/db-server",
      "args": ["--read-only"],
      "env": { "DB_URL": "${DB_URL}" }
    }
  }
}

${VAR} expands from your environment, so the committed file holds no secrets; ${VAR:-default} supplies a fallback. To add a server from a JSON snippet without editing files, use claude mcp add-json <name> '<json>'. If you already set servers up in Claude Desktop, claude mcp add-from-claude-desktop imports them on macOS and WSL.

Listing and removing servers

claude mcp list            # every configured server
claude mcp get issues      # details for one
claude mcp remove issues   # delete it (add --scope project for .mcp.json)

Inside a session, /mcp shows which servers connected and which failed, and lets you switch one off without deleting it. That is the quickest fix when a server is misbehaving and you want it out of the way for an afternoon.

Claude Desktop and claude.ai

Claude Desktop reads local servers from claude_desktop_config.json. Open it from the Claude menu: Settings, then Developer, then Edit Config. It lives at ~/Library/Application Support/Claude/claude_desktop_config.json on macOS and %APPDATA%\Claude\claude_desktop_config.json on Windows, and uses the same mcpServers shape as .mcp.json. Quit and reopen the app after editing it; Desktop only reads the file at start-up.

claude.ai in the browser cannot start local processes, so it uses remote servers only, which it calls connectors. Open Settings, choose Connectors, click Add and then "Add custom connector", paste the server URL and sign in. The connector's settings let you turn individual tools on or off, which is worth doing for anything that can write.

Security in three habits

An MCP server runs with real credentials and, for local servers, with your user account's permissions. Three habits cover most of the risk:

Least privilege. Give a server the narrowest scope that does the job: specific folders, a read-only database user, a token limited to one repository.
Trust before you connect. Anthropic's own docs warn that servers which fetch outside content can expose you to prompt injection. Install servers from their publisher, not from a random fork.
Allow-list the risky tools. Leave read tools on, and keep tools that delete, send or pay behind a confirmation, or switched off.

The design side of this, deciding which actions a server should expose at all, is most of what MCP vs API is about.

Claude MCP vs Skills

MCP gives Claude access to a system; a Skill teaches it a procedure. The two fit together: the server fetches the data, the Skill says what to do with it. For the full comparison, including when to reach for each, see Claude Skills vs MCP servers.

Claude MCP: common questions

What is Claude MCP? Claude MCP is Claude using the Model Context Protocol, Anthropic's open standard for connecting AI assistants to other systems. Claude connects to MCP servers, and each server exposes tools, resources or prompts from one system, such as GitHub or a database, so Claude can read live data and take actions there.

How do I add an MCP server to Claude Code? Run claude mcp add. For a local server, put the launch command after two dashes: claude mcp add my-server -- npx -y some-mcp-server. For a remote server, pass the transport and URL: claude mcp add --transport http my-server https://example.com/mcp. Then run /mcp inside Claude Code to check it connected.

Where is the Claude MCP config file? It depends on the app. Claude Code keeps local and user servers in ~/.claude.json and shared project servers in .mcp.json at the project root. Claude Desktop uses claude_desktop_config.json, under ~/Library/Application Support/Claude on macOS and %APPDATA%\Claude on Windows. claude.ai stores connectors in your account settings.

How do I remove an MCP server from Claude? In Claude Code, run claude mcp remove followed by the server name, adding --scope project if it lives in .mcp.json. To pause a server without deleting it, open /mcp and turn it off. In Claude Desktop, delete its entry from the config file and restart the app.

What is the difference between Claude MCP and Claude Skills? MCP gives Claude access to another system; a Skill teaches Claude a procedure. An MCP server fetches the customer record, and a Skill says how to turn it into your standard report. They solve different problems, and most real workflows end up using both together.

Where to start

Pick one system you already open every day, add its server at local scope, and give Claude one real task in it. If the server earns its place, move it to project scope so the rest of the team gets it too.

Which system would you connect first?

Originally published on nulltensor.com.

My Claude Code chief of staff: one VM, eight specialists and one Telegram chat

Rohan Sen Sharma — Sat, 10 Oct 2026 03:00:00 +0000

For the last week, most of my work outside meetings has gone through one conversation. I send a text or a voice note to a Claude Code session on Telegram, and it hands the work to one of eight other Claude sessions, each with its own job. By evening I have an experiment measured, an inbox turned into a task list and a research note, and I have not opened a terminal.

That sounds tidier than it has been. The setup broke in new ways almost every day, and most of what I know about it came from those breaks. This post walks through the agent architecture, what it costs and the failures that shaped it, at the level of detail where you could build your own.

The shape of the system

Everything runs on one Azure VM with a permanently logged-in desktop, because some work needs a real browser. On it run three kinds of process:

A chief of staff. One long-running Claude Code session I talk to. I named it Elaine, mostly so voice notes would sound like a person rather than a process. It never does the work itself. It routes each task to a specialist and relays the result.
Specialists. Background Claude Code sessions (claude --bg --name <role>), each with a written brief: content, SEO, video, inbox triage, a sysadmin for the VM itself, and engineers for the product repositories. They are separate processes rather than subagents, so they survive the chief of staff restarting.
Timer loops. systemd user timers that run headless claude -p jobs on a schedule, with no conversation at all.

Why the chief of staff never does the work

The first version let the chief of staff help with small tasks. Claude Code handles one turn at a time, so every minute it spent grepping a log was a minute my next request sat in a queue.

So the rule became strict. On any incoming task it makes at most one quick read to decide who should do it, sends the brief to that specialist, and ends its turn. If no specialist fits, it launches one. Verification is delegated too: a claim of "done" gets a request for evidence or a second specialist's check.

A side benefit I did not plan for: every specialist's context stays about its own domain. The content session knows the content experiment ledger in detail and nothing about database migrations, which keeps its turns short and its mistakes easy to trace.

Telegram as the only interface

I am rarely at the VM's terminal, so anything that shows up only there never reaches me. Claude Code's channels feature connects a session to a Telegram bot, and that bot became the only way the system talks to me.

Voice notes both ways. Incoming voice notes are transcribed with a speech-to-text deployment on Azure, and the chief of staff can reply as a native voice bubble, all for about a dollar a month. The one trap: Telegram sends .oga files, and the endpoint rejects that extension until you rename it to .ogg.
Tappable decisions. A decision arrives as a reply keyboard with the options as buttons in Telegram. Inline buttons looked nicer, but their taps never reached the session, because the plugin forwards messages, not callbacks.
One chat, long reads elsewhere. Every message names its project, and anything long, such as a draft or a plan, goes to a Notion page with a link in chat. I briefly tried one topic per project but for one person and one assistant it cost more attention than it saved.

The watchdog, and what it taught me about Telegram

Telegram allows exactly one poller per bot token, and the Claude Code plugin enforces that by killing any existing poller when a new one starts. That is reasonable for one session and dangerous for a machine running nine. In the first two days the Telegram bridge went down four ways:

Another session started the plugin, killed the chief of staff's poller and took over, so my messages arrived in a specialist's session.
A health check spawned a second copy. claude mcp list starts each MCP server to check it, which starts a second poller, which kills the live one and then exits. The result was no poller at all.
Deleting old sessions in the agent view killed the live poller within a minute.
Idle retirement stopped the session after about an hour; pinning it exempts it.

The watchdog is a systemd timer that runs a short shell script every two minutes: 30 to 40 milliseconds of CPU and about 5 MB of memory per healthy tick. It checks that the poller is alive, runs from the plugin's directory, and has the chief of staff session as its ancestor. That last check came from failure 1: for three ticks the poller was alive and healthy and belonged to the wrong session. "A poller exists" and "my session owns the poller" are different health signals, and only the second tells you whether messages will reach you.

After two failed checks it restarts the chief of staff, with a lock and cool-downs. The restart has its own trap: claude --bg --resume <session-id> wakes the session in place only with no other flags. Pass any flag, even the original ones, and you get a copy with a new id, no name and no Telegram channel.

Timer loops: move the deterministic work out of the model

Each loop runs in two stages. The first is plain bash with no model: check whether anything is due, gather the inputs, and exit if there is nothing to do. Only then does the runner start claude -p on a cheaper model, with the inputs already in the prompt.

The lesson we kept relearning: anything deterministic belongs in a script, not in the model's instructions.

A headless run cannot call sleep, so waiting lives in a helper.
A browser helper re-reads the page before any retry, so it never submits twice.
A skill file must never point at memory: a headless run from another directory loads none, and the reference silently does nothing.

Memory as markdown files

Every session reads a shared directory of markdown files, one fact per file with a name, a one-line description and a type, linked with [[name]] into a small graph. A one-line-per-memory index loads into every session at start.

Two rules keep it from rotting: specialists may correct facts in their own domain but must say so, and nobody deletes a memory without the chief of staff agreeing, because a stale-looking memory is often the only record of why a rule exists. The weak spot was the decision log, which grew to 188 KB in five days and was re-read at every session start. It is now a short live queue plus an archive.

What it costs

On one day early on, the Anthropic dashboard showed about $195, and I did not know why. The audit found that 88% of spend was the model re-reading its own context: 48% cache reads and 40% cache writes, with output under 12%. The chief of staff was re-reading around 440,000 tokens on every turn, because on a 1M-token window auto-compaction never triggered.

What helped was context and turns, not clever prompts. A standard window cut the chief of staff's cost per turn from about 28 cents to 13. It runs on Claude Opus 5.5, and I capped its context window by hand at 256,000 tokens, so auto-compaction kicks in early, keeping turns cheaper and the thread responsive. The specialists run on Claude Opus 5.5 too, the scheduled jobs on Claude Sonnet 5.5 at a few cents a turn, and nothing uses Fable. And one complete brief per task, because cost follows turns more closely than anything else we measured. It is still not cheap: daily spend ranges from about $30 on a quiet day to about $200 on a heavy build day.

The failures worth copying

Most bugs in this system failed quietly: clean logs, exit code 0, and nothing happened.

Stale checkouts. The timers run scripts from each repository's main checkout, while specialists work in separate git worktrees. One day five fixes were pushed and none of them ran for five hours, because nobody had fast-forwarded the main checkout. Every report now states that the main checkout matches the remote.
Rules that read correctly. Before one loop went live, simulating every timer fire of its first day against the real data found four bugs, including a lock the loop could never acquire. Each read fine on paper and would have exited 0.
Checks that confirmed instead of tested. Grepping a log for the lines you expect confirms your expectation; it does not test the code.
A test that was not a test. A dry run once sent me a real notification because only the model call was stubbed. Runners now have a switch that silences every side channel.

The habit behind all of these: we do not trust a rule until it has run against the real state, and we do not trust a report until the result is on disk.

Why I built my own

There are good off-the-shelf options: xAI's Grok Bot, Meta's Muse, OpenAI's Codex, Anthropic's Claude Cowork, and open-source alternatives to each. Every one of them held me back in some way.

I wanted something wired completely around how I work. A custom build lets me take the best of each tool and change the setup whenever I need to without switching providers.

This is not the final build, and I am excited about what I will build into my personal assistant next.

If you built your own, what is the first thing you would wire it into?

Originally published on nulltensor.com.

Claude Code Review in Production: How Our Review Rules Evolved Over Five Months

Rohan Sen Sharma — Fri, 09 Oct 2026 03:00:00 +0000

Claude Code review started at my company as a single prompt in a GitHub workflow, and five months later it is a playbook, six specialist agents and a verifier that throws out any finding it cannot prove. Almost every step between those two points was a reaction to something that went wrong: false positives while the rules were still settling, findings on code the pull request never touched, severity that depended on the model's mood, and a reviewer that invented problems when we gave it too many rules at once.

This is how the rules evolved and what each change fixed.

Claude Code's built-in review, and why we wrote our own rules

Claude Code now has a review of its own. Anthropic's managed Code Review, in research preview for Team and Enterprise plans, runs through the Claude GitHub App: several agents look for different classes of issue in parallel, a verification step filters out false positives, and findings arrive as inline comments tagged Important, Nit or Pre-existing. It never approves or blocks a pull request, and you tune it with a REVIEW.md file. Locally, /code-review does the same for your branch.

We kept our own because we already had a customised flow. The built-in review hunts for correctness bugs by default; ours enforces a written architecture, cites a rule for every finding, and votes on the pull request. If you want a second pair of eyes for bugs, start there; if you want documented rules enforced with a verdict, you need a playbook of your own.

Version 1: one prompt, run on request (April)

The first version went in on 20 April. A workflow listened for a pull request comment containing "@claude" and "review", and ran Claude Code with one prompt that did three things.

For files in our layered modules (routers, services, repositories and DTOs), it read the architecture rule documents for each layer and treated any violation as blocking.
Everything else got a general review, explicitly without the layering rules.
It grouped findings by file, each with a severity (BLOCKING, WARNING or SUGGESTION), a location, the problem and a concrete fix.

That structure survived every later version; the trigger did not. While the rules were settling it produced false positives, so it ran only when someone asked.

Version 2: every pull request, with a verdict (May)

Between 12 and 15 May the workflow changed four times.

It ran on every pull request, on open, on each new push and on reopen.
It submitted a real GitHub review: request changes if there was any blocking finding, approve otherwise.
It skipped draft pull requests until they were marked ready for review.
It skipped long-lived project branches, where every intermediate push would have drawn a review.

The lesson of that week: once a reviewer votes, noise stops being a nuisance and becomes a cost. A wrong "request changes" blocks a colleague. Everything that followed was about precision.

Version 3: load only the rules that matter, and review only what changed

The next rewrite introduced three ideas that are still the backbone of the system.

Load rules by path, not all at once. The reviewer lists the changed files, then reads only the rule documents those paths need: router rules only if a router changed, and so on. Loading every rule at once caused hallucinations: the reviewer reported violations that were not there.

Review only changed lines. The prompt gained a sentence we have never removed: review only changed or added lines, never flag pre-existing code. Without it, a one-line fix in an old file could come back with twenty findings about code written years earlier.

Severity by definition, not judgement. The severity section was headed "apply exactly, no judgment calls". BLOCKING meant a layering-rule violation, a correctness bug, a security issue or a data-loss risk. WARNING meant correct architecture that was still risky. SUGGESTION meant style or naming that broke no rule.

Version 4: one playbook for local and CI reviews

On 18 May the rules moved out of the workflow and out of CLAUDE.md into a single docs/review.md. Both the GitHub Action and a local /review command follow the same file, so a local review before pushing applies the same rules the pull request will face.

Ask for the base branch. Locally, the review asks which branch the work will merge into instead of assuming main.
Skip data files. SQL, CSV and JSON files are excluded; reviewing migrations and fixtures as code produced findings nobody could act on.
Every finding cites a rule. BLOCKING must cite an architecture rule file and section, WARNING a style rule from CLAUDE.md. SUGGESTION is for anything no rule covers.
Never block legacy code on style. Style caps at WARNING everywhere.

Tying severity to a citation changed the character of the reviews. A finding either points at a rule the team agreed, or it is a suggestion the author can ignore.

Version 5: specialists and a verifier

On 19 May the single reviewer became an orchestrator that reviews nothing itself. It routes each changed file to the specialists that own it, runs them in parallel and assembles the result.

Specialist	Reviews	Owns
Router reviewer	API and router files in the layered modules	Router-layer rules
Service reviewer	Service files in the layered modules	Service-layer rules
Repository reviewer	Repository files in the layered modules	Repository-layer rules
Architecture reviewer	Any file in the layered modules	Cross-cutting architecture rules
Python style reviewer	Every Python file	Style rules from `CLAUDE.md`, capped at WARNING
Verifier	Every finding	Nothing: it only checks the others' work

Each specialist reads only its own rule document, so a service file reviewed by four specialists does not get the same finding four times. They all return findings in one fixed format, with the offending code quoted verbatim.

The verifier is the Claude Code review agent I would copy first. For every finding it checks four things, and rejects the finding if any fails:

The cited file exists.
The quoted code appears within five lines of the cited line.
That line was added or changed in this diff.
The cited rule section exists and says what the finding claims.

Its instructions open with the principle behind the whole design: false positives are far worse than false negatives. It may not re-grade severity or add findings of its own, and rejected findings are dropped silently.

Adding a specialist without touching the workflow (September)

In September we added a sixth reviewer for database performance, because our most expensive production problems are queries that are fast in development and slow at scale. It blocks queries inside loops, N+1 access to related fields and per-row writes where a batch write exists, and warns on unbounded fetches and aggregation done in Python instead of the database. Where a pattern is deliberate, a # db-perf: allow comment with the reason suppresses the finding.

Two decisions stand out. Database performance findings can block legacy code too, the one deliberate exception to "never block legacy", because a query in a loop costs the same wherever it lives. And the workflow file did not change at all: adding a specialist meant one rule document, one agent definition and one row in the routing table.

What we would tell another team

Start with the output format and the severity definitions. They survived every version; the trigger and the architecture did not.
Make every finding cite a rule. It turns arguments about taste into a conversation about whether the rule is right.
Never flag pre-existing code. Nothing destroys trust in an automated reviewer faster.
Verify before you post. A second pass that checks the quote, the line, the diff and the rule is cheap compared with a wrong "request changes".

This connects to six feedback loops for AI coding agents: the review stage only improves the next change if its findings are precise enough to act on, and it guards against the problem in why generated tests miss bugs.

Claude Code review: common questions

Can Claude Code PR review run automatically on every pull request? Yes. It runs in GitHub Actions through Anthropic's official action, triggered on pull request events or by an "@claude" comment.

Claude Code review skill, command or agent: which should I use? They fit together: a slash command as the local entry point, sub-agents so each reviewer holds one set of rules, and a skill for the reusable procedure. To package ours as a skill, put the steps in a SKILL.md with the rule documents and finding format beside it, and keep the specialists and verifier as sub-agents. How to create Claude Skills walks through the structure.

Does it replace human review? No. It catches rule violations reliably when the rules are written down and cited, which is most of what a human reviewer spends time on, but it is weaker at judging whether a change is the right change. A human approval is still required on every pull request. Review effort is also the cost headline numbers hide: a 90% agent PR merge rate can still describe a weak workflow.

Where this leaves us

The system we run today is not clever. It is a playbook, a routing table, a handful of reviewers that each know one set of rules, and a verifier that refuses anything it cannot prove. Every piece exists because a simpler version failed in a specific way, and if you are starting now you can skip most of those failures.

The playbook itself, with our company-specific rules removed, is here: a Claude Code review template you can copy.

Originally published on nulltensor.com.

MCP vs API: When to Build an MCP Server, and When the API Is Enough

Rohan Sen Sharma — Thu, 08 Oct 2026 03:00:00 +0000

MCP vs API is rarely an either-or choice: MCP is a protocol that lets an AI agent discover and call tools at runtime, and an API is usually the service those tools wrap, so most teams building agents end up with both. The real decision is narrower. When does an agent need its own interface to your system, and when is the API you already have enough?

We ran into this at my company when we wanted Claude and our internal agents to work with the same data our web and mobile apps use every day. We already had a REST API. We built an MCP server anyway. Ours runs on Google Cloud, and the reasons for building it were not the ones I expected. Let us walk through what MCP is, what it adds, what it costs, and a simple way to decide.

The short answer: MCP vs API

An API, usually a REST API, is a contract for software a developer writes. The developer reads the documentation, picks the endpoints and ships code that calls them in a known order. An MCP server is a contract for a model. The agent asks the server what it can do, reads the descriptions, and decides at runtime which tool to call and with what arguments.

	API (usually REST)	MCP server
Who decides what to call	A developer, at build time	The model, at runtime
How capabilities are found	Documentation or an OpenAPI file a person reads	`tools/list`: names, descriptions and input schemas returned by the server
Transport	HTTP	JSON-RPC over stdio (local) or Streamable HTTP (remote)
Auth	API keys, OAuth or service tokens	Optional; OAuth 2.1 for HTTP servers, environment credentials for local ones
State	Stateless per request	Stateless per request in the current spec; each request carries its own version and capabilities
Versioning	URL or header versions you define	The protocol is versioned by date; your tool names, schemas and descriptions are your contract
Context cost	None until your code puts data in a prompt	Tool definitions occupy the model's context window
Latency	One network call	Your API call, plus a hop to the MCP server, plus the model's decision
Clients	Your own code	Any MCP host: Claude, Claude Code, Cursor, VS Code and others
Best for	Known workflows, high volume, integrations you control	Open-ended requests where you cannot predict the sequence of calls

So the MCP server vs API question is really a question about who is calling. If the sequence of calls is known in advance, write code against the API. If people will ask an agent for things you cannot list in advance, and the agent has to reach your system to answer, that is when an MCP server starts to earn its keep.

What MCP actually is, and what it isn't

The Model Context Protocol is an open protocol, currently specified in its 2026-07-28 revision, for connecting AI applications to tools and data. It uses JSON-RPC 2.0 messages between three roles: a host (the AI application, such as Claude Code), a client inside that host, and a server that offers capabilities.

A server can offer three things: tools the model can call, resources that provide context, and prompts, which are templates a user can invoke. Most servers start with tools, and so did we.

There are two standard transports. stdio runs the server as a local subprocess of the host, which suits developer tools on a laptop. Streamable HTTP sends each message as an HTTP POST to a single endpoint, with the reply as a JSON object or a stream scoped to that request, which suits a shared, remote server.

The current revision made the protocol stateless. Earlier revisions opened a connection-scoped session with an initialize handshake, and many articles still describe that model, along with the older HTTP-plus-SSE transport. Today every request carries its protocol version and client capabilities, so a remote MCP server can sit behind an ordinary load balancer like any other HTTP service.

What MCP is not: a replacement for your backend, or a source of new capability. Every tool does its work by calling something underneath, usually the same API your apps use. If the API cannot do it, the MCP server cannot either.

Can an API be an MCP server? Wrapping one in about 30 lines

Yes. The most common MCP server is a thin, carefully described layer over an existing API, and wrapping one endpoint is the quickest way to learn how to create an MCP server. Here is the job of finding a customer's overdue invoices, first as a direct API call. The endpoint is illustrative, not ours.

import os

import httpx

resp = httpx.get(
    "https://api.example.com/v1/invoices",
    params={"customer_id": "C-1042", "status": "unpaid", "overdue_days_gte": 30},
    headers={"Authorization": f"Bearer {os.environ['API_TOKEN']}"},
    timeout=10,
)
resp.raise_for_status()
for invoice in resp.json()["items"]:
    print(invoice["number"], invoice["amount_due"])

Now the same capability as an MCP tool, using the official Python SDK. It calls the same endpoint. What changes is that a model can find it, read when to use it, and supply the arguments itself.

import os

import httpx
from mcp.server.mcpserver import MCPServer
from mcp.types import ToolAnnotations

API_BASE = os.environ.get("API_BASE", "https://api.example.com/v1")

mcp = MCPServer("invoices")


@mcp.tool(annotations=ToolAnnotations(read_only_hint=True))
async def get_overdue_invoices(customer_id: str, days_overdue: int = 30) -> list[dict]:
    """List a customer's unpaid invoices that are more than `days_overdue` days past due.

    Use this when asked who owes money or which invoices to chase.
    Returns an empty list when nothing is overdue; that is an answer, not an error.
    """
    async with httpx.AsyncClient(
        base_url=API_BASE,
        headers={"Authorization": f"Bearer {os.environ['API_TOKEN']}"},
        timeout=10,
    ) as client:
        resp = await client.get(
            "/invoices",
            params={"customer_id": customer_id, "status": "unpaid", "overdue_days_gte": days_overdue},
        )
        resp.raise_for_status()
        return resp.json()["items"]


if __name__ == "__main__":
    mcp.run(transport="streamable-http")  # serves http://127.0.0.1:8000/mcp

I ran this with version 2.2.0 of the SDK against a small local stand-in for the invoice API. A client connected over Streamable HTTP, listed the tool with its description, input schema and read-only hint, and the call returned the stand-in's invoice. If you are on version 1 of the SDK, the same class is called FastMCP and lives in mcp.server.fastmcp; version 2 renamed it.

To use it from Claude Code, register the endpoint (the syntax is in Claude Code's MCP documentation):

claude mcp add --transport http invoices http://127.0.0.1:8000/mcp

Three details are worth copying. The docstring is the description the model reads, so it says when to use the tool and what an empty result means. The read-only annotation tells hosts the tool changes nothing, although a host should only trust that from a server it trusts. And the tool is shaped around a question someone asks, "which invoices are overdue", rather than around an endpoint with a dozen filters.

When to use MCP vs API

Work down the list and stop at the first yes.

Is the sequence of calls known in advance? Call the API from code. A nightly job that pulls orders, asks a model to classify them and writes the labels back is a script; the model sits inside your workflow and has nothing to discover.
Is there exactly one agent, in one framework, using a handful of tools? Define the tools in that framework's function calling. Move them into an MCP server when a second consumer appears.
Do you need high volume or tight latency? Keep the model out of the request path and call the API directly.
Will people ask open-ended questions that need live data from your system, from more than one AI host? Build an MCP server on top of your API.
Does the agent need to change data? Build the server with read tools first, then add narrowly scoped write tools that ask for confirmation.
Does the agent need a method rather than access, such as how your team writes a release note? That is not an MCP problem. It is a skill, and Claude Skills vs MCP covers where that line sits.

For us, question four decided it. Our one MCP server serves an engineer asking Claude Code about a customer's configuration and a colleague asking Claude a business question. Building a separate wrapper for each host would have meant maintaining the same integration several times.

What we changed after putting an MCP server in front of our APIs

The first version of our MCP server mapped endpoints to tools one to one, which exposed around 150 functions to the model. It worked, and the agent was poor at using it. With that many fine-grained tools, it chained the wrong ones together and spent context on definitions it never needed.

What worked better:

Fewer, task-shaped tools. One tool that answers "what is the status of this order" can call three endpoints underneath. The model makes one decision instead of three.
Read before write. We started with read-only tools and added writes one at a time, each narrowly scoped and confirmed by the user in the host. The server has write tools today, and each has a well-defined input and output schema, so what goes in and what comes back is fixed rather than left to the model.
The user's permissions, not the server's. A call made for a person should see only what that person could see in the app. The specification forbids an MCP server from simply passing the client's token through to your API, so the server has to work out what the user is allowed to do itself. Ours calls an IAM endpoint that returns which functions the signed-in user has permission to use, and works within that list.
Descriptions under review. A change to a tool description goes through review like an API change, because it changes behaviour.

MCP vs function calling vs CLI

These three get compared constantly, and they sit at different layers.

Function calling is the model-side mechanism. You send tool definitions in an API request, the model replies asking to call one, your code runs it and returns the result. Every MCP host uses function calling underneath. MCP standardises where the definitions come from, so they arrive from a server at runtime instead of being written into each application.

A CLI is the other practical option for coding agents. Claude Code can run shell commands, so a well-built command-line tool over your API, like gh for GitHub, is often enough for developer workflows. It needs no new server, and its help text doubles as documentation. It also needs a shell, which chat applications do not give the model, and every user needs the tool installed and authenticated on their own machine.

MCP is the right layer when the same capability has to work across several hosts, including ones without a shell, with per-user authorization and descriptions written for a model. If you are choosing between MCP and skills rather than these three, the Skills vs MCP comparison covers that decision.

Cost and latency: what MCP adds

MCP adds two costs, and neither is large for a well-designed server.

Context. Tool definitions sit in the model's context window. Every tool's name, description and schema is text the model reads before it does anything useful. A server with a handful of precise tools costs little. A server that exposes every endpoint as a tool can take a meaningful share of the window before the user has asked a question. Hosts are starting to mitigate this; Claude Code, for example, can search for tools and load definitions on demand instead of all at once. Keeping the tool list short remains the most reliable fix.

Latency. Is MCP faster than an API? No. A tool call is your API call plus a hop to the MCP server, and in an agent it also waits for the model to decide which tool to call. That is the right trade for a person asking a question in plain language and the wrong one for a service handling thousands of requests a minute. I have not published timings here, because they depend far more on the model and the API behind the tool than on the protocol.

MCP vs API security

An API exposes operations to code you control. An MCP server exposes them to a model that decides what to call, based partly on text it did not write. That changes the threat model.

Prompt injection through tool results. A record, email or ticket can contain instructions, and the model may follow them. Treat every tool result as untrusted input, and never let tool output alone authorise a write.
Tool poisoning and over-broad descriptions. A description tells the model when to use a tool, so a malicious or careless one can steer it. The specification tells hosts to treat descriptions and annotations as untrusted unless they come from a trusted server. Connect only servers you trust, and write your own descriptions narrowly.
Least privilege. Prefer separate read and write tools over one tool with a mode flag. On HTTP servers, the specification's OAuth flow lets a server ask for the smallest scope an operation needs and request more only when a call requires it.
Tokens bound to the server. Tokens must be issued for the specific MCP server, and the server must not pass them through to other services. This closes the "confused deputy" route where a server acts with more authority than the user has.
Consent and audit. The specification requires hosts to get user consent before invoking a tool. Keep a human confirmation step for anything irreversible or customer-facing, and log every call with the user, arguments and result.

Maintenance belongs here too. An MCP server is another service to deploy, monitor and version, and changing a description can change agent behaviour as much as changing code. Test it the same way: keep a small set of realistic requests and check the agent still picks the right tools. It is the same idea as loop engineering, applied to your tool layer.

Will MCP replace APIs?

No. An MCP server normally calls your existing APIs, and your apps, partners and scripts keep using those APIs directly. MCP adds a second front door, designed for models, on top of the same backend. The APIs that matter more in an agent world are the ones with clean, well-scoped operations, because those are the easiest to turn into good tools.

MCP vs API: common questions

Is MCP just an API? In a sense: it is a protocol, built on JSON-RPC, that a client uses to call a server. The difference is the audience. MCP standardises how a model discovers and calls tools, so one server works across many hosts without custom wrappers.

When should I use MCP instead of an API? When the caller is a model rather than your own code, the requests are open-ended, and more than one AI host needs the same capability. For fixed workflows and high-volume integrations, call the API.

Is MCP faster than an API? No. A tool call is still your API call, plus a hop to the MCP server and, in an agent, the model's decision about which tool to use. For a fixed call made from code, the API is always faster. MCP's value is discovery and reuse across hosts, not speed.

Do I need an MCP server to use Claude with my API? No. You can pass tool definitions to Claude directly through function calling, which is often the right place to start. An MCP server becomes worthwhile when several hosts or teams need the same tools.

How do I turn an API into an MCP server? Pick the two or three questions people will actually ask, write one tool for each that calls your API underneath, describe each tool as precisely as you would brief a new colleague, and run it with the official SDK as in the example above. Start read-only. If you are new to extending Claude, how to create Claude Skills covers the lighter-weight option first.

Is MCP secure? It can be, but the security is in your design rather than the protocol: narrow tools, least-privilege scopes, tokens bound to the server, untrusted tool output and a human in the loop for writes.

Where this leaves us

MCP vs API is a question about who is on the other end. When your own code is, the API is enough and simpler. When a model is, deciding at runtime what to do for someone who asked in plain language, an MCP server gives it a way to find your tools, understand them and use them within that person's permissions. For us, the server was not a replacement for the API but a second, carefully designed front door on top of it. The work that mattered most was deciding which few tools deserved to exist, and writing their descriptions as carefully as the code.

Originally published on nulltensor.com.

How I Halved My Python Backend’s Memory Usage

Rohan Sen Sharma — Wed, 07 Oct 2026 03:00:00 +0000

We’ve all been there. Your monitoring dashboard turns red. Memory is spiking. The immediate temptation is to double the instance size—bump that t3.medium to a t3.large and move on. But today, I want to show you why that is the lazy way out.

We have a standard startup stack: FastAPI for the backend and Celery for background tasks, all running on a cost-effective EC2 instance with 4GB of RAM. The vital metrics for the server looked bizarre: Memory was screaming at 75% utilization, yet the CPU was barely waking up, sitting idle at 2-3%.

This specific signature—High RAM, Low CPU—is the hallmark of an architecture that is over-provisioned for concurrency but under-optimized for memory. Let me show you the three specific knobs I turned to cut our memory footprint in half, without spending a single extra cent.

Lesson 1: The Math of Processes vs. Threads

First, let’s look at the “Before” snapshot of the system. We were running:

API: 4 Gunicorn workers
Celery: 8 workers across 3 queues (2 normal queues and a priority queue)
Total: 12 Python processes.

Now, here is the expensive part. Each Python worker process consumes about 250MB of memory on average.

On a 4GB instance, that leaves almost no room for the OS. We needed to consolidate.

The Solution: Threads over Processes

In Python, a Process has its own memory space (it loads the whole interpreter, imports, and app code). A Thread shares memory with other threads in the same process.

I switched our FastAPI configuration from workers=2 (processes) to workers=1 with threads=4.

Why does this work?

Since our app is I/O bound (waiting on Database queries, BigQuery, and OpenAI calls), we don’t need multiple CPU cores fighting for the GIL (Global Interpreter Lock). Threads can handle concurrent requests efficiently because they release the lock while waiting for I/O.

Old Config (Processes): ~500MB usage for equivalent concurrency.
New Config (Threads): ~250MB usage. We just saved 50% memory on the API layer.

Lesson 2: The “Leaking Bucket” Theory (Handling Memory Leaks)

Even if your code is perfect, Python processes (especially those using libraries like Pandas or Numpy) tend to hold onto memory over time. It’s called memory fragmentation, or sometimes, just a plain old leak.

If you process a large Dataframe in a Celery task, that memory might not be fully released back to the OS immediately.

The Fix : tuning the --max-tasks-per-child parameter

This setting tells the Celery worker: “After you finish X number of tasks, kill yourself and let a fresh process take your place.”

We experimented a bit with this parameter and finally settled on a value of 100.

Here is a Trade-off Matrix for the different values:

Too Low (e.g., 10): Great for memory, but you waste CPU restarting workers constantly (~15% overhead).
Too High (e.g., 1000): Low overhead, but memory leaks accumulate.
The Sweet Spot (100): With our autoscaling enabled, workers naturally shut down when idle. This setting is just a safety net to ensure no single worker stays alive long enough to become bloated.

Lesson 3: The “Thundering Herd” (Gunicorn Jitter)

Just like Celery, we want our Gunicorn (API) workers to restart occasionally to clear out memory “gunk.” We set --max-requests 1000 to restart the worker after 1000 HTTP requests.

But there is a danger here.

Imagine you have 2 workers. If they both start at the same time, and both hit their 1000th request at the same time, they will both restart at the same time.

Result: Your API goes down for 2 seconds.

The Fix: tuning the --max-requests-jitter parameter

We added --max-requests-jitter 50.

This adds randomness. One worker might restart at 980 requests, the other at 1040. This staggering ensures that while one worker is recycling, the other is still serving traffic.

The Final Report Card

By understanding the difference between I/O bound and CPU bound tasks, and acknowledging that Python processes are memory-heavy, we drastically changed our footprint.

Before:

12 Total Workers
~3GB Memory Usage (75% Load)

After:

6 Total Workers (API threads + reduced Celery concurrency)
~1.5GB Memory Usage (40% Load)

So the next time your server metrics are turning red, before vertically scaling your servers try tuning your app and worker configurations. Go check your supervisor.conf or Dockerfile and answer these questions -

Are you running multiple Gunicorn workers for an I/O bound app? Try threads.
Do your Celery workers live forever? Add max-tasks-per-child.
Are you monitoring RSS memory? watch -n 5 ‘ps aux | grep celery’ is your friend.

STOP THROWING MONEY at your cloud provider.

Originally published on nulltensor.com.

The Day My "Index Optimization" Melted Our Database 🔥

Rohan Sen Sharma — Tue, 06 Oct 2026 03:00:00 +0000

It was a regular Monday morning. I was sipping my chai ☕, scanning through our monitoring dashboards, when I noticed something that had been bugging me for weeks - a SQL condition that was obviously wrong.

Many queries in our multi-tenant SaaS application had this filter:

WHERE (tenant_id = 42 OR tenant_id IS NULL)
  AND org_id = 100
  AND is_active = 1
  AND is_deleted = 0

That “OR tenant_id IS NULL” was sitting in every single query for a lot of tables. And I knew from years of database experience that OR conditions are index killers. Every DBA will tell you:

“An OR with IS NULL prevents MySQL from using indexes efficiently.”

So I did what any reasonable engineer would do. I removed it.

I pushed the fix. Deployed it. And then watched our RDS CPU climb from a comfortable 60% to a screaming 100% 📈💀.

I had just optimized our database into a meltdown 🔥.

🏗️ The Setup: A Multi-Tenant Nightmare

Let me give you some context. We run a B2B SaaS platform - think of it like a simplified ERP system. Every customer (tenant) has their own data, but it all lives in the same database. The data model looks something like this:

CREATE TABLE orders (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    tenant_id INT,
    org_id INT,
    order_date DATE,
    amount DECIMAL(10,2),
    is_active TINYINT(1) DEFAULT 1,
    is_deleted TINYINT(1) DEFAULT 0,
    -- ... 30 more columns
    INDEX (tenant_id),
    INDEX (org_id),
    INDEX (is_active)
);

The orders table had ~200,000 rows. Not huge, but not trivial either. And here’s the kicker, for any given tenant, a query would typically return about 50,000 rows out of those 200,000. That’s a 25% selectivity ratio. Remember this number. It’s going to matter a lot.

Now, some tables in our system had “shared” or “global” records, things like default payment terms, standard tax rates, common bank entries. These records had tenant_id = NULL because they belonged to everyone. But somewhere along the way, a developer had added the OR tenant_id IS NULL condition to every model’s query, not just the ones with shared records.

It was sloppy. It was clearly wrong. And it was my job to clean it up.

💥 The “Fix” That Broke Everything

The change was surgical. I went through the codebase and updated the query builder:

Before (every model):

WHERE (tenant_id = 42 OR tenant_id IS NULL)
  AND org_id = 100
  AND is_active = 1
  AND is_deleted = 0

After (transactional models only):

WHERE tenant_id = 42
  AND org_id = 100
  AND is_active = 1
  AND is_deleted = 0

Clean. Precise. The kind of change that should make indexes happy.

I deployed it on a Tuesday morning. By Tuesday afternoon, our Slack was on fire 🔥.

“The dashboard is crawling.”

“API timeouts across the board.”

“RDS CPU is pegged at 100%.”

I checked New Relic. Query times were actually slightly faster . 1-3 seconds vs the previous 2-5 seconds. But CPU had gone through the roof. How could faster queries use more CPU?

I reverted the change. CPU dropped back to ~60%. Queries went back to being slow but stable.

I sat there staring at my screen, feeling like I’d just witnessed a violation of the laws of physics 🤯.

🕳️ Down the Rabbit Hole: InnoDB’s Dirty Secret

Here’s where things get interesting. To understand what happened, you need to understand something most developers never think about: how InnoDB actually stores your data.

InnoDB doesn’t just dump rows into a file. It maintains two separate data structures for every table:

Structure 1: The Clustered Index (a.k.a. “The Actual Table”)

In InnoDB, the primary key isn’t just an index - it is the table. Rows are physically stored sorted by the primary key. When you do a full table scan, you’re reading this structure sequentially, cover to cover, like reading a book.

Structure 2: Secondary Indexes (Your Regular Indexes)

Every other index like INDEX(tenant_id), INDEX(org_id) is a secondary index. And here’s the crucial part: secondary indexes don’t contain the full row data. They only contain:

The indexed column(s)
A pointer back to the primary key

That’s it. No org_id, is_active or amount. Just the indexed column and a PK reference.

This design choice has a profound consequence.

💸 The Two-Lookup Tax

When MySQL uses a secondary index to find rows, it has to do two lookups for every single match:

Look up the secondary index → finds matching PKs
Look up the clustered index → retrieves the actual row data using each PK

This second step is called a “bookmark lookup” or “clustered index lookup.” And it’s not a sequential read- it’s a random seek. The primary keys from the secondary index are scattered across the clustered index. So for each match, InnoDB has to jump to a completely different location on disk.

Let me paint this picture clearly.

What Happened BEFORE My Change (Full Table Scan)

With “OR tenant_id IS NULL” in the query, MySQL’s optimizer looked at the query and thought: “This OR condition makes the index useless. I’ll just scan the whole table.”

Full Table Scan:\
Read row 1 → check conditions → keep/discard\
Read row 2 → check conditions → keep/discard\
Read row 3 → check conditions → keep/discard\
... (200,000 rows, read sequentially)

I/O Pattern: Sequential(Like reading a book from start to finish)\
CPU cost: LOW (just comparing values)\
Disk cost: HIGH but efficient (sequential reads)\
Wall time: 2-5 seconds

Sequential I/O is remarkably efficient. The disk head moves in one direction, reading contiguous blocks. The CPU barely breaks a sweat it’s just checking if (tenant_id == 42) for each row. Most of the time, the CPU is waiting for the disk to deliver the next chunk.

This is an I/O-bound operation. The bottleneck is disk speed, not CPU.

What Happened AFTER My Change (Index Scan)

Without the OR condition, MySQL suddenly had a usable index on tenant_id. So the optimizer switched strategies: “Great, I can use the tenant_id index! Let me find all rows where tenant_id = 42.”

Index Scan + Clustered Index Lookups:\
Step 1: Scan tenant_id index → find 50,000 matching PKs\
Step 2: For EACH of those 50,000 PKs:\
→ Random jump to clustered index\
→ Read full row\
→ Check: org_id = 100? is_active = 1? is_deleted = 0?\
→ Keep or discard

I/O Pattern: Random (Like flipping to 50,000 random pages in a book)\
CPU cost: HIGH (coordinating 50,000 random seeks)\
Disk cost: 50,000 individual random reads\
Wall time: 1-3 seconds

Each of those 50,000 random seeks requires the CPU to:

Navigate the B-tree index structure
Calculate the physical disk location
Issue the I/O request
Handle the I/O completion interrupt
Manage the InnoDB buffer pool

This is a CPU-bound operation. The CPU is constantly busy coordinating random I/O.

🧮 The Counter-Intuitive Math

Here’s where it clicks. Let me lay out the numbers:

The index scan was faster in wall-clock time therefore the query returned results quicker. But it consumed 5x more CPU doing it. Under concurrent load (dozens of these queries running simultaneously), the CPU saturated, everything started queuing, and the whole system ground to a halt.

🎯 The Selectivity Problem

This leads to one of the most important rules in database performance that many engineers miss:

Selectivity = (Rows returned) / (Total rows in table)

< 5% selectivity: Index scans are clearly better. Few lookups = low overhead.
5-15% selectivity: Grey zone. Depends on hardware, data distribution, index type.
> 15% selectivity: Full table scans often beat single-column index scans.

Our queries had 25% selectivity. We were firmly in “table scan wins” territory for single-column indexes. MySQL’s optimizer actually knew this and that’s why it chose table scans when the OR condition was present. We accidentally forced it into an inferior plan by making the index “usable.”

✅ The Real Fix: Composite Indexes

So if single-column indexes are the problem, what’s the solution? You don’t go back to table scans, instead you use composite indexes.

A composite index includes multiple columns in a single index structure:

CREATE INDEX idx_orders_tenant_org_active_deleted
ON orders(tenant_id, org_id, is_active, is_deleted);

Here’s what this changes. Instead of a secondary index that only knows about tenant_id:

Single-column index:\
Find tenant_id = 42 → 50,000 rows\
Then: 50,000 random PK lookups to check other conditions

The composite index knows about all four filter columns:

Composite index on (tenant_id, org_id, is_active, is_deleted):

Navigate to tenant_id = 42

└─ Navigate to org_id = 100

└─ Navigate to is_active = 1

└─ Navigate to is_deleted = 0

└─ Only 1,000 rows match ALL conditions!

Then: Only 1,000 PK lookups (not 50,000!)

The filtering happens inside the index itself. By the time MySQL needs to do PK lookups to fetch full row data, it’s already narrowed down from 50,000 to 1,000 rows. That’s a 50x reduction in random I/O.

🐘 Wait, But Would This Happen in PostgreSQL?

This is where things get engine-specific. Everything I’ve described is a characteristic of InnoDB’s clustered index architecture. PostgreSQL handles things differently, and the same scenario would play out differently there.

InnoDB (MySQL): Clustered Index Architecture

The table is the primary key index (clustered)
Secondary indexes store PK pointers, not row data
Every secondary index lookup requires a second hop to the clustered index
This “double lookup” is the root cause of our CPU spike

PostgreSQL: Heap-Based Architecture

The table is stored as a heap where rows are stored in insertion order, not sorted by PK
All indexes (including the primary key) are secondary indexes
Indexes point to a physical row location (tuple ID / ctid) on the heap
There’s no “clustered index lookup” step because the index points directly to the row’s physical location

InnoDB :

Secondary Index → PK value → Clustered Index → Row Data

(Two B-tree traversals per row)

PostgreSQL:

Index → Physical Row Location (ctid) → Row Data

(One B-tree traversal + one heap fetch per row)

So in PostgreSQL, our 50,000 index matches would each require one hop to the heap, not two hops through the clustered index. The CPU overhead would be lower.

But PostgreSQL has its own trade-off. Since the heap isn’t sorted by any index, all index lookups involve random heap access. There’s no concept of a “clustered scan” where the index order matches the physical data order (unless you explicitly use CLUSTER, which is a one-time operation that doesn’t persist).

In InnoDB, if your query scans the primary key in order, you get beautiful sequential reads because the data is physically sorted by PK. In PostgreSQL, even a primary key scan can involve random heap access.

For a different PostgreSQL failure mode, see how we used AI to investigate a search query that bypassed its trigram index. That investigation connects Django's generated SQL, the index definition and the query plan, then checks the proposed fix against both timings and search results.

🧠 Key Takeaways

“OR kills indexes” is true but that might be saving you. If your single-column indexes would cause worse performance than a table scan, the OR condition is accidentally keeping you in the better execution plan.
Selectivity is everything. If your query returns more than 15% of a table’s rows, a single-column index scan can be worse than a full table scan due to random I/O overhead.
Understand your storage engine. InnoDB’s clustered index architecture means every secondary index lookup pays a “double-hop” tax. This tax is invisible when selectivity is low but devastating when selectivity is high.
Composite indexes are the real answer. They don’t just make queries faster, they fundamentally change the math by filtering inside the index before touching the main table.
Monitor CPU, not just query latency. A query that’s 2x faster but uses 5x more CPU is a net negative under concurrent load.
Know your engine’s differences. The same schema optimization can have very different effects on InnoDB vs PostgreSQL due to their fundamentally different storage architectures.

The next time someone tells you “just add an index,” ask them: “What kind of index, and what’s your query’s selectivity?” The answer might surprise you.

Originally published on nulltensor.com.

If leaders want AI speed, they must share the production risk

Rohan Sen Sharma — Mon, 05 Oct 2026 03:00:00 +0000

A recent post on X from an engineer named v0xium took the social media by storm. He described an organisation where AI generated much of the specifications, code, tests, tickets, and reports surrounding software delivery. His deeper complaint was organisational. People were pushed to ship faster without enough time to understand the work, while engineers still carried responsibility when it failed.

Being an engineering leader running a fast paced AI powered engineering team , this problem struck close to home. That concern is legitimate. But it also describes only one side of a difficult transition. AI is changing the economics of software production quickly, and leadership cannot wait for the industry to discover a perfect operating model. So today the useful question is how can a company take larger bets without making engineers the sole owners of the resulting risk.

Both sides are responding to real responsibilities

An engineer who reviews a change has to imagine what happens when it reaches production. Does the generated test check the business outcome? Can we roll the change back? Will an incident wake the same team at 2 a.m.? If the system fails, the organisation will expect engineers to diagnose it and restore service. Asking for stronger quality controls is a rational response to that accountability.

Leadership has a different responsibility. A founder or CTO has to decide which changes could materially improve the business, how quickly the company must learn, and which risks are worth taking. Incremental improvements may keep an existing system healthy while a competitor discovers a much more efficient way to operate. Refusing every uncertain experiment can therefore be as consequential as shipping an unsafe one.

The tension becomes sharper because AI increases the amount of work a team can start. DORA's 2025 research on AI-assisted software development reported a positive relationship between AI adoption and delivery throughput, alongside a negative relationship with delivery stability. The report's broader conclusion is useful here: AI amplifies the system of work around it. Faster generation exposes weak testing, unclear ownership, slow feedback, and fragile architecture sooner.

Neither side can solve that by winning an argument about whether AI is good or bad. They need an agreement about how risk will be taken and who will carry it.

Speed without shared risk creates defensive behaviour

Suppose leadership asks a team to use an agent to change how invoices are matched to payments. The experiment could reduce manual work across finance and support. It could also attach a payment to the wrong invoice, create misleading account balances, or require a difficult correction later.

If the instruction is simply to move faster, the incentives diverge. Management receives the potential business gain. Engineers inherit the production failure, investigation, and repair. The safest personal response for an engineer is to slow the change down or resist it entirely.

The opposite arrangement also fails. If engineers can reject every experiment until uncertainty disappears, leadership remains accountable for growth without the ability to explore a new operating model. The company protects the current system at the cost of learning what could replace it.

Shared accountability changes the conversation. Leadership still decides that the experiment is worth attempting. Engineering still defines what is technically safe enough to attempt. Both sides agree on the boundaries, observe the outcome, and own the consequences.

Make the experiment reversible before making it fast

For the invoice-matching example, the first decision should not be whether the agent is accurate enough to run everywhere. The first decision should be how to learn without creating an uncontrolled failure.

The team could begin with historical records, then run the agent in shadow mode on live work without allowing it to update an invoice. Once its errors are understood, a limited/canary rollout could permit changes for one low-risk class of transactions. A feature flag or workflow switch should provide a tested path back to the previous process. High-value or ambiguous matches could remain subject to human review.

These controls do more than reduce technical risk. They make the leadership decision explicit. Everyone can see which failure modes the company accepted, which it refused, and what evidence is required before expanding the rollout.

A useful operating agreement would answer several connected questions:

What customer or business outcome justifies the experiment?
Which records, tenants, or workflows are inside its initial boundary?
What must remain unchanged even if the agent makes a mistake?
Which signals cause the rollout to pause or reverse?
Who has the authority to stop it, and who owns the recovery?

The answers should be recorded before the rollout. A short decision record is enough if it captures the assumption, risk boundary, evidence, owner, and review date. The goal is not process for its own sake. It prevents the organisation from rewriting the original decision after seeing the result.

Reliability has to be a leadership constraint

Operational reliability cannot remain an engineering concern that leadership supports only after an incident. It has to constrain how the company takes bets.

Google's example error-budget policy shows one way to make this trade-off explicit. When a service remains within its reliability objective, releases proceed. When it exceeds the agreed budget, reliability work takes priority. The policy is not framed as punishment. It gives the organisation a shared rule for deciding when further change is acceptable.

An AI-first team can apply the same principle without copying the policy literally. Leadership can ask for rapid experiments while accepting limits on blast radius, rollout pace, and accumulated operational risk. Engineers can support larger bets while accepting that zero risk is not the goal. The boundary is agreed in advance and adjusted using production evidence.

Observability is part of that agreement. Technical metrics such as errors, latency, and failed jobs matter. The business outcome matters too. An agent that produces no exceptions while silently increasing invoice corrections has not succeeded. A dashboard should make both dimensions visible to the people who authorised the experiment.

Incident ownership must follow decision authority as well. Engineers will lead diagnosis because they understand the system. Product and business leaders should remain present for customer impact, operational trade-offs, and follow-up priority. When leadership participates in the consequences, reliability becomes a company decision rather than a burden delegated to the on-call team.

Trust requires permission to stop and permission to fail

Transparency cannot mean announcing a decision after it has already been made. Engineers need enough context to understand why the company is taking the risk and enough authority to stop a rollout when the agreed boundary is crossed. Leaders need honest technical judgement, including uncertainty, rather than a demand for guarantees that software teams cannot provide.

The same trust must run in the other direction. Founders and managers need room to test ideas that may fail. Engineers can help make those failures contained, observable, and informative instead of treating every unsuccessful experiment as evidence that it should never have been attempted.

This transition will be messy because nobody has a settled answer for how AI changes software delivery. Every side will make poor calls. The way through is to expose assumptions, define reversible boundaries, share the operational consequences, and turn each result into the next decision.

Engineers should not carry an AI strategy's production risk alone. Leadership should not carry the responsibility for transformation without room to experiment. A team can move quickly when both sides are willing to be transparent about what they do not know, vulnerable about mistakes, and accountable for what happens next.

Originally published on nulltensor.com.

A 90% agent PR merge rate can still describe a weak engineering workflow

Rohan Sen Sharma — Sun, 04 Oct 2026 03:00:00 +0000

When a coding agent opens ten pull requests and reviewers merge eight, an 80% acceptance rate looks like a useful performance measure. It tells us that most submitted changes cleared review. It does not tell us whether the agent worked on the problems that mattered, reduced delivery time, or left the product in a better state.

The denominator contains only the work we gave the agent and allowed it to submit. Once teams begin optimising that ratio, they have an incentive to route small, well-specified changes to the agent and keep ambiguous, high-impact work elsewhere. The metric can improve while the agent contributes little to the team’s real constraint.

Acceptance measures the submitted portfolio

Pull-request acceptance rate is straightforward:

Accepted agent pull requests ÷ submitted agent pull requests

Both parts are shaped before review begins. Someone selects the task, decides whether the agent’s attempt is presentable, and chooses whether to open a pull request at all. A failed attempt abandoned in a local branch may never reach the denominator.

Task type changes the result too. A 2026 paper analysed 7,156 agent-generated pull requests from the AIDev dataset. Documentation changes had an 82.1% acceptance rate, while new features had 66.1%. Across all categories, the reported range ran from 84.0% for chores to 55.4% for performance work. The authors describe task type as a major factor and also note an alternative explanation: review standards may differ by category. This is observational evidence from public repositories, not proof that the same gap exists inside every company.

Another study examined 567 Claude Code pull requests across 157 open-source projects. It reported that developers tended to use the agent for refactoring, documentation and testing. Although 83.8% of the pull requests were merged, only 54.9% of those merged needed no further modification.

These studies do not establish the business value of any individual change. They show why an unsegmented acceptance rate is hard to interpret: the mix of work and the human correction behind the merge both matter.

A perfect score can come from a narrow assignment

Consider a hypothetical B2B SaaS backlog with three changes:

Correct an API example in the documentation.
Add validation for a missing optional field.
Change subscription cancellation across billing, access control, stored files and outbound webhooks.

The first two tasks are bounded and easy to verify. The third crosses systems, contains unresolved product decisions and carries a larger rollback cost. Suppose an agent completes the first two, both pull requests are accepted, and the team handles the cancellation change manually.

The agent’s acceptance rate is 100%. That is accurate. It is also incomplete.

Sure, the agent removed two useful pieces of work, so we should not dismiss the contribution. But the problem appears when we use the score to answer a different question: how much valuable engineering work can this workflow take on? The metric has no representation of the unattempted third task, its relative impact, or the effort reviewers spent correcting the accepted changes for the 1st and 2nd scenarios.

This is a broader measurement problem. In an exploratory analysis of coding-agent transcripts from seven METR technical staff during January 2026, METR explicitly warned about task selection and task substitution. People used agents where they expected help and sometimes completed useful but lower-value tasks they might not otherwise have done. The author therefore treated observed task-level time savings as a soft upper bound on productivity improvement, not a productivity multiplier.

Review effort sits outside the numerator

A merged pull request can require a five-minute check or several rounds of correction. Acceptance counts both as one success.

For agent work, the hidden effort may include clarifying the ticket, restoring an assertion the agent weakened, checking tenant boundaries, rerunning a flaky test, or rewriting the change so it fits an existing design. The final merge tells us that a reviewer accepted the resulting code. It does not attribute how much of that result came from the agent.

Rejection has a similar ambiguity. A valuable attempt at a difficult migration may uncover an undocumented dependency even if its pull request is not merged. A trivial change can be accepted immediately. If we reward teams only for the ratio, the safer strategy is to submit more of the second kind.

The practical response is not to stop tracking acceptance. It is to retain the diagnostic signal and add the context needed to interpret it.

Measure the work before, during and after review

I would evaluate an agent workflow through five connected questions:

Question	Evidence to collect
What work reached the agent?	Task category, expected impact, uncertainty, affected systems and a rough human effort estimate
Did it satisfy the task?	Reviewed acceptance criteria and end-to-end checks against the resulting behaviour
What did the human add?	Review time, number and type of corrections, and unresolved decisions returned to the author
Did it improve delivery?	Lead time from task start to production, including waiting and rework
Did the change remain healthy?	Rollbacks, escaped defects, operational incidents and the intended product or business result

The categories do not need an elaborate scoring model. Start by separating documentation, tests, fixes, features, migrations and performance work. Compare acceptance only within similar groups and show the number of attempted, abandoned, submitted and merged tasks at each stage.

For task success, use the behaviour the change was meant to produce. OpenAI’s SWE-Lancer illustrates one way to make this concrete in an evaluation: it includes more than 1,400 real freelance software-engineering tasks with actual payouts, while independent tasks are graded using end-to-end tests reviewed three times by experienced engineers. It is a benchmark, not a template for measuring an internal team, but its design separates task outcome and economic context from whether a patch merely looks acceptable.

Delivery measures can then connect the agent’s contribution to the surrounding system. DORA’s current software-delivery measures include change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate. These are team-level measures and should not be attributed entirely to the agent. They help reveal whether more accepted pull requests coincide with safer, faster delivery.

Keep acceptance rate in its proper place

Pull-request acceptance rate is useful for finding friction. Segment it by task type, inspect rejection reasons, and track whether agent output routinely needs the same corrections. It can tell us where the workflow produces reviewable changes.

The larger decision requires a wider denominator. Include the tasks the agent attempted, the tasks we chose not to give it, and the human work needed to turn its output into production software. Then connect those results to delivery and product outcomes.

If I had to start over again, I would begin with one month of agent-assisted work. Label every task by type and uncertainty, record whether an attempt reached review, and sample the corrections behind accepted pull requests. We can then decide whether the agent is expanding the work the team can complete or becoming very efficient at the work that was already easiest to accept.

Originally published on nulltensor.com.

Your customer support AI is probably going to fail in production!

Rohan Sen Sharma — Sat, 03 Oct 2026 03:00:00 +0000

We were looking to implement a customer support AI agent. Our customer experience (CX) team evaluated several products, with varying levels of confidence in their suitability. A support-AI demo can produce a convincing answer to a product question. The harder test is whether it gives the right answer when the customer has omitted a detail, the documentation is incomplete, or the request needs to be escalated to an engineer.

To avoid choosing a product based on its demo alone, we needed to formalise an AI agent evaluation process. We zeroed in on starting with a combination of 50 resolved tickets and a human-scored rubric. That is a manageable first evaluation, not a statistically established threshold for deployment.

I wanted to understand which kinds of tickets AI could help us handle, how much review its answers would require, and where it would need to hand the work back to a human. We experimented and had several back-and-forth discussions between the engineering and customer success teams. I wanted to summarise what we learned to save other teams time.

This is not a comprehensive guide. We are still learning how to operate this way, and success depends on several factors specific to each business. However, it can serve as a starting point for adapting the process to your own support workflow.

Based on that experience, here is how I would structure an initial pilot.

Start with one job for the AI

“Automate support” leaves too much undefined.

For this pilot, I would give the AI one job: draft the next customer-facing response using the ticket history and approved support information. It could propose a clarification or escalation when appropriate. A human would review every draft.

That scope matters. We would be evaluating response drafting, not autonomous ticket resolution. A useful reply might move an investigation forward without resolving the problem.

Take a customer who says:

My export has been processing for an hour. Can you restart it?

A useful response depends on what we know. Is the job still running? Has it failed? Can restarting it create duplicate work? Does the support team have permission to restart it?

A confident “I’ve restarted your export” would be unacceptable if the AI had neither the tools nor the authority to do so.

Before selecting tickets, write down what the AI can inspect, what it can recommend, and when it must escalate.

Build the sample without giving away the answer

Select 50 recent resolved tickets from the support queue, covering the work the proposed assistant would encounter.

Include common how-to questions, troubleshooting, missing information, and cases that required escalation. Sample across customers and product areas and remove duplicates that would give one recurring incident disproportionate influence.

If we deliberately include extra difficult cases, report them separately. Their failure rate should not represent the normal queue.

For each ticket, choose a point where the AI would have been asked to help. Build two separate records:

Input: the customer message, preceding conversation, and permitted context available at that point.
Reviewer reference: the eventual resolution, relevant evidence, and acceptable next steps.

Keep subsequent replies and resolution notes out of the AI’s input. Otherwise, we would be testing whether it can restate an answer already present in the context. Giving the AI information that became available later introduces lookahead bias and makes the evaluation unreliable.

Check the retrieval source too. Removing the resolution from the prompt achieves little if the AI can retrieve the complete closed ticket.

Use an approved environment and remove customer identifiers that are unnecessary for the task. Preserve details that materially affect the answer, such as the product version or relevant account configuration.

Define the rubric before reading the outputs

I would score each draft across five dimensions, using a simple 0–2 scale.

Dimension	0: unacceptable	1: needs correction	2: meets expectations
Factual correctness	Gives incorrect or invented information	Contains a material ambiguity or imprecision	Claims agree with the available evidence
Appropriate next step	Recommends an unsuitable action	Direction is useful but incomplete	Answers, clarifies, or escalates appropriately
Use of context	Ignores a relevant fact	Uses some context but misses an important detail	Accounts for the details needed at this stage
Policy and authority	Crosses a boundary or claims an unperformed action	Leaves a permission or policy condition unclear	Stays within the defined boundaries
Customer usability	Confusing or unusable	Requires avoidable rewriting	Clear, specific, and actionable

The rubric needs examples from our own support workflow. “Appropriate escalation” means little until we define which requests require it and what information the handoff must contain.

I would record critical failures separately from the total score. Exposing another customer’s information or falsely claiming that an action was completed should fail the draft regardless of how well it scores elsewhere.

Anthropic’s guidance on evaluating agents distinguishes the agent’s transcript from the actual outcome. That distinction applies here: a response claiming that an export was restarted is not evidence that the restart happened.

The historical support reply should also be treated as a reference, rather than the only acceptable wording. A different response might be equally valid, or better supported by the information available.

Calibrate the reviewers, then freeze the setup

Within the 50 tickets, I would use 30 for development and reserve 20 for evaluation after the setup was fixed. This is similar to how backtests are run in financial simulations.

Start by having two experienced support reviewers independently score the same five development cases. Compare their scores and discuss disagreements.

If one reviewer rewards a direct answer while another expects clarification, the issue may be an undefined support rule. Resolve that before judging the remaining outputs.

OpenAI’s evaluation guidance recommends clear, task-specific criteria and human calibration. For this pilot, calibration gives us a shared interpretation of what a good response looks like.

Use the development tickets to improve instructions, retrieval, and escalation guidance. Then freeze the model, prompt, knowledge sources, and settings before running the reserved tickets.

If we tune against those reserved results, they become development data. We need fresh tickets for the next independent check.

For every output, retain the draft, rubric scores, critical-failure flag, reviewer correction, and time spent reviewing and editing. Where practical, have both reviewers independently score the reserved set and reconcile disagreements.

Measure the work left for the human

A high average rubric score would be insufficient if every answer still required substantial checking.

Alongside the scores, classify each draft as:

Usable without edits.
Usable after minor edits.
Requiring substantial correction.
Unusable.

Record appropriate escalations separately from unnecessary ones. Otherwise, an assistant that escalates everything could appear dependable while contributing little.

For a time comparison, ask another reviewer to draft responses without AI from the same input context. Avoid having someone write the baseline immediately after seeing the AI’s answer. Compare human preparation time with the time a human spends reviewing and editing the AI’s draft, and record generation latency separately.

The comparison remains a small offline exercise. It cannot establish customer satisfaction, a reduction in reopened tickets, or end-to-end resolution time.

Report counts as well as percentages. “Three of six troubleshooting drafts needed substantial correction” is more useful than a single average across all 50 tickets.

Use the result to choose a narrower next step

Before running the next iteration, agree on the conditions for progressing: acceptable answer quality, no observed critical failures, manageable review effort, and clear escalation behaviour.

Passing those conditions would justify a limited, human-reviewed live trial. It would not establish readiness for automatic sending.

In our case, the results supported using AI to draft routine how-to answers while leaving troubleshooting with the support team. Another team might find that missing documentation is the main constraint. Either finding gives us a concrete next action.

I would finish the pilot by assigning each recurring failure to an owner and a correction: improve a source, change an instruction, clarify a rule, or restrict the supported scope. Preserve those cases as regression checks, then evaluate the revised system on fresh tickets before expanding its role.

Originally published on nulltensor.com.

Using AI to Investigate a Slow Query: Why Django icontains Skipped Our pg_trgm Index

Rohan Sen Sharma — Fri, 02 Oct 2026 03:00:00 +0000

We used AI to find out why Django's icontains filter was quietly bypassing our PostgreSQL pg_trgm trigram index while product searches slowed to several seconds.

We had recently migrated our database from MySQL to PostgreSQL and were benchmarking the new database’s performance. During this exercise, we found that product searches were taking several seconds despite having trigram indexes in place. The code looked straightforward. The indexes existed. However, the generated SQL was asking PostgreSQL to search a different expression from the one we had indexed.

To debug this I sought the help of my trusted claude and asked it to analyse different parts of my stack. Let us walk through that investigation and what it tells us about using AI when the behaviour of a system is unfamiliar.

An unfamiliar error becomes easier to investigate when we can connect it to something concrete: the request that failed, the code it executed, and the work the underlying system performed.

Start with what the system is doing

Our application stores products belonging to different companies. During the investigation, we recorded the following comparison at approximately 4,908 requests per minute, using the same db.r6i.large instance class:

Metric	MySQL comparison	PostgreSQL
Database CPU utilisation	42.7%	94.8%
Average database time per request	72 ms	444 ms
Product-list database time during the peak window	112 ms	4.67 s

These numbers established the scale of the problem. They did not establish why it was happening. Similar throughput and instance size still leave differences in query mix, configuration, and execution plans.

In investigations like this, New Relic helps us locate the affected operation, while CloudWatch and RDS metrics provide the surrounding resource context. We then need to connect that operation to the actual code and database query. Before asking AI for a fix, give it the exact error, the affected operation, the timestamp, and the relevant application code. Include what you expected to happen and what happened instead.

A useful starting question is:

What does this evidence establish, which explanations remain possible, and what should we inspect next to distinguish them?

This encourages an investigation that can progress as new evidence arrives.

Look below the abstraction

Our product search used Django’s case-insensitive lookup:

query = query.filter(product_name__icontains=value)

In the configuration we investigated, the generated PostgreSQL expression was:

UPPER(product_name::text) LIKE UPPER('%term%')

We had created trigram indexes on the raw columns, using these indexed expressions:

product_name gin_trgm_ops
itemid gin_trgm_ops

The important detail was the UPPER() transformation.

The product-name index covered the raw column, while the query searched its uppercase representation. Our EXPLAIN output showed that this query form bypassed the product-name trigram index. PostgreSQL read the tenant’s candidate products and applied the text filter row by row.

This explains why checking that an index exists is insufficient. We need to understand whether the query can use it.

There is a second question: what work does using that index actually require? In the MySQL index change that sent our database CPU to 100%, the problem was the cost of the selected access path. Here, the query expression prevented PostgreSQL from using the intended index. Both investigations needed the query plan to explain what the database was doing.

It also gives AI a much more specific problem. With the ORM code alone, it has limited evidence. With the generated SQL, index definition, and query plan together, we can ask it to examine the relationship between them.

So I followed up with a prompt along these lines:

Compare the expression being filtered with the expression being indexed. Explain which plan nodes support an index mismatch, and identify any assumptions that still need verification.

The useful principle is to ask the assistant to connect its explanation to observable details. We want to understand what happened beneath the abstraction that reported the failure.

Make a change that preserves behaviour

Our fix aligned the query with the existing index.

Moving from icontains to contains removed the UPPER() transformation. We also configured case-insensitive collation on the product-name column to preserve the intended search behaviour.

Both parts mattered. A query becoming faster would not be sufficient if it stopped returning products that users expected to find.

I would treat an AI-generated recommendation with the same requirement: explain both the performance mechanism and the behaviour that must remain unchanged.

In this case, that means checking case sensitivity, representative search terms, and the returned results alongside the execution plan. The lookup and collation changes describe our tested configuration; they are not a general recommendation to replace every icontains call.

PostgreSQL supports trigram-based indexing for LIKE and ILIKE, but effectiveness depends on the query and the trigrams that can be extracted from its search pattern. PostgreSQL pg_trgm documentation.

The plan and the result checks are what let us move from a plausible change to a supported one.

Validate the explanation as well as the fix

We validated the revised queries in PostgreSQL staging against a company with approximately 1.15 million products.

The raw-column LIKE queries used the existing GIN trigram indexes and completed in 77–90 ms. The previous UPPER(...) LIKE UPPER(...) queries exceeded our 12-second database timeout.

That gives a lower-bound improvement of more than 130× for these measured cases. The exact speed-up is unknown because we stopped the old queries before they completed(timed out).

This is where an AI-assisted investigation needs a feedback loop. Return the new plan, timings, and correctness results to the assistant. Ask whether they support the proposed explanation and what remains unresolved. Sometimes an anomaly can be attributed to a combination of reasons. AI helps you better calibrate if the new data conforms to the fixes we introduced.

Carry the method into the next error

The final finding was SPECIFIC to this scenario : our query expression and index expression did not match. However the AI assisted debugging method applies more broadly.

When using AI to investigate an unfamiliar error or stack trace, start with the failing operation. Gather the relevant code and runtime evidence. Ask for explanations that can be tested, then bring the results back into the investigation.

Google’s Effective Troubleshooting chapter describes this process as forming hypotheses and testing them against observations. AI fits naturally into that process when its explanations remain tied to evidence. Google SRE: Effective Troubleshooting.

For a lean engineering team, the lasting value is in making the investigation repeatable. Preserve the symptom, the evidence, the change, and the validation checks. The next unfamiliar failure then starts with a better set of questions and a tested way to answer them.

Originally published on nulltensor.com.