DEV Community: Emery Huang

Your Runbook Is a Graph, Not a Document: Proving Every Alert Has a Way Out

Emery Huang — Tue, 27 Oct 2026 13:08:59 +0000

The most expensive runbook bug is not a wrong command. It is a step nobody can reach, or an alert that lands on a page with no exit. We keep reviewing runbooks as prose, though, so the structural defects stay invisible until 3am.

So here is the reframe I want to argue for: treat your runbook as a directed graph, audit it like one, and let a machine tell you which alerts are stranded. That single change catches a class of failure that grammar, lint, and even careful human review will happily miss. Does your runbook currently survive that test? Most don't, and mine didn't either on the first pass.

Three Tables Beat One Long Document

The graph model needs only three collections, and you can hold all three in one YAML file that lives next to the service it describes.

Alerts — the name, the source, the severity, and the entry step where a human starts.
Steps — the owner, the commands, the allowed next steps, and whether the step is terminal (we are done).
Rules — freeze windows, who can lift them, and which steps are only safe while a freeze is active.

That is it. No prose narrative, no screenshots of dashboards, no "see the wiki". If a step has no owner, it is not a step, it is a hope.

# runbook.yaml
alerts:
  - name: API5xxHigh
    source: prometheus
    severity: page
    entry: triage_api
  - name: DiskFullNode
    source: prometheus
    severity: page
    entry: triage_disk

steps:
  triage_api:
    owner: oncall-platform
    commands:
      - kubectl -n prod rollout status deploy/api --timeout=30s
      - kubectl -n prod get pods -l app=api
    next: [rollback_api, escalate_platform]
  rollback_api:
    owner: oncall-platform
    commands:
      - kubectl -n prod rollout undo deploy/api
    terminal: true
  escalate_platform:
    owner: oncall-platform
    commands:
      - page the platform secondary
    requires_freeze: true
    next: []
  triage_disk:
    owner: oncall-infra
    commands:
      - df -h /var/lib
    next: [escalate_platform]

Spot the defect yet? escalate_platform has no terminal: true and no outgoing edge, so it is a dead end — the page goes to a human who has no recorded next action. You only see it because the shape is now inspectable, and that is exactly the point.

Four Checks Worth Automating Today

Once the runbook is a graph, a short script answers questions that reviews cannot answer consistently.

Orphan alerts: every alert's entry must exist as a step, otherwise the page renders a blank page at the worst moment.
Dead ends: every reachable non-terminal step needs at least one outgoing edge, or the responder stalls.
Hop distance: the nearest terminal step should be within a small number of hops, because a five-hop path is a five-decision path under stress.
Freeze traps: any step with requires_freeze must also declare an unfreeze_trigger, or the runbook can strand a live incident inside a change freeze.

I would also flag unreachable steps, which is dead code written in English. Nobody deletes them, everybody reads them, and they quietly teach the wrong procedure.

Listing 2: The Reachability Checker

This is a scaffold, not a finished product — stdlib plus PyYAML, BFS over the step graph, non-zero exit code so CI can gate on it. I have not executed this exact copy, so run it against your file and adjust before you trust it.

#!/usr/bin/env python3
"""runbook_lint.py - reachability audit for an on-call runbook graph."""
import sys
from collections import deque
import yaml

MAX_HOPS = 3

def load(path):
    with open(path, encoding="utf-8") as fh:
        return yaml.safe_load(fh)

def walk(steps, start):
    """Return {step: hops} for everything reachable from start."""
    dist, queue = {start: 0}, deque([start])
    while queue:
        cur = queue.popleft()
        for nxt in steps.get(cur, {}).get("next") or []:
            if nxt not in dist:
                dist[nxt] = dist[cur] + 1
                queue.append(nxt)
    return dist

def audit(rb):
    steps, alerts, problems = rb["steps"], rb["alerts"], []
    reachable = set()
    for alert in alerts:
        entry = alert.get("entry")
        if entry not in steps:
            problems.append(f"{alert['name']}: entry step {entry!r} missing")
            continue
        dist = walk(steps, entry)
        reachable |= set(dist)
        terminals = [s for s in dist if steps[s].get("terminal")]
        if not terminals:
            problems.append(f"{alert['name']}: no terminal step reachable")
        elif min(dist[s] for s in terminals) > MAX_HOPS:
            problems.append(f"{alert['name']}: terminal too far (>{MAX_HOPS} hops)")
        for step in dist:
            if steps[step].get("requires_freeze") and not steps[step].get("unfreeze_trigger"):
                problems.append(f"{alert['name']}: freeze at {step} has no unfreeze trigger")
    for name, step in steps.items():
        if not step.get("owner"):
            problems.append(f"{name}: no owner")
        if name not in reachable:
            problems.append(f"{name}: unreachable from any alert")
        elif not step.get("next") and not step.get("terminal"):
            problems.append(f"{name}: dead end")
    return problems

if __name__ == "__main__":
    issues = audit(load(sys.argv[1]))
    print("\n".join(issues) if issues else "ok: runbook graph is reachable")
    sys.exit(1 if issues else 0)

Run it on the YAML above and you get three complaints in one pass: the dead end, the missing unfreeze trigger, and the disk alert whose only route runs through a freeze-gated step. That is a real design smell, and it took four seconds.

Let a Free Model Draft the Graph, Keep the Machine as Judge

Hand-writing YAML for forty steps is boring, so I treat the conversion as a drafting task rather than a design task. Give a model the existing prose runbook and a strict output contract, and ask only for structure — never for new commands.

Convert the runbook below into YAML with exactly three top-level keys:
- alerts: [{name, source, severity, entry}]
- steps: {<id>: {owner, commands[], next[], terminal?, requires_freeze?, unfreeze_trigger?}}
- rules: {freeze_windows[]}
Rules: copy commands verbatim, never invent a command, mark a step terminal
only if the text says the incident is resolved, and output YAML only.

This is where MonkeyCode fits the workflow rather than the marketing: it advertises free model access and a free server option, which makes the drafting pass cheap enough to run on every runbook PR instead of once a quarter. The free-token allowance I was given for this article was described to me as 10 million tokens, and I have not independently benchmarked it or the server, so check the project's current terms before you design a pipeline around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The important discipline is that the model's output is a draft, and the checker is the judge. Never let the drafter be the reviewer, because a model that invents next edges will produce a graph that looks beautiful and routes nowhere.

Where the Free Server Actually Earns Its Keep

Be honest with yourself: a 90-line static checker does not need a cloud box, and a laptop cron is fine. The scheduled part that benefits is the diff — running the audit weekly, committing the report, and letting the exit code fail a PR when a new alert arrives without an entry step.

# crontab -e on the free server, Monday 06:17
17 6 * * 1 /opt/runbook/venv/bin/python /opt/runbook/runbook_lint.py \
  /opt/runbook/runbook.yaml >> /var/log/runbook_lint.log 2>&1 || \
  /opt/runbook/notify.sh "runbook graph audit failed"

Two lines, one log, one notification, and the runbook stops rotting silently between incidents. That is the whole pitch — not intelligence, just regularity.

What This Does Not Prove

A reachability audit is a structural check, and it is important to say plainly what it cannot see.

It does not verify that any command is correct, safe, or idempotent — a perfectly connected graph can still run the wrong kubectl.
It does not know whether your owners still exist; a named team that disbanded three months ago looks fine to BFS.
It is only as fresh as the YAML, so the YAML has to be the source of truth, not a copy of the wiki.
MAX_HOPS is a judgment call, not a law of nature, and parallel steps flatten into a single path in this model.
Free tiers change; if the drafting pass becomes expensive, the checker still works with a human doing the conversion.

Who Should Skip This

If you have four alerts and one service, this is overhead — a two-line README beats a graph. If your runbooks are already generated from infrastructure code, you likely get reachability from that generator and should not build a parallel one. And if nobody will run the checker in CI, the YAML will decay into a second stale document, which is worse than the prose you started with.

Your situation	Does the graph audit pay off?
10+ alerts, 3+ services, shared on-call rotation	Yes — orphans and dead ends appear fast
Compliance demands freeze windows be respected	Yes — the unfreeze-trigger check is cheap insurance
Single service, single responder, weekly pages	Probably not yet
Runbook is generated from IaC tooling	No — extend the generator instead

Start with the four checks, wire the exit code into one pipeline, and let the first failing run tell you which alert has been pointing at a blank page all along. If you want to try the drafting half without spending tokens, the project's free model access and free server option are documented on the MonkeyCode site — but run the checker against whatever it produces before you trust a single step.

Your Runbook Has No Deadlines: Budgeting Every On-Call Step Before the Page Goes Late

Emery Huang — Mon, 26 Oct 2026 11:38:49 +0000

Most runbooks I read fail for a boring reason: nothing in them has a deadline. Every step says what to check, almost none say how long you are allowed to check it before you stop and page a human. That missing number is why a five-minute incident becomes a forty-minute one — not because the steps were wrong, but because nobody was told when to quit.

So here is the takeaway up front. Treat each runbook step as a small contract with three fields: a time budget, a stop condition, and a timeout action. Then check the sum of those budgets against your escalation SLA in CI. If the worst case exceeds the SLA, the runbook is broken before the incident starts, and you find out on a laptop instead of at 03:00.

This matters more now, not less. Everyone is wiring models into ops tooling, and the current wave of AI-assisted runbooks mostly produces commands. Generated commands without generated deadlines is just a faster way to be busy. The harness below is where I put the guardrail.

The seam nobody budgets: between "I have a hunch" and "I page someone"

Look at a typical first step: kubectl logs ... | grep 500. How long is that allowed to take? Two minutes? Ten? If the answer is "until I understand it", you have written a step with an unbounded budget, and the escalation clock is now controlled by whoever is holding the pager.

I keep the arithmetic explicit because humans are bad at it under stress. Six steps that each feel like "about two minutes" is twelve minutes of declared work, plus the two to four minutes it takes to actually reach a human after you decide to. That is already past a fifteen-minute first-page target, and nobody wrote down a wrong word.

Three questions are worth asking of every step you own:

What is the budget, in seconds, and who says so?
What observation means "this step is done, move on"?
When the budget expires, do we escalate or abort?

If any of those is unanswerable, the step is a wish, not a procedure.

Give every step a deadline, a bail-out, and an owner

The format below is deliberately boring YAML, because boring survives review. It describes a payments API and a runbook for my least favourite alert, a 5xx spike.

# runbook.yaml — payments-api, alert: PaymentsApi5xxSpike
alert: PaymentsApi5xxSpike
service: payments-api
escalation_hold_s: 240      # measured: decide-to-page until a human answers
steps:
  - id: check-error-mix
    owner: on-call
    command: "kubectl -n payments logs deploy/api --since=5m | jq -r '.status' | sort | uniq -c"
    budget_s: 120
    stop_if: "5xx share below 1% on the dashboard for two consecutive minutes"
    on_timeout: escalate
  - id: check-dependency
    owner: on-call
    command: "curl -sS -o /dev/null -w '%{http_code} %{time_total}\\n' http://psp.internal/healthz"
    budget_s: 90
    stop_if: "psp answers 200 in under 300ms twice in a row"
    on_timeout: escalate
  - id: rollback-canary
    owner: payments-lead
    command: "./deploy rollback --env prod --to previous --dry-run"
    budget_s: 300
    stop_if: "canary error rate matches the previous release for two minutes"
    on_timeout: abort
freeze:
  reason: "ledger schema migration window"
  owner: payments-lead
  expires_after_s: 1800
  unfreeze_check: "SELECT count(*) FROM payments.schema_migrations WHERE dirty;"
  unfreeze_command: "./freeze lift --id ledger-migration"

Worst case here is 120 + 90 + 300 + 240 = 750 seconds, which leaves 150 seconds of slack against a 900-second target. That is a defensible number, and it is written down where a reviewer can argue with it.

The validator: twenty lines of arithmetic that pay for themselves

Arithmetic in a document rots, so I check it mechanically. The script is small on purpose: it validates required fields, sums the declared budgets, and refuses to pass when the total plus the escalation hold exceeds the SLA you pass in.

#!/usr/bin/env python3
# runbook_budget.py — fail a runbook whose declared worst case blows your SLA
import argparse, sys
import yaml

STEP_FIELDS = ("id", "owner", "command", "budget_s", "stop_if", "on_timeout")
FREEZE_FIELDS = ("reason", "owner", "expires_after_s", "unfreeze_check", "unfreeze_command")

def check(doc, sla):
    errs, steps = [], doc.get("steps") or []
    if not steps:
        errs.append("no steps declared")
    seen = set()
    for n, s in enumerate(steps, 1):
        sid = s.get("id", f"#{n}")
        if sid in seen:
            errs.append(f"{sid}: duplicate id")
        seen.add(sid)
        for f in STEP_FIELDS:
            if s.get(f) in (None, "", []):
                errs.append(f"{sid}: missing {f}")
        if s.get("on_timeout") not in (None, "escalate", "abort"):
            errs.append(f"{sid}: on_timeout must be escalate or abort")
        b = s.get("budget_s")
        if not isinstance(b, int) or b <= 0:
            errs.append(f"{sid}: budget_s must be a positive integer")
    freeze = doc.get("freeze") or {}
    if freeze:
        for f in FREEZE_FIELDS:
            if freeze.get(f) in (None, "", []):
                errs.append(f"freeze: missing {f}")
    worst = sum(s.get("budget_s") or 0 for s in steps) + (doc.get("escalation_hold_s") or 0)
    if worst > sla:
        errs.append(f"worst case {worst}s exceeds SLA {sla}s by {worst - sla}s")
    return errs, worst, steps

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("path")
    ap.add_argument("--sla", type=int, default=900)
    a = ap.parse_args()
    with open(a.path) as fh:
        doc = yaml.safe_load(fh) or {}
    errs, worst, steps = check(doc, a.sla)
    print(f"{a.path}: {len(steps)} steps, declared worst case {worst}s, SLA {a.sla}s")
    for i, s in enumerate(steps, 1):
        print(f"  {i}. {str(s.get('id','?')):<18} {str(s.get('owner','?')):<15} "
              f"{str(s.get('budget_s','?')):>5}s  on_timeout={s.get('on_timeout','?')}")
    if doc.get("freeze"):
        fr = doc["freeze"]
        print(f"  freeze: {fr.get('reason','?')} — expires_after_s={fr.get('expires_after_s','?')}")
    if errs:
        print("FAIL")
        for e in errs:
            print("  -", e)
        return 1
    print(f"OK — {a.sla - worst}s of slack before the page is late")
    return 0

if __name__ == "__main__":
    sys.exit(main())

Usage is two commands, and the second one belongs in CI next to your lint job:

pip install pyyaml
python runbook_budget.py runbook.yaml --sla 900

The output format for the file above looks like this — run it on your own file to confirm before you trust the numbers:

runbook.yaml: 3 steps, declared worst case 750s, SLA 900s
  1. check-error-mix      on-call             120s  on_timeout=escalate
  2. check-dependency     on-call              90s  on_timeout=escalate
  3. rollback-canary      payments-lead       300s  on_timeout=abort
  freeze: ledger schema migration window — expires_after_s=1800
OK — 150s of slack before the page is late

Now raise rollback-canary to 600 seconds, which is a very realistic edit after one bad rollback, and the same command fails with a sentence you can paste into a review: rollback-canary: worst case 1050s exceeds SLA 900s by 150s. That is the whole value proposition — the argument happens in a diff, not in a bridge call.

Freeze rules rot unless they have an expiry and an unfreeze check

A freeze without expires_after_s is a permanent policy that everyone quietly ignores within two weeks. The validator requires four things for any declared freeze: a reason, an owner, an expiry in seconds, and a command that proves the unfreeze condition. A fifth field, unfreeze_command, closes the loop so the next person does not have to reconstruct intent from chat history.

Is that overkill for a 30-minute migration window? Possibly. But the failure mode is asymmetric: an expired freeze that nobody lifted blocks unrelated deploys for days, and no dashboard will ever tell you.

Where a free model actually helps — and where it must not decide

Here I should mention the tooling I have been pointing this workflow at. MonkeyCode is an open-source project that offers free model access and a free server option; both are availability claims from the project side, so verify current model names, quotas, and server specs on its own pages rather than trusting a blog post, including this one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

In this workflow I use the free model access for exactly two narrow jobs, and I keep both of them outside the decision path:

Critique pass, not authoring pass. Feed it one step at a time and ask whether the budget is plausible for that command and what could make it hang. I accept or reject each suggestion in a PR; the model never writes to budget_s directly.
Stop-condition drafts from real incidents. Paste a redacted timeline and ask for candidate stop_if predicates in the shape the schema expects. Reviewing three bad suggestions is often faster than staring at a blank field.

What it must not do is decide on_timeout. Whether a step escalates or aborts is a risk decision owned by the service owner, and no model has the context to make it for you.

Where the free server earns its keep

The validator is only useful if it runs when nobody remembers to run it. A scheduled check is the obvious fit: clone the runbook repo onto the free server option, run the validator on a timer over every runbook*.yaml, and post failures to the on-call channel. That is a read-only job over text files, which is the least risky thing you can put on shared infrastructure.

Please do not put the free server in the incident response path. It should not hold production credentials, and it should not be the thing you rely on at 03:00. If the scheduled job is down, you should lose a warning, not a recovery option.

# sketch of the scheduled job (cron, every weekday morning)
0 8 * * 1-5 cd /srv/runbooks && for f in runbook*.yaml; do \
  python runbook_budget.py "$f" --sla 900 || echo "BUDGET FAIL: $f"; done

Who should not do this

A decision table is more honest than a blanket recommendation, so here is mine.

Your situation	Verdict
You have incident timelines with real step durations	Good fit — the budgets can be grounded in data
You have no incident history yet	Skip the numbers, keep `stop_if` and `on_timeout`
Your paging is fully automated and sub-minute	Marginal — the SLA math is already enforced elsewhere
Nobody reviews runbook changes	Do not add a schema nobody maintains
You need approvals and audit trails per step	Use your existing change-management tool instead

Limitations I will not paper over

budget_s is a declared number, not a measured one, and it will drift the moment someone changes a command. Re-derive it from real incident timelines every quarter, or it becomes fiction with a schema around it. The validator also cannot tell whether your stop_if is correct — it only proves the field exists and the arithmetic closes. And YAML runbooks have a habit of turning into a shadow configuration system, so keep the file small enough that a tired human can read all of it.

Start with one alert, one file, and one failing build. If your runbook cannot survive python runbook_budget.py, would you really want to discover that from a page?

The Alert Renamed a Label, the Runbook Rendered Blank: Contract-Testing Payloads Before 3am

Emery Huang — Tue, 20 Oct 2026 09:46:58 +0000

Your runbook rarely fails because somebody wrote the wrong command. It fails because the alert that fires it quietly stopped carrying the label that command depends on, and nobody noticed until a template rendered an empty string during an incident. I want that discovery to happen in a CI log at 2pm, not in a terminal at 3am with six people watching. So the artifact below is a payload contract test: assert the labels, render the commands, resolve the owner, and refuse to emit anything that mutates while a freeze is active.

The quiet break: a template that renders to nothing

Most runbooks I read are Markdown with a YAML header, and the commands inside are string templates over alert labels. That design is good, because it keeps one runbook for every service instead of forty near-copies. It is also fragile, because a template has no idea whether the label it references still exists in the alert rule.

# runbooks/checkout-error-rate.yaml
alert: CheckoutErrorRateHigh
first_commands:
  - name: pod health
    cmd: "kubectl -n {{ $labels.namespace }} get pods -l app={{ $labels.service }}"
    class: read
  - name: recent deploys
    cmd: "kubectl -n {{ $labels.namespace }} rollout history deploy/{{ $labels.service }}"
    class: read
escalation:
  owner: "{{ $labels.owner }}"

Now let someone rename namespace to k8s_namespace in the rule file during a refactor. The rule still evaluates, the alert still pages, and the test suite around the alerting rules still passes, because no test ever asked what the runbook needs. The command renders as kubectl -n get pods -l app=checkout, and an empty namespace flag usually collapses to whatever your current kubeconfig context points at. Is that a good way to find out which cluster you are actually talking to?

Three things the runbook actually needs from the alert

I reduce the contract to three assertions, because every other check I have tried was either unenforceable or theatre.

Identity labels — the values that make a command target a specific workload (service, namespace, cluster).
Routing — an owner or team label that resolves to a real rotation, not a dead alias.
Action text — a non-empty first command per severity, so the page is executable rather than decorative.

Everything below is in service of those three, and each check fails the build rather than warning.

Check 1: assert the label set in a rule unit test

If you already use Prometheus-compatible rules, promtool can pin the exact label and annotation set an alert emits. The test file feeds synthetic series, evaluates the rule, and compares the emitted alert to what you declared.

# tests/checkout.alerts.test.yaml
rule_files:
  - rules/checkout.alerts.yaml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="checkout",code="500"}'
        values: '0+10x20'
      - series: 'http_requests_total{job="checkout",code="200"}'
        values: '0+100x20'
    alert_rule_test:
      - eval_time: 12m
        alertname: CheckoutErrorRateHigh
        exp_alerts:
          - exp_labels:
              severity: page
              service: checkout
              namespace: checkout-prod
              owner: payments-oncall
              runbook: https://runbooks.example.com/checkout-error-rate
            exp_annotations:
              summary: "checkout 5xx above 5%"
              first_command: "kubectl -n checkout-prod get pods -l app=checkout"

promtool test rules tests/checkout.alerts.test.yaml

When someone drops or renames a label, this goes red in the same PR that would have broken the runbook. If your alerting stack is not Prometheus-shaped, do not port promtool; instead export one alert payload per alert name as JSON and assert the same key set with a fifteen-line script. The tool matters less than the habit of treating the payload as an interface.

Check 2: render every command against a captured payload

The unit test proves the alert can emit the labels. A renderer proves the runbook actually consumes them without producing empty strings, and it is the check I trust most because it exercises the exact string a human will paste.

#!/usr/bin/env python3
"""Render a runbook's first commands against a captured alert payload."""
import json
import re
import sys
from pathlib import Path

import yaml

LABEL = re.compile(r"\{\{\s*\$labels\.(\w+)\s*\}\}")


class MissingLabel(Exception):
    pass


def render(template: str, labels: dict) -> str:
    def sub(match: re.Match) -> str:
        key = match.group(1)
        value = labels.get(key)
        if not value:
            raise MissingLabel(f"$labels.{key} is empty in {template!r}")
        return value

    return LABEL.sub(sub, template)


def main(manifest: Path, payload: Path, freeze: Path | None) -> int:
    spec = yaml.safe_load(manifest.read_text())
    labels = json.loads(payload.read_text())["labels"]
    frozen = bool(freeze and json.loads(freeze.read_text()).get("active"))
    failures = 0

    for command in spec["first_commands"]:
        if frozen and command["class"] != "read":
            print(f"SKIP (freeze active): {command['name']}")
            continue
        try:
            rendered = render(command["cmd"], labels)
        except MissingLabel as exc:
            print(f"FAIL {command['name']}: {exc}")
            failures += 1
            continue
        print(f"OK   {command['name']}: {rendered}")

    return 1 if failures else 0


if __name__ == "__main__":
    args = sys.argv[1:]
    sys.exit(main(Path(args[0]), Path(args[1]), Path(args[2]) if len(args) > 2 else None))

Capture one fixture per alert from a real non-incident delivery, redact anything sensitive, and commit it:

{
  "labels": {
    "alertname": "CheckoutErrorRateHigh",
    "service": "checkout",
    "namespace": "checkout-prod",
    "owner": "payments-oncall"
  }
}

python tools/render_runbook.py runbooks/checkout-error-rate.yaml fixtures/checkout.json

The class field is small but load-bearing: read commands render during a freeze, anything else is skipped with an explanatory line. That is the whole freeze rule — no fake clock, no simulation framework — just a renderer that will not hand you a mutating command while the freeze file says the account is locked. You still need a human decision about when the freeze ends; the script only refuses to pretend it does not exist.

Check 3: escalation owners must resolve

A runbook that pages the wrong rotation is worse than no runbook, because it manufactures the belief that someone is coming. Keep an owners.yaml next to your runbooks and fail when a fixture references an owner that is not in it.

#!/usr/bin/env bash
set -euo pipefail
missing=0
for owner in $(jq -r '.labels.owner' fixtures/*.json | sort -u); do
  if ! grep -qx -- "- ${owner}" owners.yaml; then
    echo "unroutable owner: ${owner}" >&2
    missing=1
  fi
done
exit "${missing}"

Check	What it proves	Fails the build when	Cost
`promtool` rule test	The alert emits the promised label set	A label or annotation is renamed or dropped	Minutes per alert
Render pass	The runbook consumes labels safely	A template interpolation is empty	Minutes per command
Owner resolution	Escalation reaches a real rotation	An owner alias is stale or misspelled	One script
Freeze flag	No mutating command appears mid-freeze	`class` is missing on a command	One if-statement

Where the check runs when nobody's laptop is awake

None of this needs a laptop. The whole job is a shell script, a Python file, and a promtool binary, so it fits on almost anything that can run a scheduled container. One option I have been pointing people at is MonkeyCode's free server option, which is enough for a nightly render of a handful of runbooks, paired with its free model access for the tedious first pass of drafting fixtures and candidate first commands.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use the model for the boring part only — proposing label keys and a first read-only command per alert — then review every line by hand before it lands in a repo. Free tiers and quotas change often, so check the current limits on the project's own page instead of trusting any number quoted in a blog post, including this one. And never let a generated command with class: mutate merge without a named human reviewer attached to the PR.

Who should skip this

Teams with fewer than five alerts. A checklist in your head is cheaper than a renderer, and honesty about scale beats cargo-culting a pipeline.
Stacks where rules are not machine-readable. If alerts live in a GUI and cannot be exported, start there; this contract needs a file to test.
Anyone without an owner file. Check 3 is the one that keeps people safe, and it is also the one that requires organisational agreement, not code.
Runbooks with no commands. If yours is prose only, fix that first; a renderer over zero commands is a very well-tested no-op.

What I would do this week

Pick the alert that scared you most recently and save one real payload as a fixture.
Add class: read or class: mutate to each existing command, and delete any command you cannot classify.
Run the renderer locally and fix every empty interpolation it reports.
Wire the three checks into CI and make the nightly job fail loudly, not silently.

If a label rename can break your first five commands, how confident are you that the next refactor will be the polite kind? Start with one alert and one fixture, and let the runbook prove it still works before the pager has to ask.

Your Escalation Ladder Belongs in a Table, Not in Prose

Emery Huang — Mon, 19 Oct 2026 08:27:59 +0000

Your escalation ladder should be a decision table that a test suite can replay, not four paragraphs a tired engineer has to interpret at 03:00. That single change turns your on-call runbook from a document people read into an artifact your CI can disagree with. It also gives you something far more valuable than confidence: a failing test when the freeze rule and the first commands drift apart.

Below is the schema, the router, and the test cases I would ship first. The code is an illustrative blueprint rather than a benchmarked system, so treat it as a starting shape and adapt the field names to your own paging stack.

Why prose ladders break at exactly the wrong moment

A paragraph like "escalate to the platform team if the error rate stays high" hides at least four decisions inside one sentence. Who counts as the platform team at 03:00 on a holiday? How high is high, and measured over what window? Does the freeze window change the answer? Has anyone confirmed the freeze is actually cleared?

Ask five engineers on your rotation to answer those questions from the same page, and you will usually get three different answers. Is that a training problem, or is it a representation problem? I think it is mostly representation: prose is a lossy encoding for a decision, and on-call is where lossy encodings get expensive.

A decision table fixes this by making each row answerable. Every row names the alert, the severity band, the read-only commands to run first, the owning rotation, the escalation timer, and the freeze behaviour.

The artifact: one YAML table that owns escalation and freeze

Keep the table in the same repository as your alert rules, so a rename in one place forces a change in the other. The schema below is deliberately small; each field exists because some incident taught someone that it was missing.

# runbook/escalation.yaml — illustrative schema, adapt field names
version: 1
defaults:
  max_first_command_seconds: 30
  escalate_after_minutes: 10
routes:
  - id: api-5xx-spike
    match:
      alert: ApiHighErrorRate
      severity: [critical, warning]
    first_commands:
      - kubectl get pods -n api -o wide
      - kubectl logs -n api -l app=api --tail=200
      - aws cloudwatch get-metric-statistics --namespace AWS/ALB \
          --metric-name HTTPCode_Target_5XX_Count --period 60 --statistics Sum \
          --start-time 2026-10-19T00:00:00Z --end-time 2026-10-19T00:30:00Z
    owner: api-oncall
    escalate_after_minutes: 10
    freeze:
      applies: true
      during: [change_window, month_end_billing_freeze]
      unfreeze_requires:
        - two read receipts from api-oncall
        - page acknowledged by incident commander
    handoff_fields: [customer_impact, last_deploy_sha, error_budget_burn]

Three rules keep this table honest over time. First, first_commands must contain only read-only verbs, because the first five minutes are for gathering evidence. Second, any mutating step lives in a separate list that requires explicit clearance. Third, handoff_fields is mandatory, since an escalation without a handoff packet just moves the confusion to another rotation.

A router that refuses writes while a freeze is active

The router is the only thing anyone runs during an incident, so it stays boring and short. It resolves the row, filters the commands, and raises an error instead of guessing when the freeze evidence is incomplete.

# runbook/router.py — illustrative, not executed for this post
from __future__ import annotations

import shlex
from dataclasses import dataclass

import yaml

MUTATING_VERBS = {
    "apply", "create", "delete", "patch", "replace", "scale",
    "rollout", "drain", "cordon", "uncordon", "set", "exec",
}


class FreezeNotCleared(RuntimeError):
    """Raised when a write-class step is requested without freeze clearance."""


@dataclass(frozen=True)
class Plan:
    route_id: str
    commands: tuple[str, ...]
    owner: str
    escalate_after_minutes: int
    frozen: bool
    handoff_fields: tuple[str, ...]


def load_runbook(path: str) -> dict:
    with open(path, encoding="utf-8") as fh:
        return yaml.safe_load(fh)


def is_read_only(command: str) -> bool:
    tokens = shlex.split(command)
    if len(tokens) < 2:
        return False
    return not any(token in MUTATING_VERBS for token in tokens[1:3])


def plan_for(runbook: dict, alert: str, severity: str,
             freeze_active: bool, evidence: list[str] | None = None) -> Plan:
    route = next((r for r in runbook["routes"]
                  if r["match"]["alert"] == alert
                  and severity in r["match"]["severity"]), None)
    if route is None:
        raise LookupError(f"no route for {alert}/{severity}: page the platform owner")

    offenders = [c for c in route["first_commands"] if not is_read_only(c)]
    if offenders:
        raise FreezeNotCleared(f"{route['id']}: write verb in first_commands: {offenders}")

    freeze_rule = route.get("freeze") or {}
    frozen = bool(freeze_rule.get("applies")) and freeze_active
    if frozen:
        missing = set(freeze_rule.get("unfreeze_requires", [])) - set(evidence or [])
        if missing:
            return Plan(route["id"], tuple(route["first_commands"]), route["owner"],
                        route["escalate_after_minutes"], True,
                        tuple(route.get("handoff_fields", [])))

    return Plan(route["id"], tuple(route["first_commands"]), route["owner"],
                route["escalate_after_minutes"], frozen,
                tuple(route.get("handoff_fields", [])))

Notice what the router does with an unknown alert: it raises LookupError and names a default owner. A silent empty plan is worse than an exception, because an empty plan looks like calm.

Table-driven tests: incidents as data rows

A rehearsal only proves something if it can fail. Encode the questions your rotation argues about as parameterised rows, then let the suite answer them consistently on every pull request.

# tests/test_runbook.py — illustrative
import pytest

from runbook.router import FreezeNotCleared, is_read_only, load_runbook, plan_for

RUNBOOK = load_runbook("runbook/escalation.yaml")
EVIDENCE = ["two read receipts from api-oncall",
            "page acknowledged by incident commander"]


@pytest.mark.parametrize(
    "alert,severity,freeze_active,evidence,expected_route,expected_frozen",
    [
        ("ApiHighErrorRate", "critical", False, [], "api-5xx-spike", False),
        ("ApiHighErrorRate", "critical", True, [], "api-5xx-spike", True),
        ("ApiHighErrorRate", "critical", True, EVIDENCE, "api-5xx-spike", False),
        ("DiskPressure", "warning", False, [], "node-disk", False),
    ],
)
def test_route_and_freeze(alert, severity, freeze_active, evidence,
                          expected_route, expected_frozen):
    plan = plan_for(RUNBOOK, alert, severity, freeze_active, evidence)
    assert plan.route_id == expected_route
    assert plan.frozen is expected_frozen


def test_first_commands_are_read_only():
    for route in RUNBOOK["routes"]:
        for command in route["first_commands"]:
            assert is_read_only(command), f"{route['id']}: {command}"


def test_every_route_has_handoff_fields():
    for route in RUNBOOK["routes"]:
        assert route.get("handoff_fields"), route["id"]


def test_unknown_alert_raises_instead_of_guessing():
    with pytest.raises(LookupError):
        plan_for(RUNBOOK, "SomethingNobodyNamed", "critical", False)

Run it with python -m pytest tests/test_runbook.py -q in CI, and add one more test that fails when an alert rule exists without a matching row. That coverage check is the part most teams skip, and it is the one that catches the alert nobody documented during the last incident.

Freeze and unfreeze, written as evidence rather than judgement

The freeze rule is the field people argue about most, so make it boring. A freeze that applies during change_window returns read-only commands only, and it names the exact evidence strings that lift it.

When the evidence is incomplete, the router still returns the diagnostic commands. That is intentional: gathering evidence during a freeze is allowed, and blocking it would push people to work around the tool. Unfreeze is a separate, explicit act with two named receipts, which means the audit trail lives in the incident channel rather than in someone's memory.

One caveat worth stating plainly: this table proves consistency, not correctness. If your severity bands are wrong, the tests will faithfully protect the wrong behaviour.

Where free model access and a free server fit

MonkeyCode is an open-source coding assistant that the team offers with free model access and a free server option, and both are relevant here in narrow ways. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Drafting candidate rows. Paste two or three sanitised incident summaries and ask for candidate routes entries, then rewrite every command yourself and check it against your own CLI before it lands in the table.
Generating the boring tests. The parametrised cases are mechanical, and a model is genuinely good at proposing edge rows such as severity mismatches or missing handoff fields.
Running the rehearsal harness. The free server option is enough to run pytest and a weekly cron job that re-validates the table, without paying for a VM that sits idle.

Two cautions. Do not let a model write commands that touch production without a human reading them line by line. And do not treat free-tier availability as permanent, since the terms as of October 2026 should be re-checked on the project page before you build a dependency on them. If the limits change, the YAML and the tests move with you; only the harness host needs replacing.

Limitations and who should skip this

This approach earns its keep when you have several services, a real rotation, and at least one freeze window per month. It earns a lot less elsewhere.

Single-service teams get little from a routing table with one row; a short checklist is cheaper.
Teams without versioned runbooks should write the runbook first and encode it second.
Heavily regulated environments may still require human approval for every write, in which case the router should only ever print plans.
Anyone expecting the tests to validate reality will be disappointed; they validate agreement between your alerts, commands, and freeze policy.

Ship it this week

Extract every alert name from your rules into a plain list.
Write one YAML row per alert, with read-only commands and an owner.
Add the three invariant tests before you add the router.
Wire the suite into CI and let it fail once on purpose.

What would your ladder look like if the next argument about escalation had to be settled by a failing test instead of a loud channel?

Let a Model Draft Your First Five On-Call Commands — Then Prove They Stay Read-Only

Emery Huang — Sun, 18 Oct 2026 07:09:12 +0000

The riskiest lines in a runbook are rarely the paragraphs; they are the first five commands, because one wrong flag turns a page into a second incident. I now treat that command block as untrusted input: a model may draft it, a linter must prove it is read-only, and a replay step must show it still matches the alert payload that fired. The whole loop is one YAML schema plus roughly 120 lines of Python, and you can wire it into CI before your next rotation.

Why the first commands carry the risk

Four failure modes show up in almost every public runbook review, and none of them are about missing knowledge.

A read-looking command with a write side effect. Something like curl -X POST https://api.internal/health/refresh returns JSON, reads like a probe, and quietly enqueues work at 03:00.
A leftover mitigation. Someone added a restart command mid-incident because it helped, and it never got demoted into the escalation section afterwards.
A pipe that executes. A formatting convenience ends in | sh or xargs ... kubectl delete pod, and now the runbook mutates state.
Stale flags. The command was correct for the previous API version, so it fails in a way that looks like a brand-new incident symptom.

So the fix is not better prose. The fix is a three-stage gate: draft, verify, replay. Anyone can do stage one; stages two and three are what make the runbook safe to hand to a tired human.

A runbook schema that separates drafting from execution

The schema matters more than the model, because the schema decides what a machine can check. Here is the version I would start from:

service: checkout-api
owner: payments-oncall

alerts:
  - name: CheckoutApiHighLatency
    signal: p99_latency_ms > 900 for 10m
    severity: page
  - name: CheckoutApiErrorBudgetBurn
    signal: burn_rate_1h > 14.4
    severity: page

first_commands: |
  date -u
  kubectl get pods -n checkout -l app=checkout-api -o wide
  kubectl logs -n checkout deploy/checkout-api --tail=80 --since=15m
  aws cloudwatch get-metric-statistics --namespace AWS/ELB \
    --metric-name TargetResponseTime --period 60 --statistics p99

escalation:
  - after: 10m
    if: error_rate still above 2%
    to: payments-oncall-secondary
  - after: 25m
    if: no mitigation applied
    to: payments-incident-commander

freeze:
  declared_by: incident-commander
  scope: [deploys, schema_migrations, feature_flags]
  max_duration: 60m
  unfreeze_requires:
    - two consecutive green 5xx SLO windows
    - follow_up_ticket_id present
    - secondary_oncall_ack within 15m

A few deliberate choices. first_commands is a single block string so it can be diffed and linted as one artifact, not scattered across prose. escalation is structured data with after, if and to, so an on-call can read it as a table instead of hunting for sentences. And freeze is data, which brings me to the part most runbooks get wrong.

The freeze rule should be a state, not a sentence

"Freeze deploys during incidents" is not a rule, it is a mood. Record a freeze as an object with a declarer, a scope, an expiry and explicit unfreeze criteria; then a five-line script can answer "are we frozen right now?" without asking a human. Every unfreeze_requires entry must be computable — a green SLO window, a ticket id, an ack timestamp. If a criterion cannot be computed, it is a wish, and wishes are what people argue about at 04:10.

The verifier: fifty lines that say no

The verifier is boring on purpose. It allowlists binaries and subcommands, denies write verbs outright, rejects pipes into shells, and fails closed on anything it does not recognise.

#!/usr/bin/env python3
"""runbook_lint.py - prove the first_commands block is read-only."""
import re
import shlex
import sys
from pathlib import Path

import yaml

ALLOWED_SUBCOMMANDS = {
    "kubectl": {"get", "describe", "logs", "top", "events", "explain", "version", "api-resources"},
    "aws": {"describe", "list", "get"},
    "gcloud": {"describe", "list", "get-iam-policy"},
}

PLAIN_READ_ONLY = {"date", "hostname", "uptime", "dig", "ss", "df", "free"}

DENY_TOKENS = (
    " delete ", " apply ", " patch ", " scale ", " rollout restart ", " rollout undo ",
    " exec ", " cordon ", " drain ", " taint ", " edit ",
)

DENY_SHELL = ("| sh", "| bash", "|sh", "|bash", "xargs ", "eval ", "> /", ">>")


def split_block(block: str):
    out = []
    for raw in block.strip().splitlines():
        line = raw.strip().lstrip("-").strip()
        if not line or line.startswith("#"):
            continue
        for part in re.split(r"&&|\|\|", line):
            if part.strip():
                out.append(part.strip())
    return out


def violations(command: str):
    hits = []
    padded = f" {command} "
    for token in DENY_TOKENS:
        if token in padded:
            hits.append(f"write verb {token.strip()!r}")
    for token in DENY_SHELL:
        if token in command:
            hits.append(f"pipe or redirect {token!r}")
    try:
        argv = shlex.split(command)
    except ValueError as exc:
        return hits + [f"unparsable command: {exc}"]
    if not argv:
        return hits
    binary = Path(argv[0]).name
    if binary in PLAIN_READ_ONLY:
        return hits
    allowed = ALLOWED_SUBCOMMANDS.get(binary)
    if allowed is None:
        return hits + [f"unknown binary {binary!r}: allowlist it or drop the command"]
    sub = argv[1] if len(argv) > 1 else ""
    if not any(sub.startswith(prefix) for prefix in allowed):
        hits.append(f"{binary} {sub!r} is not an allowlisted read-only subcommand")
    return hits


def main(path: str) -> int:
    doc = yaml.safe_load(Path(path).read_text())
    block = doc.get("first_commands", "")
    if not isinstance(block, str) or not block.strip():
        print(f"FAIL {path}: no first_commands block")
        return 1
    failures = 0
    for command in split_block(block):
        for hit in violations(command):
            failures += 1
            print(f"FAIL {command!r}: {hit}")
    print(f"{'PASS' if failures == 0 else 'FAIL'} {path}: {failures} violation(s)")
    return 0 if failures == 0 else 1


if __name__ == "__main__":
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "runbook.yaml"))

Run it locally, then run it over every runbook in CI:

python -m pip install pyyaml
python runbook_lint.py runbooks/checkout-api.yaml
for f in runbooks/*.yaml; do python runbook_lint.py "$f" || exit 1; done

The exit code is the contract. A failing lint blocks the merge, which means nobody can quietly add a mutation to the page path in a hurry.

Replay the block against a recorded payload

A command that lints clean can still be wrong for today's cluster, so add a replay stage with fixtures. Capture a sanitised alert payload plus the outputs of each read-only command, then assert that the runbook still produces the observations it claims to produce.

Store fixtures/CheckoutApiHighLatency/alert.json, pods.txt and logs.txt next to the runbook.
Run each command against a dry-run kubeconfig or a recorded namespace, never against production.
Assert that the fields your runbook reasons about — pods_ready, error_rate, p99_latency_ms — actually exist in the parsed output.
Assign an owner for fixture refresh, because a stale fixture is the same bug as a stale runbook.

Does this sound like a lot of ceremony for five commands? It is about the same effort as one postmortem action item, and it is reusable across every page you own.

Where free model access and a free server actually help

Two parts of this loop are tedious and low-judgement enough to delegate: drafting first_commands from an alert payload plus a service README, and summarising a long dump mid-incident into three observed facts.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project whose operator states that it offers free model access — currently advertised as a 10M-token free allowance — and a free server option, which is enough to run this loop end to end without a paid account on day one. Verify the current terms before you depend on any specific number, because free-tier numbers move, and treat every capability claim as something to re-check rather than something I can promise on their behalf.

Concretely, the free tier fits three jobs in this workflow:

Drafting. Send the alert name, the signal expression and the list of read-only tools your service exposes, and ask for a command block in the schema above.
Nightly lint. Run the linter over runbooks/*.yaml on the free server and open an issue when a command starts failing, so drift surfaces on a Tuesday instead of at 03:00.
Incident note summarising. Paste text you have already read; keep the process environment free of write credentials and long-lived cloud tokens.

The rule I would not bend: model output never reaches a human without passing the linter first. The value is a faster first draft, not an authority, and your on-call should still read each command before running it. If you want to rehearse the drafting half before wiring anything to production, the free model access and the free server option are enough to close the loop.

Decision table: what to delegate

Task	Give it to a model?	Gate before merge
Draft `first_commands` from payload + README	Yes	`runbook_lint.py` passes, human reads every line
Choose escalation thresholds	No	SLO owner decides, runbook records the number
Summarise a 40-line output dump mid-incident	Yes	Read-only text in, no credentials in the environment
Rewrite `unfreeze_requires`	No	Incident commander owns the criteria
Generate a follow-up ticket skeleton	Yes	Human signs the ticket id into the runbook

Limitations and who should not use this

A deny-list is not a sandbox. The linter catches known write verbs and unknown binaries; it cannot understand a custom CLI where status means "apply pending changes". Read-only is also not the same as safe to paste — a get secrets output is read-only and still a credential leak, so keep that class of command out of the block entirely. And if your team has no review habit, this automation mostly produces confident-looking green checkmarks.

You should skip this approach if your runbooks have no machine-checkable command block, if your environment is air-gapped and cannot use a hosted server option, or if you need a formal safety guarantee rather than a guardrail. The technique is a speed bump sized for humans who are awake at the wrong hour.

A twenty-minute adoption checklist

Move the first five commands of one page into first_commands as a single block.
Copy runbook_lint.py, install PyYAML, and fix whatever it flags.
Convert the freeze sentence into a freeze object with an expiry and computable unfreeze criteria.
Add the lint call to CI and make it a required check on runbook changes.
Capture one fixture set from a real, sanitised alert and add a replay step.
Optionally, point a drafting prompt at the free model access and diff its output against your reviewed block.

What would your runbook look like if every command had to survive a linter before it could page anyone? Probably shorter, definitely more readable, and a lot less likely to become the next incident.

Your Runbook Has No Unfreeze Test: Rehearsing On-Call Freezes With a Fake Clock

Emery Huang — Sat, 17 Oct 2026 06:14:06 +0000

Every runbook I have inherited tests the entrance and forgets the exit. We rehearse the page, rehearse the first command, rehearse the escalation, and then we ship a freeze rule whose only validation is that a human remembered to lift it. So here is the conclusion up front: a freeze without an executable unfreeze step and a rehearsal that proves it runs is not a safety mechanism, it is a scheduled outage with a nicer name.

I have watched a read-only window outlive its incident by four hours because the ticket that created it had an owner and the ticket that closed it did not. The alert that page fired was correct, the customer impact was real, and the fix was a single annotation removal nobody was assigned to type. Once you have seen that, you stop trusting "the freeze expires on its own" as a control.

False freezes outlive incidents for three boring reasons

None of these are exotic. They are process gaps that a parser can catch in under a second, which is exactly why it is embarrassing that we catch them during a postmortem instead.

The freeze has an owner but the unfreeze has no assignee. The page names whoever pressed the button; nothing names whoever releases it, so release competes with sleep.
Expiry is treated as a command. A timestamp in a ticket is not an action, and anything that silently reverts at 09:30 during a deploy is a load test you did not schedule.
The exit path is undocumented, so it is improvised. The person who unfreezes at 3 a.m. is rarely the person who froze at 8 p.m., and reverse-engineering a kubectl invocation from bash history is not a runbook.

So the question I now ask of every runbook is narrow and rude: show me the test that fails when the unfreeze step is deleted. If you cannot produce it, the freeze rule is decoration.

The runbook contract I now require

I keep the document machine-readable next to the human prose, because a contract that cannot be parsed cannot be tested. Here is the shape I have been using for a checkout service; it is deliberately small, and it is the fixture my harness reads.

# runbook.yaml
service: checkout-api
alerts:
  - name: CheckoutP99LatencyHigh
    first_commands:
      - run: kubectl -n checkout top pods --sort-by=cpu | head -6
        timeout_s: 30
      - run: curl -s localhost:9090/metrics | grep checkout_queue_depth
        timeout_s: 15
    escalate_to: payments-oncall
    escalate_after_s: 600
freeze:
  id: checkout-writes
  owner: "@oncall-primary"
  expires_at: "2026-10-17T09:30:00Z"
  unfreeze:
    command: kubectl -n checkout annotate deploy/checkout-api freeze- --overwrite
    verify: kubectl -n checkout get deploy/checkout-api -o jsonpath='{.metadata.annotations.freeze}'
    expected: ""

Notice what is mandatory in that block and what is not. The owner, the expiry, an unfreeze command, and a verify-plus-expected pair are all required; a freeze with no way to observe its own removal is rejected, because "I ran the command" is a claim and an empty annotation is evidence.

The rehearsal harness: a fake clock plus an exit assertion

The harness does two jobs, and I care about the second one far more. First it lints the contract; then it replays a page against an injected clock and refuses to pass unless the unfreeze step actually executed inside the grace window.

# drill.py — stdlib + PyYAML only
import sys
from datetime import datetime, timedelta, timezone
import yaml

GRACE = timedelta(minutes=15)
MAX_ESCALATE = timedelta(minutes=15)


def parse_ts(value):
    return datetime.fromisoformat(str(value).replace("Z", "+00:00"))


def lint(doc, page_at):
    errs = []
    alerts = doc.get("alerts") or []
    if not alerts:
        errs.append("E_NO_ALERTS")
    ladder = []
    for alert in alerts:
        name = alert.get("name", "<unnamed>")
        cmds = alert.get("first_commands") or []
        if not cmds:
            errs.append(f"E_NO_FIRST_COMMAND:{name}")
        for i, cmd in enumerate(cmds):
            if not cmd.get("run"):
                errs.append(f"E_EMPTY_CMD:{name}#{i}")
            timeout = cmd.get("timeout_s")
            if not isinstance(timeout, int) or not 0 < timeout <= 120:
                errs.append(f"E_BAD_TIMEOUT:{name}#{i}")
        after = alert.get("escalate_after_s")
        if not isinstance(after, int) or not 0 < after <= MAX_ESCALATE.total_seconds():
            errs.append(f"E_ESCALATION_BUDGET:{name}")
        else:
            ladder.append(after)
    if ladder != sorted(ladder):
        errs.append("E_ESCALATION_NOT_MONOTONIC")

    freeze = doc.get("freeze")
    if freeze:
        if not freeze.get("owner"):
            errs.append("E_FREEZE_NO_OWNER")
        if not freeze.get("expires_at"):
            errs.append("E_FREEZE_NO_EXPIRY")
        unfreeze = freeze.get("unfreeze") or {}
        if not unfreeze.get("command"):
            errs.append("E_NO_UNFREEZE_COMMAND")
        if not unfreeze.get("verify") or "expected" not in unfreeze:
            errs.append("E_NO_UNFREEZE_VERIFY")
        else:
            if parse_ts(freeze["expires_at"]) <= page_at:
                errs.append("E_FREEZE_ALREADY_STALE_AT_PAGE")
    return errs


def rehearse(doc, page_at, unfreeze_at):
    errs = lint(doc, page_at)
    freeze = doc.get("freeze")
    if freeze:
        if unfreeze_at is None:
            errs.append("E_FREEZE_NEVER_LIFTED")
        else:
            deadline = parse_ts(freeze["expires_at"]) + GRACE
            if unfreeze_at > deadline:
                errs.append("E_UNFREEZE_LATE")
    return errs


if __name__ == "__main__":
    doc = yaml.safe_load(open(sys.argv[1]))
    page_at = parse_ts(sys.argv[2])
    unfreeze_at = parse_ts(sys.argv[3]) if len(sys.argv) > 3 else None
    failures = rehearse(doc, page_at, unfreeze_at)
    print("\n".join(failures) or "OK")
    sys.exit(1 if failures else 0)

Run it against a good runbook and a bad one; the difference should be exit codes, not opinions.

python drill.py runbook.yaml 2026-10-17T08:05:00Z 2026-10-17T09:12:00Z   # prints OK, exit 0
python drill.py runbook.yaml 2026-10-17T08:05:00Z                          # E_FREEZE_NEVER_LIFTED, exit 1
python drill.py stale-freeze.yaml 2026-10-17T10:00:00Z 2026-10-17T10:20:00Z # E_FREEZE_ALREADY_STALE_AT_PAGE, exit 1

In CI I do not run the third form against production state. The harness reads a checked-in fixture of the alert and freeze metadata, so the drill is deterministic and safe to run on every pull request; the only live call it makes is the read-only verify command, and I gate that behind a label.

The three fixtures that earn their keep

missing-unfreeze.yaml — freeze block with an owner and expiry but no command. Expects E_NO_UNFREEZE_COMMAND.
unfreeze-after-close.yaml — a realistic page where the incident closed at 08:52 but the annotation was removed at 11:40. Expects E_UNFREEZE_LATE.
silent-timeout.yaml — a first command with no timeout_s, which in practice means someone waits on a hung kubectl until the escalation budget is gone. Expects E_BAD_TIMEOUT.

Each fixture is a two-line diff someone can review in ten seconds, and each one maps to a failure I have actually seen. That is the bar for adding a new check.

Where a free model actually helped, and where it did not

Drafting first_commands from a bare alert name is tedious, so I tried generating candidates with a hosted model instead of typing them. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project's page advertises free model access and a free server option, and I used the free tier for exactly this step: propose three read-only commands per alert, then let the harness grade them. I am not going to quote model names, quotas, or hardware here, because those details change and my job is the workflow; check the project page for current terms before you plan around them.

What made the free tier usable was treating model output as untrusted input, not as a runbook. A second pass rejects anything that mutates state before a human ever sees it.

# quarantine.py — deny-first, allow-by-exception
DENY = ("delete", "scale", "rollout undo", "apply", "patch", "rm ", "drain", "cordon", "secrets")

def quarantine(candidates):
    safe, needs_human = [], []
    for cmd in candidates:
        line = cmd.get("run", "")
        if not cmd.get("timeout_s"):
            needs_human.append((line, "missing timeout"))
        elif any(token in line for token in DENY):
            needs_human.append((line, "write-class verb"))
        else:
            safe.append(cmd)
    return safe, needs_human

The failures were instructive and always the same shape: the model is happy to suggest kubectl rollout undo as a "first command", which is an action, not a diagnosis. It also invents metric names that look plausible. Both defects are caught by the timeout rule and the denylist, but only because I refused to let it write directly into the YAML.

I also used the free server option to host the drill runner as a tiny internal service that accepts a runbook path and returns the failure list. It holds no cluster credentials, reads fixtures from disk, and is disposable; if it disappears tomorrow I lose ten minutes, not an incident. If you want to try the same setup, the project page is the place to check what is currently free before you commit a weekend to it.

Limitations, and who should skip this

This harness checks the shape of your runbook and the ordering of your exit path. It cannot tell you whether kubectl top is the right first command, and it will happily pass a runbook full of confident nonsense. It also assumes you can express freeze state as something observable, which rules out teams whose lockdown lives in a human conversation.

Skip it if you are a two-person team without a paging rotation, because you will spend more time maintaining fixtures than you save. Skip it if your freezes are already enforced by infrastructure that audits its own removals. And skip it if you cannot yet name an owner for the unfreeze, because that is not a tooling problem and no test will fix it.

If you take one thing from this, take the failing assertion rather than the parser. Write down who lifts the freeze, how you observe that it is lifted, and how long after the incident closing that must happen. Then delete the unfreeze line and confirm your pipeline turns red. What does your runbook do today when someone removes that one line?

Your Runbook Drifted Since the Last Incident: Audit Alert Names, Permissions, and Step Intent in CI

Emery Huang — Fri, 16 Oct 2026 05:11:09 +0000

Your Runbook Drifted Since the Last Incident: Audit Alert Names, Permissions, and Step Intent in CI

A runbook rarely fails because the idea behind it was wrong; it fails because something around it moved while nobody was looking. The alert got renamed during a dashboard cleanup, the rotation lost a cluster role in a permissions refactor, or the prose says "restart the pod" while the command quietly deletes the deployment. All three of those are mechanical, testable, and cheap to catch before the pager goes off — so why do we still discover them from a laptop at 3 a.m.?

Here is the workflow I keep coming back to: give every runbook step a small machine-readable block, then run three passes over it — alert-name resolution, permission proof, and a semantic check that the command actually matches the stated intent. The first two are deterministic and run in CI, and the third only needs a throwaway model endpoint.

Why runbooks rot quietly

Review culture is optimized for the day the runbook is written, not the six months after it. Docs get review comments about clarity, tone, and ordering, but almost nobody re-reads them when the alerting rules change or when the platform team tightens RBAC. That is why drift accumulates silently and surfaces only under pressure.

There are three drift classes worth separating, because they need different detectors:

Renamed or deleted alerts. The runbook references CheckoutLatencyP99High, but the rule is now checkout_latency_p99_high or lives in a different rule file entirely.
Revoked permissions. The step was authored by someone with cluster-admin, so it "worked" once; the on-call rotation only has namespace-scoped patch rights.
Prose that no longer matches the command. Someone edited the command during an incident and updated the title loosely, so the step now claims one outcome and produces another.

Give every step a machine-readable block

If you keep runbook steps as free-form prose, no tool can help you without an NLP project. Pick a boring convention instead: one ### heading per step, followed by a fenced runbook block of YAML. Humans still read the prose around it; the auditor reads the block.

### Checkout latency page

The p99 latency alert fires when the checkout API is slow but still serving traffic.

```runbook
alert: CheckoutLatencyP99High
namespace: prod
command: kubectl -n prod rollout restart deploy/checkout-api
requires:
  - patch deployments
intent: restart the checkout API pods so a bad in-memory cache is rebuilt
read_only: false
```

If the restart does not clear the alert within ten minutes, escalate per the contacts section.

That is it. One block per step, four fields that matter, and a requires list written in the same verb resource shape that kubectl auth can-i already understands.

Pass 1 — every alert name must resolve

The cheapest check is also the one that catches the most embarrassing failures. Parse the runbook, collect the alert fields, then parse your Prometheus rule files and confirm each name exists.

# alerts/checkout.yml
groups:
  - name: checkout
    rules:
      - alert: CheckoutLatencyP99High
        expr: histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m]))) > 1.5
        for: 10m

Run promtool check rules alerts/checkout.yml first so you are comparing against syntactically valid rules, then compare the name sets yourself. A missing name is not a style nit; it means the person paged at 3 a.m. will search for an alert that no longer exists.

Pass 2 — prove the rotation can actually run the command

This is the check most teams skip, and it is the one that converts a runbook into something executable. kubectl auth can-i is a read-only self-check: it answers what the current identity may do, without changing anything. Run it with the rotation's credentials, not your admin kubeconfig — otherwise every probe returns yes and proves nothing.

# As the on-call identity, not as an admin:
kubectl auth can-i patch deployments -n prod
# yes  -> the step is runnable
# no   -> the runbook is fiction for anyone on rotation

Do the same for whatever the step actually touches. If a step shells into a bastion, run ssh -o BatchMode=yes bastion true and check the exit code. If it calls an internal API, hit a /healthz endpoint with the token the rotation holds. The point is to turn "we think this works" into a recorded yes or no.

Pass 3 — catch prose that lies about the command

Name resolution and permissions are binary, but intent mismatch is fuzzy, which is exactly where a model earns its keep. For each step, send the intent and the command and ask one narrow question: does this command plausibly achieve that intent? Keep the prompt boring and demand JSON, so the output is parseable rather than a paragraph of encouragement.

SEMANTIC_PROMPT = """You audit an on-call runbook step. Reply with JSON only.
{"mismatch": true|false, "reason": "<one sentence>"}
Set mismatch=true when the command does not plausibly achieve the stated intent."""

This is the AI usage I actually endorse during the current wave of AI fatigue: narrow, gated, reversible, and attached to a review step rather than a production action. It flags candidates; a human still deletes the step.

Where free model access fits — and the honest caveat

MonkeyCode is an open-source project that provides free model access and a free server option, and it advertises a free allotment in the range of ten million tokens. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I have not benchmarked its throughput or model quality for this workflow, and the quota figure is the project's own claim rather than something I measured.

What makes it a reasonable fit for pass 3 is the shape of the job: a handful of short requests per pull request, an endpoint behind an environment variable, and zero tolerance for a surprise bill. Point the auditor at any OpenAI-compatible base URL and the semantic pass gets cheaper to try.

export MC_BASE_URL="https://<your-endpoint>/v1"
export MC_MODEL="<your-model-id>"
export MC_API_KEY="<your-key>"
python tools/runbook_audit.py runbooks/checkout.md \
  --rules alerts/checkout.yml --semantic

The auditor, in one file

Here is the whole thing: parse the blocks, run the three passes, exit non-zero on drift.

#!/usr/bin/env python3
"""runbook_audit.py — catch runbook drift before the next page."""
import argparse, json, os, pathlib, re, shutil, subprocess, sys, urllib.request
import yaml

STEP_RE = re.compile(r"^### +(?P<title>.+)\n```

runbook\n(?P<body>.*?)\n

```", re.S | re.M)

def load_steps(runbook):
    steps = []
    for m in STEP_RE.finditer(pathlib.Path(runbook).read_text(encoding="utf-8")):
        step = yaml.safe_load(m.group("body")) or {}
        step["title"] = m.group("title").strip()
        steps.append(step)
    return steps

def alert_names(rules):
    names = set()
    for doc in yaml.safe_load_all(pathlib.Path(rules).read_text(encoding="utf-8")):
        for group in (doc or {}).get("groups", []):
            for rule in group.get("rules", []):
                if "alert" in rule:
                    names.add(rule["alert"])
    return names

def check_alerts(steps, known):
    return [f"[alert] {s['title']}: '{s.get('alert')}' not in rules"
            for s in steps if s.get("alert") not in known]

def check_grants(steps, kubectl="kubectl"):
    if not shutil.which(kubectl):
        return ["[grant] kubectl not found; pass skipped"]
    problems = []
    for s in steps:
        for spec in s.get("requires") or []:
            verb, _, resource = spec.partition(" ")
            cmd = [kubectl, "auth", "can-i", verb, resource]
            if s.get("namespace"):
                cmd += ["-n", s["namespace"]]
            out = subprocess.run(cmd, capture_output=True, text=True)
            if out.stdout.strip().lower() != "yes":
                problems.append(f"[grant] {s['title']}: cannot {verb} {resource}")
    return problems

SEMANTIC_PROMPT = """You audit an on-call runbook step. Reply with JSON only.
{"mismatch": true|false, "reason": "<one sentence>"}
Set mismatch=true when the command does not plausibly achieve the stated intent."""

def check_intent(steps, base_url, model, api_key):
    problems = []
    for s in steps:
        payload = {
            "model": model,
            "temperature": 0,
            "messages": [
                {"role": "system", "content": SEMANTIC_PROMPT},
                {"role": "user", "content": json.dumps(
                    {"intent": s.get("intent"), "command": s.get("command")})},
            ],
        }
        req = urllib.request.Request(
            f"{base_url.rstrip('/')}/chat/completions",
            data=json.dumps(payload).encode(),
            headers={"Content-Type": "application/json",
                     "Authorization": f"Bearer {api_key}"},
        )
        with urllib.request.urlopen(req, timeout=60) as resp:
            content = json.load(resp)["choices"][0]["message"]["content"]
        verdict = json.loads(re.search(r"\{.*\}", content, re.S).group(0))
        if verdict.get("mismatch"):
            problems.append(f"[intent] {s['title']}: {verdict.get('reason')}")
    return problems

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("runbook", type=pathlib.Path)
    ap.add_argument("--rules", type=pathlib.Path)
    ap.add_argument("--semantic", action="store_true")
    ap.add_argument("--base-url", default=os.environ.get("MC_BASE_URL"))
    ap.add_argument("--model", default=os.environ.get("MC_MODEL"))
    args = ap.parse_args()

    steps = load_steps(args.runbook)
    problems = check_grants(steps)
    if args.rules:
        problems += check_alerts(steps, alert_names(args.rules))
    if args.semantic:
        problems += check_intent(steps, args.base_url, args.model,
                                 os.environ.get("MC_API_KEY", ""))
    for p in problems:
        print(f"DRIFT {p}")
    print(f"{len(steps)} steps checked, {len(problems)} problems")
    return 1 if problems else 0

if __name__ == "__main__":
    sys.exit(main())

I am labeling this as a reference implementation, not a battle-tested product: I have not run it against your cluster, and pass 2 is only as trustworthy as the identity you run it with.

Wire it into the handoff, not just CI

name: runbook-audit
on:
  pull_request:
    paths: ["runbooks/**", "alerts/**"]
  schedule: [{cron: "0 6 * * 1"}]
jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install pyyaml
      - run: python tools/runbook_audit.py runbooks/checkout.md --rules alerts/checkout.yml

The scheduled run matters more than the pull-request run, because drift arrives from other teams' changes. Alerts get edited in a rules repository you don't own, and roles get tightened in a platform repo you never touch. A weekly audit catches both without asking anyone to remember.

What this audit will not catch

It cannot tell you the step is correct, only that it is consistent. A step whose alert exists, whose permission exists, and whose prose matches the command can still be bad advice.
It ignores judgment steps entirely. "Decide whether to fail over to the secondary region" has no command and no intent to compare, so passes 2 and 3 skip it silently.
The semantic pass has false positives. Treat its output as a review prompt, and start it as a warning rather than a blocking gate until you have tuned it against your own runbooks.
It says nothing about escalation contacts. Names resolve, permissions hold, and the escalation section can still point at someone who left the company.
It does not work if you run pass 2 as an admin. You will get green checks that mean nothing, which is worse than no check at all.

Who should skip this? Teams with two or three runbooks that the same two people maintain from memory will spend more time on the YAML convention than they save. If your pages are already handled by a small group whose runbooks live in their heads, fix the documentation problem first and come back when the rotation grows.

Limitations on the model endpoint

Before you enable --semantic, decide what leaves your network. Runbook steps often embed internal hostnames, ticket prefixes, and occasionally a token that someone pasted in a hurry. Send a redacted copy, or run the model on the free server option if your data policy allows a hosted endpoint at all. Also remember that a hosted model may be revised or retired without warning, so the semantic pass should be optional and never the only thing standing between you and a broken runbook.

Pre-handoff checklist

Every step has a runbook block with alert, command, intent, and requires.
promtool check rules passes on the rule files the runbook references.
kubectl auth can-i was run as the rotation identity, not as an admin.
The semantic pass ran at least once and produced a diff someone read.
The escalation section names people who are still on the team this quarter.
The audit runs weekly on a schedule, not only when the runbook is edited.

If you want to try pass 3 without wiring a paid key, point MC_BASE_URL at MonkeyCode's free server and see how many of your steps survive the intent check — the format and the first two passes are the part that pays off regardless.

Your On-Call Runbook Should Be Executable: Alerts, First Commands, and a Freeze Rule You Can Audit

Emery Huang — Thu, 15 Oct 2026 11:20:57 +0000

Most on-call runbooks fail for a boring reason: they are readable documents, not executable procedures. When the pager fires at 03:12, nobody has the attention budget to translate prose into commands and then argue about whether a deploy counts as a "write". If your runbook cannot be run, tested, and diffed, it is a wish list with headers.

So here is the version I keep coming back to: every page has three parts — a triage block, a first-commands block, and a freeze rule that is enforced in code rather than in a wiki page nobody re-reads. The goal is not to remove human judgement; it is to make the judgement reproducible and auditable.

Alerts: collect three fields before you touch anything

An alert without context is just an anxiety generator. Before any command, I want the alert fingerprint, the affected surface, and the change window. Write those three fields into the incident channel first — not as bureaucracy, but because they decide everything that follows.

Why the fingerprint? Because "checkout is slow" and "checkout returns 5xx on one route in one region" are different incidents with different owners, and the fingerprint is what stops two responders from debugging two different problems for forty minutes.

The first five commands

Keep this list short enough to memorize. Everything else is a follow-up, not a first move.

#!/usr/bin/env bash
set -euo pipefail

# 0. Fingerprint the alert before acting.
printf 'alert=%s surface=%s region=%s\n' "$ALERT_ID" "$SURFACE" "$REGION"

# 1. What is actually serving right now?
curl -sS --max-time 5 "$STATUS_URL/healthz" | jq '{status, version, build}'

# 2. What changed in the last 60 minutes?
git -C "$REPO" log --since='60 minutes ago' --pretty='%h %an %s'

# 3. One probe per dependency, capped and timed.
for dep in "$@"; do
  printf '%-30s ' "$dep"
  curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' --max-time 3 "$dep/healthz"
 done

# 4. Rate of failure, not raw count.
curl -sS "$METRICS/api/v1/query" --data-urlencode \
  'query=sum(rate(http_requests_total{code=~"5.."}[5m])) by (route)'

Notice what is missing: no restarts, no scaling, no cache flushes. Those are interventions, and interventions belong after the freeze rule has been evaluated, not before. Restarting first is how you destroy the evidence that would have told you what broke.

The freeze/unfreeze rule, written as code

This is the part most teams skip, and it is the part that prevents a small incident from becoming a long one. "Don't deploy during an incident" is not a rule; it is a sentiment. A rule names the owner, the ticket, the expiry, and the blocked action classes — and then it refuses writes that violate it.

A policy file you can diff

# freeze.yaml
freeze:
  active: true
  owner: "@emery-oncall"
  ticket: "INC-214"
  reason: "5xx rate above SLO on checkout"
  started_at: 1792000000      # epoch seconds
  expires_at: 1792005400      # 90 minutes; no silent forever-freezes
  blocked_actions:
    - deploy
    - migrate
    - config_write
    - secret_rotate
unfreeze_requires:
  - "error rate under SLO for 30 consecutive minutes"
  - "rollback file named in the handoff packet"
  - "second human acknowledgement"

The gate that enforces it

# freeze_gate.py — deny write-class actions while a freeze is active and unexpired.
import time
from pathlib import Path
import yaml

POLICY_PATH = Path("freeze.yaml")

class Denied(Exception):
    """Raised when an action is not permitted right now."""

def assert_writable(action: str, policy: dict, now: float | None = None) -> None:
    now = time.time() if now is None else now
    freeze = policy.get("freeze") or {}
    if not freeze.get("active"):
        return
    if action not in freeze.get("blocked_actions", []):
        return
    if not freeze.get("owner") or not freeze.get("ticket"):
        raise Denied("freeze has no owner or ticket; writes stay blocked")
    if float(freeze["expires_at"]) < now:
        raise Denied("freeze expired; renew explicitly or lift it, do not ignore it")
    raise Denied(f"frozen by {freeze['owner']} for {freeze['ticket']}")

def main() -> int:
    policy = yaml.safe_load(POLICY_PATH.read_text())
    try:
        assert_writable("deploy", policy)
    except Denied as exc:
        print(f"blocked: {exc}")
        return 1
    print("allowed")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Prove it with tests, not with hope

import time
import pytest
from freeze_gate import Denied, assert_writable

BASE = {
    "freeze": {
        "active": True,
        "owner": "@emery-oncall",
        "ticket": "INC-214",
        "expires_at": 1792005400,
        "blocked_actions": ["deploy", "migrate"],
    }
}

def test_blocks_deploy_during_freeze():
    with pytest.raises(Denied):
        assert_writable("deploy", BASE, now=1792000001)

def test_allows_unlisted_action():
    assert_writable("read_logs", BASE, now=1792000001) is None

def test_expired_freeze_does_not_silently_lift():
    with pytest.raises(Denied, match="renew"):
        assert_writable("deploy", BASE, now=1792009999)

def test_ownerless_freeze_is_invalid():
    policy = {"freeze": {**BASE["freeze"], "owner": ""}}
    with pytest.raises(Denied, match="owner"):
        assert_writable("deploy", policy, now=1792000001)

The expired-freeze test is the important one. A freeze that lapses on a timer and quietly re-enables deploys mid-incident is worse than no freeze at all, because now you have the illusion of a control.

Decision table: page, note, or wake someone up

Use this when you are unsure whether the incident justifies pulling a second person out of bed. It is deliberately blunt.

Signal	Action
SLO burn rate normal, one noisy alert	Note in channel, close after 15 min
5xx above SLO on one route, no customer report	Page primary, start triage block
Any data loss or corruption path	Wake secondary + data owner immediately
Write performed during an active freeze	Stop, revert, escalate to freeze owner
Unknown cause after 30 min of triage	Escalate; do not keep guessing alone

Three sentences of instruction beat a paragraph of philosophy, and a table beats both. The escalation ladder is short on purpose: primary → secondary → service owner → incident commander. Each hop has a time budget you agreed on before the page, not during it.

Where a free model and a free server actually fit

Here is the honest part. Two things in this workflow are tedious and low-judgement: rehearsing the runbook against historical alert text, and hosting the freeze state somewhere that is not a laptop that closes when the responder walks away.

That is where MonkeyCode's free model access and free server option come in for me. I use the free model access to dry-run the triage block against redacted past alerts and to flag steps in the runbook that contradict each other, and I keep the freeze gate plus the policy file on the free server so the state survives a closed lid.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes a free token allowance alongside the free model access and free server option; I have not benchmarked either, and the current allowance is whatever the project page says today, not a number you should hard-code into a design doc.

Two boundaries I hold firmly. The model never decides whether to escalate — that stays a human call with a timestamp. And I would not paste raw customer data or secrets into any rehearsal, free or paid, without checking the project's data handling first.

Limitations, and who should not use this

You do not have an SLO yet. A freeze rule without a target to unfreeze against is just a longer outage with extra steps.
You cannot name an owner per alert. The gate will block writes and nobody will be able to lift the freeze; fix the ownership map first.
Regulated or contractual data. Check data handling before routing any incident text through a hosted model.
You want the assistant to make the call. Don't. Use it for rehearsal and contradiction-hunting, not for escalation authority.

Also treat free tiers as free tiers: useful for exercises and low-stakes internal tooling, never as the only place your production paging path depends on.

A 30-minute drill that proves the runbook is real

Pick one real alert from the last 30 days and redact it.
Run the triage block out loud with a colleague and time each command.
Flip active: true in freeze.yaml, then attempt a deploy and confirm the gate blocks it.
Set expires_at to the past, confirm it still blocks, and confirm a human has to lift it explicitly.
Record the two steps that took longest and rewrite them as a single command.

If step five produces no changes, you either have a genuinely excellent runbook or you ran the drill in your head. If you want to try the gate end to end without provisioning anything, the two files above plus the free server option are enough to get a working rehearsal today.

Hold the Write Until Five Read Receipts Match the Escalation Ack

Emery Huang — Sun, 11 Oct 2026 21:24:48 +0000

I will not let a paging model choose the first command, because the alert class should already have made that choice. The only question that matters in minute one is whether five read-only receipts exist before anyone requests a write. If those receipts are missing, I keep the shell frozen and I escalate on a clock rather than on a feeling. Have you watched a confident paragraph turn a noisy page into a restart that nobody had actually approved?

The page may change attention, not authority

A page is a request for attention, and I refuse to treat it as a grant of write access on the paged host. I want that sentence in the incident channel before the bridge call collects opinions, because opinions arrive faster than evidence. The card in this article is a proposal you can adapt, not a diary of an outage I am pretending to have run last night. If your team already runs a stricter change gate, keep that gate, and steal only the receipt idea from this card.

Three constraints sit above the commands, and I repeat them whenever someone offers a shortcut around the card.

Intake may name an alert class, but intake may not mutate hosts, flags, or queues.
The first five commands are read-only, and the class picks them before a person or a model starts improvising.
Unfreeze covers one named command, and only after receipts plus an escalation ack exist in the channel.

Faster drafts did not retire the freeze

I keep seeing discussion, including this week, about models sounding smoother while the software around them stayed stubborn. I will not quote a leaderboard I have not opened, and I will not treat a headline as a primary source for your pager. The practical lesson I am willing to keep is narrower than the headline, and it is about authority rather than fluency. A cleaner paragraph does not earn a write on a sick host, even when the alert text was copied perfectly into the prompt.

If the draft is fluent and the receipts are still empty, the freeze still wins, and the escalation clock still runs. Why would smoother wording change the owner of a production shell? I want the answer written down before the bridge call turns confidence into a typed mutation.

Pick a class before you open a shell

I use three classes, because a single generic checklist hides the check that would actually explain this page. Class A means a customer-visible failure, or any credible risk that stored data was lost or corrupted. Class B means the error budget is burning, while nobody has confirmed an actual customer-facing break yet. Class C means a detector symptom that might belong to the detector itself, rather than to the service under the page.

Why would I start in a softer class when a wrong downgrade can hide a real customer break? Downgrading the page is itself a decision, and I would rather run one extra read than miss the checkout path. If I cannot defend the class in a single line, I stay on Class A until a human explicitly moves it. That bias is intentional, which is why the table below never shows an open write posture for any class.

Class	What I need to see	First goal of the reads	Write posture
A	Errors, timeouts, or data doubt	Bound impact and name a secondary	Frozen
B	Budget burn, no confirmed break	Confirm scope and the recent trend	Frozen
C	Detector-only, no user report	Separate detector fault from host fault	Frozen

The class changes the questions I ask, and it deliberately does not change the ban on mutation during intake. Would you unlock a restart for Class C merely because a draft called the alert simple? I would not, so the linter below treats every class as frozen until the receipts actually land.

Lint the first five commands on a laptop

I keep the allowlist in a script, so a tired paste cannot smuggle a write verb into the first minute of the page. The listing is unexecuted proposal code, and I would run it against a fixture on a laptop rather than on a production shell. It classifies one alert line, prints the matching card, and returns non-zero when a suggestion looks like a mutation. You should replace the paths before you trust it, because I have not executed this file on your hosts.

#!/usr/bin/env python3
"""Proposal: require read receipts before any unfreeze. Not a prod agent."""
import re, sys

WRITE = re.compile(
    r"\b(kill|delete|drop|apply|restart|scale|patch|exec|truncate|systemctl)\b",
    re.I,
)
CARDS = {
    "A": [
        "date -u",
        "hostname",
        "uptime",
        "curl -fsS --max-time 3 http://127.0.0.1/health || true",
        "tail -n 40 /var/log/app/app.log",
    ],
    "B": [
        "date -u",
        "uptime",
        "df -h",
        "ps -eo pid,pcpu,pmem,comm --sort=-pcpu | head",
        "tail -n 30 /var/log/app/app.log",
    ],
    "C": [
        "date -u",
        "hostname",
        "id",
        "journalctl -u app -n 40 --no-pager || true",
        "tail -n 20 /var/log/app/app.log",
    ],
}

def classify(text: str) -> str:
    t = text.lower()
    if any(k in t for k in ("data loss", "checkout", "5xx", "timeout")):
        return "A"
    if any(k in t for k in ("slo", "burn", "latency")):
        return "B"
    return "C"

def main() -> int:
    alert = sys.stdin.read().strip()
    kind = classify(alert)
    print(f"class={kind} posture=frozen receipts=0/5")
    for index, cmd in enumerate(CARDS[kind], start=1):
        if WRITE.search(cmd):
            print(f"reject card[{index}] {cmd}")
            return 2
        print(f"read[{index}] {cmd}")
    suggested = " ".join(sys.argv[1:])
    if suggested and WRITE.search(suggested):
        print("reject suggestion: write seen before receipts")
        return 3
    print("hold unfreeze until five receipts and an escalation ack")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

A fixture committed beside the script should stay short enough to reread on a phone during the page.

alert: checkout 5xx ratio above page threshold for 3 minutes

The local check is supposed to feel boring, which is what I want when the bridge call is already loud.

python3 receipt_card.py systemctl restart app < fixture.txt
echo "exit=$?"

An exit status of 3 means the suggestion tried to skip the receipt gate, so I discard that suggestion immediately. An exit status of 0 means I earned a reading list, and I still have not earned permission to write. After each read, I append a receipt with a tiny helper instead of trusting memory in the middle of the night.

#!/bin/sh
# Proposal: record one read receipt. Usage: ./note_receipt.sh A "date -u" 0
printf 'RECEIPT class=%s cmd="%s" at=%s exit=%s\n' \
  "$1" "$2" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$3"

RECEIPT class=A cmd="date -u" at=2026-10-12T03:12:01Z exit=0
RECEIPT class=A cmd="hostname" at=2026-10-12T03:12:04Z exit=0

Escalation runs on a clock, not a mood

I start the escalation clock when the class line is posted, because a vague status is not a policy anyone can audit the next morning. In this proposal, Class A pages the secondary at ten minutes, Class B at twenty, and Class C at thirty unless customer chatter appears. Those durations are planning defaults for a small rotation, not measurements taken from your pager data. Edit the numbers on the card before you adopt them, and do not cite this article as if it were a study.

The order of the names matters more than the speed of any generated summary you might paste.

Primary posts the class, the freeze, and the five-command card in the incident channel.
Primary names the secondary in that same message, even when the escalation timer has not fired yet.
If the timer fires without a secondary ack, primary pages them and still refuses writes.
If impact stays unclear at the next interval, primary pulls the service owner into the channel.

Could a model draft that acknowledgement faster than I can type the same four fields myself? Yes, and the speed helps only when class, freeze, secondary, and deadline all remain intact. If the draft omits the secondary, I do not send it, even when the surrounding prose sounds calm and complete. A missing name is an escalation failure, not a style problem I should polish after the page is already late.

Unfreeze is one human-typed command

Freeze means no write-class command and no temporary scale, including a one-liner that a chatbot offers as the obvious fix. Unfreeze is not a change of mood, and a red graph does not imply permission the channel never granted. I allow one command, typed by the human who holds the page, only after every gate below is already true.

Five read receipts are in the channel, and each receipt names the same class that intake recorded.
The secondary, or the service owner, has acknowledged that same class in a written channel message.
The command is absent from the read-only card, and a human typed it rather than pasting a model line untouched.
The release names an end time, and the posture returns to frozen when that time passes without a fresh line.

I keep the release on one grep-friendly line, so a later review can see exactly what the page allowed.

UNFREEZE class=A receipts=5/5 ack=lee cmd="service app reload" until=2026-10-12T03:40:00Z by=sam

If any field is missing, I stay frozen, even when the chart looks worse than it did during intake. A worse chart is a reason to escalate the page, not a reason to skip the receipts you already required. Would you sign a blank unfreeze because a draft said the reload looked safe from the alert text alone? I want that answer recorded as no, in the same channel that received the original page.

Draft the card away from production credentials

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I would draft and lint this card on a separate machine, and the operator says MonkeyCode offers free model access plus a free server option for that drafting work. I am not stating a token quota, a hardware shape, or an end date, because no primary source for those figures was attached here. If that option is closed when you read this, run the same linter on any laptop that cannot see production credentials.

What I would send is the alert sentence and the allowlist, with secrets left out of the prompt on purpose. What I would accept back is a class label or a rejection, never a live session on the host that was paged. If the draft inserts a write verb, the script's non-zero exit is the decision, and the drafting host does not get another vote. That split is the point of keeping generation away from the shell this runbook is trying to protect.

I am not using this section to rank tools, and I am not claiming the free option will still exist next quarter. Availability language here is operator-supplied, not a measurement I reran against a dated product page on the day of this draft.

Who should skip this card

This approach fits poorly when you lack an incident channel, a secondary, or a read-only check that stays safe on a sick host. It also fits poorly on safety-critical control gear, where my sample commands could be the wrong shape entirely. I am not claiming a shorter recovery, a better model score, or a replacement for the vendor runbook you already trust.

I keep the limits beside the suggestion, so a tired reader cannot skip them on the way to the code block.

The classifier is only a keyword stub, and novel alert names will miss until you extend those lists yourself.
Read-only is not automatically harmless, so the tail and curl examples stay capped, local, and easy to delete.
A free drafting host may be down, rate-limited, or unfit for secrets, so never paste credentials into the prompt.
The clocks above are proposals, not results from a published on-call study that you should quote as evidence.

If a command in the card does not exist on your hosts, delete it before the next page instead of discovering the gap live. I would rather you keep a shorter card than run an example that only matched an imaginary layout. The receipt rule still works after you swap in commands your platform team already trusts for diagnosis.

If the paper card has never been linted, run it once against a fixture before the next rotation, on a separate machine you control. A free drafting server is optional for that pass, and the freeze rule does not depend on which editor helped you type the card.

Block Write-Class Commands Until the Page Clock Names Owner and Expiry

Emery Huang — Sat, 10 Oct 2026 17:50:19 +0000

I will not let an assistant draft become a shell command until a page clock names the owner, the severity, and a hard expiry. The opening stretch of a page should stay in a read class, because an early write is only a guess with privilege. Have you ever changed a unit file while the alert title was still a blur on the second monitor? I keep that impulse behind a command class gate, and I open the gate only when a human can point at the clock.

What the page clock is allowed to decide

A page clock is a small file that records who is on the hook, how severe the page looks, and when the read window ends. It does not diagnose the outage, and it does not bless a clever patch that a model just printed into chat. I use it as a gate in front of write-class commands, so a draft can sit on scratch storage away from the paging host. Would you trust a suggestion that arrived before anyone was willing to name the severity out loud?

I would not, which is why the clock has to exist before that suggestion is eligible for a human review. The file lives on a scratch path, not beside the unit that woke you up in the first place. I treat the production shell as frozen until a human replaces a closed write lock with an explicit unfreeze. If the clock file is missing, I keep reading and I escalate, rather than improvising a fix from a chat pane.

The clock file I want beside the runbook

The sample below is a proposal for the runbook repository, and I have not executed it against a live fleet. Copy it only into a scratch directory, then replace every placeholder with values taken from the page you received. Leave production hostnames out of this file on purpose, because a clock should never become a quiet map of targets. I would rather see an empty field than a hostname copied from a suggestion that nobody reviewed.

# page-clock.yaml: proposed artifact, not a live incident record
page_id: "replace-with-alert-id"
owner: "replace-with-human-name"
severity: "unknown"
opened_at: "replace-with-utc-timestamp"
read_window_minutes: 15
write_lock: "closed"
escalation_at: "replace-with-utc-timestamp"
draft_state: "held"
rollback_intent: "unset"

First commands that stay in the read class

I start with commands that only observe, and I refuse anything that mutates packages, units, routes, or stored data. These examples assume a Linux host you already administer, and they stay proposals until your access policy allows the reads. I also keep a tiny wrapper that classifies a command string before I consider running that string for real. Have you noticed how a draft mixes a harmless status check with a restart on the next line?

Confirm the page identity in the clock file before you open a process list, because a mismatched alert wastes the window.
Reject the snapshot when owner or severity is still a placeholder, and spend that minute naming the fields instead.
Collect service state into the scratch directory, and do not pipe that output into a restart or a package tool.
Hold any command the classifier marks unknown, because an unknown verb is closer to a write than to a safe read.

# classify.sh: proposed read/write split for a scratch shell you control.
# Unknown commands fail closed. This example has not been run on a fleet.
set -eu
classify() {
  local cmd="$1"
  case "$cmd" in
    "systemctl status "*|"systemctl is-active "*|"journalctl "*|"ss "*|"ps "*|"cat "*|"ls "*)
      printf 'READ %s\n' "$cmd"
      ;;
    "systemctl restart "*|"systemctl stop "*|"apt "*|"dnf "*|"rm "*|"kubectl apply "*|"kubectl delete "*)
      printf 'WRITE blocked until the page clock opens: %s\n' "$cmd" >&2
      return 2
      ;;
    *)
      printf 'UNKNOWN hold for a human: %s\n' "$cmd" >&2
      return 3
      ;;
  esac
}
classify "journalctl -u demo.service --since 20 min ago"
classify "systemctl restart demo.service" || true

I do not smuggle a restart into this window, even when a draft sounds sure about a cause it invented. A restart can hide the evidence the next human needed, and a package change can widen the failure without rollback intent. If the service is already down, the snapshot still comes first, and the write class stays blocked. Have you restarted first and then lost the only log line that explained why the page fired?

How the gate decides an unfreeze

When the escalation time passes and the owner has not updated the clock, I stop polishing drafts and page the secondary. The secondary should inherit a closed lock and a held draft, not a half-applied change that nobody can describe. I would rather spend the read window twice than ship an unnamed edit into the same hour as the alert. If severity is still unknown at that moment, I widen the bridge instead of letting a model guess a rank.

The unfreeze sentence

The unfreeze rule is short enough to read aloud without scrolling through a long bridge novel. A human may open the write lock only after owner, severity, and rollback intent are real values you trust. An assistant may suggest wording for the rollback line, but it may not flip the lock field by itself. If your severe-page policy demands a second person, the gate stays shut until that second name is written down.

# gate.py: proposed local check, not a production agent.
# Exit 2 while the lock is closed or required fields are placeholders.
import sys

PLACEHOLDERS = {
    '',
    'unset',
    'unknown',
    'replace-with-alert-id',
    'replace-with-human-name',
}

def load(path):
    fields = {}
    for raw in open(path, encoding='utf-8'):
        line = raw.strip()
        if not line or line.startswith('#') or ':' not in line:
            continue
        key, val = line.split(':', 1)
        fields[key.strip()] = val.split('#', 1)[0].strip().strip('"')
    return fields

def main():
    fields = load('page-clock.yaml')
    problems = []
    for key in ('page_id', 'owner', 'rollback_intent'):
        value = fields.get(key, '')
        if value in PLACEHOLDERS or value.startswith('replace-'):
            problems.append(key)
    if fields.get('severity') not in ('sev1', 'sev2', 'sev3'):
        problems.append('severity')
    if fields.get('write_lock') != 'open':
        problems.append('write_lock')
    if problems:
        print('write lock stays closed:', ', '.join(problems))
        return 2
    if fields.get('draft_state') != 'reviewed':
        print('draft still held; human review required')
        return 3
    print('human gate passed; use your normal change process')
    return 0

if __name__ == '__main__':
    sys.exit(main())

Where a free draft bench fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I bring it up here because free model access and a free server can hold this scratch clock away from the production shell. I have no measured quota or hardware sheet, so I will not invent either figure or a permanence claim. The part I actually want is a clean split between a draft bench and the host that is paging.

Would I paste production credentials onto that free server just to save a copy step during the page? I would not, and the runbook should forbid that shortcut in the same paragraph as the product note. I would use the free model only after the read commands have already produced a snapshot I can stand behind on the bridge. It may tidy four short lines, and it may not invent the owner, the severity, or the rollback intent.

If you already have a MonkeyCode login, render this clock on the free server and carry the reviewed file back by hand. That is the only invitation I will make, because a runbook should stay useful when the product name is deleted. I still want the classifier and the gate even if you never open that draft bench at all. Does your bridge note still make sense after you remove every product word from the page?

A decision table for the first twenty minutes

I read this table before I accept help from anyone, including myself at minute ten when the alert still feels urgent. The rows are states of the clock, not moods, and the last column is the only next action I allow. If a row says the lock stays closed, I do not negotiate with a confident paragraph from a model. Have you ever talked yourself out of a table you wrote while you were still calm?

Clock state	Draft state	Next action I allow
File missing	Any text	Create the clock on scratch, run read-class commands only, and escalate if no owner can be named
Lock closed and severity unknown	Held	Keep observing, widen the bridge at escalation time, and do not apply a patch
Lock closed and severity named	Held	Let a human review the note, and keep production writes blocked
Lock open and draft reviewed	Reviewed	Follow the existing change process, with rollback intent still visible
Lock open and draft still held	Any text	Fail closed, because model text is not approval to write

A local test I can run without a fleet

I would test the gate on a laptop before anyone pastes it into a shared runbook, because an untested lock is only a comment. Create a temporary directory, copy the sample clock and gate script into it, and confirm a closed lock exits with status 2. Then write real-looking values, open the lock, mark the draft reviewed, and confirm the pass line prints on your machine. Set severity back to unknown afterward, and confirm the script refuses again, since that state is not a real unfreeze.

Exit codes worth trusting

I am not claiming this rehearsal caught a production incident, and I am not publishing a timing number from a borrowed machine. The only result I want is a predictable exit code in a directory you can delete when the check is done. If the closed lock exits zero, the gate is wrong, and it does not belong in the on-call document yet. Would you hand a pager a script that passes while the lock field still says closed?

# Proposed local test. Temporary directory only.
# Expect status 2, then status 0, then status 2 again.
tmpdir=$(mktemp -d)
cp page-clock.yaml gate.py "$tmpdir/"
cd "$tmpdir"
python3 gate.py || echo "closed lock failed closed, exit=$?"
python3 - <<'PY'
from pathlib import Path
path = Path("page-clock.yaml")
text = path.read_text(encoding="utf-8")
text = text.replace("replace-with-alert-id", "alert-1001")
text = text.replace("replace-with-human-name", "bridge-owner")
text = text.replace('severity: "unknown"', 'severity: "sev2"')
text = text.replace('write_lock: "closed"', 'write_lock: "open"')
text = text.replace('draft_state: "held"', 'draft_state: "reviewed"')
text = text.replace('rollback_intent: "unset"', 'rollback_intent: "revert the last config commit"')
path.write_text(text, encoding="utf-8")
PY
python3 gate.py
python3 - <<'PY'
from pathlib import Path
path = Path("page-clock.yaml")
text = path.read_text(encoding="utf-8")
text = text.replace('severity: "sev2"', 'severity: "unknown"')
path.write_text(text, encoding="utf-8")
PY
python3 gate.py || echo "unknown severity failed closed, exit=$?"

Who should skip this pattern

This pattern is a poor fit when your charter already authorizes a break-glass write, such as a documented failover that does not need a fresh clock. It is also a poor fit when you are alone on a severe page and the existing runbook already names the immediate action to take. Do not place secrets, customer payloads, or production kubeconfigs on a free server because the draft step feels convenient today. Do not use the gate to delay a human page, and do not treat model confidence as a fill-in for the owner.

Free model access can change, rate-limit, or vanish, so the runbook still has to make sense with the assistant powered off completely. If your organization forbids third-party tools on an incident bridge, skip the product path and keep the clock file plus the classifier. I would rather keep a boring read window than defend a fast draft that nobody on the bridge can explain. Are you adopting a gate you cannot test on a laptop this week, or are you only collecting another note?

What I leave in the bridge note

I end the read window with four lines, and I do not let the note grow into a novel while the lock is still closed. Those lines are the page id, the owner, the severity, and the next human action after the escalation time you already named. An assistant may tidy the grammar once those lines exist, but it may not invent them from a vague alert title alone. If you cannot fill those four lines, are you really ready to run a write-class command on the host?

A Three-Line Scratch Journal Is the Only Ticket Past the Suggestion Fence

Emery Huang — Fri, 09 Oct 2026 14:20:32 +0000

I keep every mitigation draft behind a fence until a three-line scratch journal exists on disk. That rule is the conclusion, and the rest of this note is only the machinery that makes the rule hard to skip. Would you rather read a smooth fix than the three boring lines that prove you touched the right host? I would not, because a smooth paragraph is cheap and a wrong restart on the wrong host is expensive.

I am labeling this whole workflow as an unexecuted proposal, not as a pager story from a shift I actually worked. Please copy the checker if it helps, and please do not treat my wording as evidence that a drill already passed. If your team already freezes production with a stronger control, keep that control and borrow only the journal habit.

Why I refuse the draft first

A draft that arrives before the journal is not a head start, because it is a contaminated input I did not ask for yet. I delete that file and I start the journal again, even when the sentences look calm and specific. Would you keep an early guess just because it happened to use the right service name in the first line? I would not, since the service name is often the only true fragment hiding inside an early draft.

I want the assistant fenced into a suggestion block that a human can throw away without touching a live host. Clever wording is not the goal, because clever wording is how a quiet page becomes a loud one. The fence is local, boring, and intentionally easy to audit with the two shell scripts below. If the scripts feel too small for your estate, that smallness is a feature I am willing to defend.

Three lanes before any shell

I choose a lane before I open a shell, and I write the lane name at the top of the journal. Observe means I collect read-only lines and I do not rehearse a change on any host. Rehearse means I may use a scratch host that is not production, and I still may not unfreeze anything. Escalate means I stop drafting immediately and I call the person already named in the runbook.

Lane names I refuse to blur

The checker cannot promote a lane, because promotion is a human decision I do not want hidden inside a script. If the lane is missing, I treat the page as escalate rather than inventing a policy while I am tired. How often do you discover the real severity only after you have already typed a fix into the shell? I still want the first choice written down, so the later correction has something honest to replace.

A table I can read without scrolling

I keep five rows in the table, because a table I cannot memorize will not survive a bad night. If a signal is not in the table, the lane is escalate and the freeze stays on. Would you rather debate a sixth row while the mitigation draft is already half written on disk? I would rather lose a minute to a short table than lose an hour to a guessed command.

Signal I can verify	Lane	First action	Freeze rule
Alert text only, no host I own	Escalate	Call the named secondary	Stay frozen
Synthetic fixture on a scratch host	Rehearse	Write the three-line journal	Stay frozen
Known gap, no customer impact I can show	Observe	Record the journal, do not edit	Stay frozen
Confirmed impact on a service I own	Escalate	Call, then journal if asked	Frozen until a human writes unfreeze
Draft file exists before any journal	Refuse	Delete the draft and start over	Stay frozen

The last row is the only row I actually enforce with a script rather than with memory. The other rows remain human judgment, and that judgment stays with the person who accepted the page. I do not want a model to pick the row, even when the alert text sounds confident and complete.

The only first commands I allow in the drill

I allow only three read-only commands, and I append them to journal.txt before I create suggestion.md. I do not point this drill at production, because production is not a classroom and a classroom is not a change window. The fixture can live on a scratch host, or on my laptop if I do not have a scratch host yet. Can you recite those three lines from memory without opening a wiki or a chat transcript?

If you cannot recite them, you are not ready to accept a mitigation draft from anyone else. Swap the commands if your estate uses different read-only checks, but do not raise the count and do not add a write. A fourth line is how a drill quietly becomes a change, and I do not want that slide. Keep the journal boring enough that a secondary can read it without asking what you meant.

#!/usr/bin/env bash
# Proposal only. Not executed against production in this article.
set -euo pipefail
umask 077
journal="${1:-journal.txt}"
: > "$journal"
{
  printf 'utc=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  printf 'host=%s\n' "$(hostname -s)"
  printf 'who=%s\n' "$(id -un)"
} >> "$journal"
echo "journal_lines=$(wc -l < "$journal")"

Those three lines prove time, place, and identity, and they prove nothing about the cause of the page. That emptiness is deliberate, because I want a witness that I was on the scratch host at all. I do not want a pretend root-cause essay generated before I have looked at anything myself. If your real first commands differ, swap them, but keep them read-only and keep the count at three.

The checker that holds the freeze

I run a second script against the journal file and against the suggestion file in the same directory. It fails when the journal has fewer than three non-empty lines, and it fails when the suggestion contains a mutating verb from a short deny-list. It fails closed, so a missing file is a refusal rather than a quiet pass you might miss. Would a short deny-list stop a determined person from pasting a dangerous command somewhere else entirely?

No, and I am not selling this little script as a security boundary against hostile users on your team. It is a tripwire for a tired first draft, not a sandbox and not a privilege boundary. Run it on files you already trust enough to open in an editor. If you need isolation, use the isolation your security team already approved.

#!/usr/bin/env bash
# Proposal only. Unexecuted example for a local drill.
set -euo pipefail
journal="${1:-journal.txt}"
suggestion="${2:-suggestion.md}"
test -f "$journal" || { echo "refuse: missing journal"; exit 2; }
test -f "$suggestion" || { echo "refuse: missing suggestion"; exit 2; }
lines=$(grep -c . "$journal" || true)
if [ "$lines" -lt 3 ]; then
  echo "refuse: journal has ${lines} lines"
  exit 2
fi
if grep -Eiq '(^|[^[:alpha:]])(rm|reboot|shutdown|systemctl|kubectl|terraform|git push)([^[:alpha:]]|$)' "$suggestion"; then
  echo "refuse: mutating verb in suggestion"
  exit 3
fi
echo "hold: journal cleared, freeze still on"

A deny-list is not a boundary

The sample deny-list also blocks read-only kubectl text, so treat it as a starting point rather than a law. Replace those words with the mutating verbs your own estate actually fears in a first draft. I would rather over-block a suggestion file than under-block a restart hidden in polite prose.

What hold is allowed to mean

Please read the success string out loud before you trust the script during a tired night shift. It says hold, and it does not say unfrozen, because the checker must not sound like permission. A human unfreeze is a separate line in the handoff, written by a named person after they have read the journal. The assistant may comment on that unfreeze line, and the assistant may not author it alone.

Escalation stays a phone call, not a paragraph

I name one secondary and one time box in the runbook, and I do not let the assistant choose either of them. If the lane is escalate, I call the secondary before I spend time polishing a suggestion block. If the lane is rehearse, I still call when the scratch journal disagrees with the alert text in a way I cannot explain. How long should you stare at a polished draft before you admit that you are actually stuck?

I use the time box already written down, and a blank time box means I escalate now. Freeze means no production change, no config push, and no restart that someone might later call temporary. Unfreeze means a named human writes the unfreeze line after reading the journal, not after reading a confident suggestion. I keep those verbs apart so a helpful rewrite cannot smuggle a release into a quiet edit.

If your handoff packet has no place for that line, add the place before you add another assistant prompt. I want the unfreeze line to name a person, a time, and the journal file they read. A missing name means the freeze continues, even if the suggestion file looks finished and calm. Would you unfreeze a service because a paragraph sounded sure of itself? I would not, not on a night when I am already behind.

Where a free scratch server is allowed to help

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I would use free model access and a free server option only as a rehearsal bench for this fence. The operator supplying this draft says both options are available, and I am treating that claim as supplied rather than tested. I am not stating a token quota, a model list, a hardware size, or any duration. This draft has no primary source for those figures, so inventing them would make the runbook dishonest.

If a current pricing or quota page is your source of truth, read that page before you plan a rotation around it. Would I send production traffic, secrets, or real alert payloads to a free rehearsal server at all? No, and you should not do that either, even when the signup flow makes the server feel disposable.

MonkeyCode shows up here only as a place where I can open a scratch host and ask for wording inside the suggestion file. It does not choose the lane, it does not clear the checker, and it does not write the unfreeze line. If the free option is gone tomorrow, the runbook still works on a laptop with the two scripts above. That fallback matters more to me than any invitation to try a product on a quiet afternoon.

If you want the same fence on a scratch host, start from the operator's current access note and stop where that note stops. I am leaving the product mention there, because a second pitch would not make the checker any stricter. Would another slogan make the hold string any safer when you are tired and the page is still frozen? I do not think so, and I would rather you spend that minute rereading the journal than rereading an ad.

Who should skip this fence

Do not use this fence when the page requires immediate physical safety action, such as cutting power or isolating a live hazard. A journal gate is the wrong first move when people can get hurt while you create a file. Do not use a local bash script as an audit artifact for a control framework you have not scoped with your security team. Do not use an external model when your incident policy forbids third-party tools, even on fixtures that might echo real names.

I also skip the model when the alert already contains secrets, customer payloads, or hostnames I am not allowed to paste into a draft box. I strip the fixture until it is boring, or I skip the model and keep the checker alone. The checker is the artifact I am willing to defend, and the model is optional help for wording. If you need a vendor to be on the hook for uptime, a free rehearsal bench is the wrong dependency for your real rotation.

A six-step drill before the next handoff

I would run this on a laptop, with fake alert text, before I ever mention it in a real channel. Nothing below has been executed for this article, so treat every exit code as something you still need to see yourself. If a step surprises you, stop and fix the script before you talk about it in a handoff. A drill you have not watched fail on purpose is not a drill yet.

Write a one-line fake alert into alert.txt, and keep real hostnames and customer data out of that file.
Create journal.txt with the three-line script, and confirm the printed line count is exactly three.
Create suggestion.md only after the journal exists, and keep every mutating verb out of that file.
Run the checker on both files, and confirm a clean suggestion prints hold rather than a release word.
Paste one denied verb on purpose, rerun the checker, and confirm the process exits with code 3.
Delete the drill files when you finish, so the next page cannot reuse a stale journal by accident.

If step 5 does not fail, the fence is theater and I would not carry it into a rotation. If step 4 prints anything that sounds like permission to change production, rewrite the success string before you trust the script. I am leaving both scripts unexecuted here so you can read them as proposals before you run them anywhere.

The conclusion I am willing to keep

A page gets safer when the first artifact is a journal, not a fix written in a hurry. The assistant can help me phrase a suggestion after that journal exists, and it cannot grant an unfreeze by itself. Free model access and a free scratch server can host the rehearsal inside the limits I already stated. If your runbook already names a human unfreeze and a read-only first move, you can ignore this note, and that is a fine outcome.

Echo the Alert Fingerprint Before an Assistant Draft Leaves Quarantine

Emery Huang — Thu, 08 Oct 2026 07:40:05 +0000

I keep every assistant draft in quarantine until a rehearsal host echoes the same alert fingerprint I copied from the page. A fluent explanation can still describe the wrong incident, the wrong cluster, or a page that already closed an hour ago. Would you really execute a mutating command that cannot name the exact page it claims to repair? I would not take that shortcut, and this runbook is the gate I use before any production change.

How I treat the page when it first lands

I read the alert as a payload first and as a story second, even when the paging app shows a friendly title. The friendly title helps a human get oriented, but it is not the identity of this page. The fingerprint is the identity I will later demand from the rehearsal host, character for character. If the paging app hides that field behind a collapsed panel, I expand it before I copy anything else.

I write this gate because a tidy summary can belong to a different page than the one sitting in my hand. The service name matches, the verbs sound familiar, and the timestamp quietly belongs to a page that already ended. Have you ever accepted a paragraph because the cluster label looked right and the rest felt too boring to check? I want the echo to come from a probe I can hash, not from another sentence that merely sounds careful.

A model can rearrange an old runbook into confident steps without ever seeing the payload that opened this page. It can also blend two alerts that share a service name and differ only in a label a tired reader skips. Would a second generated paragraph catch that mistake if it only saw the same pasted note? I do not treat a second paragraph as evidence, so the rehearsal host must repeat the fields before I trust the note.

Where the draft and the echo are allowed to live

I let the explanation live in a scratch note, and I let the echo live on a host that holds no production credentials. I use MonkeyCode's free model access for the scratch note and its free server option for the echo, and neither one is an approver. The model may rewrite that note, but it may not open a shell on the rehearsal host at all. The rehearsal host may print the fingerprint, but it may not hold a write token for the real cluster.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I need from those two options is a place to draft and a place to echo, not a promise about capacity. Would I paste a production kubeconfig into a chat window just to save a few minutes of typing? I would not, even when the server costs nothing and the chat feels private enough for a quick paste. A free seat does not convert a secret into something safe to share with a drafting tool.

If the rehearsal host can still reach production, I treat that host as production and I stop the drill. Isolation is a property I have to prove, not a feature I get to assume from a price of zero. Have you checked which credentials are already sitting in that host's environment before you call it safe? I stop the echo until that check is written down, because a memory of isolation is not isolation.

What I copy before I ask for an explanation

I copy four fields into the scratch note, and I refuse to add color before those fields are complete. If any field is missing, I escalate with the raw page instead of asking a model to guess the gap. Guessing a timestamp feels helpful in the moment and becomes a lie the moment someone audits the note. Would you want your name beside a startsAt value that a chat invented from the word recently?

Four fields, nothing else

alertname, so the echo cannot quietly drift toward a sibling alert that shares a similar title.
fingerprint, so two similar series cannot impersonate each other inside an otherwise convincing note.
startsAt, so a closed page cannot reuse a summary that was written for yesterday's incident.
generatorURL, so I can see which rule produced the page before I trust any explanation of it.

I leave annotations, suggested fixes, and dashboard screenshots out of that first copy on purpose. Those extras help later, but they tempt the draft to sound finished before any echo exists on disk. Have you noticed how a bright screenshot can make an unverified guess feel like a finished observation? I keep the screenshot in the ticket until the fingerprint match is already written under oncall-state.

First commands, and only these

The first commands stay on my laptop until a minimal page file exists and a hash is written beside it. I am labeling the snippets as a proposal you should adapt, not as a transcript from a cluster I measured. Paths, probes, and field names will differ on your side, and that difference is your job to resolve. What should you do if your alert payload uses different names for the same four ideas?

mkdir -p ./oncall-state
jq '{alertname,fingerprint,startsAt,generatorURL}' page.json | tee ./oncall-state/page-min.json
jq -e '.alertname and .fingerprint and .startsAt and .generatorURL' ./oncall-state/page-min.json
sha256sum ./oncall-state/page-min.json | tee ./oncall-state/page-min.sha256

On macOS I swap sha256sum for shasum with the 256 flag, and I keep the same output file name. If the field check fails, I do not open a model chat and I do not start a rehearsal session. I escalate with the raw page attached and the words fingerprint incomplete in the very first line. Would you ask a model to invent a startsAt value from a vague just now typed in chat?

The rehearsal echo is a read-only probe that prints the same four fields and then exits cleanly. I run that probe only after the local minimal file has a hash I can compare later by eye. A free server helps only when that environment cannot reach production credentials or the paging API. If I cannot prove that isolation, I skip the server and escalate with the local file alone.

I save the probe output as echo.json with a redirect, and I never let the model write that file for me. The redirect is boring, which is exactly what I want from the first commands on a live page. A boring command is easier to read aloud to a secondary who is only half awake. Can you explain a clever one-liner to that person without accidentally opening a second incident?

# Proposal: run on the rehearsal host, not on production.
# Replace the probe with your own read-only status command.
# Save that probe output yourself as ./echo.json first.
jq '{alertname,fingerprint,startsAt,generatorURL}' ./echo.json | tee ./oncall-state/echo-min.json

The comparison I refuse to skip

I keep the comparison in a script so a tired shift cannot decide that two strings mostly match. A partial match is how the wrong series survives a glance and later becomes a restart. This script is a proposal, and I am not claiming I ran it against your pager or scored its misses. Read it, change the paths, and throw it away if your payload cannot supply stable fields.

#!/usr/bin/env bash
# Proposal only. Do not point this at production credentials.
set -euo pipefail
PAGE_MIN="${1:?page-min.json}"
ECHO_MIN="${2:?echo-min.json}"
STATE_DIR="${3:-./oncall-state}"
mkdir -p "$STATE_DIR"

page_fp="$(jq -r '[.alertname,.fingerprint,.startsAt,.generatorURL] | join("|")' "$PAGE_MIN")"
echo_fp="$(jq -r '[.alertname,.fingerprint,.startsAt,.generatorURL] | join("|")' "$ECHO_MIN")"

if [[ "$page_fp" == *"null"* || "$echo_fp" == *"null"* || -z "$page_fp" ]]; then
  echo 'escalate: incomplete fingerprint' | tee "$STATE_DIR/decision.txt"
  echo 'frozen' > "$STATE_DIR/prod.state"
  exit 2
fi

if [[ "$page_fp" != "$echo_fp" ]]; then
  echo 'escalate: echo mismatch' | tee "$STATE_DIR/decision.txt"
  echo 'frozen' > "$STATE_DIR/prod.state"
  exit 3
fi

echo "match: $page_fp" | tee "$STATE_DIR/decision.txt"
echo 'echo-matched-still-frozen' > "$STATE_DIR/prod.state"

Read the state aloud

After the script exits, I read the state file aloud before I touch any other tool on the laptop. A nonzero exit means I stop drafting and I send the decision line to the secondary on call. A zero exit still leaves production frozen, which surprises people who treat a match as approval. Why would a matching string be enough to restart a service you have not otherwise checked?

bash ./fingerprint-gate.sh ./oncall-state/page-min.json ./oncall-state/echo-min.json
cat ./oncall-state/prod.state
cat ./oncall-state/decision.txt

A drill that should fail closed

This fixture should exit on a mismatch and leave the state file frozen, which is the result I want from a drill. If your copy prints a match, you swapped the fingerprints and the drill has not taught you anything yet. Run it locally before you point any probe at a rehearsal host, so a path bug does not look like an incident. Did the frozen state actually stop you from pasting the next command, or did you override it from habit?

mkdir -p ./oncall-state
jq -n --arg alertname LatencyHigh --arg fingerprint abc123 --arg startsAt 2026-10-08T09:00:00Z --arg generatorURL https://example.invalid/rule '{alertname:$alertname,fingerprint:$fingerprint,startsAt:$startsAt,generatorURL:$generatorURL}' > ./oncall-state/page-min.json
jq -n --arg alertname LatencyHigh --arg fingerprint def456 --arg startsAt 2026-10-08T09:00:00Z --arg generatorURL https://example.invalid/rule '{alertname:$alertname,fingerprint:$fingerprint,startsAt:$startsAt,generatorURL:$generatorURL}' > ./oncall-state/echo-min.json
bash ./fingerprint-gate.sh ./oncall-state/page-min.json ./oncall-state/echo-min.json || echo 'drill failed closed, as intended'
cat ./oncall-state/prod.state

Escalation when the echo will not match

I escalate on three results, and I do not ask the model to repair any of them in the chat. A mismatch means I am holding two payloads, not a wording problem the draft can smooth over. An incomplete page means the rule or the receiver dropped a field I refuse to invent tonight. A rehearsal host that offers a write-capable shell is the wrong host, even when the text looks perfect.

Send the secondary the two minimal JSON files and the decision line, not a generated essay about the page.
Say whether the failure is incomplete, mismatched, or a write-capable shell you should abandon immediately.
Stop drafting commands until a human owner names the next read-only probe you are allowed to run.
Keep the original page open so nobody summarizes away the fingerprint while the call is still live.

If the secondary is dark, I use the owner already printed in the service catalog and I wait for that person. I stay on the read-only probe until that person answers, even if the draft already shows a tempting command. I am not inventing a new rotation policy in this note, and I am not replacing your existing roster. I am only refusing to let a generated paragraph become the path that wakes the next person.

Freeze, then a separate unfreeze line

I keep production frozen while the state file is missing, reads frozen, or still reads echo-matched-still-frozen. A fingerprint match is not permission to change anything a user can already feel in production. A match only shows that the rehearsal host repeated the page I think I am holding right now. Would a matching string justify a restart by itself if you still have not checked capacity or scope?

Unfreeze is a separate line I append by hand after that human check, and the model does not write it. The rehearsal host does not write it either, because a host that echoes data should not grant change rights. I want a name, a UTC time, and the fingerprint in that line so a later review can see who moved. If those three pieces are missing, the freeze continues and the draft remains a note, not a plan.

# Human only. Replace YOUR_NAME. Do not pipe model output into this file.
echo "unfreeze YOUR_NAME 2026-10-08T09:30:00Z $(jq -r .fingerprint ./oncall-state/page-min.json)" >> ./oncall-state/unfreeze.log

A decision table I can scan at 3 a.m.

Echo result	State file	What I do next
Incomplete fields	frozen	Escalate with the raw page and stop drafting
Fingerprints differ	frozen	Escalate with both minimal files and stop drafting
Fields match	echo-matched-still-frozen	Human reviews scope, then may write the unfreeze line
Shell can write	frozen	Abandon that host and pick one without production credentials

I can scan that table without asking a model to reinterpret my own rule back to me. If the table and the script disagree, I trust the more restrictive result and I fix the script later. A 3 a.m. argument about which artifact is canonical is how freezes quietly evaporate on a tired team. Would you rather debate the table in the channel, or keep production still until a named human decides?

Limitations, and who should skip this

This gate catches identity mix-ups, and it does not tell you whether the underlying alert is a true page. A bad exporter can stamp the same fingerprint on two different failures, and the script will happily match them. I have no measured false-match rate to offer, so please do not quote this note as a control study. Would you accept a teaching script as proof that your paging pipeline is safe enough for Friday?

Free model access does not make the scratch note true, current, or allowed to approve a production change. A free server option does not prove isolation, network distance, or the absence of copied secrets on disk. You still have to check credentials yourself before the first probe, or the echo is just production with extra steps. If your organization forbids external models from seeing alert text, do not paste the page into one.

Skip this approach if the alert payload has no stable fingerprint, start time, and generating rule URL. Skip it if you cannot place the echo on a host that is sealed off from production credentials entirely. Skip it during a life-safety incident where the existing human runbook already names the immediate action to take. Skip it if you wanted an assistant to act as approver, because this note never grants that role to anyone.

What I still do with the draft

I still read the draft, because a clear sentence can help a tired person notice a label they missed. I just refuse to let that sentence travel until the echo matches and a human writes the unfreeze line. If you already have a free model seat and a free server you can truly isolate, try the gate on a drill. Watch whether a bad echo stops your hands before you ever trust the same habit on a live page.