DEV Community: Guangyi Zhao

Wiper-Service-Agent explained

Guangyi Zhao — Sun, 01 Nov 2026 21:44:10 +0000

Wiper-Service-Agent explained

Web-Vuln-Scanner explained in

Guangyi Zhao — Sun, 01 Nov 2026 21:43:44 +0000

Web-Vuln-Scanner explained in

Go-Diagnoser-Engine explained

Guangyi Zhao — Sun, 01 Nov 2026 21:43:20 +0000

Go-Diagnoser-Engine explained

Emotion-Guard explained in Eng

Guangyi Zhao — Sun, 01 Nov 2026 21:42:56 +0000

Emotion-Guard explained in Eng

Bloom-Counter explained in Eng

Guangyi Zhao — Sun, 01 Nov 2026 21:42:32 +0000

Bloom-Counter explained in Eng

AI-Node-Sentinel explained in

Guangyi Zhao — Sun, 01 Nov 2026 21:42:04 +0000

AI-Node-Sentinel explained in

SSH, systemd, kubectl: the one-page cheatsheets I keep open during incidents

Guangyi Zhao — Sun, 25 Oct 2026 22:24:48 +0000

I work on data center infrastructure. When something breaks at 2 a.m., nobody re-reads a 400-page manual — you reach for the handful of commands you half-remember and hope you get the flags right.

So I started writing one-page cheatsheets for the commands I keep looking up, and publishing them as open source. The first set covered everyday tools (git, docker, curl, jq, regex). This one is the infrastructure set: SSH, systemd, and kubectl — the three I actually keep open during incidents.

Every command below is copied verbatim from the cheatsheets, and every flag in them was checked against the real tools (OpenSSH 9.6 man pages, systemd 255, the official kubectl reference).

SSH: tunnels and jump hosts, not just login

Most people use SSH to log in. The parts that save me are the forwarding trio and ProxyJump.

Bring a remote port to your machine:

ssh -L 8080:localhost:80 deploy@example.com
# http://localhost:8080 on YOUR machine now hits port 80 on the server
ssh -L 5433:db.internal:5432 deploy@example.com
# ...or reach a third machine (db.internal) through the server you're logged into

Bake a bastion hop into ~/.ssh/config once, and every tool inherits it:

Host bastion
    HostName bastion.example.com
    User ec2-user
    IdentityFile ~/.ssh/id_ed25519

Host internal
    HostName 192.0.2.10
    User deploy
    ProxyJump bastion

ssh internal    # the hop happens automatically —
                # scp / sftp / rsync to "internal" get it for free too

And the smallest gotcha in the whole set: scp's port flag is capital -P; ssh uses lowercase -p. That mix-up is the #1 reason "scp works but ssh doesn't" (and vice versa).

Full sheet (28 entries: keys, config, forwarding, hardening): https://github.com/Ezra-Zhao/ssh-cheatsheet

systemd: read the journal like you mean it

The service commands everyone knows are start/stop/status. The ones that matter in an incident:

sudo systemctl restart nginx          # full stop + start; drops connections
sudo systemctl reload nginx           # re-read config WITHOUT restarting, if the unit supports it
sudo systemctl reload-or-restart nginx # reload if supported, otherwise restart — the safe default

After a crash or a hang, the previous boot's journal is where the answer lives:

journalctl -b         # this boot only
journalctl -b -1      # the PREVIOUS boot — where you look after a crash or a hang
journalctl --list-boots   # list recorded boots with IDs and time ranges, then pick with -b <id>

Filter by priority instead of scrolling:

journalctl -p err          # errors and worse (this boot)
journalctl -p warning -b -1  # warnings and worse, previous boot

And see the unit the way systemd actually sees it — base file plus every drop-in merged:

systemctl cat nginx   # base file + every drop-in merged, each piece labelled with its file path
# this is the ground truth — what you edited is not always what is running

Full sheet (30 entries: services, journal, unit files, timers): https://github.com/Ezra-Zhao/systemd-cheatsheet

kubectl: triage in the right order

My triage order is: describe first, events second, previous-container logs third.

kubectl describe pod web-6d9f7c5b4-x2abc   # why pending/crashing? read the Events at the bottom
kubectl describe node node-1               # capacity, allocatable, taints, conditions

kubectl get events --sort-by=.lastTimestamp        # OOMKilled, FailedScheduling, pull errors live here
kubectl get events -n prod --sort-by=.lastTimestamp

kubectl logs -p web-6d9f7c5b4-x2abc   # CrashLoopBackOff? this is where the real error is

(-p reads the previous, crashed container — the current one just shows you the restart loop.)

When a release goes wrong, the rollback is one command:

kubectl rollout history deployment/web        # list revisions
kubectl rollout undo deployment/web           # back to the previous revision
kubectl rollout undo deployment/web --to-revision=2

Full sheet (30 entries: status, deploy, logs, inside the container): https://github.com/Ezra-Zhao/kubectl-cheatsheet

The whole set

All fifteen one-page cheatsheets (git, docker, ffmpeg, curl, jq, regex, sql, ssh, kubectl, vim, tmux, systemd, and more) are MIT-licensed, each with a Chinese / Spanish / Portuguese README:

Index site: https://ezra-zhao.github.io/cheatsheet-pages/
First article in this series: https://dev.to/ezrazhao/9-one-page-cheatsheets-for-commands-you-keep-looking-up-ffmpeg-git-docker-curl-jq-regex-225d

If you find a wrong command or an outdated flag, a PR or a comment is the fastest way to fix it — the sheets only stay useful if they stay accurate.

Elder-Sentinel: Pose-Based Action Recognition for Elder-Care Safety Monitoring

Guangyi Zhao — Wed, 21 Oct 2026 14:11:55 +0000

Pose-based action recognition for elder-care safety monitoring.

A real-time pipeline that watches body movement — not identity — and flags safety-relevant events: help-seeking gestures, wandering into off-limits areas, prolonged inactivity that may signal a problem, and falls. Built for the realities of care homes, where caregivers can't be in every room at once, and where a fall detected minutes earlier can save a life.

Project status: scaffold v0.1 (honest edition).
The end-to-end pipeline runs today on procedurally generated, simulated skeleton sequences. What is real: pose data model, rule-based action detectors (hand-raise counting, loitering, fall alarm), event reporting.
What is still TODO (clearly marked in code): the field-validated action thresholds and the MediaPipe live-camera backend. This repo contains no face recognition and no identity matching — pose only.

Background

In a care home, caregivers can't watch every room at once. This system defines measurable action standards — e.g. "hand raised above shoulder N times in M seconds" as a call-for-help signal, "resident stationary near a stairwell or exit for T seconds", "sudden vertical drop consistent with a fall" — and raises alerts the moment a live pose stream matches one. The same architecture applies anywhere safety depends on body movement without identifying the person: elder-care fall detection, hospital ward monitoring, assisted-living wandering alerts.

What it does

Module	Input	Output
Pose estimation	camera frame (or simulated skeleton)	`SkeletonFrame` joint coordinates
Hand-raise counter	wrist/shoulder trajectories	count + timestamps per side
Loitering detector	body centroid trajectory	`loiter_alert` after dwell threshold
Fall detector	hip-height trajectory	`fall_alarm` (critical) on sudden drop
Event reporter	all detector outputs	console summary + JSON report

Architecture

                +------------------+
                |  Pose source     |
                |  MediaPipe cam   |  (TODO: live backend)
                |  Simulated seq   |  <-- demo runs on this
                +--------+---------+
                         | SkeletonFrame stream
          +--------------+--------------+
          |              |              |
   HandRaiseCounter LoiteringDetector FallDetector
   (config thresholds — TODO: field values)
          |              |              |
          +--------------+--------------+
                         v
                  EventReporter
                  console + report.json

Quickstart

cd Elder-Sentinel
pip install -r requirements.txt
python examples/demo.py        # simulated end-to-end demo
python -m pytest tests/ -q     # unit tests

The demo runs three simulated scenarios — hand raises, loitering, a fall — and writes examples/demo_report.json.

Configuration

All action thresholds live in config/action_thresholds.yaml. The shipped values are placeholders, not standards:

hand_raise:
  raise_margin: 0.05   # TODO(ezra): field-validated value

TODO(ezra): replace every threshold with values validated from your real deployments (raise margins, dwell times, drop ratios, zone geometry). The detectors are deliberately dumb rules — your experience is what makes them trustworthy. Do not present placeholder values as operational standards.

Live camera backend (TODO)

pose/mediapipe_backend.py is a stub. To go live:

pip install -r requirements-optional.txt (MediaPipe + OpenCV)
Implement MediaPipeBackend.estimate() mapping the 33 MediaPipe pose landmarks onto this repo's joint set, emitting SkeletonFrame(simulated=False).

Ethics & lawful use

Pose only. This repository contains no face recognition, no identity matching, and no person re-identification. It answers "what is the body doing", never "who is this".
Authorized safety use only. Intended solely for lawful, authorized safety deployments (e.g. licensed elder-care facilities, with family/guardian consent). Do not use for covert surveillance, and do not deploy without proper legal authorization and oversight.
Simulated data. Every demo frame is procedurally generated and labeled simulated: true. Nothing here was recorded from real people.

Roadmap

[ ] Field-validated thresholds (Ezra's deployment experience)
[ ] MediaPipe live backend + zone polygons on floor plans
[ ] Multi-person tracking (ID by track, not by face)
[ ] Alert routing (SMS/dashboard webhook)
[ ] Recurring-pattern analytics (e.g. agitation escalation over hours)

What a Second Model Actually Catches: Seven Articles, Forty Review Comments, Measured

Guangyi Zhao — Tue, 20 Oct 2026 01:52:32 +0000

Author: Guangyi Zhao

1. Problem and background

This article series runs under a standing rule: every draft gets reviewed by a different model before it can ship. The reviewer is told to hunt for factual errors, logical leaps, and overclaims; every comment must be resolved, one by one, before the piece is final. After seven days and seven articles, a fair question is whether the ritual earns its keep. What does the second model catch that a careful author re-read would not? And how does it fail when it fails?

No new apparatus is needed to answer that. The review records themselves are the dataset: every comment, every disposition, and every failed API call is on disk in the project's review/ directory. This piece does two things with that record. First, it codes all forty comments from the seven articles and counts what kinds of problems a cross-model review actually produces. Second, it runs a planted-error probe: a short passage with six defects buried in it, sent to the reviewer with the key withheld, to measure recall against a known denominator — something the historical record alone can never give you.

One clarification up front, because the title invites the wrong reading: of the forty comments, 29 came from the cross-model reviewer and 11 from the human three-role fallback used on the two nights the model never answered. The two sets are counted separately throughout. The headline result: the model reviewed five of seven articles and produced 29 comments, 27 of which changed the text. The other two were a false alarm of its own and a style note I kept anyway. What it catches is concentrated — not typos, but numbers that don't reconcile, causal claims that outrun their evidence, and conclusions stated without boundaries. The catch is availability: getting those five reviews took 13 API calls, of which 6 died with HTTP 503 and 2 with 429 quota exhaustion. It is a good reviewer who calls in sick a lot.

2. Method

Dataset A is the complete review log of the seven articles published October 13–18, 2026: seven review records, plus the raw reviewer receipts where they exist (JSON receipts for three successful reviews — Sentinel, Vuln-Scanner, Bloom; the Wiper receipt file was lost to a capture bug and its comments and usage figures survive in its record; the REA review survives as record text; two failure receipts document the nights the model never responded). The reviewing configuration was Gemini throughout — gemini-flash-latest as primary, with one article falling back to gemini-3.8-flash after two 503s. I coded each comment on two axes: category (numbers/definitions, causality/concepts, overclaim/generalization, scope qualification, method disclosure, structure/cross-references, style) and disposition (changed the text, verified as correct on re-check, rejected as reviewer error, already covered elsewhere, kept by author). Counting was done by a script, not by hand; the script and its JSON output ship with this article's data files.

Dataset B is the probe: a roughly 330-word excerpt about adding a cache to a diagnostics gateway, with six planted defects — a breakdown whose parts sum to 45 ms against a stated 41 ms total, a causal attribution asserted without control conditions, a boundless "no side effects, at any scale" conclusion, a stale cross-reference, a conclusion "supported" by an unverifiable internal newsletter metric that measures something else, and a success rate of 100% in the same breath as two timed-out failures. Only the passage went to the reviewer; the answer key did not. One design flaw disclosed in advance: the cross-reference defect carries an editorial note in the text admitting the numbering is stale, which half-cues that item. Discount it accordingly.

Also disclosed: the historical coding counts only the reviewer's comments. The single largest catch of the week — a draft that claimed "we tried this, it doesn't work" about an experiment that had never been run — was caught by the author's own pre-submission check, not by the reviewer. It is discussed in Section 4 but not credited to the reviewer.

3. Measurements

3.1 Availability: 13 calls, 5 reviews

The call log, article by article: REA failed once with 503, then succeeded on retry. Web-Vuln-Scanner failed twice with 503 and succeeded only after switching to the fallback model. AI-Node-Sentinel succeeded on the first call but a day late — drafted on the 14th, reviewed and published on the 15th. Wiper and Bloom-Counter succeeded first try. The remaining two never got through: Go-Diagnoser hit a 429 quota wall, and Emotion-Guard burned four calls in one evening (three 503s, one 429) before I gave up and ran the human fallback. Totals: 5 of 13 calls succeeded (38%); of the 8 failures, 6 were 503 and 2 were 429. Five of seven articles received a cross-model review; only four received it the same day. This is what a free-tier quota buys you: a 429 that says "come back in about 22 hours" and a 503 that means the provider is busy are both failures that retrying for a few minutes will not fix.

3.2 The probe: six planted, six caught, one unplanted

All six planted defects were identified explicitly. The arithmetic error came back with the sum worked out. The success-rate contradiction was quoted back in adjacent fragments. The causal claim was faulted for ignoring second-order effects — downstream relief, network and GC behavior — and for asserting complete attribution without controls. The overclaim was quoted word by word. The newsletter evidence was flagged, though filed under causality rather than sourcing: ticket volume moves with business cycles and rosters, so it cannot certify "stability." The cross-reference was caught too, subject to the half-cue disclosed in Section 2.

More interesting than the perfect score is the seventh finding, which I did not plant. The probe states an 87% cache hit rate and also books the full 19 ms origin fetch into the average per-request breakdown. The reviewer objected: if only 13% of requests reach the origin, that line should enter the average weighted (≈2.5 ms), not at full value — the metric definition contradicts the hit rate. On re-checking, the reviewer is right; the breakdown described a full request path while claiming to be an average. Planting defects, I had accidentally planted a real one. Final probe score: 6 of 6 on the key, plus 1 valid finding outside it.

3.3 What the 29 comments look like

By category: causality/concepts 6, overclaim/generalization 6, numbers/definitions 5, method disclosure 5, structure/cross-references 3, scope qualification 3, style 1. The top three families are 17 of 29 (59%).

Representative catches. Numbers: in the Bloom-Counter piece I had summarized an overexposure condition as essentially unaffected; the reviewer pointed out that its 15 false positives (precision 0.933) were bleached leaf fragments masquerading as buds — brightening manufactures targets while darkening loses them, two different failure modes my summary had flattened into one. In the Web-Vuln-Scanner piece, a target's 15 findings were itemized as 14 in my text; the reviewer counted. Causality: in Bloom again, my explanation of an over-count contradicted my own limitations section — adhesion blamed in one place, a "conservative estimator" claimed in another; the reviewer laid the two passages side by side. Overclaim: "not a bottleneck at any realistic scale" in the Sentinel piece was flagged as evidence-free extrapolation and cut down to the measured scale. The most instructive comment was method-related, in the Wiper piece: the reviewer said the rounding convention behind the quoted prices was undocumented, so a reader recomputing from the printed numbers would not balance. I re-ran all 8 records under both pricing branches — 16 combinations — and every total balanced to the cent. That comment's product was not a corrected number but a paragraph documenting the convention and the reconciliation.

Dispositions: 27 of 29 changed the article (including the Wiper comment, resolved by documenting the re-check). One was rejected: the reviewer called the Sentinel draft's date, 2026-10-14, a "future date typo." The draft was written on the 14th and reviewed on the 15th; the review prompt carried no current date, and the model's knowledge ends before 2026, so a true date read as future. Section 4 returns to this. One was kept: a style objection to a turn of phrase in the REA piece, noted for the record; I preserved the first-person register.

3.4 The fallback group: what human three-role review produces

The two fallback articles generated 11 comments: numbers 5, method 2, scope 2, overclaim 1, style 1. Their disposition profile is the mirror image of the model's: only 5 changed the text, while 5 were "verified correct on re-check" and 1 was already covered by the limitations section. The Emotion-Guard reviewer (also me, wearing a different hat) recomputed every probability in the draft — the teacher-spike figure, 1 − 0.95⁶ ≈ 26.5%, checks against the text's "more than a quarter" — and reconciled every number against the raw data file. That is audit confirmation, not fault-finding, and the reason is structural: in the fallback, reviewer and author share one brain. It is good at re-deriving its own arithmetic and poor at stepping outside its own framing. The Go-Diagnoser fallback did catch an unsupported attribution and one "tests pass" claim that had never actually been run — both are evidence-state problems, neither required an outside frame.

3.5 Cost

The direct cost is small. The four successful reviews with complete usage records: Sentinel 5,095 prompt tokens, Vuln-Scanner 5,365, Bloom 5,149, Wiper 24,372 (that receipt folds the model's internal thinking tokens into its totals, so it is not strictly comparable to the other three), with responses of 1,820, 1,764, 1,974, and 1,605 tokens — all under two thousand. The REA review's usage was not preserved, the one gap in the ledger. The probe cost 443 prompt and 1,009 response tokens. The real cost of the pipeline is not tokens but latency and uncertainty: retries against 503s, a model swap, a review that landed a day late, and two evenings of human fallback labor.

4. Analysis

Four judgments survive the data.

First, the reviewer's highest-value skill is reconciliation, not judgment. Roughly ten of the 29 comments turn on numbers, definitions, or count completeness, and nearly all were right. Reconciliation needs no theory of the whole article — it needs the stamina to collect every instance of the same quantity scattered across sections and compare them. That is exactly where a single author fatigues: the same figure appears in an abstract, a body paragraph, and a table, gets edited in one place, and drifts in the other two.

Second, it is more sensitive to unbounded language than its author is. Overclaim plus scope comments form the largest bloc at 9. The reviewer has no sunk cost in a pretty sentence. The probe reproduced the pattern in miniature: "no side effects" was quoted back and condemned in the same reflexive way "at any realistic scale" had been a few days earlier.

Third, its false alarms are context-shaped, not confabulation-shaped. The week's single rejection traces to a missing date in the prompt plus a knowledge cutoff that predates the draft's year. The defenses are procedural, not rhetorical: resolve every comment individually rather than accepting them wholesale, record rejections, and hand the reviewer the context it cannot guess. For this article's own review I injected the current date into the prompt; the failure mode did not recur — one observation, not a cure.

Fourth, the fallback path is part of the main path. Two of seven articles ended up there, so "what happens when the model doesn't answer" is not an edge case; it is a weekly event. The fallback produces a different substance (verification over reframing), and the ledger says so in plain language, so readers can discount the review strength of those two articles correctly. The one unforgivable version of this pipeline is a vague methods line that lets a fallback review masquerade as a cross-model one.

And the question from Section 1 — can author self-check substitute? Within this workflow — one author, unaided re-reads, no checklist scripts — the evidence says the two catch different species of error. The week's worst defect (a never-run experiment reported as run) was invisible to the reviewer: on the page it read perfectly. The reconciliation errors the reviewer caught had survived two of my own read-throughs. Bolting extraction scripts and checklists onto self-check would move that boundary; this dataset doesn't measure that configuration.

5. Limitations and open problems

First, the sample is seven articles, one reviewer family (one primary model, one fallback variant), one author, and one coder — me — with no second coder to check the category boundaries; the script guarantees arithmetic, not coding. Second, the probe is a single short passage with a defect density no real draft has, and one half-cued item; 6-of-6 says these defect types don't slip past it in a short text, and says little about recall in a 5,000-word article where defects hide deeper. Third, the fallback and model groups reviewed different drafts, so the contrast in Section 3.4 mixes reviewer effects with manuscript effects; it is not a controlled comparison. Fourth, the availability figures are bound to a free-tier quota and one particular week of provider load; 5-of-13 will move with a paid tier or a different week, and every conclusion here is scoped to this configuration. Fifth, nothing here measures the endpoint that ultimately matters — whether reviewed articles earn more readers or more trust. Comments changed texts; the rest of that chain has no data in this study.

6. References

Review records for the seven articles: review/2026-10-13-rea-gemini-review.md, review/2026-10-14-sentinel-review.md, review/2026-10-15-vuln-scanner-review.md, review/2026-10-16-wiper-review.md, review/2026-10-17-bloom-review.md, review/2026-10-18-emotion-guard-review.md; raw Gemini receipts in the same directory (*-gemini-raw.json).
This round's data: data/2026-10-19-cross-review-bench.json (script: data/2026-10-19-cross-review-bench.py, re-runnable); probe text, answer key, and reviewer output: data/2026-10-19-probe.txt, review/2026-10-19-probe-gemini-raw.json.
The pipeline's standing rule: the project's writing standard, quality-gate section (cross-model review and honest fallback labeling).

Author: Guangyi Zhao. The pipeline, records, and experiments described here are the author's own clean-room work; the experimental material consists of the series' own manuscripts and a synthetic probe passage. No client information is involved.

Letting an AI Agent Drive Ghidra: A Hands-On Test of REA’s 170 MCP Tools, One Recovered Password, and the Limits

Guangyi Zhao — Mon, 19 Oct 2026 05:05:58 +0000

Author: Guangyi Zhao
Date: 2026-10-13
Subject: morluto/rea (REA: Reverse Engineer Anything), npm package rea-agents 6.4.0, MIT license
Environment: Ubuntu 24.04 (Linux x64), Node 24.20; target is an ELF built from my own C source

1. Problem and background

Reverse-engineering an unfamiliar binary the traditional way means sitting in front of Ghidra, paging through assembly, following cross-references, guessing what each function does, and writing the conclusions down by hand. The slowness is not even the worst part. The worst part is how easily conclusions drift away from evidence: three days later I reread my own notes and cannot reconstruct what I actually saw when I wrote a given claim.

REA proposes a different division of labor. It does not implement decompilation itself. Instead, it standardizes the operating surface of reverse-engineering tools as MCP tools, so agents such as Claude Code, Codex, Cursor, or Gemini CLI can drive them directly: the agent asks the question, opens the binary, lists functions, pulls pseudocode, checks references, runs the target and diffs its behavior, then comes back with evidence attached. The human's job shifts from paging through assembly to asking the right questions and auditing the evidence chain.

Is that division real, or does it only demo well? I spent a day running it end to end, from install to password recovery, and I did not dodge the failures. Every number below comes from that day's logs and Evidence JSON files, and I re-checked each one against the raw files before finalizing this article. Anything I could not verify is either deleted or explicitly labeled as inference.

One ground rule up front: the only program analyzed here is one I wrote myself. The legal boundary is covered in Section 5, and it is not filler.

2. Method and architecture

REA is easiest to understand in four layers.

The first is the agent access layer. rea mcp starts an MCP server over stdio. I performed a real JSON-RPC handshake: initialize succeeded and tools/list returned 170 tools, with names like open_binary, analyze_function, batch_decompile, and inspect_binary_layout. npx rea-agents setup registers the server with the agent client you choose and installs an investigation-workflow skill that teaches the agent what to check first, what to check next, and how to follow up when the evidence is thin.

The second is the workflow layer, shared by the CLI and MCP. The CLI exposes a hundred-plus commands — analyze, function, decompile, xrefs, search for native analysis; capture-process for behavior capture; compare-process-captures for diffing two runs. The only difference is invocation: the CLI is a fresh process per call with results saved to disk, while MCP is a persistent session.

The third is the analysis-engine layer, and the most commonly misunderstood. Deep native analysis — decompiled pseudocode, function dossiers — is not implemented by REA. It bridges to one of three existing engines: Ghidra, through a Java bridge script (ReaGhidraBridge) running headless and talking over a local unix socket, requiring Ghidra 12.1.x; Hopper, through a Python bridge; and IDA, through IDA's own MCP registration, bring-your-own-license and bring-your-own-instance. REA is MIT-licensed, but that license grants you no commercial engine. IDA and Hopper still cost what they cost. Ghidra is free, which is why it is the route tested here.

A separate set of capabilities needs none of those engines: offline ELF layout parsing (through a Python environment you supply, pinned to pwntools 4.15.0, pyelftools 0.33, and Unicorn 2.1.2; read-only parsing of sections, segments, symbols, relocations, and mitigations, never executing the target), inspection of metadata embedded in Go binaries, purely static parsing of JavaScript and .NET files, and process behavior capture that actually runs the target under a PTY.

The fourth layer is the evidence layer, and in my judgment it is the real design center. Every call returns a structured Evidence record: an evidence_id, the target file's SHA-256, the provider and its version, the raw and normalized results, a confidence label such as observed or inferred, and an explicit list of limitations — what was not checked and what is inferred rather than seen. REA deliberately refuses to hand you a verdict. That is a different temperament from one-click report generators, and it is what makes automated follow-up questions defensible: every step can be traced back to specific evidence and open items.

Two further design notes. According to REA's own documentation, analysis runs entirely on your machine and target files are not uploaded to REA's servers. I did not verify that with packet capture, so treat it as a vendor claim, not a measurement — and note that what your agent does with tool results afterward is governed by your model provider's data policy, which is a separate question. The claimed target coverage is broad: beyond native binaries it lists Go, .NET, JavaScript/Electron, Android APKs, firmware, WASM, EVM bytecode, SQLite, pcap files, and crash dumps. This article tests exactly one of those lanes, native ELF, and draws no conclusions about the rest.

The target, checkme, is 30 lines of my own C: it takes a key on the command line, and check_key() first compares the length, then XORs the input byte-by-byte against a built-in constant and accumulates; only a zero sum passes. I built a symbolled and a stripped version. The correct key is 13 bytes. The question for the day: starting from "we do not know the answer," how far does this toolchain get, and at which step does a human — or an agent — still have to actually think?

3. Measurements

3.1 Readiness check: half the tool works out of the box

Installing rea-agents globally (version 6.4.0) took about 1 minute 51 seconds. rea doctor was refreshingly honest: Node and the host were healthy, but all three deep engines — Hopper, Ghidra, IDA — were missing, and the workflow skill was not installed. Out of the box you get only the built-in capabilities; deep decompilation requires bringing an engine. Doctor spelled out what was missing and how to fix each item, which already puts it ahead of tools that hand you a stack trace.

One piece of friction runs through everything: every rea command cold-starts in roughly 20 to 30 seconds, rea --help included. I did not profile where that time goes; my guess is Node loading the full tool catalog at startup, but that is a guess and I am labeling it as one. If you explore interactively at the rhythm you would use in Ghidra's GUI, this will wear on you. The tool rewards deciding your questions first and fetching evidence in batches.

3.2 Offline ELF layout: the manual jigsaw, done in one pass

I created a Python venv with pwntools — the versions landed exactly on REA's pins — pointed REA_PWNTOOLS_PYTHON at it, and ran rea inspect-binary-layout ./checkme --json. About 22 seconds later I had roughly 45 KB of Evidence JSON. I reopened that file while finalizing this article; the numbers are:

31 sections, 13 segments, 48 symbols, 11 relocations, with the file anchored by SHA-256;
mitigations matching the build reality item by item: PIE on, NX on, non-executable stack, Full RELRO, and no stack canary — correct, because the source has no local arrays. The whole set is labeled inferred rather than observed, which is the honest label;
a symbol table that names main, check_key, and strlen outright, so you know where to start;
12 limitations attached, including that link-time addresses are not runtime addresses and that relocation inventory completeness is unknown. Nothing swept under the rug.

This replaces the manual checksec-plus-readelf jigsaw with one structured pass. Repeating it on the stripped binary left only 9 dynamic symbols — the libc imports — with check_key and main gone. That is expected, and it draws the boundary of this lane precisely: layout only, no function recovery.

3.3 Behavior diffing: locate the divergence before reading code

Next, black-box capture. I ran rea capture-process once with a wrong key and once with the right one (about 30 seconds each, PTY startup included) and diffed them with rea compare-process-captures. The result names a first_divergence: terminal event 0, ACCESS DENIED on the left, ACCESS GRANTED on the right, with the exit dimension also marked changed. I checked the raw comparison JSON; its confidence is labeled derived — a conclusion computed from two captures rather than a single observation — which is the correct label.

On a toy target this looks trivial. The value is the ordering habit: confirm behaviorally which input decides success before you open the static analysis. Get the order right and you skip a lot of wandering later.

3.4 Ghidra deep analysis: it worked — the second time

First, the setup bill, because it is real: this machine had no Java. Temurin JDK 21 (198 MB, about 2 minutes) plus the Ghidra 12.1 release (542 MB, about 10 minutes), roughly 874 MB unpacked, plus GHIDRA_INSTALL_DIR and JAVA_HOME to set. Call it about 740 MB of downloads before the deep lane opens. Any honest evaluation has to count that.

Then rea analyze ./checkme --provider ghidra --json:

Attempt one: about 3 minutes 26 seconds, failed with error code cleanup_incomplete. Ghidra's own log showed import and post-analysis succeeding, but REA's teardown refused to confirm cleanup because the process tree contained orphaned or PID-reused processes, and failed the whole run. My inclination is that its process-ownership checking clashed with this containerized environment, but I did not root-cause it — container PID namespaces and process reaping were never individually ruled out — so this is recorded as an unlocalized failure, not an explained one. No workaround was applied and it is not reported as a success.
Attempt two: about 2 minutes 50 seconds, success — with a caveat. The overview's normalized result is only about 2.7 KB (the saved JSON file is about 4.4 KB): segment layout plus counts. It counted 30 procedures and 80 strings and gave pseudocode for none of them. Deep answers require naming the function you want.

So I named it: rea function ./checkme check_key --provider ghidra --json took about 2 minutes 42 seconds and returned a roughly 52 KB dossier with three things that matter. First, decompiled pseudocode matching the source logic line for line, with placeholder names like param_1 and DAT_00102004: compare strlen against 0xd — 13 — then XOR the input byte-by-byte against the constant at DAT_00102004 and accumulate; pass only if the sum is zero. Second, the full assembly listing: a 76-byte function body at 0x101189–0x1011d4, which I verified against the raw JSON. Third, the call graph edges: caller main at 0x1011d5, callee the strlen PLT thunk, with incoming reference addresses.

That closed the loop. Within this target's value range — password and constant are plain ASCII, so every XOR result lands between 0 and 127 — a zero sum is equivalent to byte-for-byte equality, so the 13 bytes at that address are the key; I read them out by file offset and confirmed. One precision worth adding: the pseudocode accumulates the XOR result truncated to a char, so with high-bit bytes, signed interpretation could in theory let several mismatches cancel out to zero. As a teaching target this check is fine; as a real password check it would be an exploitable weakness, and "sum equals zero" should never be mistaken for a cryptographic equality test. From opening an unfamiliar binary to recovering both the checking logic and the key, no human read any assembly. For completeness: with no engine configured, the same analyze command fails politely — a structured provider_unavailable error with remediation guidance instead of a crash.

3.5 The MCP channel, and the full timing table

rea mcp runs over stdio; the JSON-RPC initialize handshake succeeded, the server identifies as rea 6.4.0, and tools/list returned 170 tools. Scope discipline matters here: 170 is the catalog size. The tools I actually exercised and verified are a small subset — binary opening, layout inspection, behavior capture and comparison, Ghidra analysis, and the function dossier. The other 160-plus tools are unverified by this article, and a directory count must not be passed off as capability verification. Within the subset I used, CLI and MCP are two front doors to the same workflows.

Full run log, all from that day:

Shallow repo clone (56 MB): success, about 28 s
npm i -g rea-agents: success, about 1 min 51 s
rea doctor: success, three engines reported missing, about 30 s
Offline ELF layout, symbolled and stripped: both success, about 22 s each
Process capture twice plus comparison: success, first behavioral divergence pinpointed, about 30 s each
Analyze with no engine configured: failed cleanly with provider_unavailable, as expected, about 23 s
JDK 21 plus Ghidra 12.1 download and unpack: success, about 740 MB, about 14 min
Ghidra analyze, attempt one: failed, cleanup_incomplete (engine analysis succeeded, teardown verification failed), about 3 min 26 s
Ghidra analyze, attempt two: success, segment-level overview only, about 2 min 50 s
Ghidra function dossier for check_key: success, logic fully recovered, about 2 min 42 s
MCP initialize plus tools/list: success, 170 tools, about 23 s

4. Analysis and trade-offs

Where the division of labor genuinely holds: it splits the Ghidra learning curve into two stages. First you understand what a function does by questioning an agent that fetches evidence; then you learn the tool itself against the assembly and pseudocode you already understand. For anyone studying malware analysis or binary fundamentals, that order beats staring at an empty Ghidra window. Behavior diffing is the second habit worth stealing: behavior first, statics second is the same discipline as comparing normal and anomalous samples in dynamic analysis.

Four selection judgments for anyone evaluating this stack:

First, Ghidra versus IDA. REA does not change either engine's economics. Ghidra is free and headless-friendly; the price is the setup bill above and minute-scale calls. IDA's recovery quality and ecosystem are their own argument, but the license is yours to buy — REA's MIT license does not extend to it. My recommended order: teams that already hold IDA licenses should use that integration; everyone else should get the workflow running on Ghidra first.

Second, CLI versus MCP. For one-off scripts and auditable reruns, use the CLI; evidence on disk is naturally traceable. For multi-step investigations where an agent keeps asking follow-ups, use MCP. A persistent session should amortize repeated cold starts in theory — but I did not measure per-call latency inside a session, and the handshake alone took about 23 seconds, the same order as a single CLI call. So "MCP is faster" is an architectural expectation in this article, not a measured result; measure session latency yourself before choosing on speed. Also, a persistent session means evidence management depends on session semantics; do not mix the two modes' outputs in one folder and expect clean accounting.

Third, is the evidence contract worth its verbosity? Every result carrying limitations and a confidence label looks like overhead until you are the reviewer: you no longer have to guess which sentence was observed and which was inferred. In report-writing, delivery, or graded casework, that verbosity is net profit. For a one-off glance at a binary, it is pure cost. Know which mode you are in.

Fourth, and most important: every step that requires judgment was left to the caller, by design. Understanding the pseudocode's semantics — why a zero XOR sum means byte equality here, and when it would not — was the caller's work. Deciding to read the bytes at DAT_00102004 was the caller's work. Translating length 0xd into 13 characters was the caller's work. REA supplies evidence and open items; the conclusion belongs to whoever called it, and an agent that misreads pseudocode will produce a wrong conclusion that looks thoroughly evidenced. "Agent plus REA" does not mean unmanned reverse engineering. It means automated evidence collection with judgment still on the human side — and anyone who quietly drops the second half of that sentence will eventually be bitten by it.

5. Limitations and open questions

Measured limitations:

Cold starts are slow: 20 to 30 seconds per CLI command, and Ghidra-backed calls are measured in minutes. Function-level answers must be requested one function at a time, so cost grows roughly linearly with function count. I did not test full deep analysis of a large binary and will not extrapolate to one.
The analyze overview is shallow by design — layout and counts only. If you expect one command to hand you function-level conclusions, you will be disappointed; the design pushes you to arrive with a question and fetch named evidence.
Against stripped, packed, or obfuscated targets, the layout lane degrades to dynamic symbols, and the deep lane inherits whatever recovery ability the engine itself has. REA adds no magic. My target was an unobfuscated program I wrote, so this article has no standing to quote recovery rates on real-world samples.
The process-ownership check failed once on containerized Linux with cleanup_incomplete and passed on retry. Unattended batch runs need retries and failure logging budgeted in; do not assume a 100 percent first-pass success rate.
Only the native ELF lane was tested. APK, firmware, WASM, EVM, and the rest of the coverage list went unexercised, and unexercised means unjudged.
The ceiling on conclusion quality is the agent and the human reviewing it. Evidence chains can skip steps and pseudocode can be misread, and such errors arrive wearing Evidence IDs, which makes them look more credible, not less. That is the most dangerous failure mode in the whole setup.

Legal and compliance boundaries, which are not optional:

Analyze only software you own or are explicitly authorized to analyze: your own programs, CTF and lab targets, samples you purchased under terms that allow it, client deliverables with written authorization.
Unauthorized reverse engineering of commercial software can trip three wires at once: anti-reverse-engineering clauses in the EULA; DMCA Section 1201 anti-circumvention rules, where bypassing a technical protection measure can be unlawful in itself and legitimate purposes have only bounded exceptions; and unauthorized-access statutes. REA's own DISCLAIMER takes the same line: lawful research, education, interoperability, and software preservation.
Commercial engine licenses are yours to obtain legitimately, and trial-license restrictions on automation and sharing outputs apply in full.
Practical rule: learn on crackmes, CTF challenges, and programs you wrote — exactly what this article does. At work, with a client binary, confirm the authorization paperwork is in hand before you touch anything.

Open questions I am leaving for follow-up measurements rather than pre-spending conclusions: recovery-quality differences on the same target with the IDA provider, deep-lane behavior on a packed sample, and the context cost of accumulated evidence in long MCP sessions.

6. References and sources

REA repository and documentation: morluto/rea (MIT), npm package rea-agents 6.4.0; architecture descriptions checked against its README, docs, and the ReaGhidraBridge sources under bridge/ghidra.
Primary test materials: target source checkme.c (30 lines, my own), and the Evidence JSON files — layout_checkme.json (31 sections, 13 segments, 48 symbols, 11 relocations; provider pwntools 4.15.0 / pyelftools 0.33 / Unicorn 2.1.2), layout_stripped.json (9 dynamic symbols), capture_compare.json (first_divergence at terminal event 0), ghidra_func_checkkey.json (check_key dossier, Ghidra 12.1, 76-byte body at 0x101189–0x1011d4). Each was re-opened and re-checked before this article was finalized.
Engine versions: Ghidra 12.1 PUBLIC, Temurin JDK 21, Node 24.20, Ubuntu 24.04.
Legal reference: DMCA anti-circumvention provisions, 17 U.S.C. § 1201. Consult a lawyer for specific cases; this article is not legal advice.

Disclosure: All code, target programs, and experiments in this article are the author's own clean-room work. The experiments were run only against the author's own program. No client information, client code, or unauthorized software is involved in any way.
Author: Guangyi Zhao

“Can You Make It Cheaper?” — and the Agent Booked the Order: A State-Machine Field Test of Wiper-Service-Agent

Guangyi Zhao — Mon, 19 Oct 2026 05:03:42 +0000

Author: Guangyi Zhao
Date: 2026-10-16
Subject: Wiper-Service-Agent (my own open-source project, github.com/Ezra-Zhao/Wiper-Service-Agent), scaffold v0.2, commit 9675291, Python 3.12.3
Environment: Linux x86_64 VM, agent exercised in-process. No WhatsApp connection, no real messages sent, no real customer data. The fitment catalog, price table, and NLU are the repository's own simulated implementations, and the article says so wherever it matters.

1. Problem and background

The hard part of a WhatsApp service agent for a small shop is not wiring up a model. It is walking one conversation down a path where money changes hands: ask for the vehicle, look up the size, quote a price, answer questions, record an order intent. Misread the customer once and the cost is real — hear a haggle as an order and the shop holds an order nobody placed; hear a hesitant "I don't know" as a rejection and a nearly closed sale is shut down on the spot.

Wiper-Service-Agent is the agent I built for a wiper-blade shop. The conversation is not left to a language model. An explicit state machine runs it — GREETING → COLLECT_VEHICLE → QUOTE → DONE, four states — and the understanding layer defaults to a deterministic rule-based parser (MockLLM, marked SIMULATED in the repo). The real LLM parser (RealLLMProvider) is only ever asked to parse a message into an intent and vehicle slots; it never decides prices, sizes, or whether a deal closed. That division of labor is written into the README. Status, honestly: scaffold v0.2, a simulated catalog of 8 fitment records across 5 makes, a demo price table. This is not ready to take real orders.

So this article does not measure whether the demo runs — the bundled demo and all 56 tests pass (I re-ran them: 56 passed; the README still says 44, which is stale). It measures how the machine behaves off the demo path: 27 scripted conversations, 62 messages, 13 intent-collision probes, a full recomputation of quotes for all 8 catalog records, one persistence round-trip, and one two-customer isolation check. The headline first: the main flow is steadier than I expected, the quote math is exact to the cent, and all six FAQ topics stay inside the quoting state. But the intent layer's substring matching produces three misreadings that would cost a real shop money — the worst being the title of this article. Under the default rule-based parser (MockLLM), a customer in the QUOTE state who asks "可以便宜点吗" ("can you make it cheaper?") is taken straight to DONE with a confirmed order intent, at the full, undiscounted price. Scope, stated once and binding for the whole article: every intent-layer finding here is about that default parser; behavior under the real LLM parser is a separate question, revisited in §5.

2. Method and architecture

The core package, tools, and WhatsApp boundary total 1,659 lines, including a 258-line webhook server. Conversation state lives in a Conversation object: current state, three vehicle slots (make / model / year), the fitment result, the quote, a boolean haggled flag (has the one-time haggle discount been used), the outcome (confirmed / declined), detected language and confidence, and full history.

The transition rules, state by state — everything later follows from these:

GREETING: a message carrying all three vehicle slots goes straight to a fitment lookup — found means QUOTE; a miss (a year outside the catalog range, say) falls back to COLLECT_VEHICLE with a "not found" reply. A partial vehicle moves to COLLECT_VEHICLE and asks for what is missing. Anything else gets the greeting text and the state stays GREETING. This state has no branch for decline, FAQ, or price questions. None.

COLLECT_VEHICLE: only two things are handled — a decline (DONE, marked declined) and completing the slots (lookup). Confirm, price, haggle, and FAQ intents are all unhandled here and get the same "what's still missing" reply.

QUOTE: the fullest state. Haggle (once only, an extra 5%), FAQ (answered without leaving QUOTE), price question (re-present the quote), confirm (DONE / confirmed), decline (DONE / declined), a message carrying vehicle info (re-run the lookup — treated as a vehicle change), and anything else re-presents the quote with a nudge.

DONE: intercepted in handle_message. Any new message after DONE discards the conversation and starts a fresh one — vehicle, quote, and haggle flag all reset. I verified the reset by round-trip.

Intent parsing (MockLLM.parse) is where this article lives. The precedence order in code is: decline > confirm > haggle > FAQ > price > greeting > provide-vehicle > unknown. Keywords match by substring (w in low); only three short tokens — sí, да, نه — get word-boundary matching. The confirm list contains everyday words: "买" (buy), "好的" (okay), "可以" (can/sure), "ok", "yes". The decline list contains a bare "no". Vehicle extraction only recognizes catalog vehicles: a (make, model) pair from the simulated catalog must both appear as substrings; the year is the first 4-digit number matching (19|20)\d\d.

The two tools: wiper_db.lookup searches the 8 simulated fitments by make, model, and year range. pricing.build_quote prices blades in three length tiers (up to 20" $12.99, 21–24" $14.99, 25"+ $16.99), takes 10% off the subtotal (the code comment calls it a bundle discount), and a further 5% after a successful haggle.

Method: repository code unmodified, conversations driven in-process under different "phone numbers" (arbitrary string IDs), logging every message, resulting state, quote total, haggle flag, and full order intent. Quotes for all 8 fitments were recomputed independently and compared. Collisions were first classified by calling parse directly, then the three worst were replayed inside real conversations to observe the consequences. The bench script and raw JSON are archived with this article and can be re-run.

3. Measurements

3.1 The main flow: split vehicle entry, a full Spanish run, exact quote math

The standard path (hello → Toyota Camry → 2018 → how do I install it → okay, I'll take it) walks GREETING → COLLECT_VEHICLE → QUOTE → QUOTE → DONE and produces a confirmed intent with vehicle, sizes (26/20"), and a $26.98 total. All three slots in one message jump straight to QUOTE. A full Spanish run (Hola → Corolla 2021 → ¿Cuánto cuesta? → ¿Incluye instalación? → Vale, sí) answers in Spanish at every step, locks the language to es on the first message, and records detected_language: es on the final intent.

All 8 quote recomputations matched. Camry, worked in full: $16.99 + $12.99 = $29.98 subtotal, $26.98 after 10% off, $25.48 after the haggle (85% of subtotal). The RAV4, the only record with a rear blade: $42.97 subtotal, $38.67 quoted, $36.52 haggled. One bookkeeping discrepancy worth recording: the comment describes the 10% as a driver+passenger bundle discount, but the discount base is the whole subtotal including the rear blade — the RAV4's $4.30 discount is computed on all three blades. The error favors the customer; the books should still say what the code does. A rounding note, so nobody's calculator disagrees: the Quote's discount field is rounded to cents on its own (the Camry's exact 10% is 2.998, stored as 3.0), while the total is computed from the unrounded discount and then rounded. I reconciled all 16 combinations (8 records, with and without haggle): discount plus total reproduces the subtotal exactly in every case.

3.2 Haggle once, and only once — until the vehicle changes

Altima 2019 quotes at $26.98. "太贵了" (too expensive) drops it to $25.48 and sets the flag. "再便宜点" and "还能便宜吗" both get "that's already my lowest," and the total does not move. As a one-shot allowance enforced by a boolean and a recomputation, there is no path around it. That part holds.

What the tests never covered is haggling across a vehicle change. Haggle the Altima down to $25.48, then name a Civic 2020: the lookup rebuilds the quote without the haggle, the new total is back to $26.98 — and the haggled flag stays set. Haggle again and the agent replies "this is already the lowest price" about a price that was never discounted. The customer is told something factually false. The cause is granularity: the flag lives on the conversation, not on the quote. A vehicle change neither carries the discount over nor clears the flag; of the two coherent choices, the code makes neither.

3.3 FAQ only lives inside QUOTE

In QUOTE, all six FAQ topics (install, warranty, lifespan, payment, pickup, shipping) answer correctly, stay in QUOTE, leave the quote untouched, and close with an ordering prompt. That matches the documented design — FAQ must not break the sales flow — and it held in every probe.

Ask before the quote and the question disappears. At GREETING, "怎么安装" (how do I install it?) gets only the greeting; the question is neither answered nor logged (no FAQ note is recorded). At COLLECT, the same question gets a repeat of "which year is it?", the question vanishes, and after completing the vehicle the customer has to ask again. Price and haggle attempts at COLLECT meet the same fate. There is no holding area in this machine for "remember this, answer it when we get there." Out-of-flow input is not refused; it is silently dropped, and the reply gives the customer no sign their question was never heard — it just reads as a non sequitur.

Declines are state-dependent too. "算了" (forget it) at COLLECT and at QUOTE correctly ends as declined. "不要了" (I don't want it) as the very first message is not handled at all — GREETING has no decline branch — and earns a cheerful greeting. Minor on its own; included because it shows GREETING's intent coverage is not partial, it is zero.

3.4 Intent collisions: 13 probes, three failure families

Feeding everyday phrasings to parse, 7 of 13 came back with a classification visibly at odds with what the speaker meant, in three families:

Confirm words are too broad and outrank everything else they co-occur with. "可以便宜点吗" (can you make it cheaper?) classifies as CONFIRM — "可以" is in the confirm list and confirm is checked before haggle. "好的，多少钱" (okay, how much?) is CONFIRM, not a price question. "我想买，先问问保修" (I do want to buy, but first let me ask about the warranty) is CONFIRM — "买" fires confirm even though the FAQ topic was correctly detected as warranty and never gets used. "yes but too expensive" is CONFIRM for the same structural reason.

The decline list contains a bare substring. "no" is matched as a substring, so "I don't know" and "another question" both classify as DECLINE — both contain "no". The codebase already has a word-boundary mechanism for short tokens (CONFIRM_TOKENS / DECLINE_TOKENS); "no" simply was not put in it.

Decline outranks haggle. "不要太贵了" (don't make it too expensive) and "不买了太贵" classify as DECLINE. Whether the first of those is really a refusal is debatable — it reads more like a haggle opening — but the machine offers no clarifying move. Between "kill the conversation" and "treat as haggle," it picks kill.

The three worst, replayed inside real conversations:

In QUOTE, "可以便宜点吗" → straight to DONE, a confirmed order intent at the full $26.98. The customer asked for a discount; the system recorded a sale.
In QUOTE, "我想买，先问问保修" → straight to DONE, confirmed, warranty question never answered. The buying intent was genuine, but the flow closed before the customer's actual question, leaving the human follow-up holding a "confirmed" record and an unanswered conversation.
In QUOTE, "I don't know" → straight to DONE, declined, logged as "declined at quote." The customer was undecided; the system recorded a rejection. The same sentence at COLLECT ends the same way.

Controls, so the word lists are not condemned wholesale: "能便宜点吗" (the same haggle without "可以") classifies correctly as HAGGLE; "先看看保修再说" goes to FAQ; "好的" and "OK" in QUOTE are legitimately CONFIRM. What breaks is the compound sentence — and real customers rarely speak in single keywords.

3.5 Changing vehicles: an off-catalog car is quietly merged into the old one

In QUOTE, "Honda Civic 2020" correctly re-quotes a new vehicle (same $26.98 total, but sizes change from 26/20 to 26/18, so the change is visible). "BMW X5 2020" produces the reply "Found it! Toyota Camry 2020." BMW and X5 are not in the simulated catalog, so vehicle extraction fails entirely and only the year, 2020, is captured; slot merging folds that year into the old vehicle, the lookup succeeds for a Camry 2020, and a fresh Camry quote is presented with total confidence. The customer thinks they asked about a BMW. The agent quoted them a Camry. At no point does anything say "I can't find that car."

The same off-catalog car at the start of a conversation behaves almost honestly: extraction fails and the agent asks which vehicle — phrased as "still missing the vehicle," which will confuse a customer who just named one, but at least nothing is fabricated. Two different failures should not be conflated here: that was an extraction failure (make and model never parsed, slots incomplete, so all it can do is ask). A lookup failure is different — "Toyota Camry 2005" as a first message parses completely, misses the catalog, and gets an explicit "not found" with a fallback to COLLECT, which I also tested. In QUOTE, the pre-existing slots mask the failure and turn it into a silent substitution. The only difference is whether old slot values exist to paper over the gap.

A year-only change has the mirror-image problem. "2025" merges into Camry, falls outside the catalog range (2018–2024), and this time the lookup properly misses: the state falls back to COLLECT and asks the customer to double-check. The handling is right; the cleanup is not — the old $26.98 quote stays attached to the conversation. When the customer then gives up ("算了"), the declined order intent carries that stale quote, with the vehicle field reading Camry 2025: a car with no fitment, wearing a 2018 quote, filed as if the customer had rejected a real offer.

3.6 Persistence and isolation: two passing grades

Mid-conversation — quoted and haggled — I restored the conversation into a brand-new agent instance from the same JSON file store. State QUOTE, vehicle, haggled total $25.48, the flag, and all 4 history entries came back exact, and the restored conversation answered a warranty question while staying in QUOTE. The to_dict / from_dict round-trip is lossless. Two interleaved customers (one haggling, one completing a vehicle) never leaked slots, quotes, or flags into each other; keying by phone number holds.

Gibberish in QUOTE ("asdf qwer") re-presents the quote with a nudge and changes nothing. Not smart, but it neither crashes nor closes anything — and in context it is the intent layer's most trustworthy behavior. Only UNKNOWN gets this treatment. Every failure above came from a match that was almost right.

4. Analysis and trade-offs

4.1 What the architecture genuinely gets right: the parser never holds the pen on a deal

Read against the design, the result worth keeping is the division of authority. The state machine owns the flow, the tools own the numbers, and the parsing layer — rules today, an LLM tomorrow — only emits intents and slots. Exact quote math, a hard one-time haggle cap, FAQ containment, and lossless persistence all follow from decisions being kept in deterministic code. Swapping in RealLLMProvider will improve parsing, but its intents still pass through the same transition table, and it still cannot mint a price or close a deal. For a small-shop agent that is the right border: language may be probabilistic; the ledger may not.

4.2 The defects are not in the machine; they are in its entry contract

None of the three measured problem families is a wrongly drawn transition. The table's flaw is that it trusts whatever the entrance hands it: whatever intent parse returns gets executed, whatever the slot merger produces gets looked up. Every business-harming action in the tests — the wrongly closed order, the wrongly quoted car, the fabricated intent record — happened at a point where input was over-trusted:

CONFIRM and DECLINE are irreversible (DONE, plus a recorded intent), yet they are triggered by the weakest evidence in the system: a single substring. Irreversible actions should demand stronger evidence than reversible ones (re-presenting a quote, answering an FAQ). The current precedence inverts that — the irreversible intents are checked first.
Slot merging overwrites field by field and cannot distinguish "the customer added the year" from "the customer changed cars and the new car failed extraction." The second case should route to clarification, not be healed with stale values.
haggled and quote have mismatched lifetimes. A quote can be replaced while its flag persists; a failed lookup retreats while its quote persists. Derived state has no rule that says "a new quote invalidates everything derived from the old one."

4.3 Fix order: gate the irreversible actions first, upgrade the NLU later

In cost-benefit order. First, move short words like "no" out of the substring lists into the existing word-boundary sets, and tighten confirm words like "可以" and "买" with co-occurrence conditions (confirm only when no haggle or price wording is present). That is an afternoon of word-list and ordering work and it dismantles most of §3.4. Second, reorder precedence so reversible intents win: FAQ, price, and haggle are evaluated before confirm and decline, and confirm only stands when the sentence carries no other request. "Wants to buy, but first the warranty" then gets its answer and stays in QUOTE instead of being closed early. Third, give CONFIRM a read-back step: before DONE, echo vehicle and total — "shall I book this?" — and record the intent only on an explicit yes. One extra CONFIRMING state holds it, and it mirrors what a human clerk does anyway. Fourth, add a merge rule: when a message carries a year or vehicle-like text but extraction comes back empty and inconsistent with the current vehicle, do not merge — clarify; and when a lookup miss retreats to COLLECT, clear the quote and the haggle flag with it. Fifth, and only then, swap MockLLM for RealLLMProvider, with this article's 13 probes plus real conversation logs as the regression set. A better brain fixes part of the compound-sentence problem; it fixes none of the merging or lifecycle problems, because those belong to the machine itself.

The judgment underneath the ordering: in a demo, the thing most easily overlooked is not answer quality but closing correctness. A wrong FAQ answer gets asked again. A wrongly recorded confirmed intent gets acted on — the owner preps stock, reaches out, and finds nobody there.

5. Limitations and open questions

Stated plainly. First, everything ran on MockLLM; the collision patterns are specific to a rule-based parser. A real LLM endpoint would very likely handle the compound sentences better, but I did not re-run the 13 probes against RealLLMProvider, so the post-swap error rate is unmeasured. Second, the catalog has 8 simulated records at demo prices. In a real catalog of hundreds of records, extraction failures get rarer but not harmless — the silent-merge mechanism is size-independent, and the claim here is about the mechanism. Third, the scripts were written backwards from the transition table, so they cover the paths I could think of; real customers' speech-to-text errors, code-switching, and two cars in one sentence were only spot-checked, not systematically tested. Fourth, the WhatsApp boundary layer (webhook signature verification, Cloud API delivery, usage counters) is out of scope — this round exercised the in-process agent API only; the repo's own tests, all 56 passing, cover parts of that layer, but no end-to-end message test was run. Fifth, beyond Spanish, the Russian and Persian flows were verified only at greeting and language-detection level, following the repo's own test coverage; the templates have not had native-speaker review, which the README also states.

Three open questions. Adding a CONFIRMING read-back state will cost some completion rate — how much is only knowable with real traffic. How fine-grained the merge-clarification rule should be (year conflicts only, or any "vehicle-like text, empty extraction") needs real conversation logs to calibrate. And two cars in one order — a customer buying for themselves and a family member — does not fit the one-conversation-one-vehicle data model at all; whether that is a notes-field workaround or a multi-intent model change is undecided.

6. References

Wiper-Service-Agent source (version tested, commit 9675291): github.com/Ezra-Zhao/Wiper-Service-Agent. Key files: wiper_agent/agent.py (state machine, slot merging, language upgrade), wiper_agent/llm.py (MockLLM precedence and word lists, the RealLLMProvider parsing contract), wiper_agent/conversation.py and store.py (state and persistence), wiper_agent/schemas.py (the four states and data models), tools/wiper_db.py (8 simulated fitments, marked SIMULATED in the file header), tools/pricing.py (three price tiers and discounts), tools/faq.py (six FAQ topics, five-language templates).
Bench script and raw data for this article: data/2026-10-16-wiper-bench.py and data/2026-10-16-wiper-bench.json (27 conversation traces, 13 intent-collision probes, 8 quote recomputations, persistence round-trip, two-customer isolation), archived with the article; the numbers can be reproduced by running the script against a clone of the repository.
Repository test suite: tests/run_tests.py — 56 passed, 0 failed in this run (the README Quickstart says 44, which is out of date).

Author: Guangyi Zhao. The code, experiments, and text in this article are clean-room work of my own: the subject is an open-source scaffold I maintain, every conversation was script-generated, no WhatsApp account was connected, and no real customer's chats, phone numbers, or orders were used. No client information, client systems, or third-party non-public data are involved. All vehicles, prices, and language templates are the repository's simulated data and are not presented as real product information; the scaffold is not presented as a production system in service.

Before You Wire an LLM Into GPU Fault Triage: AI-Node-Sentinel's Integration Points and Six Measured Failure Modes

Guangyi Zhao — Mon, 19 Oct 2026 05:01:17 +0000

Author: Guangyi Zhao
Date: 2026-10-14
Subject: AI-Node-Sentinel (open-source project by the author, github.com/Ezra-Zhao/AI-Node-Sentinel), scaffold v0.1, Python 3.12.3
Environment: Linux x86_64 VM, repository code run unmodified; LLM experiments use local fake providers implementing the project's own protocol — no real model API was called

1. Problem and background

In a GPU cluster, the first signal of failing hardware is usually one line in the kernel log: an Xid. XID is NVIDIA's numbering scheme for GPU hardware and firmware errors — a GPU falling off the PCIe bus, an uncorrectable double-bit ECC error, an NVLink fault, a GSP firmware timeout each has its own code. The on-call ritual is manual: pull dmesg from a thousand GPUs, look up what each code means, then decide — drain the node, reset the GPU, reboot, or start an RMA. Decide slowly and the training job keeps retrying on broken hardware, burning GPU-hours that cost real money.

The obvious impulse is to hand this to an LLM agent. Let the model read telemetry, look up error codes, and recommend actions — isn't that exactly what agents are for?

My own open-source project, AI-Node-Sentinel, was scaffolded around that idea: telemetry comes in, XID events are extracted, a lookup table enriches them, triage rules are applied, and a structured report comes out, with a LangChain-based slot reserved for LLM reasoning. Let me state the honest status up front, because everything after this paragraph depends on it: the project is scaffold v0.1, and its README says so. The triage decision tree is a deliberate stub — every event gets NEEDS_HUMAN_REVIEW with confidence 0.0. The LangChain provider is also a stub that raises NotImplementedError on construction. All telemetry is synthetic, produced by a bundled simulator. Only three stages actually run: ingestion, lookup, and report rendering.

So this is not a write-up of a production diagnostic system. It is the opposite: I put the scaffold on the dissecting table and ran six fault-injection experiments to measure where this pipeline would break once an LLM is actually wired in. The results generalize to anyone attaching an LLM to an operations system, because none of the six breakpoints is a model-capability problem. They are wiring problems: an output contract that doesn't match, evidence fed too thin, one malformed record poisoning the whole batch, a prompt path coupled to the working directory, stateless single-event myopia, and corroborating signals discarded by the pipeline itself. A stronger model fixes none of these.

2. Method and architecture

The codebase is small: eight core Python files, roughly 470 lines including tests and the demo. Four parts. agent/schemas.py defines the data types: TelemetryEvent (a normalized event with source, kind, and payload), XidFinding (an XID event enriched by lookup), and TriageVerdict (action, confidence, rationale). tools/xid_lookup.py is a static table mapping codes to name, category, and severity, seeded only from NVIDIA's public XID documentation and Google Cloud's GPU troubleshooting guide; a miss returns None — it never guesses. tools/triage_rules.py holds the decision tree, currently the stub. agent/orchestrator.py is the TriageAgent: ingest, extract XID events, triage each one, render the report.

The LLM integration point lives in agent/llm.py: a minimal protocol, LLMProvider, requiring one method, reason(system_prompt, evidence) -> str. The orchestrator first computes the deterministic verdict; if a provider is present, it builds an evidence string, calls the provider, and appends the returned text to the verdict's rationale with an [LLM] prefix. The draft system prompt in prompts/triage_system.md instructs the model to cite only the evidence given, to say explicitly when a code is unknown, to return exactly one action with a confidence between 0 and 1, to force NEEDS_HUMAN_REVIEW whenever confidence is below 0.7, and to answer in JSON.

Method: five experiment groups against the unmodified repository, with the scripts and raw JSON archived alongside this article. (1) Baseline scale: the bundled simulator generates telemetry windows for 4, 16, and 64 nodes; I record event composition, verdict distribution, and pipeline latency. (2) Lookup coverage: table size plus ten out-of-table probe codes. (3) Malformed-data injection: events with a missing xid key, a non-numeric xid, and xid=None, mixed into otherwise good batches. (4) Fake-LLM experiments: a local provider that returns exactly the JSON the prompt demands, to observe what actually changes in the verdict; then a provider that always raises, to observe fault propagation. (5) Failure-mode probes: repeat faults on one GPU, the fate of corroborating events, and prompt loading from a different working directory.

3. Measurements

3.1 Baseline: the deterministic part is too fast to matter, and the verdicts are the stub's

The simulator emits one thermal and one ECC baseline event per GPU, then injects one or two XID events on about 35% of GPUs:

Scale	Events	XID events	Verdicts	Unknown verdicts	Severity (critical/high/unknown)	Pipeline time
4 nodes × 8 GPUs (seed 42)	76	12	12	1	10 / 1 / 1	~0.1 ms
16 nodes × 8 GPUs (seed 42)	313	57	57	5	39 / 13 / 5	~0.2 ms
64 nodes × 8 GPUs (seed 42)	1,285	261	261	34	157 / 70 / 34	~0.7 ms
64 nodes × 8 GPUs (seed 7)	1,295	271	271	35	177 / 59 / 35	~0.7 ms
64 nodes × 8 GPUs (seed 123)	1,282	258	258	28	167 / 63 / 28	~0.7 ms

Three observations. First, every verdict is NEEDS_HUMAN_REVIEW at confidence 0.0. That is stub behavior, not measured intelligence, and I record it as the control group for what follows. Second, roughly 1,300 events process in under a millisecond — about 1.77 million events per second on the raw single-run perf_counter timing (the table rounds to 0.1 ms; sub-millisecond figures are order-of-magnitude only, not a precise benchmark) — because lookup is a dictionary read. At this article's scale the deterministic part is free; streaming ingestion and window state will need their own measurement later, but the bulk of future latency and cost will come from the LLM step. Third, do not misread the 11–13% unknown share: the simulator's injection pool has eight codes, one of which (999) is a deliberately synthetic unknown, so the unknown rate is an artifact of pool design — one eighth by construction — not a field measurement of unknown codes in a real fleet.

The repository's six bundled tests all pass when invoked directly (determinism, known-code lookup, unknown codes not guessed, stub routing to human review, report shape). This machine has no pytest installed, so I ran the test functions directly rather than through the runner; noted for accuracy.

3.2 Lookup coverage: seven codes, everything else is "unknown"

The table contains exactly seven codes: 13, 31, 48, 74, 79, 119, 120. Ten out-of-table probe codes (43, 45, 56, 63, 64, 68, 92, 94, 109, 154) all return None and are treated exactly like 999: severity unknown, known=False, routed to human review. The routing behavior is right — refusing to guess is this pipeline's most important safety property. The coverage is another matter: any XID outside those seven earns the operator exactly one piece of information, "an unknown error occurred, go look." The table was hand-seeded from public documents; extending it is unglamorous work, and section 4.3 argues it is still not the first thing to fix.

3.3 Malformed data: one bad event poisons the batch

The orchestrator extracts the code with int(event.payload["xid"]) — no validation, no per-event isolation. Mixing malformed events into good batches:

Injection	Outcome
One well-formed event	Completes, 1 verdict
Good + missing xid key + good	Entire run raises KeyError; completed verdicts are lost with it
Good + xid as a non-numeric string	Entire run raises ValueError
Good + xid=None	Entire run raises TypeError

run() processes events in a list comprehension, so the first exception discards every verdict in the window. Translate to production: a dmesg parser will eventually emit a field-incomplete event, and when it does, the failure mode is not "one event undiagnosed" but "the whole window's verdicts vanish, as a process-level exception instead of a line in the report." Per-event isolation with a bad-record counter is table stakes for this kind of pipeline, and it is absent here.

3.4 The fake-LLM experiment: the output contract is broken, and the model doesn't get a vote

This is the experiment the article exists for. I implemented a fake provider that obeys the system prompt perfectly, returning JSON that recommends DRAIN_NODE at confidence 0.95 for XID 79 (GPU fallen off the bus). Wired into the orchestrator, the verdict's actual state:

Field	Fake provider output	Final verdict
action	DRAIN_NODE	NEEDS_HUMAN_REVIEW (unchanged)
confidence	0.95	0.0 (unchanged)
rationale	The full JSON	Appended verbatim as text, prefixed [LLM]

The code makes the reason plain: the protocol returns a string, the orchestrator only concatenates it, nothing ever parses the JSON, and nothing overwrites the deterministic verdict. The prompt's output format and confidence gate are a gentlemen's agreement with the model that the code side never honors. Even if the real LangChain provider were implemented tomorrow and the model followed the format perfectly every time, the triage decision would remain 100% determined by the stub; the LLM's total influence would be a paragraph pasted into the rationale.

The second half of the experiment is evidence thickness. The exact evidence string the orchestrator assembles is: node=node-00 gpu=0 xid=79 name=GPU has fallen off the bus severity=critical ts=. That is all. The lookup entry has category and notes fields — including remediation hints and sourcing notes — and neither enters the evidence. The same GPU's thermal and ECC readings cannot enter it either, because the evidence is built inside the per-event triage function, whose signature has no window context at all. The prompt's first rule orders the model to cite only the evidence provided. The pipeline starves the evidence down to a restatement of the error name, then forbids the model from reaching beyond it. If the model gives an informative answer anyway, it can only do so by breaking the rule. That contradiction is a wiring decision, not a model defect.

Third: fault propagation. Replace the provider with one that always raises RuntimeError, and triage_event lets the exception straight through — the whole run dies, the same death as section 3.3. A provider is an external dependency; timeouts, rate limits, and network jitter are its normal weather. An optional reasoning step should degrade to the deterministic verdict, never take the batch down with it. There is no degradation path here.

3.5 The prompt path is coupled to the working directory

The orchestrator loads the prompt with a relative path, open("prompts/triage_system.md"). From the repository root it works. From /tmp, the identical code raises FileNotFoundError immediately. (The LangChain provider raising NotImplementedError on construction is an honest stub, not a defect.) The relative path is the kind of bug that bites exactly once, in production: services started by systemd, containers, or schedulers rarely have the repository root as their working directory, while a developer's terminal always does — which is why it never reproduces at a desk.

3.6 No memory, and corroboration blindness: two failure modes with one root

Repeat faults do not escalate. Three XID 79 events on the same GPU within three hours produce three verdicts that have never heard of each other: same action, same confidence, same rationale. Causality, stated precisely: the identical verdicts are directly caused by the decision tree being a stub — any architecture would produce them today. The architectural problem sits one layer down. The stub's own comments sketch the intended real rule — XID 79 twice in 24 hours on one GPU should mean drain plus reboot — but the orchestrator triages events one at a time, statelessly, so even once that rule is written, the pipeline has nowhere to put it. This is not only a missing rule; it is missing state.

Corroborating signals are filtered out by the pipeline itself. I built a three-event window by hand: one GPU reports an ECC double-bit error count of 17, then a temperature of 104 °C, then XID 13. The prompt's second rule says app-level faults like XID 13/31 should not trigger a drain recommendation without corroborating evidence — which implies corroboration should change the judgment when present. The measured result: three events in, one verdict out. The two non-XID events are dropped at the extraction stage; they produce no verdicts and are never attached to the XID 13 verdict as context. At baseline scale this is 1,024 of 1,285 events in the 64-node window. The pipeline demands corroboration-aware reasoning in its prompt while deleting the corroboration from its data path.

4. Analysis and trade-offs

4.1 Deterministic first, LLM second — and the order is not negotiable

The scaffold's best judgment call is its layering: lookup and rules in a deterministic core, the LLM as a downstream reasoning supplement, and — until the decision tree exists — everything routed to humans with an honest confidence of 0. The reasoning is simple. Drain, reset, reboot, and RMA are real operations on production hardware. Anything a table or a rule can decide should never be improvised by a probabilistic model at 3 a.m.; the LLM's value is synthesis and explanation, not authority.

Section 3.4 yields a counterintuitive corollary: the broken output contract is currently an accidental safety net. Because the model's JSON is never parsed into the verdict, a hallucinating model today can pollute a rationale but cannot drain a node. But it is an accident, not a design. The correct fix is not to preserve the break; it is to add parsing plus validation — actions must land inside the site's runbook vocabulary, confidence below 0.7 must be rewritten to human review in code (the prompt already states this rule; move it from prose into the program), and parse failure must degrade to the deterministic verdict alone. Whoever wires up the parsing without the validation has personally removed the safety net.

4.2 The failure-mode list, in fix order

None of the six failure modes needs a better model or a better prompt. All six are ordinary engineering, and I would fix them in this order:

Per-event isolation. Wrap each event's triage in exception handling; count and surface bad records in the report. Batch survival comes before everything else.
Honor the output contract. JSON parsing, vocabulary validation, and the confidence gate in code, per section 4.1.
Evidence assembly. Include the lookup's category and notes, and pass the triage function a per-GPU window summary — recent same-code counts, thermal and ECC extremes — so the prompt's "cite only the evidence" rule becomes executable.
A state layer. Repeat counting and escalation need window-level aggregation, which means restructuring from stateless per-event triage to aggregate-then-triage. It is an architectural change, so it goes after the contract and evidence fixes.
Resource paths and graceful degradation. Load the prompt as a package resource or absolute path; wrap provider calls with timeout and exception handling that fall back silently — but visibly in the report — to the deterministic verdict.
Corroboration plumbing. Stop discarding non-XID events at extraction; attach them to the same GPU's XID verdicts as context. It shares a root with item 3 but is listed separately: item 3 changes the evidence template, this one changes the data path.

Extending the lookup table is deliberately not on this list. It is data-dictionary work, not a failure mode; seven codes is thin coverage and the table should grow — after these six, because without them a 70-code table still dies batch-wide on the first malformed event.

4.3 A judgment: trustworthiness in systems like this comes from restraint, not from the model

The design choices I would defend most strongly in this scaffold are the things it refuses to do. Unknown codes are not assigned invented meanings. An unfinished decision tree issues no actions. Simulated data is labeled SIMULATED from the generator to the report header. The stub raises NotImplementedError instead of returning a fabricated rationale. Together these refusals make every output auditable: for any line in the report you can tell whether it came from the table, a rule, or a model. That is precisely what gets lost first when an LLM is bolted onto an operations system — once fluent model prose washes over the deterministic core's true state, the on-call engineer can no longer tell what was looked up from what was made up, and the report stops deserving trust. All six failure modes point at the same repair: make the model's scope, evidence, and failure behavior explicit in code, not implied by prompt wording.

5. Limitations and open questions

The limitations are material. First, all telemetry is synthetic, generated by the bundled simulator; its distributions are set by injection probabilities and the injection pool, so every proportion in this article validates pipeline behavior only and says nothing about fault distributions in a real fleet. Second, the LLM experiments use a local fake provider and verify wiring contracts and fault propagation; no real model was called, so this article makes no claim about any model's reasoning quality. How often a real model, fed evidence this thin, would reach beyond it and fabricate is a separate measurement that requires wiring one up. Third, the scale ceiling here is 64 nodes and about 1,300 events on a single VM; concurrent ingestion, persistence, and alerting channels are untested. Fourth, lookup coverage is reported as table size plus out-of-table probes; I did not diff the table against NVIDIA's complete published catalog, so no coverage denominator is claimed.

Three open questions remain. The aggregation grain for the state layer — per GPU, per node, or per NVLink domain — implies different escalation rules and false-positive costs, and calibrating it needs real fault sequences. The 0.7 confidence gate in the draft prompt was picked, not calibrated; once a real model is wired in, it should be set by replaying historical tickets. And once corroborating evidence enters triage, the baseline thermal and ECC stream is roughly four times the XID volume — what evidence assembly costs in tokens and context length has to be measured when the real model arrives.

6. References

AI-Node-Sentinel source (version measured here): github.com/Ezra-Zhao/AI-Node-Sentinel — key files: agent/orchestrator.py (orchestration and evidence assembly), agent/llm.py (LLMProvider protocol and LangChain stub), agent/schemas.py (data types), tools/xid_lookup.py (lookup table), tools/triage_rules.py (decision-tree stub), prompts/triage_system.md (draft system prompt), telemetry/simulator.py (simulator).
Experiment scripts and raw data for this article: data/2026-10-14-sentinel-bench.py and data/2026-10-14-sentinel-bench.json, archived with this article; every number can be re-checked by re-running the script.
NVIDIA, Xid Errors documentation: docs.nvidia.com/deploy/xid-errors — the authoritative source for the lookup metadata, also cited in the header of tools/xid_lookup.py.
Google Cloud, Troubleshooting GPU issues: cloud.google.com/compute/docs/troubleshooting/troubleshooting-gpus — cross-check source for the lookup table, as above.

Author: Guangyi Zhao. The code, experiments, and text in this article are clean-room work by the author: the subject is an open-source scaffold project maintained by the author, all telemetry is synthetic data from that project's simulator, and all experiments ran in the author's own local environment. No customer information, customer systems, or third-party non-public data are involved. Every number comes from the measurements listed at the top of this article; the simulated nature of the data and the limits of the conclusions are stated in the text and in section 5, and no scaffold state is presented as a production system.