DEV Community: Naser Sadrmanesh

Safe write tools for agents: preview, confirm, audit, and stale previews in a WooCommerce MCP server

Naser Sadrmanesh — Sat, 31 Oct 2026 12:59:14 +0000

Reading data is the easy part of connecting an agent to a store. Listing orders, summarizing sales, finding low stock: if the model gets one of those wrong, you get a wrong answer and you ask again. Writes are different. "Raise the price of these 40 products by 10%" is one sentence for the user and 40 irreversible changes for the store.

I maintain woocommerce-mcp, an open-source Model Context Protocol server for WooCommerce. It exposes products, orders, refunds, coupons, customers, stock, webhooks, and analytics as MCP tools, over stdio or authenticated streamable HTTP. This post is about one part of it: how the write tools are designed so that an agent cannot change a store in a single step, and what changed in v0.5.0 when I added stale preview detection.

I will also be explicit about what the design does not guarantee, because a safety feature that overstates itself is worse than none.

The problem with one-step write tools

The obvious way to write an MCP tool is: take arguments, call the API, return the result. For a write, that means the model's tool call is the change. There is no point where a human sees what is about to happen.

There are three separate failure modes hiding in that:

The model asks for the wrong thing. It picks the wrong product id, misreads "10" as a fixed amount instead of a percentage, or targets the wrong store.
The arguments change between "I showed the user" and "I ran it". A client shows a plan, the user approves, and then the model retries with slightly different arguments.
The store changes underneath the plan. The user approves "+10%" when a product costs 20.00. Before they click confirm, someone else edits the price to 25.00. The same arguments now produce 27.50, a number nobody approved.

The first one is solved by showing the user the exact request. The second and third need something that ties the confirm to the preview. That is what the rest of this post is about.

Layer 1: stores are read-only until you say otherwise

Every store starts read-only. A write tool fails with read_only before it contacts the store unless you set WOOCOMMERCE_READ_ONLY=false (single store) or "readOnly": false in a store list. Over HTTP, write tools also need the woocommerce:write scope, so a read-only token can never change anything.

This sounds trivial, but it decides the default. Someone who installs the server to ask "what sold best last week?" never has to think about writes at all.

Layer 2: two calls instead of one

With WOOCOMMERCE_REQUIRE_CONFIRMATION=true (the default), the first call to a write tool changes nothing. It returns the planned REST request:

{
  "status": "confirmation_required",
  "store": "default",
  "previewId": "pv_4f1c9a2e7b3d0856",
  "message": "The change was NOT applied. ...",
  "request": { "method": "PUT", "path": "orders/1015", "query": {}, "body": { "status": "completed" } }
}

Calling the same tool again with the same arguments and confirm: true applies it. dryRun: true always previews, even if confirmation is turned off.

The interesting engineering question is how you produce that preview without writing a second, parallel "describe what you would do" implementation for every tool. Two implementations drift. I did not want 19 write tools each with a real path and a preview path.

The planning client

Every tool talks to the store through a WooCommerceClient. The client has a planning() view that behaves normally for reads and throws on the first write instead of sending it:

/**
 * Returns a client that performs reads normally but stops at the first write
 * and throws a {@link PlannedWrite} describing the request it would send.
 */
planning(options: PlanningOptions = {}): WooCommerceClient {
  return new WooCommerceClient(this.options, this.hooks, true, this.shared, this.namespace, options);
}

Inside request(), a non-GET on a planning client becomes a PlannedWrite error carrying method, path, query, and body. The registry runs the tool's real run() function against that client and catches the error:

async function plan<S extends AnyObject>(
  spec: ToolSpec<S>,
  ctx: ToolContext,
  client: WooCommerceClient,
  args: z.infer<S>,
): Promise<Plan> {
  const reads: RecordedRead[] = [];
  const planner = client.planning({ fresh: true, onRead: (read) => reads.push(read) });
  try {
    const result = await spec.run({ ...ctx, client: planner }, args);
    return { write: null, result, state: captureState(reads, null) };
  } catch (error) {
    if (!(error instanceof PlannedWrite)) throw error;
    return { write: error.request, result: null, state: captureState(reads, error.request) };
  }
}

So the preview is produced by the same code that does the write. If bulk_adjust_prices computes regular_price from the current price, the preview contains the computed absolute value, because it is literally the body the tool tried to send. If the tool finishes without trying to write (for example every product was skipped), the response is no_change instead of a confirmation request.

The trade-off: a tool that makes several requests, such as woocommerce_fulfill_order (record tracking, complete the order, notify the customer), is previewed only up to its first write. I document that rather than pretend the preview covers the whole sequence.

Layer 3: the previewId

Two calls are not enough on their own. Nothing stops the confirm call from carrying different arguments than the preview. So in v0.4.0 every confirmation_required, dry_run, and no_change response started including a previewId:

export function previewIdFor(tool: string, store: string, args: Record<string, unknown>): string {
  const digest = createHash('sha256').update(`${tool}\n${store}\n${canonicalJson(args)}`).digest('hex');
  return `pv_${digest.slice(0, 16)}`;
}

canonicalJson sorts object keys and drops undefined members, so { "id": 1, "status": "x" } and { "status": "x", "id": 1 } hash the same. store, dryRun, confirm, and previewId itself are excluded from the hash.

On a confirm call that sends previewId, the server recomputes the id from the call's own tool, store, and arguments. If it differs, the call is refused with preview_mismatch and nothing is sent. This check reads nothing from the store, so it is cheap and runs first.

Why a fingerprint and not a token

I deliberately did not make previewId a random one-time token stored on the server. A fingerprint has properties I wanted:

No server state. The server keeps no table of issued previews. It survives restarts, works across HTTP sessions, and works the same with one replica or several.
Deterministic. The same call always gets the same id, which makes audit logs easy to join and tests easy to write.

It also has costs, and the docs list them: the server does not check that a preview was actually requested, how old it is, or whether it was already used. It is a fingerprint, not a secret. If you need single use approvals, you need state, and that is a different design.

previewId is also optional. A confirm without it still works, because breaking every existing client to add a safety check is a good way to make people turn the check off.

Layer 4: stale previews (new in v0.5.0)

The previewId in v0.4.0 answered "did the caller change what it asked for?". It did not answer failure mode 3: "did the store change?". If a price moves from 20.00 to 25.00 between preview and confirm, the arguments { mode: "percent", change: 10 } are identical, so the arguments fingerprint matches, and the server would happily write 27.50.

v0.5.0 adds a second part to the id for tools that read before they write:

pv_e2a742fe51c807fc_a3510d17b8a51b93
   |<-- arguments -->| |<-- state -->|

Recording what the plan read

The planning client now runs with fresh: true (it bypasses the read cache) and an onRead callback that records every successful read. After planning, captureState fingerprints those reads. For every record returned it keeps the id, the date_modified, the fields the write sets, and a fixed list of fields writes are commonly computed from: status, type, price, regular_price, sale_price, the sale dates, manage_stock, stock_quantity, stock_status, and total. Payloads that are not records are hashed whole.

entries.push({ resource, dateModified, values });
// ...
const stateId = createHash('sha256').update(canonicalJson(fingerprint)).digest('hex').slice(0, 16);

Why not hash the whole response? Because WooCommerce responses contain fields that change for reasons unrelated to the write, and a state id that changes on every request would make every confirm stale. Hashing only what the write could plausibly depend on, plus date_modified as a catch-all, keeps false positives low without ignoring real changes.

The preview also shows those records, with before and after values, so "+10%" is no longer abstract:

"preconditions": {
  "stateId": "a3510d17b8a51b93",
  "recordCount": 1,
  "records": [
    {
      "resource": "products/42",
      "dateModified": "2026-10-30T09:12:44",
      "before": { "regular_price": "20.00" },
      "after": { "regular_price": "22.00" }
    }
  ],
  "note": "A confirm with this previewId is refused with stale_preview if these records change before it is applied."
}

Up to 50 records are listed, with truncated: true beyond that. The state id always covers all of them.

Checking at confirm time

When a confirm sends a two-part id and the arguments part matches, the registry plans the write again on fresh reads and compares state ids:

if (supplied?.stateId) {
  // Re-run the plan on fresh reads and compare the state it was computed from.
  const { state } = await plan(spec, ctx, handle.client, args);
  const currentStateId = state?.stateId ?? null;
  link = { ...link, stateMatch: currentStateId === supplied.stateId };
  if (!link.stateMatch) {
    throw new StalePreviewError(spec.name, handle.config.id, { /* both ids and current records */ });
  }
}
const result = await spec.run(ctx, args);

A mismatch returns stale_preview with both state ids and the current values, so a client can show the user what changed:

{
  "error": {
    "code": "stale_preview",
    "details": {
      "previewStateId": "a3510d17b8a51b93",
      "currentStateId": "8bed7e78cf74f185",
      "current": [
        { "resource": "products/42", "before": { "regular_price": "25.00" }, "after": { "regular_price": "27.50" } }
      ]
    }
  }
}

The server instructions tell clients what to do with it: request a fresh preview, show it again, and confirm with the new id.

Two error codes on purpose

preview_mismatch and stale_preview mean different things and I kept them separate. A mismatch means the caller changed what it asked for, which usually points at a client bug or a model retrying with different arguments. A stale preview means the world changed, which is normal in a busy store and just needs a re-preview. Arguments are checked first, so a call with both problems reports preview_mismatch.

Layer 5: the audit log

Set WOOCOMMERCE_AUDIT_LOG to a file path or stderr and every write attempt becomes one JSON line, with an outcome of applied, dry_run, confirmation_required, refused, or failed:

{"time":"2026-10-30T14:02:11.120Z","tool":"woocommerce_update_order_status","store":"default","outcome":"applied","arguments":{"id":1015,"status":"completed"},"client":"static:3f9a1c2b7d10","previewId":"pv_4f1c9a2e7b3d0856","previewMatch":true}

Because the preview entry and the applied entry share a previewId, you can answer "who approved this change, and what did they see?" with a grep. When the id had a state part, the entry also has stateMatch, and a stale_preview refusal is logged as refused with a stale object holding both state ids.

What this is not: it is not a lock

This is the part I most want people to read. The WooCommerce REST API has no conditional writes. There is no If-Match header I can send that makes the store reject a write if the record changed. So the sequence is:

confirm arrives
  -> re-plan on fresh reads      (GET products?include=42)
  -> compare state ids           (match)
  -> send the write              (POST products/batch)

A change that lands between the fresh read and the write is not detected. The check narrows the window from "whenever the user gets around to approving" (seconds to hours) to the length of a round trip. It does not close it.

Other limits, all in the safe writes guide:

Only what the plan read is covered. Tools that write without reading, such as woocommerce_update_order_status or woocommerce_create_product, keep the one-part id and behave as in v0.4.0.
It is opt-in per call. A confirm without previewId, or with a one-part id, is not checked for staleness. Clients that want the guarantee have to send the id back.
Multi-request tools are checked against the reads before their first write.
The re-plan costs a read. A confirm with a two-part id makes the same GETs as the preview. For a 100-product batch that is one extra list request, which I think is a fair price.

Testing it

The server has 142 tests (npm test runs them with Node's built-in test runner against a mocked WooCommerce API). The stale preview tests use a one-product store whose price the test can change between calls:

it('shows absolute before and after values and a state part in the previewId', async () => {
  const store = mockStore();
  const runtime = testRuntime(store.fetchImpl, writable);
  const preview = parse(await runTool(runtime, adjustSpec, raise));
  assert.equal(preview.status, 'confirmation_required');
  assert.match(String(preview.previewId), /^pv_[0-9a-f]{16}_[0-9a-f]{16}$/);
  // ...
});

The pattern that matters is preview, mutate the mock, confirm, and assert that no write request reached the mock.

What's next

More tools on the state check. Anything that reads before writing benefits, so new write tools should be planned through the same client from the start.
Optional single-use approvals for deployments that are happy to keep state, layered on top of the fingerprint rather than replacing it.
Better client guidance. The protocol gives me tool descriptions and server instructions, but how a host renders preconditions to a human is up to the host. I want to document a reference rendering.

If you are building MCP tools for any system with writes, I think the general pattern carries over: produce the preview from the same code path as the write, fingerprint both the request and the data it was computed from, give the two failures different names, and write down exactly where the guarantee stops.

On the search side of commerce, I also maintain llm-search-eval, which asks a different question: is a ranking change actually better, or just noise?
I wrote it up in Is your new search actually better?.

Try it

npx -y @sadrmanesh/woocommerce-mcp

Or with Docker:

docker run --rm -i -e MCP_TRANSPORT=stdio \
  -e WOOCOMMERCE_URL -e WOOCOMMERCE_CONSUMER_KEY -e WOOCOMMERCE_CONSUMER_SECRET \
  ghcr.io/sadrmanesh/woocommerce-mcp:0.5.0

It is MIT licensed. Issues and feedback on the stale preview design are very welcome, especially from anyone who has found a way to get conditional writes out of WooCommerce.

Why RAG breaks on Persian text, and what I built to fix it: normalization, half-spaces, spelling variants, and a regression bench

Naser Sadrmanesh — Sat, 31 Oct 2026 12:56:37 +0000

Most RAG tutorials assume the hard parts are the embedding model and the vector database. For Persian, a lot of the damage happens before either of them sees a byte. Two strings that look identical on screen are different code points. A word can be written three ways depending on how the author typed a half-space. A PDF can give you every word backwards. None of these raise an error. Retrieval just quietly gets worse.

I built persian-rag as a Persian-first RAG toolkit for Python. The core install is pure Python with no dependencies, and models, vector stores, parsers, and the HTTP server are optional extras. This post walks through the Persian-specific problems in pipeline order, the design decision behind each fix, and a small benchmark that shows where the fixes help and where they do not.

Problem 1: the same letter, different code points

Persian is written in an extended Arabic script, and several letters have an Arabic twin with a different code point that renders almost the same:

Arabic Yeh ي (U+064A) and Alef Maksura ى versus Persian Yeh ی (U+06CC)
Arabic Kaf ك (U+0643) versus Persian Keheh ک (U+06A9)

Text copied from Arabic-locale keyboards, older websites, or some CMS exports uses the Arabic forms. A user typing on a Persian keyboard uses the Persian ones. To BM25, and to many tokenizers, they are different words.

On top of that come three digit systems (ASCII 0-9, Persian ۰-۹, Arabic-Indic ٠-٩), the tatweel (kashida) used for stretching words, optional diacritics, and Arabic presentation forms from PDFs.

normalize handles all of it, driven by one config object:

from persian_rag import NormalizeConfig, normalize

normalize("يك كتاب با اعداد ٤٥٦")
# 'یک کتاب با اعداد ۴۵۶'

normalize("کِتاب", NormalizeConfig(remove_diacritics=True, digits="ascii"))
# 'کتاب'

The order of operations inside normalize is deliberate:

if options.compatibility_forms:
    text = unicodedata.normalize("NFKC", text)

text = text.translate(_INVISIBLE)

if options.unify_arabic_chars:
    text = text.translate(_ARABIC_TO_PERSIAN)

if options.remove_tatweel:
    text = text.replace(_TATWEEL, "")

if options.remove_diacritics:
    text = _DIACRITICS_RE.sub("", text)

text = _map_digits(text, options.digits)

if options.clean_punctuation:
    text = _clean_punctuation(text, options.punctuation)

text = _apply_zwnj_policy(text, options)
text = _normalize_whitespace(text)

NFKC runs first because it folds presentation forms (contextual glyphs such as U+FEE3) into base letters, and every later step expects base letters. Invisible characters such as bidirectional marks are stripped next so they cannot hide inside words. Half-space handling runs after punctuation cleanup, and whitespace is collapsed last, so neither step can undo the other.

Two design rules hold the whole thing together:

normalize is idempotent for a fixed config. normalize(normalize(t)) == normalize(t). That makes it safe to call at every layer without worrying about double application.
Documents and queries must use the same config. build_retriever and HybridRetriever apply it to both automatically, and a saved index stores the config and restores it on load. A mismatch here is the single easiest way to lose recall without noticing.

What it deliberately does not do: آ and ئ stay as they are. Folding them would merge words that are genuinely different, and that is a spelling question, not a code point question. It is handled later, at query time.

Problem 2: the half-space

The zero-width non-joiner (ZWNJ, U+200C), which Persian speakers call the half-space, is the most Persian-specific problem in the pipeline. It separates parts of a word visually without breaking it into two words. The verb prefix می, the plural suffix ها, and the comparative suffixes تر and ترین are joined to their word with it.

In practice people type it three ways: with a half-space, with a normal space, or attached with nothing in between. A search for the plural of "book" can miss a document because the author used a space where the query used a half-space.

persian-rag does two things. First, it corrects the common cases toward the half-space form, with regular expressions anchored on known prefixes and suffixes:

_PREFIX_SPACE_RE = re.compile(r"(?<![\w\u200c])(نمی|می|بی)[ \t]+(?=[\u0600-\u06FF])")
_SUFFIX_SPACE_RE = re.compile(r"(?<=[\u0600-\u06FF])[ \t]+(هایی|های|ها|ترین|تر)(?![\w\u200c])")

Then it cleans up the half-spaces themselves: repeated ones collapse into one, spaces around them are removed, and half-spaces next to punctuation or at the edges of a string are dropped.

normalize("كتاب هاي مي روم ٤٥٦")
# 'کتاب\u200cهای می\u200cروم ۴۵۶'

Arabic letters, spaced suffix, spaced prefix, and Arabic-Indic digits all land on one canonical form.

The policy is configurable with zwnj="normalize" (the default), "remove", or "space", because some downstream systems cannot handle the character at all. That choice has consequences in the tokenizer, which is the next problem.

Problem 3: tokenization

A tokenizer that splits on whitespace and punctuation, or one that treats U+200C as a separator, turns a single Persian word into two tokens. For BM25 that is a real loss: the suffix ها (plural) is extremely common, so it carries almost no IDF weight, and the stem token is shared with the singular form.

The tokenizer in persian-rag keeps the half-space inside the token:

# Letters only. Digits, diacritics, and Arabic punctuation stay out of words
# so "سلام،" does not become a single token.
_TOKEN_RE = re.compile(
    r"[0-9۰-۹٠-٩]+(?:[.\u066b][0-9۰-۹٠-٩]+)?|"
    r"[A-Za-z]+(?:'[A-Za-z]+)?|"
    r"[\u0621-\u063A\u0641-\u064A\u066E\u0671-\u06D3\u06FA-\u06FF\u200c]+"
)

Because normalization runs first, the spaced and half-spaced forms converge:

tokenize("می\u200cروم")   # ['می\u200cروم']
tokenize("می روم")        # ['می\u200cروم']
tokenize("میروم")         # ['میروم']

The first two become the same token. The third, written fully attached, does not. I chose not to split or join attached forms in the tokenizer, because doing that reliably needs a morphological analyzer, and a wrong split is worse than a missed match. Attached forms are handled by query expansion instead.

If you choose zwnj="space", you get what most generic tokenizers give you:

tokenize("کتاب\u200cها", config=NormalizeConfig(zwnj="space"))
# ['کتاب', 'ها']

Numbers are separate tokens, including decimals with the Arabic decimal separator (U+066B), and Latin tokens are lowercased so mixed Persian and English product names still match.

Problem 4: PDFs that store Persian backwards

Many Persian PDFs, especially older ones, store lines in visual order: the glyphs are in the order they appear on screen from left to right, so every Persian word comes out of the extractor reversed, while numbers and Latin words inside the line read normally. Indexing that text produces an index of reversed words that no query will ever match.

The loader detects this per page with a score built from two signals:

def visual_order_score(text: str) -> int:
    score = 0
    for word in _WORD_RE.findall(text):
        if len(word) > 1:
            first, last = _form(word[0]), _form(word[-1])
            if first == "FINAL" or last == "INITIAL":
                score += 1
            if first == "INITIAL" or last == "FINAL":
                score -= 1
        folded = unicodedata.normalize("NFKC", word)
        if folded in _REVERSED_COMMON:
            score += 2
        elif folded in _COMMON_WORDS:
            score -= 2
    return score

The first signal uses presentation forms: a word that starts with a final-form glyph or ends with an initial-form glyph is almost certainly reversed. The second uses a short list of very frequent words (از, به, که, در, این, and so on): seeing them reversed is strong evidence. Positive means visual order.

from persian_rag.loaders import clean_extracted_text, visual_order_score

line = "دش جراخ پاچ زا 1402 لاس رد باتک نیا"
visual_order_score(line)        # 6
clean_extracted_text(line)      # 'این کتاب در سال 1402 از چاپ خارج شد'

The line is reversed and the left-to-right runs inside it, like 1402, are restored. On the ingest CLI this is --rtl-fix auto (the default), with reverse and none for when you know better than the heuristic.

The heuristic can be wrong on very short lines or lines with no frequent words and no presentation forms. Detecting per page rather than per line is the compromise: there is more evidence on a page, and a PDF rarely mixes orders within one page.

Problem 5: spelling variants that normalization must not fix

Persian has many words with more than one accepted spelling. Room can be اتاق or اطاق. Tehran is تهران or the older طهران. Board is هیئت, هیات, or هیأت. Ticket is بلیت or بلیط. These are not encoding problems, so normalization should not touch them. Rewriting documents would destroy the original text you later want to cite.

Instead, persian-rag expands the query. QueryRewriter produces a short, ordered list of variants: the original query, spelling variants from a built-in table of groups, orthographic variants (hamza seat, alef madda, ئی and یی), half-space spacing variants of prefixes and suffixes, and optionally rewrites from a language model.

from persian_rag import QueryRewriter

QueryRewriter().variants("اطاق های هتل طهران")
# ['اطاق های هتل طهران', 'اطاق های هتل تهران', 'اتاق های هتل طهران',
#  'اطاق\u200cهای هتل طهران', 'اطاقهای هتل طهران']

The spacing variants are how the fully attached form from Problem 3 gets found.

RewritingRetriever wraps any retriever, runs every variant, and fuses the rankings with reciprocal rank fusion:

weights = [self.original_weight] + [1.0] * (len(variants) - 1)
fused = reciprocal_rank_fusion(rankings, k=self.rrf_k, weights=weights)

Two decisions here matter. The original query counts double (original_weight=2.0), so expansion adds recall without pushing exact matches down. And every result records which variants found it in metadata["matched_queries"], so you can see why a document was retrieved. Debuggable expansion is much easier to trust.

The same reciprocal rank fusion merges BM25 and dense results in HybridRetriever, with rrf_k=60 and separate weights for each side. RRF works on ranks, not scores, so I never have to calibrate a BM25 score against a cosine similarity.

Measuring it: a small Persian bench

I wanted a check that fails when a change breaks Persian-specific behavior, so the repository ships a benchmark in benchmarks/: 34 short documents (several are distractors that share words with other documents) and 42 labeled questions. Twelve questions are tagged spelling: they use a different accepted spelling than the corpus. The data is also on the Hugging Face Hub as sadrmanesh/persian-rag-bench.

python benchmarks/run_benchmark.py

These are the numbers I get with the offline hashing embedder (512 dimensions, character 2 to 4 grams), chunks of 400 characters, and k=10:

| Configuration    | Hit@1 | Hit@5 | MRR@10 | nDCG@10 | Hit@1 (spelling) |
|------------------|------:|------:|-------:|--------:|-----------------:|
| sparse           | 0.881 | 0.881 |  0.881 |   0.879 |            0.583 |
| dense            | 0.952 | 0.976 |  0.957 |   0.962 |            0.917 |
| hybrid           | 0.952 | 0.976 |  0.964 |   0.967 |            0.833 |
| sparse + rewrite | 0.952 | 0.976 |  0.964 |   0.967 |            0.833 |
| dense + rewrite  | 0.952 | 0.976 |  0.957 |   0.962 |            0.917 |
| hybrid + rewrite | 0.952 | 0.976 |  0.964 |   0.967 |            0.833 |

What I read from this:

BM25 alone misses five of the twelve misspelled queries, because a different spelling is a different token. The rewriter recovers most of them: sparse Hit@1 on the spelling subset goes from 0.583 to 0.833, and overall Hit@1 from 0.881 to 0.952. That is the effect the feature exists for.
The dense numbers flatter the embedder. The hashing embedder compares character n-grams, so it is naturally tolerant of one-letter differences like ت versus ط. A semantic embedding model behaves differently. The script takes --embedder sentence-transformers --model ... so you can measure yours.
Hybrid is not always best. On the spelling subset, hybrid (0.833) is below dense alone (0.917). Fusing in BM25, which misses those queries, pulls some correct dense results down. I left this in the docs on purpose. It is the kind of result that a benchmark is for.
Rewriting is not free. Each variant is another retrieval call. In-memory it stays in the low milliseconds, but against a remote search engine it multiplies requests.

The set is small and the scores are high by design. It is a regression check for Persian behavior, not a leaderboard, and a test runs the script offline on every change so it cannot quietly rot. For decisions that matter, extend qa.jsonl with questions from your own domain.

For retrieval metrics on your own data, evaluate_retrieval reports hit rate, MRR, recall, and nDCG at several cut-offs, and persian-rag eval does the same from the CLI. If you need statistically careful comparisons between two retrieval systems, that is the problem I tackled separately in Is your new search actually better? (llm-search-eval).

What else is in the box

Briefly, since this post is about the Persian-specific parts: loaders for text, Markdown, HTML (with main-content extraction), PDF, Word, and URLs; fixed, sentence, and recursive chunking; in-memory, FAISS, Qdrant, pgvector, Elasticsearch, and OpenSearch stores (the last two also serve BM25 with the persian analyzer); incremental indexing by content hash; cross-encoder or LLM reranking; citation-grounded answers that resolve [n] markers, including Persian and Arabic-Indic digits, to source spans; conversation memory; a FastAPI server; and a CLI for ingest, query, chat, eval, and serve.

Limitations

No morphological analysis. Attached forms, broken plurals, and Arabic-origin plural patterns are not unified by the tokenizer. Query expansion covers common spacing cases, not morphology.
The spelling table is hand-curated. It covers common groups, and you can add your own with QueryRewriter(extra_groups=[...]), but it is not exhaustive.
PDF order detection is a heuristic. It is good on normal prose and weaker on tables, short headings, and pages with little Persian text.
The benchmark is small and the default embedder is not semantic. Treat its numbers as a regression check, not as evidence about your model.

What's next

More spelling groups from real query logs, an optional morphological stemmer behind an extra so the core stays dependency-free, and larger evaluation sets with semantic embedders, published with the same honesty as the current table.

Install

pip install persian-rag
pip install "persian-rag[pdf,docx,qdrant,server]"   # pick the extras you need

from persian_rag import HashingEmbedder, RAGPipeline, build_retriever

texts = [
    "حافظ در شهر شیراز زاده شد و دیوان او مشهور است.",
    "فوتبال ورزشی گروهی با دو تیم است.",
]
retriever = build_retriever(texts, HashingEmbedder())
result = RAGPipeline(retriever, lambda prompt: "...").query("حافظ در کدام شهر زاده شد؟")

The HTTP API is also published as ghcr.io/sadrmanesh/persian-rag:0.3.1.

Repo: https://github.com/sadrmanesh/persian-rag
Docs (English and Persian): https://sadrmanesh.github.io/persian-rag/
Live demo: https://huggingface.co/spaces/sadrmanesh/persian-rag-demo

MIT licensed. If you work with Persian text and have a spelling group or a PDF that breaks the heuristics, please open an issue with an example.

Is your new search actually better? Label gaps, LLM judge panels, and paired statistics for offline search evaluation

Naser Sadrmanesh — Sat, 31 Oct 2026 12:53:50 +0000

Every search team eventually ships a ranking change on the strength of a table where the new system's nDCG@10 is a little higher than the old one's. Then it goes to production and nothing measurable happens.

Usually nothing went wrong in the code. The evaluation was answering a different question from the one people thought they were asking. I built llm-search-eval to make the evaluation answer the right one: is the new system better for shoppers, and how sure are we? This post is about the design decisions behind it, in the order the problems show up.

All numbers below come from the small bilingual electronics example that ships in the repository (10 queries, 22 products, 30 gold labels). It is far too small to make product decisions with, which is exactly why it is useful for showing where a naive evaluation goes wrong.

Problem 1: labels run out, and the gap is not neutral

Offline evaluation needs relevance labels: for each (query, product) pair, how relevant is it? Teams label a set once and reuse it. A retriever built later surfaces products that were never labeled.

The common default is to count an unlabeled result as not relevant. That sounds conservative, but it is not neutral. The system that is more different from whatever produced the labeled pool gets more unlabeled results and is penalized more. A new dense retriever is often the one being compared, and it is often the one that finds products the old pool never saw.

So the first thing llm-search-eval reports is coverage. Here is the example run with gold labels only and no judge:

llm-search-eval run \
  -q examples/electronics/queries.jsonl \
  -p examples/electronics/products.jsonl \
  --qrels examples/electronics/qrels.csv \
  -s bm25=examples/electronics/runs/bm25.jsonl \
  -s hybrid=examples/electronics/runs/hybrid.trec \
  -o reports/

## Label coverage

| System | Queries answered | Labeled in top 10 | Unknown products |
| :--- | ---: | ---: | ---: |
| `bm25` | 9/10 | 48.3% | 0 |
| `hybrid` | 10/10 | 80.6% | 0 |

Gold labels used: 29, new judgments: 0, from cache: 0, failed: 0, still unlabeled: 18.

Less than half of BM25's top 10 has a label. Any number computed on that is partly a measurement of which system the labelers happened to look at. Whatever else you do, look at this table before the summary table.

Filling gaps without touching gold labels

The fix is to label the gaps, and an LLM judge is the cheapest way to do it at scale. The resolution order for every pair in each system's top depth is strict:

Gold label from --qrels, if present. Gold labels are never re-judged.
Cached judgment, if the same judge configuration already saw this exact query and product content.
Judge, if --judge is set. Otherwise the pair stays unlabeled, and the report shows the gap.

Then every system is scored against the same merged label pool. That detail matters: the ideal ranking used by nDCG and the recall denominator are identical for every system, so a judgment added because of system B also counts when scoring system A.

Finding the gaps is a short function:

def pending_pairs(dataset: Dataset, runs: Sequence[Run], depth: int) -> list[tuple[str, str]]:
    """``(query_id, product_id)`` pairs in the runs' top ``depth`` that have no gold label."""
    pending: dict[tuple[str, str], None] = {}  # ordered set
    for run in runs:
        for qid in dataset.queries:
            for pid in run.ranking(qid, depth):
                if (qid, pid) not in pending and dataset.gold_label(qid, pid) is None:
                    pending[(qid, pid)] = None
    return list(pending)

Judges grade on a 0 to 3 scale (3 exact match, 2 relevant, 1 partially relevant, 0 irrelevant), the same scale as the gold labels, so the two can be compared directly.

Problem 2: one judge is one opinion

An LLM judge has its own biases. Some are lenient, some ignore attributes like size or color, some behave differently on Persian queries than on English ones. Swapping human labels for a single model's opinion just moves the bias somewhere you cannot see it.

llm-search-eval handles this in two ways.

Panels. Pass --judge more than once and the judges form a panel. Grades are combined with --aggregate (majority, mean, median, min, or max), and --min-votes sets how many members must answer, so one failing endpoint does not stop the run.

Agreement and calibration. The report measures how much the panel agrees with itself (Fleiss' kappa, Krippendorff's alpha, pairwise Cohen's kappa plain and quadratic-weighted) and, with --calibrate, how much each judge agrees with your gold labels. Gold labels still decide the scores; calibration pairs are judged only to measure the judge.

To show what that looks like without an API key, here is a panel of three mock judges, two of them with injected noise:

llm-search-eval run ... \
  --judge mock --judge "mock=noise=0.2,seed=1" --judge "mock=noise=0.2,seed=2" \
  --calibrate --bootstrap 2000 -o reports/

### Inter-judge agreement

| Statistic | Value |
| :--- | ---: |
| Items with every vote | 48 |
| Unanimous | 72.9% |
| Fleiss' kappa | 0.732 |
| Krippendorff's alpha (ordinal) | 0.873 |

### Agreement with gold labels

| Rater | n | Exact | Within one | Binary | Weighted kappa | Mean bias |
| :--- | ---: | ---: | ---: | ---: | ---: | ---: |
| `panel(majority: ...)` | 30 | 33.3% | 66.7% | 63.3% | 0.257 | -1.000 |

This output is a good illustration of why calibration exists. The three judges agree with each other quite well (Fleiss' kappa 0.732). They agree with the gold labels badly: 33.3% exact, weighted kappa 0.257, and a mean bias of -1.0, meaning they grade a full point harsher than the humans. The mock judge is deterministic term coverage, meant for tests and CI, not a quality signal, and the calibration table says so. High inter-judge agreement only tells you the judges share a bias. You need the gold comparison to know whether it is the right one.

With a real model the same flags apply:

llm-search-eval run ... --judge openai=gpt-4o-mini --cache .judge-cache.jsonl \
  --price gpt-4o-mini=0.15,0.60 --max-cost 2

The openai judge works with any OpenAI-compatible chat completions endpoint, including local servers such as Ollama or vLLM (--judge "openai=qwen2.5:14b,base_url=http://localhost:11434/v1").

Problem 3: a cache that lies about which prompt produced a label

A judgment cache is essential. Re-running an evaluation on every pull request should not pay for the same (query, product) pair twice. But a cache keyed on ids is subtly wrong: if a product description is edited, the old label is served for the new text.

So keys hash content, not ids:

def cache_key(fingerprint: str, query: Query, product: Product) -> str:
    material = "\x1f".join((fingerprint, query.text, query.lang or "", product.to_text()))
    return hashlib.sha256(material.encode("utf-8")).hexdigest()

The other half is the fingerprint, which identifies the judge. Until v0.4.0 it included the endpoint, model, temperature, and a hand-bumped prompt version. The flaw: if someone edits the prompt text and forgets to bump the version, every cached label from the old prompt keeps being served as if it came from the new one. Nothing fails. The numbers just quietly mix two different judges.

v0.4.0 adds a prompt hash computed from the text itself:

def prompt_hash(*templates: str) -> str:
    """First 12 hex characters of the SHA-256 of the templates, joined by a unit separator."""
    return hashlib.sha256("\x1f".join(templates).encode("utf-8")).hexdigest()[:12]

#: Hash of the rendered system prompt and the user prompt template.
PROMPT_HASH = prompt_hash(system_prompt(), USER_PROMPT_TEMPLATE)

and puts it in the fingerprint, and so in every cache key:

@property
def fingerprint(self) -> str:
    return (
        f"openai:{self.base_url}:{self.model}:t={self.temperature}"
        f":{PROMPT_VERSION}:{PROMPT_HASH}"
    )

Any edit to the prompt changes the hash, and a judgment made with a different prompt is never reused. The version (relevance-v1, pairwise-v1), hash, provider, and model are also written to report.json, the Markdown report, and the dashboard, so a result always says which prompt produced it. The trade-off is that upgrading to v0.4.0 re-judges old cache entries once. I chose a one-time cost over silent contamination.

Problem 4: averages hide noise

With labels fixed, the next mistake is reading a mean difference as a result. Ten or fifty queries is a small sample, and per-query nDCG varies a lot.

Paired randomization test

Every system is compared to a baseline with a two-sided paired sign-flip randomization test on the per-query values of the primary metric. Paired, because both systems ran on the same queries, and pairing removes the variation between easy and hard queries. Randomization, because it makes no normality assumption about nDCG, which is bounded and often skewed.

def paired_randomization_test(
    a: Sequence[float], b: Sequence[float], permutations: int = 10_000, seed: int = 13
) -> float | None:
    """Two-sided paired sign-flip randomization test on the mean difference."""
    if len(a) != len(b):
        raise ValueError("paired samples must have equal length")
    diffs = [x - y for x, y in zip(a, b, strict=True)]
    if not diffs or permutations <= 0:
        return None
    observed = abs(sum(diffs))
    if observed == 0:
        return 1.0
    rng = random.Random(seed)
    extreme = 0
    for _ in range(permutations):
        total = sum(d if rng.random() < 0.5 else -d for d in diffs)
        if abs(total) >= observed - 1e-12:
            extreme += 1
    return (extreme + 1) / (permutations + 1)

The idea: if the two systems were interchangeable, the sign of each per-query difference would be a coin flip. Flip signs at random many times and count how often the total is at least as extreme as what you observed. The +1 on both sides keeps the p-value from ever being exactly zero, which it should not be with a finite number of permutations. The seed is fixed so the same inputs give the same p-value.

Paired bootstrap intervals

A p-value says whether a difference is distinguishable from zero. It does not say how big it plausibly is. So every mean gets a percentile bootstrap interval over queries, and every delta gets a paired bootstrap interval, where queries are resampled as units so both systems see the same queries in each resample:

def paired_bootstrap_ci(
    a: Sequence[float],
    b: Sequence[float],
    *,
    confidence: float = DEFAULT_CONFIDENCE,
    resamples: int = DEFAULT_RESAMPLES,
    seed: int = 13,
) -> Interval | None:
    """Bootstrap interval for the mean paired difference ``a - b``.

    Queries are resampled as units, so both systems see the same queries in
    each resample. This is the interval that matches the paired randomization
    test used for p-values.
    """
    if len(a) != len(b):
        raise ValueError("paired samples must have equal length")
    return bootstrap_ci(
        [x - y for x, y in zip(a, b, strict=True)],
        confidence=confidence,
        resamples=resamples,
        seed=seed,
    )

On the example data, with the mock panel from above filling the gaps:

| System | Δ ndcg@10 | Δ 95% CI | Relative | Win / tie / loss | p-value |
| :--- | ---: | ---: | ---: | ---: | ---: |
| `hybrid` | +0.4470 | [+0.3401, +0.5862] | +84.3% | 10 / 0 / 0 | 0.0020 (significant) |

The hybrid system wins all 10 queries, so even 10 queries are enough. Note the p-value: with 10 queries and all differences positive, the smallest possible two-sided p-value is 2/1024, about 0.002. That is a property of the sample size, not of the systems, and it is worth knowing before you look for p < 0.001 on a small set.

The report also prints a line I think every evaluation tool should print: narrow intervals need more queries, not more judgments. Filling label gaps fixes bias. It does not fix noise. Only more queries do.

That is the honest answer to the question I get most often: "my new system is +2% on 50 queries, what now?" Check coverage first. If many top results are unjudged, fill them and re-run, because the +2% may be a labeling artifact. Then look at the paired interval. If it still crosses zero, you need more queries.

Pairwise judging as a second opinion

Pointwise grades can miss ordering differences when two systems are close. With --pairwise-judge, a judge sees the top --pairwise-k results of both systems as two lists and picks the one a shopper would prefer. Each query is judged in both orders by default, and a system wins only when both orders agree, which cancels position bias. Opposite verdicts are counted as inconsistent, a direct read on how stable the judge is. Results come with a Wilson interval on the win rate and an exact sign test (ties ignored).

Problem 5: cost surprises

Judging thousands of pairs with a large model adds up. Two features exist so that nobody learns the cost from an invoice.

Estimate before you run. llm-search-eval estimate takes the same arguments as run, works out which pairs still need a judge after gold labels and the cache, and predicts calls, tokens, and cost without calling any model:

$ llm-search-eval estimate -q queries.jsonl -p products.jsonl --qrels qrels.csv \
    -s bm25=runs/bm25.jsonl -s hybrid=runs/hybrid.trec \
    --judge openai=gpt-4o-mini --price gpt-4o-mini=0.15,0.60 --max-cost 0.01
queries: 10
result pairs: 47 (29 gold, 0 unknown products)
pairs to judge: 18 (0 already cached)
  gpt-4o-mini: 18 calls, 5624 in + 720 out tokens, $0.0013
total: 18 calls, about 6344 tokens, $0.0013 (assumes 40 output tokens per call)
budget: fits

Input tokens are counted from the real prompts that would be sent, using a rough rule (about four characters per token for Latin text and two for Persian), so the estimate is approximate. It exits with 1 when the budget is too small, which makes it usable as a CI pre-check.

Budgets during the run. --max-cost, --max-tokens, and --max-judgments stop judging when a limit is hit. The run still completes, the remaining pairs stay unlabeled, and the coverage table shows how many were skipped. Prices are never built in, because they change and differ between providers. A model without a price still counts tokens, and the total cost is reported as unknown rather than as a misleading partial number.

Putting it in CI

llm-search-eval gate turns a report into a pass or fail check:

llm-search-eval gate reports/report.json --min ndcg@10=0.6 --max-drop 0.01 --significant-only

--significant-only fails the build only when a drop is statistically significant, which stops noisy small regressions from blocking every pull request. The repository is also a GitHub Action that runs the evaluation, applies the gate, and keeps one pull request comment updated on every push.

Limitations

LLM labels are still model opinions. Calibration tells you how far off they are on the pairs you have gold labels for. It cannot tell you about query types your gold set does not cover.
The token estimate is a heuristic, not a tokenizer.
Small query sets stay small. No amount of statistics turns 10 queries into a decision. The tool reports the uncertainty; it cannot remove it.
Offline is not online. This measures relevance against labels. It does not replace an A/B test on conversion.

What's next

The roadmap in the README lists what I want to do next: pairwise results as a gate condition, active selection of the queries most worth labeling next (the direct answer to "you need more queries, but which ones?"), a Vespa connector with multi-page fetching for deep runs, and interleaving and click log metrics for online comparison.

If your retrieval is in Persian, I also wrote about the language-specific side of that problem in Why RAG breaks on Persian text (persian-rag).

Install

pip install llm-search-eval

It has zero runtime dependencies and supports Python 3.10 to 3.13. There is also a Docker image:

docker run --rm -v "$PWD:/work" sadrmanesh/llm-search-eval:0.4.0 run \
  -q eval/queries.jsonl -p eval/products.jsonl --qrels eval/qrels.csv \
  -s bm25=eval/runs/bm25.jsonl -s hybrid=eval/runs/hybrid.trec --judge mock -o reports

Repo: https://github.com/sadrmanesh/llm-search-eval
Docs: https://sadrmanesh.github.io/llm-search-eval/
Sample dashboard: https://sadrmanesh.github.io/llm-search-eval/demo/report.html

MIT licensed. If you run it on a real catalog and the calibration table surprises you, I would like to hear about it.

Treat agent skills like code: evals, a strict validator, and an npx installer for commerce skills

Naser Sadrmanesh — Sat, 31 Oct 2026 12:51:06 +0000

The most expensive webhook bug I know is boring. Your order handler receives order.paid, stores the event, creates the fulfillment, sends the confirmation email, and then takes a little too long to answer. WooCommerce times out and posts the same event again. Your handler sees a new delivery id, decides this is a new event, and ships the order twice.

A coding agent asked to "make the webhook idempotent" will often write exactly the wrong fix: store X-WC-Webhook-Delivery-ID and skip repeats. That header is a fresh log id for every attempt, so the fix does nothing for this case.

That gap, between what a capable generalist writes and what someone who has debugged a store at 2 a.m. writes, is what I tried to close with commerce-agent-skills. It is a library of 29 skills and matching rules for coding agents that build e-commerce and marketplace products: WooCommerce, Shopify, Stripe, Iranian payment gateways, inventory, returns, tax, feeds, search, accessibility, and more.

This post is not a tour of the catalog. It is about the engineering around it: how a skill is structured, how I test that it teaches what it claims to teach, how the validator keeps 29 skills consistent, and how the installer gets them into a project. I will use the webhook redelivery case throughout, because it is the clearest example of why this needs tests at all.

What a skill is

The skills follow the Agent Skills format: a folder with a SKILL.md that has name and description frontmatter, plus optional references and scripts. An agent reads the description to decide whether a skill applies, reads the body when it does, and opens references and scripts only when the body points to them.

The webhook skill looks like this on disk:

skills/order-and-webhook-reliability/
  SKILL.md
  references/woocommerce-webhooks.md
  scripts/verify-hmac-signature.mjs
  evals/tasks.json
  evals/examples/verify-raw-body.md
  evals/examples/dedupe-resource-version.md
  evals/examples/redelivery-after-timeout.md
rules/order-and-webhook-reliability.md

And the frontmatter:

name: order-and-webhook-reliability
description: "Accept order and payment webhooks with a raw-body signature check, durable deduplication, and reconciliation against the source of truth. Use when handling order events, payment notifications, or outbound webhook delivery."
license: MIT
compatibility: "The signature checker requires Node.js 20 or newer. No network access."
metadata:
  author: Naser Sadrmanesh
  version: "0.3.2"

Every skill also has a rule: a short plain-Markdown file of constraints for agents that keep always-on guidance in context. A skill is for a task. A rule is for every task in that area. The webhook rule includes this line:

- Treat a redelivery after a timeout as a duplicate: if the inbox row is already `applied`, return 2xx and run no side effect again.

Write for the failure, not the happy path

The skill body is a procedure, then the details, then Gotchas, Verification, and Evaluation sections. The part about redelivery is written around the failure, because the happy path is what the agent already knows:

The most common duplicate is a redelivery after a timeout. The handler stored and applied the event, but the sender timed out before the 2xx arrived and posts the same event again, often with a new delivery id. The retry must find the inbox row already applied, return 2xx, and run no side effect: no second fulfillment, no second confirmation email, no second stock decrement. This only holds when each side effect is recorded in the same transaction that marks the row applied, or is keyed on the event id where it runs.

It also gives a usable dedupe key instead of a principle. Use the provider event id when it is stable across retries. Otherwise, hash the topic, the resource id, and the resource's date_modified_gmt. And it records the platform facts an agent cannot infer:

WooCommerce counts a delivery as failed when the status is outside 2xx, 301, and 302.
After more than five consecutive failures it disables the webhook. The threshold comes from the woocommerce_max_webhook_delivery_failures filter, which defaults to 5.
WC_Webhook::deliver() posts with redirects disabled, so a 301 does not mean the JSON reached the next URL.

That last point matters operationally. A handler that keeps returning 500 because of a bug does not just lose events. Five more failures and the store stops sending webhooks at all, which is why the skill says to return a retryable error only when the event was not stored.

Scripts where a check beats a paragraph

Some things are easier to check than to describe. WooCommerce signs base64(hmac_sha256(secret, raw_body)), but it HTML-decodes the secret first with wp_specialchars_decode( $secret, ENT_QUOTES ). A secret containing & or a quote produces a mismatch that looks like a bug in your code. The skill ships a small checker with no dependencies:

export function decodeWebhookSecret(secret) {
  return secret
    .replaceAll("&quot;", '"')
    .replaceAll("&#039;", "'")
    .replaceAll("&#39;", "'")
    .replaceAll("&lt;", "<")
    .replaceAll("&gt;", ">")
    .replaceAll("&amp;", "&");
}

export function woocommerceSignature(secret, rawBody) {
  const key = decodeWebhookSecret(secret);
  return createHmac("sha256", key).update(rawBody).digest("base64");
}

export function signaturesMatch(expected, received) {
  if (typeof expected !== "string" || typeof received !== "string") {
    return false;
  }
  const left = Buffer.from(expected);
  const right = Buffer.from(received);
  if (left.length === 0 || left.length !== right.length) {
    return false;
  }
  return timingSafeEqual(left, right);
}

WEBHOOK_SECRET="..." node scripts/verify-hmac-signature.mjs \
  --body payload.json \
  --signature "$SIGNATURE"

It reads the secret from the environment or --secret-file and never prints it. Exit code 0 means the signature matches, 1 means a mismatch, and 2 means a usage error.

Evals: making a skill falsifiable

A skill that "covers idempotency" is not a claim you can check. I wanted every skill to make claims that can fail. Each skill has an evals/tasks.json with at least two tasks, and each task lists text that a correct answer must contain and text it must not contain:

{
  "id": "redelivery-after-timeout",
  "task": "The order.paid handler stored and applied the event and created the fulfillment, but WooCommerce timed out before the 2xx arrived and posted the same event again. Make the retry safe.",
  "expects": {
    "must_include": ["applied", "2xx", "side effect", "timeout"],
    "must_not_include": ["X-WC-Webhook-Delivery-ID"]
  }
}

The must_not_include is the important half. It encodes the specific wrong answer I described at the top. The validator requires at least one negative check per skill, because a test that only looks for good words is easy to pass with a long, vague answer.

Each task also has a worked example under evals/examples/, a short answer that shows what a passing response looks like:

Look up the inbox row by topic, order id, and `date_modified_gmt`. When its status is already `applied`, return 2xx with the original result and run no side effect: no second fulfillment, no second confirmation email, and no second stock decrement.

The scorer is deliberately simple. Every must_include and must_not_include entry is one check:

export function checkAnswer(answer, expects) {
  const errors = [];
  const text = String(answer);
  for (const needle of expects?.must_include ?? []) {
    if (!text.includes(needle)) {
      errors.push(`missing required text: ${needle}`);
    }
  }
  for (const needle of expects?.must_not_include ?? []) {
    if (needle !== "" && text.includes(needle)) {
      errors.push(`contains forbidden text: ${needle}`);
    }
  }
  return errors;
}

Running it

Without arguments, the runner scores every worked example and requires all checks to pass. That is a self-check: it proves the fixtures and examples agree. It runs in CI on every pull request.

$ node scripts/run-evals.mjs --skill order-and-webhook-reliability
 100%  order-and-webhook-reliability (3/3 tasks)

Overall 100%: 10/10 checks, 3/3 tasks passed (examples).

Across the library it reports Overall 100%: 211/211 checks, 60/60 tasks passed (examples).

The more useful mode is scoring real answers. You give your agent the task text, save each answer as answers/<skill>/<task-id>.md, and run the scorer with a threshold. Here is the wrong fix from the top of this post, scored:

$ node scripts/run-evals.mjs --answers ./answers --skill order-and-webhook-reliability --min-score 0.8
  60%  order-and-webhook-reliability (2/3 tasks)
         redelivery-after-timeout: missing required text: applied; missing required text: 2xx; missing required text: side effect; contains forbidden text: X-WC-Webhook-Delivery-ID

Overall 60%: 6/10 checks, 2/3 tasks passed (answers).
$ echo $?
1

The answer file said: "Store each X-WC-Webhook-Delivery-ID in a table and skip the request if the id was seen before. Make the handler faster so the timeout does not happen." It fails on the forbidden header, and it fails on three missing concepts. It exits with status 1 below the threshold, so you can put it in a pipeline that compares agent output with and without the skill installed.

Notice what it did not catch: the answer gets credit for timeout just by mentioning it. That is the honest limit of substring scoring, and I come back to it below.

The validator: consistency at 29 skills

With one skill, you can keep everything straight by reading it. With 29, drift starts on the first release. scripts/validate-skills.mjs runs in CI and before publish, and fails on any of these:

- metadata.version must match package.json exactly
- description must be between 100 and 320 characters
- SKILL.md must have Gotchas, Verification, and Evaluation sections
- a skill with scripts/ must set compatibility
- every relative link must resolve and stay inside the skill directory
- every file under references/ and scripts/ must be referenced from SKILL.md
- every script must pass `node --check`
- the rule must exist, be plain Markdown, start with the same heading as SKILL.md,
  and be 8 to 80 lines
- evals: at least two tasks, kebab-case ids, a worked example per task that passes
  its own checks, and at least one must_not_include per skill
- every must_include term must appear in the skill, its rule, or its references
- every skill must be in the catalog order and in exactly one website category
- no em dashes and no zero-width non-joiners in prose

Three of these deserve a word.

Every eval term must be taught. This is the check I am most glad I added. Without it, it is easy to write an eval that expects date_modified_gmt while the skill only says "use the resource version". Then the eval tests the agent's prior knowledge, not the skill. The check is a case-insensitive search through the skill, rule, and references:

export function untaughtTerms(tasks, taughtText) {
  const haystack = taughtText.toLowerCase();
  const missing = [];
  for (const task of tasks) {
    for (const needle of task?.expects?.must_include ?? []) {
      if (typeof needle === "string" && !haystack.includes(needle.toLowerCase())) {
        missing.push({ task: task.id, term: needle });
      }
    }
  }
  return missing;
}

The description length window. The description is what the agent uses to choose a skill, so it has to name both what the skill does and when to use it. Under 100 characters, it is usually missing the "use when". Over 320, it starts to crowd out other skills in the selection step.

Unreferenced files are errors. An agent opens references only when the skill points to them. A reference nobody links to is dead weight that will never be read, so the validator treats it as a bug.

The release workflow ties it together. A vX.Y.Z tag must match package.json, then the validator, evals, and tests run before anything is published. npm test runs 50 tests covering the installer, scorer, validator, and catalog generator.

The installer

The format does not name an install path, and different agents look in different places. So the package ships a small CLI with no dependencies, which copies files and downloads nothing after the package itself:

npx @sadrmanesh/commerce-agent-skills init

That copies every skill to .agents/skills/ and every rule to .agents/rules/. The rest of the surface:

commerce-agent-skills list [--json]
commerce-agent-skills init [options]
commerce-agent-skills add <skill> [<skill> ...] [options]

--target <dir>       Project directory to install into
--layout <name>      agents (.agents/skills and .agents/rules, default) or root (skills/ and rules/)
--skills-dir <path>  Skills directory, relative to --target
--rules-dir <path>   Rules directory, relative to --target
--only <a,b>         init: install only these skills
--no-rules           Do not copy the matching rules
--force              Replace skills that are already installed
--dry-run            Print what would be copied and write nothing

$ npx @sadrmanesh/commerce-agent-skills add order-and-webhook-reliability --dry-run
Would add order-and-webhook-reliability at .agents/skills/order-and-webhook-reliability and .agents/rules/order-and-webhook-reliability.md
Planned: 1 of 1 skills. Skills: .agents/skills, rules: .agents/rules

Two decisions about existing files. init is meant to be rerun, so it skips skills that are already installed. add names specific skills, so an existing one is an error unless you pass --force. Both validate every name and build the full plan before writing anything, so a typo in the third name does not leave you with two of three skills installed. Exit codes are 0 for success, 1 for a skill or file error, and 2 for a usage error.

Installed skills are copies, not links. You can edit them for your shop, and you can review them in a pull request like any other file in the repository. The trade-off is that upgrades are explicit: rerun with --force and read the diff.

Where this connects to MCP

One of the skills, mcp-store-tooling, covers exposing a store to an agent through the Model Context Protocol: tool annotations, input and output schemas, scopes, and dry runs with confirmation for risky writes. I wrote it alongside a real server, woocommerce-mcp, where risky writes go through a preview, a confirmation, and an audit log.

The design is in Safe write tools for agents.

Limitations

I want to be precise about what the evals prove.

Scoring is substring matching. It checks that the right concepts appear and that known wrong answers do not. It does not check that the reasoning is correct. An answer can mention applied, 2xx, side effect, and timeout in a wrong order and pass. The negative checks catch the common wrong answer, not every wrong answer.
The 100% figure is a self-check. 211 of 211 checks pass against worked examples I wrote. It shows the fixtures are consistent, not that any agent performs well.
The runner does not call an agent. You collect answers yourself and score them. That keeps it provider-neutral and free to run in CI, but it means a real comparison takes work on your side.
Platform facts drift. Header names, limits, and filters are documented against WooCommerce and Shopify behavior I checked, and the reference notes say to confirm against your version when you debug a live problem.

What's next

From the roadmap: graded eval rubrics beyond substring checks, more store platforms written to the same level of API detail as the WooCommerce and Shopify skills, more payment providers written like the Stripe and Zarinpal skills, and runnable reference services for the payout ledger and a feed upload, with the same checks the local scripts apply.

Install

npx @sadrmanesh/commerce-agent-skills list
npx @sadrmanesh/commerce-agent-skills add order-and-webhook-reliability stripe-payments
npx @sadrmanesh/commerce-agent-skills init --layout root

Node.js 20 or newer. The package is on npm and, under the same name, on GitHub Packages. The skills are also published as a dataset on Hugging Face at sadrmanesh/commerce-agent-skills.

MIT licensed. If you have a commerce bug that an agent keeps getting wrong, open an issue with the task and the wrong answer. That is exactly the shape of a new eval.

Four open-source tools for AI in e-commerce, and what building them taught me

Naser Sadrmanesh — Fri, 30 Oct 2026 22:01:31 +0000

I'm Naser Sadrmanesh, an AI product engineer and founder working on online stores. Over the last months I turned the tools I kept rebuilding for clients into four open-source projects. They are small, MIT licensed, and each one exists because a real project needed it. This post explains what each does and the design decision behind it.

1. llm-search-eval: is the new search actually better?

Every store eventually changes its search: keyword to vectors, a new reranker, query rewriting, synonyms. The usual offline check is to run 50 queries, compare nDCG, and ship if the number went up. That check fails in predictable ways.

Labels go stale. Gold labels were written for the old system. A new retriever surfaces products nobody labeled, so they count as misses and the new system looks worse than it is.
Small deltas are noise. +2% on 50 queries is often chance.
One LLM judge is one opinion. If you fill gaps with a model, someone has to check it against people.

llm-search-eval builds one label pool per run: your gold labels first, LLM judgments only for the gaps, and every system scored on the same pool. You can run a panel of judges and see Cohen's kappa, Fleiss' kappa and Krippendorff's alpha between them, plus how each judge agrees with your human labels. Every comparison gets a paired randomization test and bootstrap confidence intervals. Pairwise A/B judging runs in both orders to cancel position bias.

The part teams like most is boring: an estimate command predicts the number of judge calls, tokens and cost before you run anything, and budgets stop a run before it overspends. A GitHub Action posts the result as a pull request comment and fails the check when ranking regresses.

pip install llm-search-eval

It has no runtime dependencies and works with any OpenAI-compatible endpoint, including local models. There is a sample report and a live demo on Hugging Face that runs in the browser with no API keys.

2. woocommerce-mcp: let an agent touch the store, carefully

The Model Context Protocol makes it easy to give an AI agent tools. For a live store, the hard question is not "can it call the API" but "what happens when it is wrong".

woocommerce-mcp answers that with a few rules:

Stores are read-only until you allow writes.
Every write tool first returns a preview of the exact REST request. It is applied only when the tool is called again with confirm: true.
Every write attempt, applied or refused, goes to an audit log.
Customer data can be redacted, and tools can be allowed or denied by pattern, for example hiding delete_*.

Within those rules it covers 47 tools: catalog search, orders and fulfillment with tracking, refunds, bulk price and stock changes, coupons and scheduled sales, customer segments, shipping and tax, webhooks, and sales analytics. It runs over stdio or streamable HTTP with bearer tokens or OAuth 2.0, and it is on npm, GHCR, Docker Hub and the official MCP Registry. You can browse every tool and its schema in the tool explorer.

npx -y @sadrmanesh/woocommerce-mcp

The lesson: the preview and confirm loop costs one extra call, and it is the single feature store owners ask about first.

3. commerce-agent-skills: teach agents the domain, not just the syntax

Coding agents write plausible checkout code that misses things an experienced e-commerce engineer would never miss: idempotent webhooks, refunds that do not match the original currency unit, stock that oversells across warehouses, inaccessible product pages.

commerce-agent-skills is a library of 29 skills written in plain Markdown, each with rules, reference notes, gotchas, a verification checklist and evaluation fixtures. Topics include Stripe and Iranian payment gateways, returns and refunds, inventory across warehouses, B2B pricing, catalog migration, WCAG 2.2 accessibility, Core Web Vitals, WooCommerce and Shopify development, and MCP store tooling. A small scorer checks answers against the fixtures, so a skill change can be measured instead of guessed.

npx @sadrmanesh/commerce-agent-skills init

The skills are tool neutral: they are just files in .agents/skills/ that any agent can read. They are also published as a Hugging Face dataset.

4. persian-rag: retrieval that respects the language

Persian breaks naive pipelines quietly. Arabic and Persian Yeh and Kaf look identical but are different code points. Half-spaces come and go. Three digit systems appear in one document. PDFs often store Persian in reversed order. Common words have several accepted spellings.

persian-rag handles those first, then provides a normal RAG toolkit: loaders, chunking, BM25 plus dense retrieval with rank fusion, a spelling-variant query rewriter, reranking, answers with citations resolved to source spans, chat memory, evaluation, a FastAPI server and a Docker image. The core install is pure Python with no dependencies.

On its small bundled benchmark, the rewriter raises BM25 Hit@1 on misspelled queries from 0.58 to 0.83. That set is tiny, so I treat it as a regression test rather than a claim. You can try it in the live demo, and the benchmark is on Hugging Face.

pip install persian-rag

What ties them together

All four follow the same habits: docs sites for every project, CI on every change, tagged releases that publish to PyPI, npm, GHCR, Docker Hub or the MCP Registry automatically, a GitLab mirror, and honest limits written in the README. None of them is finished. If one of them solves a problem you have, I'd be glad to hear about it, and issues and pull requests are welcome.

Site: https://sadrmanesh.github.io/
GitHub: https://github.com/sadrmanesh