DEV Community: AumCreate

One page, one question: how to structure headings and FAQ blocks so an answer engine can quote a passage

AumCreate — Mon, 02 Nov 2026 04:38:43 +0000

Someone asks an assistant a question. The assistant fetches a handful of pages, splits them into chunks, and keeps the chunk that looks like an answer. If your page buries the answer under four paragraphs of context, the chunk that survives is the context, and you don't get quoted.

The unit that gets retrieved and quoted is the passage, not the page. Everything below follows from that: write pages whose passages survive being cut out and read alone.

What a chunk looks like by the time it reaches the model

Retrieval systems split HTML into pieces of a few hundred tokens, usually cutting near headings. The piece arrives without your navigation, without the paragraph before it, without the table of contents. A sentence that begins "as mentioned above" or "this approach" arrives with no referent.

Test any paragraph by asking: if I paste only this block into an empty document, does it still say something true and complete? If not, it is a weak chunk.

Should a page answer one question or several?

One. The heading, the opening line and the FAQ should all point at the same question, phrased the way a person would type it. A page that answers "how much does a 3-bedroom move cost in Singapore" will not also answer "how do I pack a kitchen" well enough to be quoted for either.

Splitting is not a rule about word count. A short page answering two questions is worse than two short pages, because the retrieval system may pull half of the first answer and match it against the second question.

When two of your pages answer the same question, merge them. Duplicate answers compete with each other and neither becomes the reference.

The first sentence under a heading has to be the answer

Readers who scan and extractors that chunk both want the answer in the first sentence, before the justification.

Which version would you quote?

## Why is my WordPress site slow?

Performance is a broad topic and there are many factors to consider. In this
article we will look at several areas that affect how quickly a page loads,
starting with the hosting environment and then moving on to caching...

## Why is a WordPress site slow?

A WordPress page is slow when PHP rebuilds the page on every request and the
finished HTML is never stored. The usual causes are no page cache, an expensive
uncached query on each load, and images served at full resolution.

The second version answers inside one sentence and stays understandable out of context. The first version only announces that an answer is coming.

Delete the announcement paragraph entirely rather than shortening it. Openings like "in this article we will cover" have no value as a chunk.

Write chunks that name their own subject

A passage that gets pulled out of your page loses its title. Pronouns lose their reference and the chunk becomes useless.

Instead of "it handles retries automatically", write "the queue handles retries automatically".
Instead of "this approach scales", write "the queue-based import scales".
Instead of "most people prefer the second option", name what the second option is.

Numbers need units and a date. "5 working days" beats "fast turnaround". "Checked in October 2026" tells the reader the number has an owner and an expiry, and both are things a model can repeat without distorting them.

How should FAQ blocks be arranged?

An FAQ block is a set of small, self-contained answer chunks, and it works when you treat each item as its own page in miniature.

One question per item, phrased the way it gets searched. "Do you charge a yearly fee?" beats "Pricing notes".
The first sentence answers. Detail, caveats, exceptions come after, and they can be dropped without breaking the answer.
No nested sub-questions. If an item needs a follow-up question, it needs its own page.
No filler openings. "Great question!" and "That's a common concern" are the first thing a chunk carries and the first thing a reader skips.
Only questions someone actually asked. Six invented questions that pad a section produce six useless chunks and dilute the ones that matter.
Keep the answers in the HTML source. Content injected by JavaScript after load is invisible to a fetcher that does not run scripts. A <details> element is fine because the text is in the markup; a tab widget that loads its panel by request is not.

<h2>Frequently asked questions</h2>

<details>
  <summary>Does a heading have to be phrased as a question?</summary>
  <p>No, but a question heading matches how people search and gives the
  extractor a clean boundary. "Pricing" is a weaker heading than "How much
  does a delivery cost?"</p>
</details>

<details>
  <summary>Can an extractor read text inside a collapsed element?</summary>
  <p>Yes, when the text is in the HTML. A details element ships its content in
  the markup even while collapsed; a panel fetched by JavaScript after load
  does not.</p>
</details>

Structured data for FAQ is optional. Markup can describe a page, but the text is what gets extracted and repeated. Adding schema to thin answers produces well-labelled thin answers.

A page skeleton you can copy

# How much does X cost?          <- the question, in the searcher's words

One or two sentences that answer it, with the number, the currency and
what is included. No preamble.

## What is included in the price?

The first sentence states the inclusion. Then the detail.

## What changes the price?

Named factors, each with a short explanation.

## How do I start?

The concrete next action, one step, not a menu of five.

## Frequently asked questions

Four to six items, each answer-first and self-contained.

Notice the heading is not "Pricing". A heading that repeats the question gives the extractor a clean boundary and gives the reader a reason to stop scrolling.

Small things that break otherwise good passages

Comparisons that say "the first option" without naming it, in a table where the column headers were cut off with the chunk.
Answers that begin with "it depends" and never reach the actual answer. Put the dependency in the second sentence.
Numbers with no date and no scope. A figure that was true last year, or true only for one project size, gets repeated as if it were general.
A single page that is really six questions stapled together, each half-answered.
Repeated boilerplate under every heading, so that three different chunks all start with the same sentence about the company.

Before you publish, read the first sentence of every section on its own

Scroll through the page and read only the first sentence under each heading, with the heading attached. If those sentences form a coherent answer to the page's question, the page will survive chunking. If any of them needs the preceding paragraph to make sense, rewrite it.

That check takes two minutes and catches most of what makes a page unquotable.

Cleaning a product CSV before import: encoding, image URLs, variant rows, and finding the line that broke

AumCreate — Mon, 02 Nov 2026 04:33:32 +0000

An 800-row product sheet looks like a 20-minute import until the importer stops somewhere past the halfway mark and says the SKU is duplicated. It does not say which line. That is usually the moment people start deleting rows until something passes.

Most of those failures come from the file, not the store. The encoding, the price column, the image URL column, the variation rows and the SKU column account for nearly all of them. Clean those in a script first and the import turns into a non-event.

Everything below is plain Python 3 and works on any CSV you are about to feed into an admin importer. Only the column names are WooCommerce-flavoured.

pip install charset-normalizer requests

Read the file as bytes and let the library guess

A CSV exported from Excel on a Chinese or Japanese Windows machine is GB18030, not UTF-8. A CSV exported from Excel on macOS often has a UTF-8 BOM. Open a GB18030 file with encoding="utf-8" and you get either a UnicodeDecodeError on line 400 or, worse, mojibake that imports fine and renders as Ã¤Â¸Â on the product page three days later.

import csv, io, re, sys, unicodedata
from pathlib import Path
from urllib.parse import urlparse
from charset_normalizer import from_bytes
import requests

ZERO_WIDTH = dict.fromkeys(map(ord, "\u200b\u200c\u200d\ufeff"), None)
ODD_SPACES = {"\u00a0": " ", "\u2007": " ", "\u202f": " ", "\u3000": " "}

def decode_any(path):
    raw = Path(path).read_bytes()
    guess = from_bytes(raw).best()
    if guess is None:
        sys.exit(f"no usable encoding for {path}")
    return str(guess).lstrip("\ufeff")   # drop the BOM if there is one

def clean(value):
    if value is None:
        return ""
    v = unicodedata.normalize("NFKC", value)
    v = v.translate(ZERO_WIDTH)
    for bad, good in ODD_SPACES.items():
        v = v.replace(bad, good)
    return re.sub(r"[ \t\r\n]+", " ", v).strip()

NFKC folds full-width letters and digits to ASCII, which is what you want in a SKU (ＡＢ－１２ becomes AB-12). It also rewrites some typographic punctuation, so skip it on the description column if the client cares about curly quotes.

Non-breaking spaces and zero-width characters are the other silent killer. They survive copy-paste from a chat app, look identical on screen, and make SKU == "AB-12" false for no visible reason.

Prices need one decision: which separator is the decimal one

1.299,00, 1,299.00, 1299, $1299.00, 1299,00 EUR all arrive in the same file when several people filled in the sheet. The rule that survives contact with real data: if both separators appear, the rightmost one is the decimal separator. If only one appears and it is followed by exactly three digits, it is a thousands separator. Everything else is decimal.

def parse_price(raw):
    s = clean(raw).replace(" ", "")
    if not s:
        return None, "empty"
    s = re.sub(r"[^\d.,\-]", "", s)        # drop currency symbols and letters
    neg = s.startswith("-")
    s = s.lstrip("-")
    if not any(c.isdigit() for c in s):
        return None, f"no digits in {raw!r}"

    last_dot, last_comma = s.rfind("."), s.rfind(",")
    if last_dot > -1 and last_comma > -1:
        dec = "." if last_dot > last_comma else ","
        thou = "," if dec == "." else "."
    elif last_dot > -1 or last_comma > -1:
        sep = "." if last_dot > -1 else ","
        digits_after = len(s) - s.rfind(sep) - 1
        if s.count(sep) == 1 and digits_after in (1, 2):
            dec, thou = sep, None
        else:
            dec, thou = None, sep
    else:
        dec = thou = None

    if thou:
        s = s.replace(thou, "")
    if dec and dec != ".":
        s = s.replace(dec, ".")
    try:
        value = round(float(s), 2)
    except ValueError:
        return None, f"unparsable {raw!r}"
    return (-value if neg else value), None

Run this before the import and eyeball the rows where the input contained a separator. A 1.299 that should have been 1.299 euros and got read as 1299 is a refund, not a rounding error.

Image URLs fail in ways a browser hides

A URL that opens in your browser can still break the import. The three that bite: a content-type that is text/html because the host serves a "hotlinking not allowed" page, an unescaped space from a filename like red shoe.jpg, and an http URL on an https store (mixed content, image silently dropped).

Check with a HEAD request, fall back to a streamed GET because plenty of CDNs answer HEAD with 403 or 405, and cache the result so 40 variations sharing one parent image do not make 40 requests.

SESSION = requests.Session()
SESSION.headers["User-Agent"] = "csv-preflight/1.0"

def check_image(url, cache):
    if url in cache:
        return cache[url]
    problem = None
    p = urlparse(url)
    if p.scheme not in ("http", "https"):
        problem = "not an absolute http(s) URL"
    elif " " in url:
        problem = "unescaped space in URL"
    elif "." not in p.netloc:
        problem = "host has no dot"
    else:
        try:
            r = SESSION.head(url, timeout=10, allow_redirects=True)
            if r.status_code in (403, 405, 501):
                r = SESSION.get(url, timeout=10, stream=True, allow_redirects=True)
                r.close()
            if r.status_code >= 400:
                problem = f"HTTP {r.status_code}"
            else:
                ctype = r.headers.get("content-type", "").split(";")[0].strip().lower()
                if not ctype.startswith("image/"):
                    problem = f"content-type {ctype or 'missing'}"
        except requests.RequestException as exc:
            problem = exc.__class__.__name__   # Timeout, SSLError, TooManyRedirects
    cache[url] = problem
    return problem

Split gallery columns on commas before checking, and strip quotes around each part — "img1.jpg, img2.jpg" pasted out of a spreadsheet carries both.

Variation rows are where the SKU logic breaks

A variable product in a CSV is two kinds of row: the parent (Type=variable) and one row per combination (Type=variation) whose Parent column holds the parent SKU. Four things go wrong, in this order of frequency:

a variation points at a parent SKU that exists in the shop but not in the file, or was renamed in one place only;
two rows share a SKU, usually the parent's price row copied down into the variations;
the variation has no attribute value, so the importer creates a combination that cannot be selected on the front end;
the parent has no price and no variation has one either, so the product imports at 0.

Build the parent index first, then validate the variations against it. Do not validate in file order — a variation can legally sit above its parent.

def check_skus_and_variants(rows, header):
    problems, seen, parents = [], {}, {}
    for line, row in rows:
        sku = row.get("SKU", "")
        if sku:
            if sku in seen:
                problems.append((line, "SKU", sku, f"duplicate, first seen on line {seen[sku]}"))
            else:
                seen[sku] = line
        if row.get("Type", "").lower() == "variable" and sku:
            parents[sku] = line

    attr_cols = [h for h in header if h.lower().startswith("attribute")]
    for line, row in rows:
        if row.get("Type", "").lower() != "variation":
            continue
        parent = row.get("Parent", "")
        if parent and parent not in parents:
            problems.append((line, "Parent", parent, "parent SKU not present in this file"))
        if attr_cols and not any(row.get(c) for c in attr_cols):
            problems.append((line, "", "", "variation has no attribute value"))
        if not row.get("Regular price") and not parents.get(parent):
            problems.append((line, "Regular price", "", "no price on the variation"))
    return problems

Finding the line that actually failed

The importer's error message counts data rows, starting at 1 for the first row after the header. csv.reader.line_num counts physical lines in the file, starting at 1 for the header. If any field contains a line break inside quotes — a description with <br> typed as a real newline — the two numbers drift apart and you end up staring at the wrong record.

Report both, and let the reader pick.

There is a second, sharper signal: the column count. An unquoted comma inside a description pushes every later field one column to the right, so the row arrives with more fields than the header. Catch that before anything else, because every field-level check after it is meaningless.

def load_rows(path):
    text = decode_any(path)
    reader = csv.reader(io.StringIO(text))
    header = [clean(h) for h in next(reader)]
    good, bad = [], []
    for rec in reader:
        line = reader.line_num
        if not any(clean(c) for c in rec):
            continue
        if len(rec) != len(header):
            bad.append((line, "", "", f"expected {len(header)} columns, got {len(rec)}"))
            continue
        good.append((line, {h: clean(v) for h, v in zip(header, rec)}))
    return header, good, bad

def main(src, report="import-report.csv"):
    header, rows, problems = load_rows(src)
    problems += check_skus_and_variants(rows, header)
    cache = {}
    for line, row in rows:
        for col in header:
            if "image" not in col.lower():
                continue
            for url in [u.strip().strip('"') for u in row.get(col, "").split(",") if u.strip()]:
                why = check_image(url, cache)
                if why:
                    problems.append((line, col, url, why))
        if "price" in " ".join(header).lower() and row.get("Regular price"):
            _, why = parse_price(row["Regular price"])
            if why:
                problems.append((line, "Regular price", row["Regular price"], why))

    problems.sort()
    with open(report, "w", newline="", encoding="utf-8") as fh:
        w = csv.writer(fh)
        w.writerow(["physical_line", "column", "value", "problem"])
        w.writerows(problems)
    print(f"{len(rows)} clean rows, {len(problems)} problems -> {report}")

if __name__ == "__main__":
    main(sys.argv[1])

A note on what the report means. physical_line is the line in the file; the importer's row number is that minus one, minus any multiline rows above it. Sort the report by physical_line, open the sheet, and look at the columns named — nine times out of ten the value in the value column is visibly wrong once you are looking at the right cell.

Fix the sheet, re-export as UTF-8 with all fields quoted, run the script again until the report is empty, then import. A clean file imports in one pass and, more importantly, the data that lands in the database is the same data you reviewed.

We publish free WordPress plugins (translation, structured data, crawler visibility, digital-goods checkout) at https://aumcreate.com/plugins?utm_source=devto&utm_medium=content&utm_campaign=devto-csv-import-field-cleaning

Auditing wp_options autoload, revisions and transients before you clean a WordPress database

AumCreate — Mon, 02 Nov 2026 04:28:26 +0000

Cleaning a WordPress database is easy to do in the wrong order. The commands that drop revisions and expired transients are one line each, and none of them ask whether the option you just un-autoloaded is read by a plugin on every page load. Audit first, delete second — that order is what lets you undo a mistake without restoring last night's backup.

Everything below runs against a real database. Substitute your own table prefix wherever wp_ appears.

Take a copy you can actually restore from

wp db export pre-cleanup-$(date +%F).sql

Move that file off the server. A dump sitting in the same web root is not a backup. If the site takes inquiries, do this at the quietest hour you have; a dump of a busy InnoDB database is a long read and it holds locks you don't want during business hours.

Then look at where the space actually is:

wp db size --tables --human-readable

SELECT table_name,
       ROUND((data_length + index_length) / 1024 / 1024, 1) AS mb
FROM information_schema.tables
WHERE table_schema = DATABASE()
ORDER BY (data_length + index_length) DESC
LIMIT 15;

If wp_options is near the top of that list on a site with a few hundred posts, the autoload column is the first thing to open.

wp_options: find out what ships with every request

Autoloaded options are read into memory on every request — front end, admin, REST, cron. A large autoloaded payload is a cost you pay whether the visitor needed it or not.

Newer schemas store autoload as on / off / auto-on / auto-off; older ones use yes / no. Check which you have before writing anything else:

SELECT autoload, COUNT(*) FROM wp_options GROUP BY autoload;

Then the total, and the biggest individual rows:

SELECT COUNT(*) AS options,
       ROUND(SUM(LENGTH(option_value)) / 1024, 1) AS kb
FROM wp_options
WHERE autoload IN ('yes', 'on', 'auto-on');

SELECT option_name,
       LENGTH(option_value) AS bytes
FROM wp_options
WHERE autoload IN ('yes', 'on', 'auto-on')
ORDER BY bytes DESC
LIMIT 25;

For every row near the top you need three answers before changing anything: who owns it, whether it is read on the front end, and whether it regenerates itself.

grep -rn "the_option_name" wp-content/plugins wp-content/themes --include=*.php | head -20

Crude, but it answers the ownership question faster than reading a settings screen. If nothing matches, the option probably belongs to something uninstalled. Note it down and handle it in a second pass, after you have confirmed the site still works without it.

Turning autoload off is one update:

UPDATE wp_options SET autoload = 'off' WHERE option_name = 'the_option_name';

On a schema that still uses yes / no, write 'no'. Then flush:

wp cache flush

Re-run the total query and write the number down. Come back in a week — some plugins call update_option() on save and autoload can get set back on, so an option you turned off today may be back on after the next settings save. Worth knowing before you tell anyone the database is clean.

Revisions: measure by parent, not by total

A revision is a full copy of post_content in wp_posts. The count is less interesting than which posts own the copies.

SELECT COUNT(*) AS revisions,
       ROUND(SUM(LENGTH(post_content)) / 1024 / 1024, 1) AS mb
FROM wp_posts
WHERE post_type = 'revision';

SELECT r.post_parent AS post_id,
       p.post_title,
       COUNT(*) AS revisions,
       ROUND(SUM(LENGTH(r.post_content)) / 1024 / 1024, 1) AS mb
FROM wp_posts r
JOIN wp_posts p ON p.ID = r.post_parent
WHERE r.post_type = 'revision'
GROUP BY r.post_parent, p.post_title
ORDER BY mb DESC
LIMIT 20;

The top of that list is usually one or two long pages that get edited on every sales call. Before deleting anything, note two things:

Autosaves are stored as revisions too, identified by autosave in post_name. Deleting them does not touch published content, but it will surprise someone who has a draft open in the editor right now.
Custom post types and page-builder content keep their revisions in the same table, and are deleted the same way.

Delete in chunks so a large site does not exhaust memory or hit command-line argument limits:

while ids=$(wp post list --post_type=revision --format=ids --posts_per_page=200) && [ -n "$ids" ]; do
  wp post delete $ids --force
done

Cap future growth in wp-config.php:

define( 'WP_POST_REVISIONS', 5 );

That is a ceiling going forward. It does not remove what already exists, and it does not shrink the table — see the rebuild step at the end.

Transients: expired is safe, orphaned needs a decision

With no persistent object cache, transients live in wp_options as a _transient_<key> row plus a _transient_timeout_<key> row.

Count the expired ones:

SELECT COUNT(*) AS expired
FROM wp_options
WHERE option_name LIKE '\_transient\_timeout\_%'
  AND option_value < UNIX_TIMESTAMP();

Those are dead. WordPress clears them on its own schedule eventually; doing it now is safe:

wp transient delete --expired

Orphans are a different case — a value row with no timeout row at all:

SELECT COUNT(*) AS orphans
FROM wp_options o
WHERE o.option_name LIKE '\_transient\_%'
  AND o.option_name NOT LIKE '\_transient\_timeout\_%'
  AND NOT EXISTS (
    SELECT 1 FROM wp_options t
    WHERE t.option_name = CONCAT('_transient_timeout_', SUBSTRING(o.option_name, 12))
  );

Some were set without an expiry on purpose; some were left behind by a plugin that stopped writing to them. Deleting an orphan costs a cache miss the next time the plugin asks for the value, so the risk is low. If the number is large, find the owning prefix first — a plugin that keeps state in a no-expiry transient may behave differently right after the flush.

One check worth doing before you count these as waste: if the site runs Redis or Memcached, transients live in the object cache and these option rows are leftovers from before it was enabled. Flush and re-count to find out which world you are in.

wp cache flush

Rebuild the tables and verify

Deleting rows does not shrink the files. On InnoDB tables with a lot of free space, a rebuild reclaims it:

wp db optimize

This is not free. It rebuilds tables and takes time proportional to their size, so run it in a quiet window rather than at 2pm on a Tuesday.

Verify in this order:

wp option get siteurl
wp post list --post_type=page --fields=ID,post_title,post_status --posts_per_page=5
wp plugin list --status=active

Then in a browser, with a cache-busting query string: the home page, the page that had the most revisions, the inquiry or checkout form, and the admin dashboard. Watch the browser console for 404s, and open the settings screen of any plugin whose options you touched — that is where a wrongly deleted option shows up first.

Re-measure what you measured at the start:

SELECT COUNT(*) AS options,
       ROUND(SUM(LENGTH(option_value)) / 1024, 1) AS kb
FROM wp_options
WHERE autoload IN ('yes', 'on', 'auto-on');

Compare the two numbers. If the autoloaded total barely moved, the database was not your bottleneck and the next place to look is the slow query log, not another cleanup plugin.

The order that holds up: export, measure, identify the owner, change one class of rows at a time, verify the front end, rebuild, re-measure. Every step you skip is a step you redo from the backup.

If you want an outside read on what the site actually serves after the rebuild — broken assets, redirect chains, pages that no longer resolve — a site check at https://aumcreate.com/tools/site-check?utm_source=devto&utm_medium=content&utm_campaign=devto-wp-db-cleanup-order covers that from the visitor's side.

Which AI crawler is actually loading your server, and which one to rate-limit first

AumCreate — Sun, 01 Nov 2026 14:28:31 +0000

A site slows down, someone opens the access log, sees a wall of bot requests, and blocks them. Two days later the site is slow again and the list of bots has changed. The step that got skipped is the one that makes the answer stable: pricing what each crawler actually costs before you change anything.

This is the sequence I'd run on a WordPress box, and it works the same on any LAMP-style stack. It ends with a ranked list and one change to make, instead of a firewall full of guesses.

First check whether the web tier is even the bottleneck

cat /proc/loadavg
top -b -n1 -o %CPU | head -15

If the top CPU consumers are php-fpm and mysqld, HTTP requests caused this and the access log holds the evidence. If you see heavy kswapd, kcompactd, or %wa near 100, memory or disk is the problem — rate-limiting a crawler buys you a few days and nothing else. Look at the PHP-FPM pool as well. All children busy with a non-zero listen queue means requests are queueing, so continue. Idle children while load is high means the time is going somewhere else: cron, a CLI import, a backup job.

Get timing into the log

The default combined format gives you bytes and user agent but no duration, and duration is what separates a crawler making 200 cheap requests from one making 200 expensive ones.

nginx:

log_format main_ext '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" "$http_user_agent" '
                    'rt=$request_time';

access_log /var/log/nginx/access.log main_ext;

nginx -t && systemctl reload nginx

Apache: add %D to a custom LogFormat; it's the request duration in microseconds. Existing lines keep the old format, so give it an hour before reading.

Slice by user agent

Both nginx and Apache combined quote the request, the referer and the user agent. That makes " a reliable field separator, and the UA the sixth field:

awk -F'"' '{print $6}' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn | head -20

You'll get the usual mix: uptime monitors, search engines, your own scripts, and whatever AI crawlers your content attracted. Note the candidates, then move to cost.

Price each bot in worker seconds

Hit counts flatter crawlers that are noisy but cheap. Multiply by time instead:

awk -F'"' '
  {
    split($3, s, " ")        # s[2]=status, s[3]=body bytes
    ua = $6
    rt = $NF; sub(/^rt=/, "", rt)
    hits[ua]++
    bytes[ua] += s[3]
    secs[ua]  += rt
  }
  END {
    for (u in hits)
      printf "%10.1f %8d %13d  %s\n", secs[u], hits[u], bytes[u], u
  }' /var/log/nginx/access.log | sort -rn | head -20

Columns: seconds of request handling, hits, bytes served, user agent. A crawler at 8,000 hits and 40 seconds is background noise. One at 900 hits and 600 seconds is your problem, and it's probably hitting something expensive.

If the file is too large to parse in one pass, sample it:

shuf -n 500000 access.log > sample.log

Ratios survive sampling. Absolute counts don't, so don't quote them anywhere.

Look at the paths the top bot requests

awk -F'"' '$6 ~ /BotName/ {split($2, r, " "); print r[2]}' access.log \
  | sort | uniq -c | sort -rn | head -20

Replace BotName with a token from the previous step. What you're looking for:

Search results, faceted filters, cart and checkout URLs. Uncacheable, database-backed, and often the entire cost.
Pagination deep in the archive, or tag pages that nothing links to.
A single URL requested thousands of times, which is a crawler bug or a redirect loop more than a load problem.

If the expensive paths are the same across several bots, fix the paths first. It's a smaller change and it applies to every visitor.

Prove causality before you throttle

A user agent is a claim, and a throttle is cheap to test. Short-circuit one UA temporarily:

# temporary, inside the server block
if ($http_user_agent ~* "BotName") {
    return 403;
}

Reload, wait 20–30 minutes, and compare load average, PHP-FPM queue depth and database CPU against the same window before the change. If nothing moved, that bot wasn't the cause and you can drop the rule. Then check whether the UA is genuine rather than a scraper wearing it:

dig +short -x 203.0.113.10        # reverse lookup
dig +short crawl.example.com      # forward lookup, should return the same IP

Forward-confirmed reverse DNS is what separates the real crawler from someone running curl with a copied header. It matters most right before you write a permanent rule.

Order to act in

Order	Change	Fixes	How to verify
1	Cache or deny the expensive paths	Every bot at once	Re-run the path query, seconds column drops
2	Per-bot rate limit	The highest-cost bot only	Hit count falls, 429s appear for that UA
3	`robots.txt` rules	Crawlers that honour it	Read the log again after 24–48h
4	Firewall block	One persistent offender	Requests stop entirely

A per-UA rate limit looks like this:

map $http_user_agent $bot_key {
    default          "";
    ~*GPTBot         "gptbot";
    ~*ClaudeBot      "claudebot";
}

limit_req_zone $bot_key zone=ai_bots:10m rate=30r/m;

server {
    limit_req zone=ai_bots burst=20 nodelay;
    limit_req_status 429;
    # ...
}

Requests with an empty key aren't counted, so ordinary visitors are untouched. Use the tokens you actually saw in your own log, not the two above. If your build exposes $limit_req_status, add it to the log format and you can confirm the limits are firing instead of assuming.

robots.txt sits third for a reason: it's advisory. Some crawlers honour it, some don't, and the only way to know which happened is to read the log a day later. Write the file, then verify it, rather than reporting it as done.

Close the loop with the numbers you started with

Run the cost query again over a window after the change. You want the same shape of answer as before: which UA is first, and how many seconds it now costs. If the top entry has changed, you fixed the last problem and this one was already there waiting.

Wrap the whole sequence in a small script and keep it. One command that prints seconds, hits and bytes per user agent, sorted, is worth more than a firewall rule nobody can explain six months from now.

If you'd rather not maintain the parser: AumCrawl is a free plugin that records which AI crawlers are hitting a WordPress site and lets you allow or block them individually from the admin. It sits next to the log work above rather than replacing it — the log is still what tells you what each bot costs.

Splitting B2B export keyword research by part number, application and industry

AumCreate — Sun, 01 Nov 2026 14:25:04 +0000

The output of this process is a spreadsheet: one row per query, one column for the vocabulary the query belongs to, one for where you found it, one for the page it maps to, one for what to do with it. Everything below is about filling those columns honestly. The research part is mostly deciding which rows deserve a page of their own.

The vocabulary column decides everything else

B2B buyers search in one of three vocabularies, and which one they use tells you where they are in the buying process and what kind of page can answer them.

Vocabulary	What the buyer already knows	Query shape	Page that answers it
Part number	the exact part	`ABC-1234`, `ABC1234 datasheet`, `replacement for XYZ-45`	family page with parameters, variants, documents
Application	the job the part has to do	`600V mosfet for LED driver`, `valve for 180C steam line`	application page with an app note or a test
Industry	the regulated world they sell into	`automotive grade ...`, `medical device ...`	segment page with certifications and references

Sort every row before you do anything else. A list that mixes the three gets prioritised by whatever has the biggest volume number, which is usually the vaguest query and the hardest page to write.

Part numbers: the shortest path from search to enquiry

Start by collecting the strings. They're already written down somewhere:

your catalogue and order history
customer emails and RFQs — the exact string is in there, punctuation included
distributor spreadsheets and price lists
the cross-reference tables printed in datasheets, both yours and the ones for parts you replace
drawings, packaging, spare-part lists

Judging demand works differently here. Volume tools return zero or a dash for most of these strings, and that means "not enough data", not "nobody searches". Better evidence, in rough order of strength: the string appears in an actual enquiry; the string shows impressions in your search console query report; a distributor asked for it by name.

Normalise before you count. The same part arrives as ABC 1234, ABC-1234, abc1234, ABC/1234. Counted raw, that's four queries where there is one.

With a CSV that has a query column, this groups strings into families and shows you where the spellings cluster:

import csv
import re
import sys
from collections import defaultdict

FAMILY = re.compile(r"([A-Za-z]{1,5})[\s\-_/]?(\d{2,6})")


def family(query):
    match = FAMILY.search(query)
    if not match:
        return None
    prefix, digits = match.group(1).upper(), match.group(2)
    return f"{prefix}{digits[:3]}"


rows = list(csv.DictReader(open(sys.argv[1], encoding="utf-8")))
by_family = defaultdict(set)
for row in rows:
    key = family(row["query"])
    if key:
        by_family[key].add(row["query"].strip().lower())

for key, queries in sorted(by_family.items(), key=lambda item: -len(item[1])):
    sample = "; ".join(sorted(queries)[:5])
    print(f"{key}\t{len(queries)}\t{sample}")

The heuristic is deliberately crude — prefixes like LED or AC will pull unrelated strings together, so read the output before you build anything from it. What you're looking for is families with several distinct spellings. Those are the ones where buyers are already searching by part number. A family with exactly one spelling, and that spelling matches your internal code, is usually a string you invented.

A family earns its own page when you sell it, you can fill the page with parameters, variants and documents, and more than one spelling shows up. A single SKU with no documents belongs in a row of the family table, not on its own URL. Hundreds of near-identical pages end up competing with each other for the same query, which is worse than one complete page.

Applications: queries from buyers who don't know your part numbers

These buyers describe the job. The query pattern is a part type plus "for" or "used in" plus a condition: temperature, voltage, IP rating, package, certification.

Where they show up: support tickets, notes from sales calls, trade-show conversations, distributor application pages, industry forums, and the applications section of your own datasheets. That last one is underrated — it was written by engineers in the buyer's language rather than in your category tree.

Build a grid rather than a list. Applications down one side, constraints across the other.

Application	Constraint	Query example	Evidence you can show
LED driver	600 V, high switching frequency	`600v mosfet for led driver`	app note, efficiency curve
Steam line	180 °C continuous	`valve for 180c steam line`	test report
Battery charger	reverse polarity protection	—	nothing yet

Fill a cell only when there's a document, a test result, or a reference customer behind it. Empty cells are information: they tell you what to ask engineering for before you write anything.

An application phrase without proof belongs in a short FAQ block on the family page. Giving it its own URL produces a page that says the same thing as the family page in slightly different words, which helps nobody.

Industry: the vocabulary the buyer's regulator uses

Automotive, medical, lighting, appliance, industrial automation each have their own words for the same component, and those words come from standards and procurement departments, not from your product taxonomy. That's why industry rows often look unfamiliar when you first see them in a query report.

Industry queries are the vaguest and the most contested. One product page won't hold them. What holds them is a segment page — the parts you ship into that segment, the certifications, the documents, one or two references — with the family and application pages underneath it linking up. If you have no references in a segment, don't build the segment page yet; the page will be a paragraph of adjectives.

Before a term goes into the sheet, test it out loud: would a buyer say this on a phone call, or is it a literal translation of your internal category name? Translated category names are the most common source of rows that sit for a year with no impressions.

Ordering the work

Score each row on two things only: how specific the query is, and how much evidence you can put on the page. A row with zero evidence is a dead row regardless of how attractive the keyword looks.

Part-number families first if you sell by part number. The queries are narrow enough that one complete page can answer them, and the person searching already knows what they want.
Applications second. More reach, more work, and it needs documents you may not have yet.
Industry last, unless you already have references in that segment.

Monday: the first twenty rows

Take the top twenty rows by that score. For each one, write the single sentence that answers the query, exactly as it would appear on the page. If you can't write the sentence from material you already have, the row isn't a page — it's a request to engineering for a datasheet, a test, or a photo.

Two checks before anyone opens an editor:

Every row maps to a page, and every page has at least one row whose query a real buyer typed.
No two rows with the same intent map to different URLs. That's how a site ends up with three pages quietly competing for one search, and it's much cheaper to catch in the sheet than after publishing.

Internal linking order: which links to place first, and how to control the anchor text

AumCreate — Sun, 01 Nov 2026 14:21:09 +0000

A product page with 46 blog posts behind it and three internal links pointing at it. Two of those sat in the footer, so they appeared on all 46 pages and said the same thing 46 times. The third was in a post about a variant that no longer ships. Every post ended with four "related reading" links chosen by tag, so most of the blog's links pointed at other blog posts.

Nothing there was broken. The site just never had an order: links were added as pages were published, and nobody asked which page should be receiving them. Decide the order first, then choose the words — editing anchors on a link map you haven't settled on yet wastes the pass.

Put the target pages in a file first

Pick the pages that have to be found — the ones that answer a question someone types before they contact you. Somewhere between five and twelve. More than that and you spread the work so thin that nothing moves.

Write them one path per line:

/services/emergency-plumbing
/products/heat-exchanger-120
/guides/sizing-a-pressure-valve

Save it as priority.txt. The script at the bottom reads it.

Order the link types by what a single edit buys

Links from pages that already have weight. Pages with external backlinks or steady organic clicks have something to pass on. A link from one of those, in the body of the page, does work that a link from an orphaned post nobody visits doesn't. Find them in your search console's links report, or — with no backlink data — in the pages report sorted by clicks. Write the donor list next to each target before you touch anything.

Links between a target page and the pages that support it. The target links down to the detail pages: variants, pricing, documents, a case. Each detail page links back up in the body, not just in a breadcrumb. That's what makes a cluster readable — a crawler can see which page is the parent, and a reader can get from a specific question to the main offer without using the menu.

Links between sibling pages that answer the next question. Someone reading about a 120 kW unit asks about the 160 kW one. Someone reading about installation asks about maintenance. These cost almost nothing and they pick up the long-tail queries the parent page can't hold.

Work in that order: a couple of weeks of donor edits, then the cluster, then the siblings.

How many in-body links belong on a page

There's no published cap, and anyone quoting one is guessing. The constraint that bites is reader attention: past roughly one link per 100–150 words of body copy, the page reads like a directory and people stop following any of them.

Navigation and footer links don't belong in that count — they're template, not editorial. Which is also why they do less work than people expect.

Find the posts that don't link to the target

On WordPress, this returns published posts whose content never mentions the target path:

SELECT p.ID, p.post_title, p.post_date
FROM wp_posts p
WHERE p.post_type = 'post'
  AND p.post_status = 'publish'
  AND p.post_content NOT LIKE '%/services/emergency-plumbing%'
ORDER BY p.post_date DESC;

Run it in your DB client or with wp db query < missing-links.sql. Check one row before you trust the result: the editor stores absolute URLs, so the pattern has to match what's actually in post_content. If your permalinks carry a trailing slash, include it.

The output is a candidate list, not a to-do list. Pick the three or four posts that already get impressions for something adjacent to the target and edit those. The rest can wait.

Control the anchor text instead of the link count

The anchor is the part of a link that tells anything downstream what the target is about. Most sites spend it on "read more" or on the same three words repeated sitewide.

Anchor	What it carries	How often
Exact product or service term	the target's main query, unambiguously	once per target, from the strongest donor
Descriptive phrase lifted from the target's own H2	the specific question the target answers	the default; vary the phrasing per page
Sentence-natural ("a walkthrough of the sizing steps")	context from the sentence around it	wherever the sentence wants it
"click here", "read more", "this page"	nothing	replace on sight

Four rules that hold regardless of how ranking systems change:

One anchor, one URL. If two pages could both take the same phrase, one of them is the wrong target — usually the older, thinner one. Merge or choose.
Read the sentence with the link text spoken out loud. If the anchor only makes sense because it's blue and underlined, rewrite the sentence.
Pull anchors from the target page's own headings. If the target has an H2 "Sizing a pressure valve for a 3-inch line", that phrase beats your internal category name, because the page can actually answer it.
Use the exact term once, from the strongest donor, and vary it elsewhere. Ten identical anchors from ten pages read as one recommendation repeated, and you lose the vocabulary that would have told anything downstream what the page covers.

Never reuse one anchor for two different targets across the site. It splits the signal and it annoys readers who click the same words and land somewhere new.

Sitewide links: keep the template small

A link in the header appears on every crawlable URL. That's correct for humans — it's how people navigate — but it does very little to differentiate pages: links are processed per page, and the same target with the same anchor from 400 URLs is one anchor repeated.

Practically: keep the nav to the handful of pages a first-time visitor needs, keep the footer to policy and contact pages, and put the differentiation into body copy where the anchor can change per page.

Audit it with a script

This walks your sitemap, counts inbound internal links per path, and separates links inside <main> or <article> from nav and footer links. Standard library only, no dependencies.

#!/usr/bin/env python3
"""Internal link audit.
Usage: python3 audit.py https://example.com/sitemap.xml priority.txt > report.csv
"""
import csv
import re
import sys
import urllib.parse
import urllib.request
from collections import Counter, defaultdict
from html.parser import HTMLParser

SITE = "{0.scheme}://{0.netloc}".format(urllib.parse.urlsplit(sys.argv[1]))
HOST = urllib.parse.urlsplit(SITE).netloc
UA = "internal-link-audit/1.0"


class LinkParser(HTMLParser):
    MAIN = {"main", "article"}

    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.links = []
        self._href = None
        self._text = []
        self._main = 0

    def handle_starttag(self, tag, attrs):
        if tag in self.MAIN:
            self._main += 1
        if tag == "a":
            href = dict(attrs).get("href")
            if href:
                self._href = href
                self._text = []

    def handle_data(self, data):
        if self._href is not None:
            self._text.append(data)

    def handle_endtag(self, tag):
        if tag == "a" and self._href is not None:
            text = " ".join("".join(self._text).split())
            self.links.append((self._href, text, self._main > 0))
            self._href = None
            self._text = []
        elif tag in self.MAIN and self._main:
            self._main -= 1


def fetch(url):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=30) as resp:
        body = resp.read()
        charset = resp.headers.get_content_charset() or "utf-8"
    return body.decode(charset, "replace")


def path_of(url):
    return urllib.parse.urlsplit(url).path.rstrip("/") or "/"


def internal_path(href, base):
    if href.startswith(("#", "mailto:", "tel:", "javascript:")):
        return None
    parts = urllib.parse.urlsplit(urllib.parse.urljoin(base, href))
    if parts.netloc != HOST:
        return None
    return parts.path.rstrip("/") or "/"


sitemap = sys.argv[1]
priority = {line.strip() for line in open(sys.argv[2], encoding="utf-8") if line.strip()}
paths = sorted({path_of(loc) for loc in re.findall(r"<loc>\s*(.*?)\s*</loc>", fetch(sitemap))})

inbound_body = Counter()
inbound_all = Counter()
anchors = defaultdict(Counter)

for path in paths:
    try:
        html = fetch(SITE + path)
    except Exception as exc:
        print(f"skip {path}: {exc}", file=sys.stderr)
        continue
    parser = LinkParser()
    parser.feed(html)
    for href, text, in_main in parser.links:
        target = internal_path(href, SITE + path)
        if target is None or target == path:
            continue
        inbound_all[target] += 1
        if in_main:
            inbound_body[target] += 1
            if text:
                anchors[target][text] += 1

rows = []
for path in sorted(set(paths) | priority):
    rows.append({
        "path": path,
        "priority": "yes" if path in priority else "",
        "in_body_inbound": inbound_body[path],
        "all_inbound": inbound_all[path],
        "top_anchors": " | ".join(f"{a} ({n})" for a, n in anchors[path].most_common(3)),
    })

rows.sort(key=lambda r: (r["priority"] != "yes", r["in_body_inbound"]))
writer = csv.DictWriter(sys.stdout, fieldnames=list(rows[0].keys()))
writer.writeheader()
writer.writerows(rows)

Run it with python3 audit.py https://example.com/sitemap.xml priority.txt > report.csv. It reads a flat sitemap; if yours is a sitemap index, expand the <loc> list first or point it at the child sitemap.

The sort puts priority pages with the lowest in-body inbound count at the top. Those rows are your donor-edit list, in order.

Two caveats worth knowing before you act on the numbers. Links injected by JavaScript won't show up — the script only sees what's in the HTML, so spot-check one page in the browser. And a path in priority.txt that isn't in the sitemap still gets a row, usually with zeros; that's often the most useful row on the sheet, because it means the page has no inbound links from anywhere the crawler reaches.

After the edits

Re-run the same script three weeks later and diff the report. Then look at whether the priority URL is indexed and whether it's picking up impressions for the phrase you used as its anchor. Give it weeks, not days — an internal link is a hint about structure, not a switch.

If you're on WordPress and would rather not maintain the script: AumViso handles sitemap output, schema and internal links, and it's free. I still run the audit above afterwards, because it counts what's actually in the rendered HTML, which is the only version that matters.

For me it comes down to this: a site with 46 pages and no link order is 46 pages competing to be noticed. A site with 46 pages and one decided order is a hierarchy. The difference is a few hours of editing, not a redesign.

How many pages a company site needs before launch, and what goes on each one

AumCreate — Sun, 01 Nov 2026 13:53:04 +0000

A template sitemap arrives with twelve entries on it, the domain is already bought, and the launch date is three weeks out. The useful question at that point is not how many pages a company site should have. It is which pages this particular sale requires, and what has to be true on each one before someone can act on it.

Working backwards from a page count gives you pages nobody maintains.

Let the purchase path set the floor, not the template

Four inputs decide the minimum: how a customer decides, the size of a typical deal, where trust has to come from, and whether the buying happens in one session or over several weeks.

Business shape	Pages that must exist at launch	Reasonable to add later
Local service, one offer, phone-first	Home, Contact	Separate service page, gallery
Quoted B2B, a few distinct offers	Home, one page per offer, About, Contact, legal	Process, compliance, case pages
Self-serve product	Home, product and pricing, docs or FAQ, Contact, legal	Comparison pages, changelog
Long cycle, high value	Home, offer pages, About, process, Contact, legal	Case studies, security page, team

Legal means whatever your jurisdiction requires — privacy notice, terms, and a company imprint where one is mandated. Those are not marketing pages and they are not optional.

The table is a starting position you argue with, not a rule. If a page sits in the "later" column and nobody can name the sentence that would go on it, it does not go on the list.

What has to be on each page before it goes live

Page	Its one job	Must contain before launch
Home	Say what this is, for whom, where, and what to do next	One sentence naming the offer and the customer; the primary action visible without scrolling; one concrete proof point; a link to contact
Offer page	Answer "can they do my version of this"	Who it is for, what is included, what is not, turnaround, what the buyer has to supply, next step
About	Give a reason to trust the people	Who runs it, where they operate, what they do not do, how to verify the claims
Contact	Remove the last excuse	A form that works, an email address, a phone number someone answers, hours, a response time you can keep
Legal	Keep you out of trouble	Privacy notice, terms, required registrations
Thank-you and 404	Catch people at the edges	Confirmation of what happens next; links back to the main offers

The first screen is not a slogan slot

Many home pages open with a positioning line and a stock photo. A visitor arriving from a search result needs three facts before anything else: what you sell, who it is for, and what the next step costs them in effort. Put those in the first paragraph, as plain text, in the words buyers use when they describe the problem.

One primary action. Two equally weighted buttons — "Contact us" next to "Download brochure" — split the decision and usually reduce both.

If geography matters, say where you serve in the first screen, not in the footer. If you work remotely and location does not matter, say that too, because a visitor who cannot tell will assume you are local to someone else.

When one offer deserves its own page

The default is one page per offer only if these hold:

A different buyer, or genuinely different questions from the same buyer.
Different proof is needed — a certificate, a lead time, a spec sheet the others do not have.
Enough to say to fill a page without padding. A screen and a half of real content is a fair bar.
Someone will maintain it.
People search for it under a different phrase than the parent offer.

Two or more failing means one page with anchored sections. Splitting because "more pages is better for SEO" produces thin pages that compete with each other for the same query, and they usually cannibalise rather than help. If you split later, use a URL that survives a redesign: /services/name, not /our-services-2026.

Filling the trust gap with no testimonials, cases or photos

Real options that require inventing nothing:

Publish the process step by step, including what you need from the customer and how long each step usually takes.
Say what you do not do. A page that excludes some work reads as more credible than one claiming everything.
Show real work with the client's written permission, even if it is one photo and a paragraph.
Give prices or ranges where the business allows it. "Contact us for pricing" costs you the visitors who only wanted a number.
Name the person responsible and how to reach them.
Link to things a reader can check independently — registration, certification, address, a partner's own site.

What you should never do is write your own testimonial, use a stock face captioned as a customer, or publish a case with figures you cannot back up. It reads fine until a buyer asks one question about it.

The pre-publish pass, page by page

The title contains the offer, in the buyer's words.
The first paragraph names who the page is for.
Exactly one primary action, repeated at the end.
No placeholder text, no "coming soon", no dead items in the menu.
The form was submitted end to end by a human and the confirmation email arrived.
Phone numbers are tappable; addresses are text, not baked into an image.
The page loads on a phone over a slow connection without the layout jumping.
There is a link to it from at least one other page.

Site-wide: a sitemap, a robots file that blocks nothing you need indexed, and analytics that record which page a form was submitted from — not only that a form was submitted.

Leaving room for page seven

Navigation that is a flat list of every page becomes a redesign the moment you add two. Group by what the buyer is trying to do and keep the top level short.

When you merge, split or rename after launch, redirect the old URL and keep the redirect. Internal links inside other pages are the usual thing that gets missed — the redirect works for search engines and for readers, but paragraphs that reference the old page still point at the old URL.

Deciding after launch on something other than taste

Give it a month, then read entrance data per page: which pages people land on, which pages nobody enters, which pages earn internal clicks, and where form submissions start and stop.

A page with no entrances, no internal links and no form activity is a candidate to merge into a page that does have traffic, not to delete. A page where two different questions keep showing up in the same visit is a candidate to split. New pages should come from a question you have seen more than once, not from a keyword tool's estimate.

Set a review date when you launch. Otherwise the next decision about the site gets made by whoever is annoyed enough to raise it, and by then the fix is a redesign.

Domain, business email and DNS: the order to configure them so mail doesn't land in spam

AumCreate — Fri, 30 Oct 2026 04:26:22 +0000

A zone file that serves a website and a zone file that can send mail are the same file. That is why the usual failure looks like this: the site is live, an MX record got added, everything looks configured, and then the first invoice lands in a customer's spam folder.

Deliverability comes down to order more than to any single record. DMARC published before DKIM exists rejects legitimate mail. Two SPF records left behind by a mail migration return permerror, which is worse than having no SPF at all. A sending subdomain with no DMARC record of its own quietly inherits the parent policy.

Here is the sequence I use. Each step ends with a check that has to pass before the next one starts. Nothing below depends on a particular registrar or mail provider.

0. Write the sending map before touching DNS

List every system that will put your domain in a From: header, and give each one its own DKIM selector:

Sender	Envelope domain	Auth
People in the mail client	example.com	SPF + DKIM (`s1`)
App / transactional mail	example.com	DKIM (`s2`)
Newsletter platform	mail.example.com	DKIM (`s3`), own DMARC
Website contact form	example.com	DKIM (`s2`)

The map matters because selectors are per-sender, and you need all of them before you configure DMARC. Anything you forget here becomes the thing that breaks in step 6.

1. Register the domain and keep DNS in one place

Use the registrar's nameservers or delegate the zone to a DNS host, but pick one location and keep the whole setup there. Splits, where some records sit at the registrar and some at the hosting panel, are where "the record exists but nothing resolves" comes from.

Lower the TTL on the records you are about to create to 300. You will edit them several times; a 3600 TTL turns each edit into an hour of waiting.

2. Publish and verify the web records first

Do the boring part before mail, so a broken A record is never confused with a broken mail setup.

example.com.      300 IN A     203.0.113.10
example.com.      300 IN AAAA  2001:db8::10
www.example.com.  300 IN CNAME example.com.

dig +short example.com A @1.1.1.1
dig +short example.com A @8.8.8.8
dig +short www.example.com CNAME @1.1.1.1

No CNAME at the apex. If your DNS host offers ALIAS or ANAME, that is a different record type and it is fine; a literal CNAME on example.com is not.

If the web half is being handed to someone else, that is a separate job from the mail half, and it is worth knowing which one you are buying. AumCreate's Setup service ($399, what's included) installs a WordPress template on the client's own hosting, replaces every page with the client's content, sends a preview link with one round of revisions, and goes live within five working days of the material being complete. Domain and hosting stay in the client's name and are paid to the third party. The records in this article still have to be published by whoever owns the zone — a site handover does not touch mail.

3. Create MX records only after the mail host exists

The MX target has to be a hostname that already accepts mail for the domain, and its own A record has to resolve.

example.com.  300 IN MX 10 mx1.mailhost.example.
example.com.  300 IN MX 20 mx2.mailhost.example.

dig +short example.com MX @1.1.1.1

Two MX records at the same priority is a coin flip per delivery attempt. Different priorities give you a fallback instead.

If you run your own MTA on a VPS, the PTR for the IP has to resolve forward to the same hostname you use in HELO. The PTR is set at the hosting provider, not in your zone, so start that request now — it is the one item in this list you cannot fix from the zone file.

4. SPF: one record, at the apex, under ten lookups

example.com.  300 IN TXT  "v=spf1 include:_spf.mailhost.example include:_spf.newsletter.example -all"

The rules that break people:

Exactly one v=spf1 record per name. Two TXT strings starting with v=spf1 is a permerror, not a merge. When you migrate, delete the old record in the same change that adds the new one.
The ten-DNS-lookup limit counts include, a, mx, ptr, exists and redirect. ip4 and ip6 are free. Nested includes count everything they pull in, which is where the limit actually bites.
-all means everything else is unauthorized. ~all is a soft fail you use during a migration. Pick one on purpose.
SPF is evaluated against the envelope sender (MAIL FROM, falling back to HELO), not the From: header. A perfect record for the wrong domain does nothing for DMARC alignment.

Check with dig +short example.com TXT. If the string appears twice, stop and fix it.

5. DKIM for every sender, verified before DMARC goes on

The provider generates a key pair. You publish the public half as a TXT record at <selector>._domainkey.example.com.

s1._domainkey.example.com. 300 IN TXT "v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A..."

Use 2048-bit keys.
A long p= value exceeds 255 characters. In raw zone text you split it into quoted strings that the resolver concatenates; most control panels do this for you, and if you paste a key into a panel that does not, you get a truncated key.
Verify the record resolves and is complete before anything depends on it:

dig +short s1._domainkey.example.com TXT @1.1.1.1 | wc -c

Compare the length against the key the provider shows you. A truncated p= fails silently — mail still goes out, just unsigned.

Then send a real message and read the Authentication-Results header at the receiving end. You want dkim=pass with d= equal to your domain, not the provider's.

6. DMARC: start at none, read the reports, then enforce

_dmarc.example.com. 300 IN TXT "v=DMARC1; p=none; rua=mailto:dmarc@example.com; adkim=r; aspf=r; fo=1"

p=none tells receivers to evaluate and report without acting on the policy. Leave it there through a couple of weeks of real traffic, including the newsletter and the form mail.

Then move in this order, a few days apart:

p=quarantine; pct=25, then pct=50, then pct=100.
p=reject.
Only then tighten to adkim=s; aspf=s if everything is genuinely aligned.

Two things that catch people here:

A subdomain without its own _dmarc record inherits the organizational policy, including sp=, which defaults to p=. Bulk mail sent as mail.example.com with a From: on that subdomain needs either alignment or its own policy record.
rua has to point at a mailbox that exists and that someone reads. The reports arrive as XML attachments — ugly, and the only way to see which sender is failing.

dig +short _dmarc.example.com TXT. It has to be a TXT record; a CNAME at _dmarc resolves at some resolvers and fails at others.

7. What comes after mail already flows

Not required for deliverability, but cheap once the zone is clean:

MTA-STS: TXT at _mta-sts.example.com with v=STSv1; id=<timestamp>, plus the policy file served over HTTPS at https://mta-sts.example.com/.well-known/mta-sts.txt. That host needs a valid certificate, so do this after TLS is settled.
TLS-RPT: TXT at _smtp._tls.example.com with v=TLSRPTv1; rua=mailto:tls@example.com.
BIMI last, and only with a VMC. Skip it until DMARC has been at p=reject for a while.

8. The failures that show up weeks later

A recipient forwards your mail. SPF breaks at the forwarder; DKIM survives if the body was not rewritten. This is the main reason DKIM has to be in place before enforcement.
A sender is retired and its include stays in SPF. You keep authorizing an IP range you no longer control.
The newsletter moves to a new provider with a new selector and nobody publishes the matching DKIM record. DMARC keeps passing on SPF until the day alignment slips.
TTL stays at 300 forever. Harmless, but raise it once you are done, or a slow resolver somewhere will hold your setup-time mistakes longer than you would like.

The check to run before calling it finished

check() {
  for ns in 1.1.1.1 8.8.8.8 9.9.9.9; do
    printf '%-26s @%-8s ' "$2 $1" "$ns"
    dig +short "$1" "$2" @"$ns" | tr '\n' ' '
    echo
  done
}

check example.com MX
check example.com TXT
check _dmarc.example.com TXT
check s1._domainkey.example.com TXT

If all three resolvers return the same answers, the zone is consistent and the rest is sending real mail and reading the headers at the other end.

How to build an export case-study page a buyer can check: background, specs, results

AumCreate — Fri, 30 Oct 2026 04:23:26 +0000

The weakest case-study pages on export sites fail in the same place: they describe the seller's effort instead of the buyer's order. "Professional solution, strict quality control, timely delivery." A buyer sourcing a component or a container of goods is not grading effort — they are checking whether the order they are about to place looks like one you have already shipped.

That reframes what goes on the page. Background, specifications, results, and enough detail that a skeptical reader can confirm the case is real. Here is how to build each block.

Start with one order, not the customer's whole history

One case equals one order, or one product family shipped to one market. A page that covers everything the company has ever made reads like a brochure and gives the buyer nothing to compare against their own requirement.

Pick the case by its constraint. The interesting cases are the ones where something was awkward: a tolerance nobody wanted to hold, a certification the buyer needed in a specific market, a packaging format that had to survive a long inland leg, a deadline tied to a retail season. That constraint is the spine of the page.

The background block: what was ordered, and what made it awkward

Four to six sentences, no adjectives. Write:

the market and channel the buyer sells into (importer, distributor, own-brand retailer, contractor)
the order shape — how many SKUs, roughly what volume band
the constraint that made the order non-standard
what the buyer had already tried, if you know and can say it without guessing

What you do not need is the customer's name. "A distributor of kitchen hardware selling into the Benelux" is enough for a reader to place the case, and it is a sentence you can publish without a permission fight.

Leave out anything you cannot state as fact. If the buyer never told you why they left a previous supplier, do not write a reason.

The specs block carries most of the page

This is a table, not prose. A buyer will scan it before reading a single sentence of the background.

Item	Value	Standard / how it was checked
Material	304 stainless, 1.2 mm	Mill certificate on file
Dimension	480 × 320 × 210 mm	Drawing rev. C
Critical tolerance	±0.3 mm on the flange	Measured with a caliper at QC
Surface	Brushed, Ra 0.8	In-house sample retained
Certification	CE, report on file	Third-party lab, dated
Packaging	5-ply carton, 6 pcs per carton	Drop test per internal spec
MOQ / lead time	As quoted	—

Rules that keep the table honest: put units on everything, name the standard or the drawing revision, and state tolerances as numbers rather than "high precision". Every row has to be backed by a document you could produce if asked. If a row cannot be backed, delete the row — an empty cell is less damaging than a claim that collapses under one follow-up email.

The results block: separate what was delivered from what changed

Two short sections work better than one paragraph.

Delivered is pure logistics and easy to defend: quantity, production window, port of loading, container type, packaging used, date the samples were sent.

Outcome is whatever was actually measured. Dimensional pass rate from your own QC record, units replaced under warranty, a test report number, the date a repeat order came in — but only if that is true and you can point to where it is written down.

If nothing was measured, do not invent a percentage. Say what was delivered and let the specs table carry the weight. A page with no invented numbers reads more credible than one with a suspiciously round "30% cost saving".

Details a buyer can verify

This is the block most pages skip, and it is the one that separates a story from a case.

Certificate body, certificate number and issue date. A buyer can query the issuing body.
Third-party test report number, laboratory, and test date.
The drawing revision and the instrument used for the critical dimension.
Port of loading and shipment month. Combined with the destination, that narrows a lot.
Container type and count, if you ship FCL.
Sample retention: whether a reference sample still exists and when it was sealed.
Factory audit date and scope, if a buyer audit was done.
Photo dates. A QC photo with a visible date stamp is worth more than ten clean studio shots.

Put a short note next to the list saying which items are available on request and that customer names, prices and drawings are redacted. Buyers expect redaction. What they do not expect is a blurred scan where the redaction accidentally hides the certificate number too.

Getting permission, and what to do when you cannot name the buyer

Ask at order confirmation, in writing, not after the page is drafted. Ask specifically: may we describe the order, the market, the specifications and the shipment dates, and may we use your logo. Those are separate permissions and buyers answer them differently.

If the answer is no, anonymise and say so on the page: "Customer name withheld at their request." That sentence costs nothing and explains the gap before the reader wonders about it. Never use a customer logo you were not given, and never assemble a logo wall from images found online.

Photos and files

Shoot the goods themselves, the packaging in place, the label, the loading. Keep the originals; compress for the page. Renders presented as photographs get spotted quickly by anyone who buys hardware for a living, and the loss of trust is larger than the gain in polish.

Publishing it and keeping it alive

One URL per case, a plain page, a visible date. Update the date when specs change, and take the page down if the product line is discontinued — a stale case that describes a part you no longer make wastes a buyer's time and yours.

Produce a PDF from the same content. Buyers forward files internally, and a link that requires anything to open dies in that chain. Then put the case link in the quotation email and in the follow-up two weeks later, where it does more work than it ever will on the site itself.

A skeleton to fill in

# <Product family> for <market / channel> — <year>

## Background
- Buyer profile: <channel>, <market>, <volume band>
- The constraint: <the awkward part>

## Specifications
| Item | Value | Standard / how checked |
|---|---|---|
| ... | ... | ... |

## What was delivered
- Quantity / SKUs:
- Production window:
- Port of loading:
- Packaging:

## Verifiable details
- Certificate: <body> <number>, issued <date>
- Test report: <number>, <lab>, <date>
- Drawing: <revision>, critical dimension <value> ± <tolerance>
- Sample retained: <yes/no>, sealed <date>

## Available on request
Redacted QC photos, full test report, buyer reference on request.

The check before publishing

Read every sentence and ask whether a buyer could demand proof of it. Anything that fails that question gets cut, moved to "available on request", or rewritten as a specification. The page does not need to be impressive. It needs to survive one procurement manager reading it carefully.

What a first WooCommerce store actually needs configured before it takes an order

AumCreate — Fri, 30 Oct 2026 04:16:30 +0000

The settings that break a first store are the boring ones. A customer reaches checkout, the address does not match any shipping zone, the page says no shipping methods are available, and nobody finds out because the first person to try it was a real buyer.

Below is the set of things that have to be right for money to come in, then the things that can wait until there is an order history to argue with.

Products: enough to sell, not a catalogue project

Enter a real price and, if you track units, tick Manage stock on the product itself, or enable stock management store-wide under Products → Inventory. Without that tick, stock never decrements and you will oversell the one item you have.

Weight and dimensions only matter if a shipping method calculates from them. If you are charging a flat rate, skip them for now; adding them later does not change anything already configured.

Use a variable product only when price, stock or image genuinely changes per variant. "Gift wrap" is a checkout field, not a variation, and each variation is another thing that can be out of stock at the wrong moment.

Two or three categories with a purpose. Categories are navigation; a taxonomy built before the tenth product is a taxonomy you will redo.

Shipping: the zone is the part people get wrong

A shipping zone matches the customer's address. If no zone matches, WooCommerce shows no shipping methods at all and the order cannot be placed. So the minimum is one zone that covers everywhere you will actually ship, even if it is the catch-all "rest of the world" zone with a flat rate on it.

One zone, one flat rate, for the countries you really deliver to.
Free shipping as a second method in the same zone, with a minimum order amount, if that is the offer.
Local pickup if you hand-deliver or the buyer collects.

Do not build a dozen zones from guesses about what shipping will cost next year. Every zone you add is another one whose rates go stale silently.

Tax: one decision before any rate exists

Decide whether your entered prices include tax or exclude it, before you type prices. Changing that setting later re-computes every price shown on the front end, and the catalogue you checked last week is suddenly wrong.

Then enable tax calculation and add standard rates for the countries you sell into. If you sell across borders, get local advice on registration thresholds before you invent rates; the software will accept any number you give it and print it on an invoice.

Payments: one gateway live, tested end to end

Enable a single card gateway and complete a real transaction for a small amount, then refund it. A sandbox run does not exercise the same path as a live authorization, and the first failed live payment is a bad way to learn that.

Confirm the site is served over HTTPS before the gateway goes live, and read the order notes after the test. An order that looks paid in the admin but has an empty gateway response is usually a callback that never arrived, and no amount of theme work fixes it.

The two emails, tested in a real inbox

The customer receipt and the new-order notice to you are separate emails and both can fail independently. Test both, delivered to an inbox you actually read, with the from-address on your own domain.

If neither arrives, the site cannot send mail at all. That is a hosting and DNS problem, not a store setting, and it is worth knowing before launch rather than after the first support message.

Pages that have to exist

Refund and returns policy, privacy policy, terms, a contact route, and a statement of delivery times. These are jurisdiction-specific. Read them against your local consumer rules instead of copying another store's text.

Read the settings back before you announce

A quick read-only pass with WP-CLI, which is faster than clicking through screens and harder to misread:

wp option get woocommerce_currency
wp option get woocommerce_default_country
wp option get woocommerce_calc_taxes
wp option get woocommerce_prices_include_tax
wp option get woocommerce_enable_guest_checkout

PREFIX=$(wp db prefix)
wp db query "SELECT zone_name FROM ${PREFIX}woocommerce_shipping_zones;"
wp db query "SELECT zone_id, location_type, location_code FROM ${PREFIX}woocommerce_shipping_zone_locations;"

What you are looking for: taxes actually enabled, the include/exclude decision matching your price list, at least one zone with at least one location that covers your buyers, and guest checkout set to whatever you decided rather than the default you never looked at.

The test order, run twice

Place an order on a phone, with a real card, for the cheapest thing you sell. Then check:

stock went down by one, and went back up after the refund;
both emails landed, and the customer one is not in spam;
the thank-you page shows the order and the admin shows the same total;
the shipping line matches the zone you expect for that address.

Then do it again from a country in your second zone, if you have one. Cross-border is where zone overlap and tax display disagree, and it is better to find that on a test order than on a real one.

What can wait

Multi-currency, a second language, subscriptions, reviews, wishlists, a coupon engine beyond one launch code, loyalty points, warehouse or accounting sync, per-product shipping rules beyond what a flat rate handles, abandoned-cart tooling, A/B testing, and a blog.

None of these are bad. Each of them adds a surface that can break during a checkout, and none of them are needed to prove that the store can take money and deliver the thing.

Setting	Ship with it now	Can wait
Products	Real price, stock tracked, one usable image each	Rich descriptions, cross-sells, attribute filtering
Shipping	One zone with a flat rate, corrections after the first five real shipments	Rate tables per region, weight bands, carrier integration
Tax	Include/exclude decided, tax enabled, rates for your main market	Every destination you might ship to in three years
Payments	One gateway, live, refunded once	A second gateway for a market you do not have yet
Email	Both transactional mails in a real inbox	Styled templates, marketing automation
Legal pages	Refund, privacy, terms, contact, delivery time	Anything you cannot read and defend yourself

The minimum is small and unglamorous: money in, correct totals including shipping and tax, stock that reflects reality, and a buyer who gets told what happened. Everything else is a second release.

How to check which AI crawlers actually hit a new WordPress site in week one

AumCreate — Fri, 30 Oct 2026 04:13:19 +0000

A site goes live on Monday. On Friday the question arrives: have the AI crawlers found it yet? If the access log format was picked years ago to debug PHP, there is no answer in the file, and the first week is already gone.

Two decisions have to be made before the first request: capture enough per request, and be able to tie a user agent to an IP you can check. Everything below assumes a self-hosted WordPress site where you control the web server.

Set the log format before launch

nginx, in a file included from the http block:

# /etc/nginx/conf.d/aiwatch.conf
log_format aiwatch '$time_iso8601 $remote_addr "$http_user_agent" '
                   '$request_method $request_uri $status $body_bytes_sent '
                   '"$http_referer" $request_time';

access_log /var/log/nginx/aiwatch.log aiwatch;

Apache httpd:

LogFormat "%{%Y-%m-%dT%H:%M:%S%z}t %a \"%{User-Agent}i\" %m %U %s %B \"%{Referer}i\" %D" aiwatch
CustomLog /var/log/apache2/aiwatch.log aiwatch

Put it at the server level, not inside one vhost, or the next site you add silently logs in the old format. Then check what logrotate does to that file: if it rotates weekly and compresses, copy week one somewhere before it is replaced.

The fields that change a decision

Field	What it decides
`time_iso8601`	Contains the UTC offset. Without it, cross-checking hits against the moment you published or changed robots.txt is guesswork.
`remote_addr`	Must be the client IP. Behind a CDN or proxy it is the edge, and your verification step will confirm the CDN as a crawler. Fix with `set_real_ip_from` plus `real_ip_header` for whatever header your edge sets.
`http_user_agent`	Store it whole. Truncating to 60 characters drops the version and the `+https://...` info URL that operators put there on purpose.
`status`	200, 304, 403 and 429 mean four different things. A crawler getting 403 is a different problem from a crawler that never came.
`body_bytes_sent`	A 200 with 0 bytes is a HEAD request or an empty body. A status-only report calls both a success.
`request_uri`	Did it fetch `/robots.txt` first, and did it ask for `/llms.txt` at all.
`request_time`	A crawler that times out on your slowest page may not come back for the rest of the site.

Count what actually arrived

The UA is field 2 when you split the line on double quotes, because the format wraps it in quotes.

LOG=/var/log/nginx/aiwatch.log

awk -F'"' '
{
  ua = tolower($2)
  if      (ua ~ /gptbot/)          c = "GPTBot"
  else if (ua ~ /oai-searchbot/)   c = "OAI-SearchBot"
  else if (ua ~ /chatgpt-user/)    c = "ChatGPT-User"
  else if (ua ~ /claudebot/)       c = "ClaudeBot"
  else if (ua ~ /claude-user/)     c = "Claude-User"
  else if (ua ~ /perplexitybot/)   c = "PerplexityBot"
  else if (ua ~ /perplexity-user/) c = "Perplexity-User"
  else if (ua ~ /ccbot/)           c = "CCBot"
  else if (ua ~ /bytespider/)      c = "Bytespider"
  else if (ua ~ /amazonbot/)       c = "Amazonbot"
  else if (ua ~ /applebot/)        c = "Applebot"
  else if (ua ~ /googlebot/)       c = "Googlebot"
  else next
  n[c]++
}
END { for (k in n) printf "%-16s %d\n", k, n[k] }
' "$LOG" | sort -k2 -nr

A zero column for a crawler is not yet evidence of anything. Read the days before you read the total.

To see whether a crawler walked the site in order, print the first hit per IP with the path it asked for:

grep -i 'gptbot' "$LOG" | awk -F'"' '{split($1,a," "); print a[1], a[2], $3}' | sort | head -20

With this format, $1 is timestamp + space + ip, and $3 is the request line (method, path, status, bytes). If robots.txt and the home page appear in that list before anything else, the crawl is behaving; if the first hit is a deep product URL, it came in through a link, not through your sitemap.

Verify the IP before you trust the user agent

A user agent is a string the client chooses. Anyone can send GPTBot. Forward-confirmed reverse DNS is the cheap check: resolve the IP to a hostname, resolve that hostname back, and require the original IP to appear in the result.

#!/usr/bin/env python3
"""Forward-confirmed reverse DNS. Reads IPs from argv or stdin."""
import socket
import sys


def check(ip):
    try:
        host = socket.gethostbyaddr(ip)[0]
    except (socket.herror, socket.gaierror, OSError):
        return ip, None, False
    try:
        addrs = socket.gethostbyname_ex(host)[2]
    except (socket.gaierror, OSError):
        return ip, host, False
    return ip, host, ip in addrs


def main():
    ips = sys.argv[1:] or [line.strip() for line in sys.stdin if line.strip()]
    for ip in ips:
        addr, host, ok = check(ip)
        print(f"{addr}\t{host or '-'}\t{'confirmed' if ok else 'NOT confirmed'}")


if __name__ == "__main__":
    main()

chmod +x fcdns.py
awk -F'"' 'tolower($2) ~ /gptbot|claudebot|perplexitybot/ {split($1,a," "); print a[2]}' "$LOG" \
  | sort -u | ./fcdns.py

Confirmed PTR means the IP belongs to whoever controls that reverse zone. It does not prove the request came from the crawler you have in mind, so compare against the operator's published IP ranges where they publish one. A hit that fails both checks is an unknown client with a borrowed user agent, and it should not go into your report as a crawler visit.

Tokens you will never find in a log

Google-Extended and Applebot-Extended are robots.txt user-agent tokens, not strings any client sends. Requests from those operators show up as Googlebot and Applebot. So "did Google-Extended fetch my page" has no answer in the access log; the only thing you control is whether robots.txt permits that token, which is a separate check against the file itself.

Which brings up the other half of the question. A user agent in the log does not mean your robots.txt was obeyed. Pull the file you served that week and see what it said:

curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' https://example.com/robots.txt
curl -sS https://example.com/llms.txt | head -20

Run it from outside your own network. A WAF rule that allows your office IP and blocks everything else looks identical to "no crawlers came" once you are reading the log.

What week one can and cannot tell you

A fetch is not an inclusion. A confirmed GPTBot request for the home page says a crawler read the page; it says nothing about whether the page will appear in an answer later, and there is no field in the log that closes that gap.

The reverse is worth stating too: no hits in week one does not distinguish "blocked" from "has not found it yet". A brand-new domain with no inbound links gives a crawler very little to follow, so an empty column is the expected starting state, not a malfunction. Do not rewrite robots.txt on the strength of one quiet week; check what the file actually serves first, then wait.

Keep the week

Print the per-crawler count once a week and keep the output next to the raw log. A single snapshot after launch tells you nothing that changes a decision. The number that matters is the change between week one and week four, and whether the paths being fetched moved from the home page into the pages you actually want quoted.

If maintaining this pipeline on every site is not the job you want, AumCrawl records crawler hits per crawler and lets you allow or block them individually. It is free and on wordpress.org; the log format above is still worth setting, because the plugin's data is only as good as what the server keeps.

Finding the strings a multilingual site forgot to translate

AumCreate — Fri, 30 Oct 2026 03:58:53 +0000

The German product page reads well. Then a visitor submits the contact form with the email field empty and gets "This field is required" sitting under a German label. The order confirmation that follows has a German subject line, an English footer, and a button that says "View order".

None of that shows up in a page-by-page review, because a rendered page is assembled from a lot of places at once: theme templates, plugin templates, rows in the database, JavaScript bundles, and — for anything that leaves by mail — a renderer that never runs in a browser. Translating the pages you can see leaves the rest untouched, and the rest is exactly what appears when something goes wrong, which is when a customer is most likely to notice.

The surfaces, and where each string actually lives

Surface	Where the string lives	How to catch it
Theme and plugin templates	`.po` / `.mo` files, or literal text that was never wrapped in a translation call	extract with `wp i18n make-pot`, then grep templates for bare text
Menu labels	database rows for menu items; one menu per language unless the theme registers separate locations per locale	switch language, read the menu, check each item's `lang` attribute
Widgets, customiser values, block content	database, usually stored once in the source language	search the target-language HTML for the source string
Form labels, placeholders, help text	the form plugin's per-form settings in the database	open the form in the target language and read every field
Validation and error messages	browser constraint validation, the client-side validator's catalogue, and the server handler	submit invalid input in each language and read the response
Cart and checkout copy	template files, settings, and JS-rendered fragments	complete a purchase end to end in the target locale
Transactional email	template files plus settings text, rendered server-side at send time	capture the raw message for every template
`title`, meta description, `og:` tags, `alt`, `aria-label`, `placeholder`	templates and database	crawl both locales and compare attributes
Structured data, sitemaps, feeds	generated from the same source strings	view source, check `inLanguage` and the URLs
PDFs — invoices, packing slips	template files, and font files that may not carry the glyphs	generate one in the target language and look at it

Start from the catalogue, not the page

Every string that is wrapped for translation ends up in a .pot file. Generate it and compare against the target language's .po:

wp i18n make-pot . languages/site.pot --exclude=node_modules,vendor,dist
msgattrib --untranslated languages/de_DE.po
msgattrib --only-fuzzy languages/de_DE.po

Fuzzy entries are worth reading. A fuzzy match is not counted as translated in the admin, and whether it reaches the front end depends on how the files are compiled.

The catalogue covers only the strings that were wrapped. Bare text inside a template never appears in it:

grep -rn --include='*.php' -E '>[^<>{}]*[A-Za-z]{4,}[^<>{}]*<' \
  wp-content/themes/mytheme \
  | grep -v -E '__\(|_e\(|esc_html__\(|esc_html_e\(|_x\(|_n\('

This is a heuristic and it will produce noise — text inside comments, inline SVG, printf fragments that are assembled elsewhere. Read what it returns instead of piping it into a fixer.

Then check the rendered page against the source page

A string can be translated in the .po file and still render in the source language, because something else overwrote it: a cached fragment, a JS bundle that ships its own copy of the text, a database value that took precedence.

The check that catches this: take the strings that appear on the source-language page and look for them on the target-language page. Anything that shows up unchanged is either untranslated or not being loaded. That is a script, not a review:

// npm i playwright && npx playwright install chromium
// node find-missing.mjs https://example.com/de/ https://example.com/en/
import { chromium } from 'playwright';

async function harvest(page, url) {
  await page.goto(url, { waitUntil: 'networkidle' });
  return page.evaluate(() => {
    const found = [];
    const walker = document.createTreeWalker(document.body, NodeFilter.SHOW_TEXT);
    let node;
    while ((node = walker.nextNode())) {
      const t = node.nodeValue.replace(/\s+/g, ' ').trim();
      if (t) found.push({ kind: 'text', value: t });
    }
    for (const el of document.querySelectorAll('[placeholder],[aria-label],[alt],[title]')) {
      for (const attr of ['placeholder', 'aria-label', 'alt', 'title']) {
        const v = el.getAttribute(attr);
        if (v && v.trim()) found.push({ kind: attr, value: v.replace(/\s+/g, ' ').trim() });
      }
    }
    return found;
  });
}

const [tgtUrl, srcUrl] = process.argv.slice(2);
const browser = await chromium.launch();
const page = await browser.newPage();
const target = await harvest(page, tgtUrl);
const source = await harvest(page, srcUrl);
await browser.close();

const sourceValues = new Set(source.map(x => x.value));
const seen = new Set();
for (const item of target) {
  if (!sourceValues.has(item.value)) continue;
  if (!/[A-Za-z]{3}/.test(item.value)) continue;
  const key = item.kind + '|' + item.value;
  if (seen.has(key)) continue;
  seen.add(key);
  console.log(`${item.kind.padEnd(12)} ${item.value}`);
}

Brand names, product codes, numbers and words that are genuinely the same in both languages will come back as suspects. So will anything that only ever existed on one locale's page, because it has no counterpart to compare against.

Run it over a URL list rather than the home page alone: a product, a category, the cart, checkout, a search with no results, a 404. Empty states and error pages are where templates nobody reviewed turn up.

Email is a separate application

Each transactional message is rendered on the server, and the locale at render time is whatever is active when the mail is built. When the trigger is an admin action or a scheduled job, that is frequently the admin's language rather than the recipient's.

Capture raw messages instead of reading them in one inbox. Point the site's mail transport at a local catcher, then trigger each template once with a test account whose language is the target one. Read the subject, the preheader, every heading, the button label, the footer, and the plain-text alternative — that part is often generated from a separate template and drifts out of sync with the HTML one.

In WordPress the locale used during rendering can be switched with switch_to_locale(), and mail-building code varies in whether it does that around the render. Testing with a real recipient account tells you what actually happened; reading the template files only tells you what is in them.

Validation messages come from more than one layer

<input required> produces a message from the browser, in the browser's UI language, not the site's. A visitor with an English browser reading a German page gets an English bubble, and no translation file is involved.

If that matters, set the message yourself:

const email = document.querySelector('#email');

email.addEventListener('invalid', () => {
  email.setCustomValidity(
    email.validity.valueMissing ? t('field_required') : t('field_invalid_email')
  );
});

email.addEventListener('input', () => email.setCustomValidity(''));

t() stands for whatever your catalogue lookup is called. After that, the server-side handler is a different set again: post the form with an empty required field and with a malformed value in each language, and read the JSON or the re-rendered page that comes back.

Keep it from drifting

Put the catalogue diff in CI so a release can't quietly add untranslated text to a translated page:

wp i18n make-pot . /tmp/now.pot --exclude=node_modules,vendor,dist
diff \
  <(grep -o '^msgid ".*"' /tmp/now.pot | sort -u) \
  <(grep -o '^msgid ".*"' languages/site.pot | sort -u)

A non-zero exit means the catalogue changed. Whether that is a problem depends on whether the new strings were translated, which is what msgattrib --untranslated answers.

While the crawler is already running, assert <html lang> and the hreflang pairs on every page. A document whose lang says en while the body is German gets hyphenated, read aloud and indexed as English.

If the site runs on WordPress and you would rather not assemble the translation layer yourself, AumLang is a free plugin on wordpress.org that handles AI translation, hreflang and WooCommerce product-page translation. Whatever you end up using, the checks above still apply — where a string lives decides whether any tool can reach it.