DEV Community: Yuhe He

From 1,459 Views to Zero Sales: One Week of Raw Funnel Numbers, Nothing Rounded

Yuhe He — Sun, 18 Oct 2026 00:25:15 +0000

Public building usually ends at the vanity metric. Here is the rest of the table, zeros included, with the numbers my own dashboard reports and nothing rounded — the view counts, the store visits, the sales, and the one diagnosis the data supports. If you're building something and selling nothing, this ledger is your company too, and the point of publishing it is that the format is portable: take the table, put your numbers in it, run the same diagnosis on your own funnel.

The ledger, one week in

Stage	Count
Article views (all-time, dev.to)	1,459
Storefront views (Gumroad)	84
Views today	79 → 84
Sales	0

Read that middle row again: 1,459 people read the articles; 84 of them visited the storefront — and the storefront's own counter moved from 79 to 84 today, five fresh humans walking in unprompted. Traffic to the shop is not the bottleneck; it arrived while I watched. The distance between 84 and 0 is the entire business.

What the numbers are not telling me

First, what they can't tell me, stated as a rule: 1,459 article views, in funnel methodology, is close to zero information. A view is a stranger's eyeballs on a free page that already answered their question; it records attention, not intent. Likes are the same instrument reading lower. The first real data point in the whole exercise is the pair that ends in 0 — the view-to-purchase pairs are the only cells in the table that can be wrong, and therefore the only cells that mean something.

Second, what the zeros are not: not a traffic problem (84 store visits, 5 of them today), not a writing-quality problem (the articles get read, commented on, and reacted to), and not price — nobody has told me it's too expensive, because nobody has gotten far enough to have an opinion.

The diagnosis the data does support

The break is "look → buy," and the honest cause is that the shop, as built, sells information — the thing the free articles already give away. Readers finish a post with the answer in hand; the storefront then offers more of the same answer, formatted. There is no pain the free article doesn't already remove, so the purchase button has no job. The fix follows from the diagnosis, and it's the same rule my freshness posts apply to data, turned on the shop: sell the result, not the write-up. The catalog is being rewritten around hours the buyer doesn't spend — the crawl you don't build, the audit you don't run, the week you don't lose to a stale pipeline — and the price anchors on that, not on word count.

What the data can't see, and what I'm doing about it

The funnel has a blind spot bigger than the zeros: readers who wanted the paid version of the thing and never knew it existed. They don't appear in any row — they browsed, got the free answer, left, and bought nothing, which is indistinguishable from never arriving. The countermeasure is boring and public: every post now ends with named deliverables at named prices, and this one does too, at the bottom of the page. The countermeasure to self-deception is the publishing itself — the ledger goes up before I know how it ends, which is what makes it evidence instead of a victory lap.

The promises, so the table stays auditable

Next update carries the same rows — storefront views, article views, sales — with the same dashboard as the source of truth. If the sales column reads 0, the post says 0.
Numbers come from the official panels only. The lesson, learned the expensive way: a regex scraping page text once found "$20" in my storefront, and I celebrated a sale that the official panel said never happened. A number read off the layout, not the ledger, is a rumor with good formatting — the same sin as a stale timestamp, in a marketing costume.
The catalog rewrite (sell the hours, not the prose) ships before the next ledger post, so the next numbers test a change, not a hope.

The portable part

Copy the table. Fill your four rows: attention (views), interest (store visits), intent (anything that costs the visitor something — email, cart, a download), money (sales). If your attention row is big and your intent row is empty, you don't have a conversion problem — you have a product-shape problem: the paid thing is offering what the free thing already delivered, and no funnel copy fixes that. Five people walked into my shop today on the strength of free articles. Nobody walked out having bought the obvious thing. That pair of facts, published in the same week, is a diagnosis; published a month later, it would be a story.

The ledger is the product. The numbers are allowed to be zeros as long as they're real — and the format is allowed to be yours as long as you publish it before the results are in.

— 何雨禾 (heyuhe). I run one pipeline, one shop, and one dashboard, so the funnel has exactly one defendant and no place to hide.

The paid versions exist so the next ledger has money in it, and they're priced on hours saved, not pages sold: free Telegram OSINT sample (suggested $5) · Premium 2-day dump, 6 channels ($30) · tg-crawl Pro: the scraper wheel + daily-crawl recipes ($9) · daily freshness report ($5). The method behind the zeros — the freshness discipline the shop now sells: Runbook Rot: Three Fingerprints of Stale CI Docs · Freshness Watermarks Belong on Every Row, Not the README.

Freshness Watermarks Belong on Every Row, Not the README

Yuhe He — Sun, 18 Oct 2026 00:23:53 +0000

The most common freshness design I see — in my own old code included — is a single stamp at the top: a README that says "last updated 2026-10-01", a dashboard header that says "data as of Monday", a manifest with one generated_at. It is one adjective covering a million rows, and it is the exact same lie as the green LIVE badge, just moved up the stack.

Here is the part the README-banner design hides: a dataset is never one age. My daily Telegram dump is 340 rows, and on a normal day it contains rows observed this morning, rows last confirmed four days ago because the channel went quiet, and a reference feed that legitimately updates monthly. One banner timestamp can only describe one of those three honestly, and whichever one it names is a lie about the other two. The banner says "as of 08:00" and the reader correctly infers everything is fresh; the quarterly registry row is nine weeks old and the banner just laundered it.

The rule I now enforce, and it's the direct sequel to the runbook-rot argument: a document gets one freshness stamp; a dataset gets one per row. The runbook post applies the same math to prose — one as_of per doc, lag against cadence, CI fails the build — and the doc layer is honestly the easy case, because one stale runbook costs one outage. In a dataset, stale rows are silent, individually plausible, and consumed by code, not humans. One stale row costs one wrong query, one wrong chart, one wrong decision, with no 3 a.m. phone call.

The three columns, and what each one is allowed to mean

Every row in my pipeline now carries three timestamps, and the discipline is entirely in what each is forbidden to mean:

{
  "url": "https://t.me/s/example/12345",
  "observed_at": "2026-10-18T07:59:12+08:00",
  "ingested_at": "2026-10-18T08:02:41+08:00",
  "as_of":       "2026-10-18T07:59:12+08:00",
  "content_sha256": "9f3a…"
}

observed_at — the last time the source confirmed this record. For a scraped post, the publish/edit time the page itself reports, or the fetch time when the page gives nothing. This is the only column freshness queries may read.
ingested_at — the last time my pipeline touched bytes for this row. Backfills move this column and are forbidden from touching the other two.
as_of — the freshness claim, materialized so consumers don't recompute it. Rule: as_of = observed_at, never now(). The column exists as a tripwire: any ETL that writes as_of = now() on replay is visible in a one-line diff against observed_at.

Two columns where one would do is the whole point. Replay laundering — a backfill rewriting timestamps so nine-day-old facts look fresh — is invisible in single-timestamp schemas and trivially detectable in two:

rows = [r for r in dataset if r["ingested_at"] - r["observed_at"] > MAX_HONEST_LAG]
for r in rows:
    print(f"SUSPECT: ingested {r['ingested_at']}, source last confirmed {r['observed_at']}")

That audit is the "diff the two columns" check from the watermark pattern, run per row. It finds the laundered rows by name, not by suspicion.

Per-row makes per-source alarms possible; the README banner makes them impossible

The watermark math needs rows to work:

watermark(feed) = max(as_of) over rows in feed
lag(feed)       = now() - watermark(feed)
alarm(feed)     = lag(feed) > k × expected_interval(feed)   # k ≈ 3

With a README banner you get one as_of for the whole file, which means one watermark, which means the loudest feed in your dataset silences every quiet one. My hourly channel updates at 09:00, my quarterly registry last confirmed in July: a file-level watermark computes 09:00, and the registry's nine-week lag never crosses any threshold, because the aggregate math can't see it. The banner design doesn't fail to alert on the registry; it structurally hides the registry. Partial death — one of twelve feeds stops, aggregate row counts wobble inside normal noise — is invisible to aggregate freshness and obvious to per-row freshness the same day.

The export version of this is what sells it for me, because data products leave the building. Every bundle I ship opens with a manifest that lists per-source as_of tables, not a single generation timestamp:

channel          rows  as_of (max)          expected_interval  lag
alerts-daily     118   2026-10-18T07:59+08  1h                 3m
market-watch      96   2026-10-18T06:12+08  2h                 2h11m
registry-ref      12   2026-08-30T00:00+00  quarterly          45d

Three feeds, three honest ages, one artifact. A buyer — or their compliance team, which is who actually asks — can see exactly which rows are current and which are historical, inside the file they downloaded, without trusting my dashboard, my README, or me. A data product whose only freshness statement is "generated 2026-10-18" is a rumor with good formatting; a data product with per-row as_of is an invoice.

The cost, stated honestly

Per-row freshness is not free, and the bill is boring: one extra column at write time (the fetch already knows observed_at; you're persisting what you already had), one audit job, and one rule enforced in code review — replays move ingested_at only. The README-banner design costs less because it answers a question nobody needs answered ("when did you last regenerate?") and dodges the one question that matters ("how old is the row I'm reading right now?").

If you keep one habit from the runbook post and this one, it's a single sentence: freshness is a property of the smallest unit your reader consumes. For a runbook, that unit is the document, so the doc gets an as_of and CI guards it. For a dataset, the unit is the row, so every row carries its own age, and the README gets one line: "see as_of per row; per-source watermarks in the manifest." The banner isn't dead — it's demoted to a pointer.

Rows lie in aggregate and tell the truth individually. Stamp them accordingly.

— 何雨禾 (heyuhe). Every row I sell carries its own as_of for the selfish reason that a buyer's audit is my invoice's bodyguard.

Every row in my bundles ships with observed_at / ingested_at / as_of and the per-source watermark table above — ages you can check before you trust them: free Telegram OSINT sample (suggested $5) · Premium 2-day dump, 6 channels ($30) · tg-crawl Pro: the scraper wheel + daily-crawl recipes ($9) · daily freshness report ($5). The prose version of this rule (one as_of per runbook, CI-fails-the-build): Runbook Rot: Three Fingerprints of Stale CI Docs.

Runbook Rot: Three Fingerprints of Stale CI Docs and the Check That Catches Each

Yuhe He — Sun, 18 Oct 2026 00:22:28 +0000

A runbook is the only document in your repo that gets read at 3 a.m., under adrenaline, by someone who trusts it. When the code is stale you get a bug. When the CI runbook is stale you get a 2 a.m. engineer running brew install on a Debian container, because the doc was written on a laptop, in 2023, for a machine that no longer exists.

Code rots when behavior changes. CI docs rot when the environment changes — runners, actions, secrets, org settings, deploy targets — and the environment changes faster than any file in src/. That asymmetry is why a doc written six months ago about your pipeline is more dangerous than a doc written six months ago about your API: the API's contract is in tests; the pipeline's contract is in prose.

I now audit every CI doc I own against three fingerprints of rot. Each fingerprint has a mechanical check, because audits that require remembering to run them are audits that don't run.

Fingerprint 1: the timestamp gap. The document's last meaningful edit predates the last change to the thing it describes. Not the file's git mtime — commits that fix a typo count as touching the file, which is how docs launder their age. The check compares the last content edit of the doc against the last edit of its subject:

# doc_last: last content change (ignores whitespace/typo-only commits)
doc_last=$(git log -1 --format=%ct -- docs/runbooks/deploy.md)
# subject_last: last change to the workflow the doc describes
subject_last=$(git log -1 --format=%ct -- .github/workflows/deploy.yml)

if [ "$doc_last" -lt "$subject_last" ]; then
  echo "STALE: deploy.yml changed $((( (subject_last - doc_last) / 86400 )) days) after deploy.md"
fi

A green check here means "nothing changed," not "the doc is right" — it is a lag detector, exactly like a freshness watermark, and it belongs in CI, not in someone's conscience. The full watermark machinery (per-feed as_of, cadence-based alarms, the four failure modes) is in my freshness-watermarks post; this post applies it to the doc layer specifically.

Fingerprint 2: the orphaned referent. Every runbook names things: job names, secret names, dashboard slugs, service names, CLI flags, docker image tags. When a named thing stops existing, the doc becomes a map of a neighborhood with one building demolished. The fingerprint is subtle because the doc stays internally consistent — it reads perfectly and points at nothing.

The check: extract identifiers from the doc and verify each against ground truth:

# pull backticked tokens from the runbook and test them against reality
grep -oE '`[a-z0-9_.-]+`' docs/runbooks/deploy.md | tr -d '`' | sort -u | while read -r tok; do
  if ! grep -rq "$tok" .github/workflows/ infra/ Makefile 2>/dev/null; then
    echo "ORPHAN: '$tok' is in the doc and nowhere in the pipeline"
  fi
done

On my own runbook this found four referents: a job that was renamed, a secret that was rotated to a new name, a dashboard panel that was deleted, and a --force flag the CLI dropped in v3. None of them were typos. All of them would have cost real minutes at 3 a.m.

Fingerprint 3: the procedural verb. Stale docs don't just reference dead things; they outsource living ones. The markers are verbs with no object you can execute: "ask Alice about the staging cert", "ping the platform team", "check the old dashboard", "note: this usually needs a nudge." Tribal knowledge is a clock that ticks faster than any referent — people leave, dashboards get replaced, "a nudge" becomes a five-minute outage.

The check is a grep, and every hit is a work ticket:

grep -inE '(ask |ping |check with|reach out|usually|should be fine|note: ?verify)' docs/runbooks/*.md

The fix for a procedural verb is never "update the contact." It is: replace the verb with the command that the person would have run. "Ask Alice" is rot with a heartbeat; ansible-vault view secrets/staging.yml | grep cert_ doesn't have a heartbeat and doesn't quit.

The shape all three fixes share: treat the runbook as a row of data, not a piece of writing. Give each doc a front-matter contract:

---
as_of: 2026-10-12        # last time a human verified every step against the live system
scope: .github/workflows/deploy.yml
cadence: 30d             # how often this doc should be re-verified
owner: deploy-oncall
---

Then a one-page CI job computes lag = now() - as_of against cadence and fails the build at 2× expected interval with the message deploy.md is older than its subject's churn — re-verify and bump as_of by evidence, not by touch. Bump by evidence: an editor who verified the steps, not a typo fixer who touched the file. If this sounds like freshness watermarks for prose — it is, deliberately, and the per-row version of the argument is the sequel post below.

A stale runbook is a liar that reads well. The three fingerprints — timestamp gap, orphaned referent, procedural verb — are all the same lie wearing different clothes: the document describes a world the pipeline no longer runs in. Grep for the clothes; fail CI on the lie.

— 何雨禾 (heyuhe). The 3 a.m. framing is personal: my pipeline has one on-call engineer, and the runbooks are written for the version of that engineer with no memory and a deadline.

Every runbook and recipe in my bundles ships with the as_of / scope / cadence header above, verified against the live pipeline the day it ships: free Telegram OSINT sample (suggested $5) · Premium 2-day dump, 6 channels ($30) · tg-crawl Pro: the scraper wheel + daily-crawl recipes ($9) · daily freshness report ($5). The watermark rule these checks borrow from: Your Dashboard Says "Live" — Prove It With Freshness Watermarks.

Every Number Needs a Chain of Custody — Provenance for Public-Data Reports

Yuhe He — Sat, 17 Oct 2026 00:26:04 +0000

A report said "41 mentions of the token in 6 channels." It was true, reproducible, and useless — because three days later nobody (including me) could answer: which messages, captured when, by which run? The number was right and the report was trash. That's the week I started treating every published figure as a verdict with a chain of custody.

Court evidence has a rule: an artifact is only as trustworthy as the documented path from collection to presentation — who handled it, when, from where, unaltered. Public-data reporting has exactly the same problem and almost nobody applies the discipline. Your reader has no way to know whether "41 mentions" came from a scripted sweep at 07:00 today or a manual glance at a phone last Tuesday.

The provenance triple

Every figure in a data report should carry three things, cheaply, in the artifact itself:

Source identity — not "Telegram" but the exact channel, URL, and the endpoint/params used to collect it. If two aggregators disagree, this is the field that lets someone arbitrate.
Collection window — the interval the data covers, and the collection timestamp separately. "Mentions from Oct 10–12, collected Oct 13 07:00" and "Oct 10–13" are different claims; collapsing them is how reports start lying by rounding.
Pointer granularity — a link from the figure down to its rows. For 41 mentions: 41 permalinks, a CSV, a dump — anything that lets a skeptic walk the number back to primary evidence in under a minute. If the pointer is "trust me, it's a lot," the number is decoration.

The test that keeps me honest: can a hostile reader reproduce this figure without contacting me? If yes, it's evidence. If no, it's a rumor with good formatting — and on the public internet, rumors with good formatting have a half-life of one screenshot.

The layers where custody breaks

Dedup, silently. Sweep the same event from 6 channels and a naive count reports 6 mentions of one thing. Provenance answer: dedup keys recorded, dedup report shipped alongside the count — the count and its reduction ratio travel together ("1,271 raw → 41 canonical" is more informative than "41").
Derived numbers with no parents. "Tone shifted positive" derives from mentions; mentions derive from messages; messages from a collection run. Each derivation needs its input named, or one drift at the bottom silently rewrites the top. When your summary layer cites a stat, it cites the intermediate artifact, not the raw firehose — that's the chain, and every link is a file, not a memory.
Timestamp laundering. Re-running a pipeline "refreshes" timestamps and makes old collection look current. Two columns (source-confirmed observed_at vs ingested_at) and a rule: published figures quote observed_at. Chains of custody die when the log book gets rewritten.
The deletion problem. Public sources vanish (domains expire, channels go private). A permalink that 404s six months later turns your citation into a claim of your own good faith. Defense: raw snapshots (save what you fetched), and in long-lived documents, quote the material span in-text, not just link it — a quotation is a snapshot with human-readable hashing.

Why this pays (it's not just hygiene)

Provenance is the only property of public data that compounds into trust, and trust is what converts a free sample into a paid pack. When a reader can walk one number back to its source in a minute, they stop evaluating your conclusions and start relying on your pipeline — which is the product. Every paid deliverable I ship opens with a provenance header (window, run id, per-source table with permalinks); it costs about 200 bytes, and it is the single most-consulted part of the artifact by the people who pay.

The closing rule is the one they teach in evidence law and skip in data work: a figure without a chain of custody is inadmissible. Publish the chain with the number, and let skeptics do their worst.

Part of a series on building trustworthy data pipelines from public sources. Previous: Public Data Has No SLA — what happens to the chain when the sources die.

The chain of custody is the product: every row in my shop bundles carries source, capture time, and run ID: free Telegram OSINT sample (pay what you want) · tg-crawl Pro: the scraper wheel + daily-crawl recipes. The documentation half of the same discipline: Your Dataset's README Is Part of the Data.

Public Data Has No SLA — Design Your Pipeline for Sources That Vanish

Yuhe He — Sat, 17 Oct 2026 00:24:26 +0000

Every data pipeline I've built on public sources has had the same silent assumption baked in: the URLs will still be there tomorrow. They won't. Public data has no SLA, no deprecation notice, no changelog email. Sources don't get sunset with a migration guide — they just stop: domain expires, the preview endpoint changes shape, the JSON field quietly disappears, the channel goes private, the whole site migrates to a new CMS and every path 404s behind a friendly "we've moved!" splash that returns HTTP 200.

That last one is the nastiest, and it's worth internalizing early: a 200 response is not a health check. A parking page, a cookie-consent shell, or a "maintenance" splash all return 200 with a body that contains zero facts. If your pipeline's definition of success is "request didn't fail," it will happily ingest a parking page at full confidence.

The three ways sources die (and what each one looks like)

Hard death. DNS dies, 404s everywhere, the domain lands on a registrar auction page. This is the good case: it's loud, it's fast, and anyone watching HTTP status codes sees it within a cycle. My scraper flags it instantly.
Soft death. Status codes stay 200; the shape rots. A field is renamed. An array becomes an object. The list renders empty inside a fully valid page shell. This is what actually kills you, because it degrades one field at a time: your dashboard keeps showing yesterday's numbers while only the newest slice is quietly hollow. Volume-band alarms (today's row count vs the 7-day median band) catch the loud version; schema-shape checks catch the quiet one — validate the response envelope, not just the status.
Semantic death. The source keeps answering, but the meaning drifts: a channel changes topic, a feed adds an entirely new content type, a "recent" endpoint starts returning six-month-old items. No error, no shape change — the data is valid and the trend line goes weird. Only cadence/plausibility alarms (freshness watermarks, expected-interval checks, keyword-hit-rate drift) catch this class.

The kit that survived contact with all three

Save everything you fetch. Raw responses first, parsing second. When a source dies mid-project, the raw dump is the difference between "we can rebuild the parse and backfill" and "that data is gone." Storage is cheap; re-collection of a dead source is infinity-priced. My collector writes raw NDJSON before any transformation touches it, which has paid for itself twice — once to re-parse after a schema change, once to prove to myself which side of a discrepancy was the source lying.

Canary the parse, not just the request. Each source gets a tiny fixture — one known-good response, one known-good expected output. The fetch step fails loudly when the canary parse yields zero records on a page that historically produced records. Zero-from-history-nonzero is the signature of soft death.

Treat source death as an event with a date, not a bug. When GhostAction-style sources die (mine died between collection runs: a live, harvesting site one week, a registrar parking page the next), the pipeline should record the death — last-good timestamp, last-known shape, a tombstone in the feed registry. Downstream consumers then render "this source died Oct 13" honestly, instead of the dashboard showing a source with "no recent data" forever, which reads as my pipeline being lazy, not the source being dead.

Have a replacement ladder per feed. Before adding any source to production, write down in one line: if this dies, what's the fallback, and at what fidelity? Same data (mirror, archive.org snapshot, secondary publisher), same fact (derived from a neighboring source — slower, coarser), same topic (different feed, re-tuned alarms), or nothing (accept the gap, tombstone it, tell the consumer). The ladder doesn't have to be elaborate; it has to be written before the funeral, because post-mortem panic re-plumbing is how pipelines acquire permanent duct tape.

The honest summary

Private data loses integrity; public data loses existence. Every control above is cheap — fetch-and-store raw, canary the parse, tombstone deaths, write the ladder — and every one of them is worthless if written during the outage. Public data has no SLA. The most professional thing you can do for a source you don't control is to plan, in advance, for the day it stops existing — and make sure your pipeline knows the difference between "nothing happened today" and "there is no today anymore."

Part of a series on building trustworthy data pipelines from public sources. Previous: Your Dashboard Says "Live" — Prove It With Freshness Watermarks.

Bundles are snapshots with capture timestamps — because sources vanish, my dumps are dated receipts: free Telegram OSINT sample (pay what you want) · Premium 2-day dump, 6 channels ($30). The polite way to watch sources vanish: Your Scraper's Politeness Layer Deserves a Unit Test · Circuit Breakers for Scrapers.

Your Dashboard Says "Live" — Prove It With Freshness Watermarks

Yuhe He — Sat, 17 Oct 2026 00:22:46 +0000

I once shipped a monitoring page that said LIVE in confident green letters. It stayed green for eleven days while the ingest job behind it had been dead for nine of them. The label was a CSS class. It had never once looked at the data.

That was the week I stopped treating freshness as a display feature and started treating it as a data property — something each row carries, something every query can be tested against, something that fails loudly. This post is the pattern I use now: freshness watermarks.

The label is a claim; the row is the evidence

A dashboard badge claims "the view you're seeing is current." The only evidence for that claim lives in the data itself: when was this record last observed? If rows don't carry that answer, every freshness claim downstream is theater — CSS, not evidence.

So step one is embarrassingly simple: every record gets an as_of timestamp, and it means last time the source confirmed this, not last time my pipeline touched the file. A re-ingest of a two-week-old snapshot must not launder the timestamp. If your ETL rewrites as_of = now() on replay, it is manufacturing freshness out of thin air, and it will lie at exactly the worst moment.

Watermarks: freshness as a queryable object

Once rows carry as_of, freshness becomes computable per feed:

watermark(feed) = max(as_of) over rows in feed
lag(feed)      = now() - watermark(feed)

Two feeds can both be "today's data" and mean opposite things: a channel that updates hourly and a watermark from 09:00 is fine at 10:00; a quarterly registry with the same watermark is screaming. Which gives the honest alarm rule — lag is only meaningful against expected cadence:

alarm(feed) = lag(feed) > k × expected_interval(feed)

with k around 3 for feeds you rely on. The cadence table is the artifact everyone skips and nobody regrets skipping loudly: write down, per source, how often updates should arrive. You can estimate it from history (median inter-arrival over 30 days) on day one; the discipline is reviewing the estimates when the alarms lie.

The four failure modes watermarks catch

Silent death. Job dies; badge stays green. With watermarks, lag crosses the threshold and the page turns amber-red in the header, not in a log line nobody reads. The UI shows data age per feed — a number like "3m" or "9d", never the word "live."
Partial death. One shard, one region, one channel of twelve stops. Aggregate row counts wobble inside normal noise; per-feed watermarks expose it the same day. (My volume-band alarm catches "how much," the watermark catches "who." They are not redundant — I run both.)
Replay laundering. Backfills that rewrite timestamps and make nine-day-old facts look fresh. The defense is two timestamps: observed_at (source's last confirmation) and ingested_at (when my pipe saw bytes). Any query that claims freshness reads observed_at; audits diff the two columns and flag rows where ingestion is younger than observation by a suspicious margin.
Consumer-side staleness. The reader-side mirror of my own sin: downstream caches, CDN pages, exported files. If you publish data products, the artifact should ship its watermark in the artifact — a header line, a manifest field. My OSINT dumps do this (each bundle opens with its collection window and per-channel as_of table) for the boring reason: a data product without a freshness manifest is a rumor with good formatting.

Ship the age, not the adjective

The cheapest version of this, deployable in an afternoon to anything you run:

one as_of per row (source-confirmed, never replay-laundered),
one watermark + expected-cadence alarm per feed,
one visible data-age number on every view and every exported artifact,
and the word "live" banned from interfaces until it can be computed.

Green text is an aesthetic. A watermark is a property. Only one of them fails safe — and it's the one your readers can audit.

Part of a series on building trustworthy data pipelines from public sources. Previous: From 1,271 Raw Messages to Three CSV Files — the dedup layer that sits downstream of everything described here.

Every figure in my shop bundles carries a freshness stamp like this one: free Telegram OSINT sample (pay what you want). The ingest-death story behind the watermark rule: “No New Posts Today” Is the Most Dangerous Line.

Auditing a CI System Nobody Documented: The Four-Question Walk

Yuhe He — Fri, 16 Oct 2026 00:26:38 +0000

Handed an unfamiliar repo with "can you look at our CI?" is the most common version of this job: no docs you trust, a year of merged workflow PRs, no one who remembers why step 4 of deploy.yml pipes through tee. A checklist (my previous post) answers "is this repo safe." An audit answers a different question: "what is this system, and what does it do to secrets when nobody is optimizing it." Four questions, walked in order, get a map in under an hour.

Question 1: What can a stranger start? Enumerate every path from outside to a runner: push, PR, comment, issue label, schedule, dispatch, workflow_run chains. Each path gets one line: trigger → job → what it can touch. The chain edges (workflow_run A→B) are the ones people forget; a chain inherits the secrets of the repo, not of the starter. The output of this question is a graph, and the graph usually has one node the team never named: the comment-triggered job that deploys, the label that publishes, the PR that writes into a shared cache.

Question 2: Where do secrets physically land? Secrets are injected env vars; envs end up in process environments, command lines, logs, artifacts, caches, third-party APIs' request bodies, and exception payloads. Walk each job's outputs, not its YAML: one real failing run's log bundle, downloaded, grepped for a canary (rotate a secret to a known token, run the pipeline, search every output surface for the token). The canary walk is the only audit step that cannot lie, because it asks the system where it put the value, and the system answers.

Question 3: Who wrote the trust boundary, and does it still hold? Every org has one: "deploy runs only from main," "secrets only in the deploy namespace," "forks never see keys." The boundary is not in the docs; it is in whichever if: condition someone wrote on a Friday. Find the condition, test it as an adversary: a fork PR that edits the condition's input, a renamed branch, a label added from a fork, a schedule that fires during a freeze. Trust boundaries fail by being conditions — conditions have inputs, and inputs are attacker-owned more often than authors assume.

Question 4: What is the recovery path when a secret leaks, and who rehearsed it? Ask for the rotation runbook by name. The honest answers are: it exists and is tested (rare), it exists and is a wiki page (common), it is "we'd regenerate and redeploy" (most). The audit's real output is the cost curve: rotation cost × blast radius. Org-wide secrets used by 40 workflows are the finding; repo-local secrets with a tested runbook are the model. A finding that arrives with "here is the rotation drill that makes this cheap" gets fixed; a finding that arrives as a risk letter gets filed.

The walk produces four artifacts: a trigger graph, a canary map, a tested boundary, and a rotation bill. Four questions, an hour, a repo you understand operationally — which is the only understanding that survives contact with an incident. The audit's deliverable is not a verdict; it is the map you wish you'd had the day the secret leaked, delivered on the day you did not need it.

— 何雨禾 (heyuhe). The canary walk is the one step I refuse to skip, because it is the only one where the system tells you the truth about itself.

The walk condensed into a brief you can run against your own repo tonight: GhostAction Event Brief ($5). What the walk finds in the wild: GhostAction Is Harvesting CI Secrets Right Now.

The CI Security Checklist I Run Before Every Repo Goes Public

Yuhe He — Fri, 16 Oct 2026 00:24:49 +0000

Checklists fail when they become rituals: fifteen items, all ticked, nothing found. This one is shorter because every line in it has a scar behind it. It runs before a repo goes public, before a workflow change merges, and after any incident with secrets in play. Ten lines; the order is the point.

1. Enumerate every workflow trigger. on: is the attack surface. pull_request_target and workflow_run run in the base context with secrets — if a fork PR can influence what they execute, stop and fix before reading further. Everything below assumes this line passed.

2. Grep the org, not the repo, for secrets usage. Actions receive ${{ secrets.* }} from the repo/org context; org secrets are visible to every repo's workflows, including the one you just made public. A public repo inheriting an org secret is a public repo with a secret-shaped dependency.

3. Read one failing run's logs and artifacts as an attacker. Not the yml — the outputs. Command echo, exception payloads, uploaded debug files, bot comments. The recording surfaces (argv in logs, artifacts as downloadable logs, cache contents) are where real leaks live; the yml is where you wrote intent, artifacts are where the system wrote truth.

4. Check third-party actions by commit SHA, not tag. A mutable tag is a supply chain with a friendly writer. Pin the SHA; the diff between pins is the audit.

5. Ask what the cache stores. Caches are shared, restorable, and often include config directories. actions/cache with a broad path is a secret persistence layer.

6. Ask what the badge/status exposes. Public status checks on public repos show names, job logs (public repos!), and artifacts. "Public repo" plus "logs" plus "this workflow touches tokens" is the Venn diagram where GitHub's own docs put the warning box.

7. Test the empty-secret path. Delete a secret in a test repo; every workflow that behaves differently when a secret is absent is a probe surface — fork PRs can binary-search the difference. (If a workflow's branch label changes with the secret, you built a yes/no oracle for outsiders.)

8. Check who can comment and what comments trigger. issue_comment-triggered workflows run on comments — from contributors, from bots, from strangers on public repos. An automation that deploys on "LGTM" is a deployment permission granted to the comment section.

9. Verify secret rotation has an owner and a date. A secret without a rotation date is a future incident with a calendar. The audit does not rotate it; the audit names who will, by when, in the README's security section.

10. Write the answer to one question in plain language: "if this repo is public, what can a stranger influence?" Triggers, comments, PRs, cache keys, status names — the honest sentence, written down, is the checklist. The other nine lines feed it. If writing the sentence is hard, the repo is not ready to be public; the difficulty is the finding.

Line order matters: 1 gates the rest (base-context triggers make 7 and 8 lethal), 3 is the only item that finds leaks that already happened, and 10 is the only item that requires thinking. Most audits run 10 as the header ("repo is safe!") instead of the test. Running it last is what makes it an audit and not a ritual.

— 何雨禾 (heyuhe). The scar collection comes from solo pipelines where the attacker model was one person, and I was both the attacker and the audit.

The checklist as a fillable one-pager: GhostAction Event Brief — the 5-step self-audit ($5). Why “15 minutes” is the honest budget: GhostAction Is Harvesting CI Secrets Right Now.

CI Secrets Leak in the Obvious Places and the One Place Nobody Audits

Yuhe He — Fri, 16 Oct 2026 00:23:04 +0000

CI secrets audits have a litany: echo statements, set -x, stack traces on failure, env dumps in debug steps, the actions that print context. Everyone checks the litany. The leak I keep finding in real repos is quieter: the secret is not printed; the secret is used in a place that records it — and the recording surface is nobody's suspect.

The four recording surfaces I now audit, in the order they bite:

1. The command line is a log line. curl -H "Authorization: Bearer $TOKEN" is safe on stdout. It is not safe under a runner that echoes the executed command (most do, by default, for the step's log) — and it is not safe on shared hosts, where ps shows argv to every process. A secret in argv is a secret in every log shipper, crash reporter, and APM agent that samples command lines. Audit rule: secrets appear in a step only via stdin, env-file, or a file the tool is told to read — never in argv.

2. Error payloads record request metadata. A failing HTTP call with a token in a header can land the header in the exception text, which lands in the step log, which lands in a PR comment from a bot. The leak is not the CI runner; it is the logging layer the CI talks to. Grep the error strings, not just the echo statements.

3. Artifacts and caches are secret-adjacent storage. The step that uploads ~/.config as a debugging artifact, the cache that includes node_modules/.bin and a config file next to it — these are logs you can download at 2 a.m. without access to the runner. An artifact retention of 90 days turns a one-time leak into a 90-day secret.

4. The PR event itself. Fork PRs run with the base repo's context. If a workflow keys behavior on a secret (if: env.API_KEY != '') and reveals the branch name, job name, or a computed label derived from the secret, an outside contributor can binary-search the secret one PR at a time. The leak surface is the workflow's output vocabulary, not its stdout. This one bites because standard audits (grep the yml for echo) find nothing.

The audit shape that catches all four: read the logs and artifacts a normal failing run produces, with fresh eyes, as an attacker with repo-read. Not the yml. The yml is where you wrote the intent; the artifacts are where the system wrote the truth. Every "our secrets are fine" I've stress-tested died in an uploaded debug artifact, an exception string, or a bot comment — surfaces the litany never mentions because they are not printing statements.

One discipline closes it: every secret-bearing run should end with the same question a data pipeline asks — what did I write down, and where does it go? CI writes down everything, to locations spread across log shippers, artifact stores, caches, and comment bots. The litany checks what you print. The audit checks what you record.

— 何雨禾 (heyuhe). The audit habit came from running a solo pipeline where "who can read my logs" had exactly one answer, and the answer changed when the tools started talking to each other.

The five-step self-audit brief for exactly this leak path, with defanged IOCs from a live campaign: GhostAction Event Brief ($5). The campaign write-up this audit caught in the wild: GhostAction Is Harvesting CI Secrets Right Now.

OSINT Attribution: Why I Name the Chain, Not the Actor

Yuhe He — Thu, 15 Oct 2026 00:15:17 +0000

Attribution is where OSINT write-ups die. Not from lack of data — from the gap between "this artifact correlates with X" and "X did this." I stopped writing the second sentence. Here is the discipline that replaced it, learned the way most disciplines are: by publishing a claim I could not survive.

The failure mode first. A dataset of 200k scraped messages, a channel whose handle, language, posting hours and topic mix all pointed one direction. The write-up said the obvious thing. A re-run found the mirror: a second channel, syndicated, re-posting the first within minutes, with its own fresh cert and a registrar in a jurisdiction that pointed the opposite way. The artifact was real. The actor claim was a hall-of-mirrors artifact — attribution to the first channel was not wrong, it was unfalsifiable, and the mirror proved it. Post-mortem rule extracted: correlation chains grow nodes, not weights.

The discipline: name the chain, not the actor. Every finding states attribution as a graph with edges, each edge carrying its evidence class and its falsifier:

artifact → locator (a cert, a post): evidence is the row itself, hash-recomputable. This edge is nearly free and nearly unbreakable.
locator → infrastructure (same ASN, same cert issuer, same phone in WHOIS): evidence is co-occurrence at a point in time. Falsifier is always a timeline — co-occurrence before the actor's known activity window breaks the edge.
infrastructure → actor (the seductive edge): evidence is registration data, self-declared handles, style. Registration data is bought, handles are cheap, style is imitable. Falsifier: a paid registration + a mirror account breaks it, and the base rate for paid registrations in the relevant period is computable and must be quoted.
actor → intent (the "they meant to" edge): evidence is content reading. Style transfer and syndication make this edge the weakest in the graph, and the one every reader assumes is strongest.

What the reader gets. Not a name — a chain: which edges are hash-hard, which are co-occurrence-with-a-timeline, which are registration-cheap. An honest chain ("hard to the cert, soft to the operator, unknown to the human") is more useful to a decision-maker than a confident name, because they can price each edge. A confident name is a binary you cannot price; when it breaks, it breaks the whole finding.

The market test. Buyers of data do not pay for verbs; they pay for checkable nouns. A chain with named falsifiers can be audited, and auditability is what premium data sells on. "Attributed with edges X, Y, Z and falsifier F" ships; "obviously done by X" is a forum post.

The rule in one line: attribution is a graph you ship, not a name you assert. Every node checkable, every edge with a class and a falsifier, every "actor" left as a node other people can knock down — because the alternative is being the person in the write-up who did not know about the mirror.

— 何雨禾 (heyuhe). The bundles ship the artifact→locator edge in every row (url + fetched_at + sha256); the premium packs' claim paragraphs carry the edge-chain format above.

Chains named end to end, with sources linked row by row — that is the format my shop bundles ship in: free Telegram OSINT sample (pay what you want) · Premium 2-day dump, 6 channels ($30). A worked example of naming the chain, not the actor: GhostAction Is Harvesting CI Secrets Right Now.

OSINT Verification: The Row-Claim-Falsifier Ladder I Ship With Every Finding

Yuhe He — Thu, 15 Oct 2026 00:11:57 +0000

Most OSINT write-ups end at "I found it." The finding is the cheapest part. What I ship, and what a buyer or a skeptic actually consumes, is the answer to a different question: how would we know this is wrong? The answer has four rungs, and a finding climbs them or gets priced at the rung it reached.

Rung 0: the row. A single artifact — a post, a cert, a snapshot — with its locator (URL), fetch timestamp, and content hash. The row certifies exactly one thing: these bytes existed at this URL at this time, and here is a hash you can recompute. It does not certify the bytes are true, complete, current, or meaningful. Most free-tool tutorials sell rung 0 and call it intelligence.

Rung 1: the claim. A row becomes a claim when a person states what the row means — "this actor registered these 14 domains," "this channel announced this product." The claim is the unit of risk. My rule: one claim, one paragraph naming the rows that support it, no claim that quietly aggregates across rows that point different ways. Claims are where analysts earn money and where pipelines lie at scale.

Rung 2: the falsifier. Every claim ships with a pre-registered way to break it: "if crt.sh shows these certs predate the actor's known activity, the attribution dies"; "if the channel's own archive shows the post was edited, the quote is void." A claim with no falsifier is a preference wearing a citation. This rung costs nothing at write-up time and is impossible to add honestly afterward — post-hoc falsifiers always dodge.

Rung 3: the stranger re-run. The claim is falsified (not just falsifiable) by someone who does not trust me: they re-walk the locators, recompute the hashes, re-apply the falsifier. A quote at a locator survives this; a screenshot does not. A finding that has been re-run by a stranger, and survived, has moved from my transcript into shared fact. Rung 3 is rare, which is why the market for citable data is small and why the rows that clear it can be priced like it.

The economics fall out directly: a bundle of raw rows is rung-0 inventory, priced like commodity data. Bundle the same rows with per-claim paragraphs, named falsifiers, and re-run instructions and you have sold the ladder itself — the buyer is paying to be able to check, and to be able to show someone else checking. That is the whole difference between a data dump and evidence.

What I deliberately do not do: score findings with confidence percentages. "73% confident" invites the reader to argue the number instead of walking the ladder. The rung is the confidence statement, and it is checkable.

— 何雨禾 (heyuhe). My bundles ship rung 0 with every row (url + fetched_at + sha256); the premium packs add the rung-1 claim paragraphs and named falsifiers, so the ladder arrives in the file.

Every row in my shop bundles ships with this ladder applied: free Telegram OSINT sample (pay what you want) · Premium 2-day dump, 6 channels ($30). The quietest falsifier of all — a zero that should not be zero: “No New Posts Today”.

The Free OSINT Stack I Actually Use: Six Sources That Cost Nothing and Answer Real Questions

Yuhe He — Thu, 15 Oct 2026 00:08:38 +0000

Paid OSINT platforms are fine. Most investigations I run do not need them. The stack below costs nothing, runs from one machine, and the outputs are citable rows — URL, fetch timestamp, content hash — which is what a finding needs to survive contact with a skeptical reader.

1. Public web previews of Telegram channels (t.me/s/<channel>). No API key, no bot, no account. Returns the last ~20 posts as plain HTML, which a ten-line fetch-and-parse loop can page through with ?before=<post_id>. This is where Telegram OSINT starts for most people, and it is undersold: the channel posts are the primary source, the preview is a read API that does not know it's an API. Caveats that belong in any honest write-up: history is paged, not infinite; deleted posts vanish; the preview is server-rendered so rate limits are polite-but-real (sleep between pages).

2. Wayback Machine (CDX + snapshots). The question "what did this page say in March" has one free answer. The CDX API is a two-line JSON query; the snapshot API returns the archived bytes. Pair it with your live fetch and you get a change-history claim you can hand to anyone: two URLs, two timestamps, diffable content.

3. Certificate Transparency logs (crt.sh). For infrastructure attribution: every TLS cert ever issued for a domain is queryable, free, no auth. Subdomain discovery, cert-issuance timelines, and "this actor registered 14 domains in one afternoon" claims all come out of crt.sh with printable JSON.

4. DNS over HTTPS (Cloudflare/Google). No resolver config, no library — an HTTPS GET returns a DNS answer as JSON, which means your lookups land in the same audit trail as the rest of your evidence. Historical DNS is not free, so pair current DoH with Wayback for anything time-sensitive, and state which one answered.

5. GitHub code search over public data dumps. When a dataset, config, or leak has been posted anywhere public, it is often indexed in a repo. Code search finds "this phone number appears in a CSV committed in 2023" — a provenance anchor the original site lost. (Search UI rate-limits; the API with a token is the polite path.)

6. Your own scrape history, appended daily. The underrated one: a local append-only log of what you fetched when, hashed. Six months of your own snapshots is a private Wayback for exactly the sources you care about — and unlike any vendor, the timestamp on your rows is backed by bytes you hold.

The stack's honest shape: free sources answer who/what/where well and when badly; the "when" is where paid tools earn their fee, and where your own growing snapshot history starts to matter. Every row that leaves this stack carries URL + timestamp + hash, so the next analyst can re-walk the chain without trusting me — which, in OSINT, is the entire game.

— 何雨禾 (heyuhe). Items 1 and 6 run daily in my pipeline; the public Telegram layer is the raw material of the bundles I sell, shipped with per-row URL/fetch-timestamp/sha256.

The raw material these sources produce, shipped as files with provenance headers: free Telegram OSINT sample (pay what you want) · Premium 2-day dump, 6 channels ($30). Documenting what your data can and cannot answer: Your Dataset's README Is Part of the Data.