DEV Community: Hoelee

The Forward-Auth Gate That Verified Perfectly — and Wasn't Live

Hoelee — Mon, 02 Nov 2026 04:25:39 +0000

Originally published on my blog — read it there with the full context, code and updates.

I finished the gate, ran the check, and it passed.

HTTP/1.1 302 Found
location: https://app.example.com/outpost.goauthentik.io/start?rd=...
x-powered-by: authentik

An unauthenticated request was being pushed to my authentik login page. Branded, correct, working. I was about to move on — except that my own rule for anything user-facing is to hit it the way a user hits it, from outside, over the public internet. So I did.

GET https://app.example.com/          → 200
x-powered-by: Express

The app answered directly. No login page, no redirect, no gate. The exact same hostname, one request path gated and one wide open, depending on where the request entered my network.

That was the whole day: not the SSO setup, which worked, but the fact that a hostname can be served by more than one ingress layer, and I had only gated one of them.

Why this matters beyond my one app

If you self-host anything behind a tunnel, a reverse proxy, or both, you have probably already made this assumption: "the reverse proxy is what serves my domains." It's an assumption that holds right up until it doesn't, and it fails in the most dangerous direction — everything you change looks correct, because the path you're inspecting is correct.

The failure mode is silent by construction. There is no error, no log line, no hint. Your configuration check passes because you're checking the configuration you edited. Meanwhile the real traffic — the traffic you were protecting — never touches it.

I've since found two other gates on the same host with the same latent problem, built the same way, waiting for someone to trust them.

The setup: SSO for an app that has no SSO

The app was a self-hosted database UI on my NAS (Docker, published on a host port, sitting behind nginx and a Cloudflare Tunnel — the usual shape of the infrastructure I manage). I wanted a login wall in front of it so that "reachable from the internet" wouldn't mean "anyone who guesses the hostname sees the login form and can start guessing passwords".

The app's own documentation advertises SSO. I'll keep this post focused on the ingress trap and put the licensing story in its own write-up, but the short version is that the advertised SSO is a licensed feature, the open-source build ships the code path but not the permission to use it, and the community fork that used to fill the gap hasn't been touched in two years. Twenty minutes of research, two abandoned approaches, one conclusion: the app would never do SSO for me, so the gate had to live in front of it.

Which is a normal, well-trodden thing to do. authentik calls it a proxy provider, and the flow is:

A login flow for this app — its own title, its own background, so a user who lands on the login page knows which system they're being asked to authenticate for.
A proxy provider — the object that knows the app's external hostname (https://app.example.com) and its internal address (http://app-container:8080).
An application linking the two, attached to the embedded outpost. The outpost is the component that actually receives traffic, checks for a session, and proxies the request to the app's internal host after login.
The ingress — pointing that hostname at the outpost instead of at the app.

Steps 1–3 live in authentik and are configuration. Step 4 is the one that decides whether any of it is real, and it's the one I got wrong.

Two smaller things from steps 1–3, since they cost me time:

A partial PATCH on a proxy provider is rejected. Updating one field returns:

{"internal_host": ["Internal host cannot be empty when forward auth is disabled."]}

The serializer re-validates the object, and a proxy provider without an internal host is invalid — so a one-field update fails as if you'd cleared a field you never touched. Send the identity fields together with the field you're changing:

curl -s -X PATCH "$AUTH/api/v3/providers/proxy/$PK/" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"mode":"proxy","external_host":"https://app.example.com",
       "internal_host":"http://app-container:8080","skip_path_regex":""}'

authentik API tokens can expire mid-session. After working for an hour, every call started returning 403 {"detail":"Token invalid/expired"}. Note the status code: 403 means "we know who you are, and you may not do this" — the response body is the only thing that tells you the token died. If you cache a token in a script, re-read it from the database when calls start failing instead of assuming a permissions problem.

The trap: I verified the layer I changed

Here is the check I ran, from the host itself:

curl -sk -o /dev/null -w '%{http_code} -> %{redirect_url}' \
  -H 'Host: app.example.com' https://localhost/
# 302 -> https://app.example.com/outpost.goauthentik.io/start?rd=...

This is a good check. It proves the nginx vhost serves the outpost, that the outpost recognises the hostname, that the provider is linked to the application, and that the login flow is alive. Everything I had configured was proven correct in one command.

It says nothing whatsoever about whether public traffic reaches that vhost.

Here's why. My hostnames are served by two independent ingress paths:

Path	What serves it	Where it lands
Direct WAN IP → nginx (443)	the vhost I edited	the outpost ✅
Cloudflare Tunnel (`cloudflared`)	a rule inside the tunnel's own config	the app's port, directly ❌

The tunnel's ingress is a small ordered list evaluated per hostname:

app.example.com     -> http://app-container:8080     ← straight to the app
*                   -> http_status:404

That rule was written years earlier, when the goal was simply "make this reachable". It never mentioned nginx, because it never needed to. So my beautiful vhost sat off to the side of the real request path, gating a route that public traffic doesn't take.

The two paths even answer on the same hostname, so nothing about the URL hints at the difference.

What finally made it obvious was a response header. The gated check returned x-powered-by: authentik. The public check returned x-powered-by: Express. Same URL, same second, two different servers answering. When you suspect you're looking at the wrong layer, read the headers — they name the software that answered, which is exactly the question you're asking.

To see your own tunnel's rules rather than guessing (Cloudflare's API, not the dashboard):

curl -s "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/cfd_tunnel/$TUNNEL_ID/configurations" \
  -H "Authorization: Bearer $CF_TOKEN" \
  | python -c "import sys,json; [print(r.get('hostname','*'), '->', r.get('service','')) for r in json.load(sys.stdin)['result']['config']['ingress']]"

git.example.com   -> http://10.0.0.10:6880
app.example.com   -> http://app-container:8080
bot.example.com   -> http://bot-container:8080
*                 -> http_status:404

There it is, in one line, after a day of looking at nginx.

The permission wall on the way to the fix

Editing that rule needs a token with Account → Cloudflare Tunnel → Edit. Mine didn't have it, and no zone-scoped token can substitute: tunnels are account-scoped objects, so a Zone-scoped token fails the read as well, with 1001 Not authorized. If you only have zone-level tokens, open the dashboard instead: Zero Trust → Networks → Tunnels → your tunnel → Public Hostnames and change the rule's service to your outpost.

For me the correct target was the outpost on the same Docker network as cloudflared:

app.example.com  ->  http://authentik-server:9000

The outpost reads the Host header (which the tunnel preserves) to pick the application, then proxies to the internal host from the provider config. That's the whole change: one service value. Everything I'd built in authentik was already correct — it was simply never on the path.

Optional SSO: gate the UI, leave the API alone

Once the gate is really in front of the traffic, the next problem is the API. A hostname-level gate can't tell a script from a browser, so token-only API calls get redirected to a login page — they don't fail with a JSON error, they get HTML.

If you want SSO for humans and no SSO for machines, authentik's proxy provider has a per-path exemption. The UI calls it Unauthenticated Paths; the API field is skip_path_regex:

{"skip_path_regex": "^/api/.*"}

Verified result on my instance, same hostname, same session:

Request	No session	Result
`/` and `/dashboard`	yes	`302` → authentik login
`/api/v1/health`	yes	`200` from the app
`/api/v2/tables/.../records` + API token	yes	`200` with real data

Two caveats worth knowing before you rely on it. This exemption applies at the outpost, so unlike a path split at the CDN it works on every ingress path. And it is not per-user: the gate applies to the request, not the person. There is no "some users can skip SSO" — either a path is exempt for everyone, or everyone needs a session.

I used a simpler form of the same idea first: move every machine caller off the public hostname before the gate exists. My automation was calling the app through the public URL and being routed out to Cloudflare and back into my own network for no reason. Pointing it at the container name (http://app-container:8080) shortened the path, removed a failure mode, and meant the gate couldn't break it. If a login wall takes down your integration, the integration was depending on the door being unlocked.

The honest part: this is not single sign-on

I have to be straight about what a forward-auth gate gives you, because "SSO" oversells it.

The outpost knows who you are. The app does not. Unless the app can consume an upstream identity — header-based auth, or native OIDC/SAML — it will show its own login form after the gate lets you through. So the user experience is: authentik login, then the app's login.

For a user who is already signed in to authentik, the first step is often silent. But it's still two systems, two session lifetimes, and two places a password can be wrong. In my case the app has no header auth and its own OIDC is a licensed feature, so the gate would have been an extra door bolted in front of the existing one — no reduction in logins at all.

That's a legitimate trade for some setups: keep the app's login for its own users, and stop the internet from reaching the login form. It's a poor trade if what you actually wanted was "my team types one password". Which meant the honest answer for me was to turn the whole thing back off.

Reverting it properly (and the arithmetic that saved ten other apps)

Undoing the gate is where I nearly caused a much bigger outage than the one I was preventing.

The nginx change was one line in a large generated vhost file. The obvious revert is to substitute the port back:

# DON'T — this is the version that would have hurt
sed -i 's#proxy_pass http://localhost:10000;#proxy_pass http://localhost:10380;#' vhost.conf

Before running it, I counted how many server blocks point at that same port:

grep -c 'proxy_pass http://localhost:10000;' vhost.conf
# 11

Eleven. My app was one of them. Ten other applications are gated through that outpost, and a blanket substitution would have silently un-gated every one of them at once — including the ones doing real authentication work. The "fix" would have looked successful and removed authentication from most of my infrastructure.

What I did instead:

Snapshot the current state first, so the revert was itself reversible: cp -a vhost.conf vhost.conf.bak-$(date +%Y%m%d-%H%M)-revert.
Find the exact line by diffing against the backup taken before the original change. That diff showed one changed line, which is proof of exactly how much I had touched.
Edit that line only, then re-diff against the original backup and require the result to be identical before going further.
Update the second file that mirrors the same setting — the NAS's own reverse-proxy database, which regenerates the vhost on DSM events. I edited the single key and asserted the hostname before writing, never restoring the whole file.
nginx -t then reload.

And then a detail that cost me another twenty minutes of doubt:

nginx -s reload && sleep 2 && curl -sk -H 'Host: app.example.com' https://localhost/
# still 302 -> /outpost.goauthentik.io/start

The revert looked like it had failed. It hadn't. A request issued microseconds after the reload can still be answered by the old worker while it drains. The re-test a few seconds later returned 200 x-powered-by: Express — direct to the app, gate gone.

If a config change appears not to apply, re-test after a beat before you go looking for a second cause. With a request that fast, "the config is wrong" and "the old process is still answering" are indistinguishable.

What I'd do differently

Enumerate every ingress path for a hostname before building anything on top of it. My checklist is now four questions, asked in this order:
- Is the hostname proxied by a CDN/tunnel? If yes, its rules are the real ones — check them first, not last.
- What does the reverse proxy vhost say?
- Does the platform mirror that vhost somewhere else that can regenerate it?
- Is there another hop (VPS, second proxy, vendor tunnel) with its own copy of the routing table?
Verify from outside, and verify which software answered. curl from the host with a Host: header proves your config is self-consistent. Only a request from the public internet, plus x-powered-by/server headers, proves it's in force.
Find out whether the app can consume an upstream identity before you build the gate. If it can't, you're adding a login, not removing one — a decision to make deliberately, not to discover afterwards.
Count before you substitute. Any port or hostname you're replacing globally probably appears in more places than you think. grep -c costs a second and prevents an outage.
Snapshot, diff, edit one line, re-diff. The backup's only job is to answer "how much did I change?" — and a diff answers it exactly.

The result

The gate worked. It was also invisible to every real user, and after working out what it actually delivered — a second login in front of an app that already had one — I turned it off again: one field cleared, one application unlinked, one line restored, verified by a request from outside my network.

The two checks that would have caught the whole thing on day one take about thirty seconds:

curl -sI https://app.example.com/ | grep -i '^x-powered-by'   # who is actually answering?
curl -s  https://app.example.com/ -o /dev/null -w '%{http_code}\n'  # what does a stranger get?

If the answer isn't what your configuration says it should be, you're editing the wrong layer. That's worth knowing before you spend the day on it — and definitely before you tell anyone the gate is live.

Want this for your business?

If you need single sign-on in front of internal tools, a self-hosted app exposed safely to the internet, or someone to audit which ingress path is actually serving your domains, that's the work I do.

WhatsApp: +60 12-797 2969 — tap to chat
Email: me@hoelee.com
Website: hoelee.com

I set up authentik single sign-on, hardened reverse proxies and Cloudflare tunnels, and self-hosted Docker stacks for small businesses in Malaysia — and I write up what I learn while doing it.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Your Disk-Full Alert Is Lying: Stop Alerting on Percentages

Hoelee — Sun, 01 Nov 2026 04:25:47 +0000

Originally published on my blog — read it there with the full context, code and updates.

I built a monitoring stack, wrote a reasonable set of rules, and then spent an
afternoon deleting most of my own thresholds. Every one of them was a percentage, and
every one of them was wrong in the same specific way.

The three alerts that taught me

1. "Filesystem above 90% used." It fired on a 17 TB volume that had ~500 GB free.
Nothing was wrong. Nothing was about to be wrong. On a volume that size, a single
large backup can swing the percentage by several points — the ratio is noise, and the
fact that 500 GB is still available is the actual state of the world.

2. "Memory above 90% used." Fired constantly on a host with 4.8 GB available.
Linux uses free RAM for page cache and gives it back on demand; MemTotal - MemFree
describes your filesystem cache, not your risk. The number that predicts an OOM is
MemAvailable. (Even if one alerts on MemAvailable exceeding 90% capacity, on a 64 GB host it still fires an alert when 4.8 GB of usable headroom remains.)

3. "CPU steal above 25%." Fired on a VPS that had been running its mail stack
perfectly at 41% steal. Steal means the hypervisor is busy — it is a capacity signal,
not a failure signal, and 41% steal on a 7-vCPU box was simply what that host costs.
The rule produced alerts that were always true and never actionable.

The common thread: I was alerting on a ratio, and the ratio doesn't know how big the
thing is.

Alert on the consequence, not the ratio

The test I now apply to every threshold: if this condition persists, what will actually
happen? That answer is almost always expressible in absolute units.

Instead of	Alert on	Because
Filesystem > 90%	Free space < N GB	"Can I still write?" is the question
Memory > 90% used	`MemAvailable` < 512 MB	about to OOM, not "cache is warm"
CPU steal > 25%	Steal > 50%	capacity cost vs. starvation
Load average > N	`load1 / cores > 2`	load is meaningless without core count

In PromQL, before and after:

# BEFORE — percentage of a 17 TB pool; fires with half a terabyte still free
(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 90

# AFTER — capacity left, on the volumes that hold data
min by (instance, mountpoint) (
  node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"}
) < 25e9

# BEFORE — page cache counted as "pressure" (misusing MemFree)
(1 - node_memory_MemFree_bytes / node_memory_MemTotal_bytes) * 100 > 90

# AFTER — the actual OOM precursor based on kernel-reclaimable MemAvailable
node_memory_MemAvailable_bytes < 512 * 1024 * 1024

Two details that make these rules survive contact with a real dashboard:

Keep the comparison in the threshold condition, not inside the expression. Query
min by (mountpoint) (node_filesystem_avail_bytes{...}), then let the alert rule
compare it to 25000000000. If you bake < 25e9 into the PromQL and leave an
inherited > 90 in the rule, the rule still evaluates — but it displays a nonsense
threshold, and the next person to read it has to reconstruct your intent from two
contradictory conditions.

Exclude the mounts that are views of other mounts. A "free space" rule is only as
honest as its selector. On my own setup, one NAS volume appears three times (as
/volume1, as /opt, and again on another host as a CIFS re-mount), and unRaid's shfs
union mount reports avail=0 — a rule without exclusions would either fire forever or
double-count capacity. One explicit selector beats a clever generic one:

node_filesystem_avail_bytes{mountpoint=~"/volume[0-9]+|/mnt/ssd|/mnt/disk[0-9]+"}

Noise is not harmless

The cost of a bad threshold isn't the alert itself — it's the training. Every alert
that fires while nothing is wrong teaches you to skim, then to ignore, and then the one
that matters arrives in a channel you've stopped reading. My rule count went down
while coverage went up: fewer, each tied to a named consequence.

Thresholds are personal, shapes are not

The specific numbers above are mine — 25 GB free before a NAS volume bothers me, 512 MB
available before I care about memory, 50% steal before a VPS is being starved. Yours
will differ with your hardware and your tolerance.

What transfers is the shape:

absolute units over ratios,
the consequence written into the alert's summary,
a selector that states exactly which mounts it speaks for,
and a rule count small enough that you still read every message.

The result

The same stack that was producing three permanent, uninformative warnings now runs 16
rules that stay silent when the infrastructure is healthy — and one of them immediately surfaced a
NAS volume sitting at 21 GB free, which is the kind of thing a percentage rule had been
hiding behind a 99%-full giant that I'd learned to ignore.

Want this for your business?

If your monitoring sends you alerts you've learned to ignore, or you have none and would
rather find out about a full disk from a rule than from a failed backup, I set up
self-hosted monitoring and alerting (Prometheus + Grafana, alerts to Telegram and email)
with thresholds tied to what actually breaks.

WhatsApp: +60 12-797 2969 · Email: me@hoelee.com · hoelee.com

Website design and development is my main line of work; server hardening and
self-hosted infrastructure is the other half of it.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Why Your Grafana Dashboard Shows No Data (When Prometheus Is Fine)

Hoelee — Sat, 31 Oct 2026 04:25:13 +0000

Originally published on my blog — read it there with the full context, code and updates.

You open the dashboard you use every day and it's empty. Not "one panel broke" empty —
every panel, top to bottom, says No data. You check Prometheus and everything is
healthy. This post is the specific, non-obvious cause I hit, and the way my own
verification convinced me the dashboard was fine while it wasn't.

The searchable question

My Grafana panels show No data but the data is in Prometheus — why?

The usual answers (wrong datasource, wrong time range, rate() on a counter that
resets, missing scrape target) did not apply. Here is the checklist that narrowed it
down, in the order that costs the least time:

# 1. is the target actually being scraped?
curl -s http://prometheus:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job, health}'

# 2. does the metric exist right now?  (bypasses Grafana entirely)
curl -s -G http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=node_uname_info{job="unraid-host"}' | jq '.data.result[0].metric'

# 3. is the panel's expression sound?
curl -s -G http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=count(node_cpu_seconds_total{mode="idle",job="unraid-host"})'

All three were green: 7 targets up, 25k series ingested, and the panel's raw
expression returned data when I ran it by hand. So the problem was not the metrics,
the scrape, or the PromQL — it was the variable layer between them.

The actual cause: a variable that filtered on itself

The dashboard had a host picker. Two variables chained: $nodename (which machine)
and $node (its instance label). The definition of the first one was:

# BROKEN — the variable filters on its own value
label_values(node_uname_info{job="unraid-host", nodename=~"$nodename"}, nodename)

On first load $nodename has no value, so the selector becomes
nodename=~"" — which matches nothing. A variable whose query returns no rows has
no options, so it stays empty. $node is then defined against $nodename:

label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance)

…which is also empty, so every panel filtering on $node matches no series. Grafana
isn't lying when it says No data: the panel query genuinely has no data, because the
variable that scopes it resolved to nothing.

The fix is to stop the self-reference and give each variable an explicit saved default:

# FIXED — no self-reference, one hop per variable
# $nodename
label_values(node_uname_info{job="unraid-host"}, nodename)
# $node
label_values(node_uname_info{job="unraid-host", nodename="$nodename"}, instance)

Then save current values for both so the dashboard opens populated instead of
depending on a click. (This is also why the same 41-panel layout can be cloned to
three hosts and just work.)

Why my verification missed it

This is the part worth stealing. I "verified" the dashboard by expanding each panel's
expression by hand, substituting the variable values I knew were correct
($nodename=unRaid, $node=host.docker.internal:9100), and asserting the query
returned points. Thirteen of those checks passed. The dashboard was still blank,
because the bug was upstream of the substitution: Grafana's own resolution of the
variables was empty.

A verification that substitutes the very value under suspicion cannot detect that
value failing to resolve. To catch it, read the values Grafana actually saved and
expand with those:

curl -s -u admin:"$PW" http://grafana:4010/api/dashboards/uid/rYdddlPWk \
  | jq '.dashboard.templating.list[] | {name, current: .current.value}'

The correct check is on the variable query itself — run what Grafana runs:

# what does $nodename's label_values() actually return?
curl -s -G http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=count by (nodename) (node_uname_info{job="unraid-host"})' | jq '.data.result'

Empty there means the dashboard cannot possibly render, no matter how healthy the data is.

The second trap: `$__all` is not `.*`

While automating panel checks, a variable set to All reports its current value as the
sentinel string $__all — not a regex. Expanding panel expressions naively turns
name=~"$__all" into a literal that matches nothing, so a perfectly healthy
container dashboard "fails" every panel. Resolve $__all to the variable's allValue
(usually .*) before expanding, or you'll chase a bug that only exists in your checker.

Not every empty panel is a bug

After the fix, out of the 41 total layout panels, 21 of the 25 core metric panels returned data. The remaining four were empty by design and
worth naming honestly in the panel description:

PSI panels (Pressure, Pressure Stall Information) need /proc/pressure, which the on-premise NAS storage node's kernel doesn't expose. The metric cannot exist there — the same panels on a host that does export node_pressure_* render fine.
Root filesystem panels whose expression carries fstype!="rootfs" can never match on a host whose / is a ramdisk, which is what unRaid's root is.

Distinguishing "broken because of a bug" from "empty because the metric cannot exist on
this host" is the difference between a real fix and a wild goose chase.

What I'd do differently

Never define a template variable that filters on itself. One hop per variable.
Always save explicit current values for variables a dashboard depends on, so a fresh load doesn't rely on a populated dropdown.
Verify with Grafana's saved values, not my idea of them. The masking is invisible in the test result and obvious in the UI.
Describe intentional gaps in the panel description. Future-you will otherwise spend an afternoon "fixing" a panel that was never meant to work there.

The result

All three hosts now run the same instrumented layout — a 41-panel host dashboard (with 21/23/25 of the 25 core metric panels returning data respectively) and a
10-panel container dashboard each — and the two structurally-empty classes documented in place instead of silently
confusing whoever opens them next.

Want this for your business?

If you have Grafana dashboards that nobody trusts — or servers and a NAS with no
monitoring at all — I build self-hosted Prometheus + Grafana stacks (host and container
metrics, sensible alert rules, alerts to Telegram and email) and I'll fix the ones that
have quietly stopped showing data.

WhatsApp: +60 12-797 2969 · Email: me@hoelee.com · hoelee.com

Website design and development is my main line of work; server hardening and
self-hosted infrastructure is the other half of it.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Stop Writing Agent Prompts for Deterministic Work

Hoelee — Fri, 30 Oct 2026 04:25:11 +0000

Originally published on my blog — read it there with the full context, code and updates.

One of my scheduled jobs failed 29 times in a row. Nobody told me, because
it was designed to speak only when it had something to report — and a run
that dies before it can speak looks exactly like a run that found nothing
wrong.

The job was a version check: ask npm for the latest published version of a
CLI I use, compare it with the one installed here, and tell me if there is a
newer one. That is the entire task. It is two strings and an inequality.

I had built it as an agent job, because that was the pattern I had in my
head at the time. Every tick, the model would read the injected script
output and decide whether to write a notification. It worked for a while.
Then my global default model changed, and the job — which records the model
it was created with — stopped running entirely. It printed [drift_skip]
and exited. Twenty-nine times.

The fix was not a better prompt. It was deleting the model from the loop.

The rule

If the stop condition is a comparison — a string, a number, a status code — a model in that loop can only add failure modes.

There is no judgment to make in "is 3.8.50 newer than 3.8.50". The model
was not deciding anything the code could not; it was adding a dependency (a
model route, a provider, a config snapshot, a token bill) and a new way to
fail. So I rewrote the checker as 125 lines of Python with an explicit exit
path for every branch, and flipped the job to script mode: the script's
stdout is delivered as the notification, with no LLM anywhere in the path.

It has run clean ever since. Anthropic's own guidance says the same thing in
one line — use scripts for deterministic work; running a script is cheaper
than reasoning through the steps — but the cost was never the interesting
part for me. The reliability was.

Where the model does earn its place

Of the eleven loops I run, eight now have no model in the path. The three
that do all handle something I genuinely cannot express as code:

Is this email important? A mailbox produces promos, DMARC reports, container alerts and a customer asking for a quote. Fetching mail is deterministic. Ranking it is not.
Is this shop voucher about to overspend? The arithmetic is trivial; deciding what a finding means for money I have already committed is not.
Is this project stalled? Reading two markdown files is a script's job. Noticing that the plan has not moved in three weeks is not.

The test I apply to a new loop: write down what the check would print on
success and on failure. If both answers are a number or a string, it is a
script. If the answer is "it depends what it says", it needs a model — and
then it needs a verifier I trust more than the agent, because I am no longer
able to predict the output.

The part that actually bit me

The failure mode was not the drift. It was the silence.

A loop that speaks only when something is wrong is the only kind of loop I
can tolerate — I do not want a daily status report from eleven jobs. But
"silent because healthy" and "silent because it crashed before it could
speak" are the same observation from my side of the phone. Twenty-nine runs
of evidence were sitting in the job's execution log the whole time; I was
trusting the absence of a notification instead of reading the history.

So every script in the fleet now has a second exit path: exit 0 means
measured, here is what I found, and exit 1 means I could not measure —
and exit 1 is never silent. Missing config, unparseable state, an unexpected
exception: all of them print a line and exit 1. That one change would have
surfaced this bug on the first run instead of the twenty-ninth.

If you have a loop that has been "quiet for a while", go and read its last
ten runs before you conclude it is working. That advice cost me three weeks.

The short version

Deterministic check → script. Judgment → model. Do not mix them for convenience.
Give the silent-on-success convention one exception: "I could not measure" must always be loud.
Read the runner's own execution history, not just your notifications.

Want a version of this for your own site or shop — uptime, certificates,
backups, mail that only pings you when it matters?
WhatsApp +60 12-797 2969 ·
me@hoelee.com ·
hoelee.com

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Running Production Infrastructure Solo: 162 Containers Across Three Hosts

Hoelee — Thu, 29 Oct 2026 04:25:39 +0000

Originally published on my blog — read it there with the full context, code and updates.

Why this matters

A business that sells software needs two things at once: an application that
does the job, and the machinery underneath that keeps it reachable, backed
up and secure. Agencies usually split those two between a developer and a
hosting company, which is how you end up with a developer who cannot see the
server and a hosting company that cannot read the code.

I run both. That means the person who wrote the application is also the
person who gets woken up when TLS renewal stops working, and who can fix the
application rather than blame the platform.

This is my own production footprint: three self-managed Docker hosts running
162 containers across 78 compose stacks, serving live public services —
client websites, a mail platform, CI/CD, workflow automation, analytics and
this blog. No cloud platform holds it together, and no team operates it.
Here is the architecture, and the two failures that changed how I check my
own work.

The shape of it

Host	What it carries	Containers	Compose stacks
Synology DSM	storage, SSO, the bulk of internal services	86	44
unRaid	CI runner, media pipeline, monitoring	37	21
Ubuntu 24.04 VPS	public entry point, mail platform, client sites	39	13

The split is deliberate rather than historical. The VPS carries what must
stay reachable and fast independent of the on-premise uplink, because that is
where public traffic and client mail land. The Synology carries what needs
disk and a stable filesystem, plus the identity provider that every other
service trusts. unRaid runs the CI runner and the heavy, spiky batch work
that would otherwise compete with the services for RAM.

What makes three hosts one system

One ingress pattern. Each host runs a Traefik reverse proxy with automated ACME TLS, so certificates renew themselves and no service owns a certificate file. Behind it there is one convention for routing and redirects, and Cloudflare sits in front of the public names.

One identity provider. authentik handles SSO with OIDC and proxy providers, so a new internal service is a new application in one place instead of a new password in a spreadsheet. Public services stay anonymous; everything else authenticates through it.

One monitoring stack. Prometheus, Grafana, cAdvisor and node-exporter across the hosts, in a single Grafana rather than three dashboards, because the interesting failures are the ones that cross host boundaries.

One delivery path. Self-hosted Gitea Action runners build and deploy. This blog ships through that pipeline — commit, runner picks it up, container rebuilds, Cloudflare serves the new build.

Backups that are tested rather than assumed. Snapshot-based backup on the NAS plus Duplicati and restic jobs, with restore procedures written down, because an untested backup is a feeling, not a backup.

Failure 1: the dashboard was lying to me

When I built the monitoring stack I trusted what Grafana showed me, and
three of the panels were wrong.

cAdvisor reported "containers" that were really cgroup pseudo-entries, which
inflated the container count. My total-storage panel counted one NAS volume
three times, because three near-identical mount points were being scraped as
separate filesystems — the number looked plausible, which is exactly what
made it dangerous. And one virtual machine exported no CPU frequency at all,
so its panel stayed empty and I read that as "no data yet" instead of "this
exporter is misconfigured".

None of those broke a service. All three would have made me report something
false to a client, because a wrong number on a dashboard is
indistinguishable from a right one. I wrote the whole thing up in
One Prometheus for unRaid, a Synology NAS, and a VPS.

Failure 2: one missing config file, three failure modes

Passbolt, my team password manager, started returning 504s and its cron job
appeared to hang. It turned out to be three separate failure modes with one
root cause: a configuration value that existed inside the running container
but never in the compose file.

A null GPG fingerprint stalled mail delivery, and because the mail send sat
inside the scheduled job, the whole job hung and took the UI with it. The
fix worked — until the container was recreated, which wiped it and restored
the original symptom, so the problem appeared to come back on its own. Then
ssl.force behind a reverse proxy produced a redirect loop that I had
introduced myself while chasing the other two.

Three symptoms, three sessions, one lesson: configuration that is not in
version control is not configuration. The full sequence is in
Passbolt UI Kept Hanging — Three Failure Modes From One Missing Config File.

How I keep the numbers honest

Numbers on a CV go stale quietly. Mine are re-derivable: a small script
reads the live Docker endpoints through the Portainer API and counts
containers and compose stacks per host, so "162 containers across 78
compose stacks" is re-measured on demand rather than remembered from the day
it was true. The same script prints the list of services that must never
appear in anything client-facing.

The rule behind it is simple. A number that cannot be re-checked becomes a
claim nobody can defend — in an interview, in a proposal, or on a call when
a client asks how you got it. If I cannot re-measure a figure, it does not
go in the document.

What I would do differently

Every configuration value in the compose file from day one, including the ones that feel like secrets of the machine rather than of the service. The Passbolt relapse existed only because a fix lived in a container.

Alert on the absence of data, not just on bad data. A blank panel and a healthy service look identical on a dashboard. If nothing arrives from an exporter for ten minutes, that is the alert.

Measure availability with something built to measure availability. A metrics pipeline tells you how a system is behaving; it does not count successful requests against total requests. If a percentage is going to appear in front of a client, it needs a probe checking endpoints on a schedule and recording the result — otherwise the honest answer is "monitored", not a number. That distinction matters more than the number.

The result

Three hosts, 162 containers, 78 compose stacks, one operator — with
automated TLS, central SSO, cross-host monitoring, tested backups and
self-hosted CI/CD. It has run client websites, a mail platform, a Telegram
support bot, scheduling automation and this blog at the same time, and it is
the reason I can take a project from "here is the requirement" to "it is
live, monitored and backed up" without handing the second half to somebody
else.

Want this for your business?

If you need someone who can build the application and run the
infrastructure it lives on — Docker and Traefik, automated TLS, SSO,
monitoring, mail, backups, CI/CD — I do exactly this, and I would rather
talk about your setup than send a brochure.

WhatsApp: wa.me/60127972969
Email: me@hoelee.com
More: hoelee.com

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Why Chrome Forgets Its Tabs in a Container — And How I Fixed It

Hoelee — Wed, 28 Oct 2026 04:25:38 +0000

Originally published on my blog — read it there with the full context, code and updates.

I run a real Google Chrome inside a Docker container on my NAS. It is not a scraping instance and not a screenshot service — it is a browser I actually use: signed into the accounts a headless bot should never touch, watchable through a web desktop from any device, and drivable over the Chrome DevTools Protocol when I want to automate part of a job.

It works well, with one exception. Every time the container restarts, the browser comes back with a single blank tab.

Chrome has a setting for exactly this — "Continue where you left off". It was already enabled. It did nothing. What follows is three fixes that failed, the actual root cause, the two landmines I hit on the way, and the ~70-line keeper that finally made restarts survivable.

Why this matters

When a container browser is your workspace, the tabs are the state. Half-finished forms, a marketplace listing in progress, a dashboard filtered to the right date range, three tabs open on a customer's infrastructure — that is the work. A browser that opens on about:blank means the first ten minutes after every restart are spent rebuilding context you already had.

I looked for this answer more than once and found nothing but "set restore_on_startup" advice that does not survive contact with a container. If you have hit the same wall, this is the wall explained.

The setup

A single Docker stack on a Synology NAS, two browsers in it (real Chrome and Brave), each behind my own SSO gate:

Piece	Value
Image	`lscr.io/linuxserver/chrome` — real Google Chrome 154
Desktop	Selkies web desktop, container `3000`/`3001` → host `3030`/`3031`
Automation	CDP published at `http://<nas>:9232/json/version`
Profile	a bind mount, `/config/profile` inside the container
Launch flags	`--remote-debugging-port=9222 --user-data-dir=/config/profile --restore-last-session`

The profile is a bind mount, so the logins, cookies and history genuinely do persist across a container rebuild — that part was never broken. Only the session (which tabs were open) was lost.

Attempt 1: the flag

Chromium ships a switch that sounds like the answer: --restore-last-session. It is documented as "restore the last session after an unexpected exit".

I added it via the image's CLI env var and confirmed it was actually applied to the running process, because these images build the command line from a wrapper script:

docker exec chrome sh -c 'tr "\0" " " < /proc/$(pgrep -f "google-chrome --" | head -1)/cmdline' | tr ' ' '\n' | grep -E 'restore|user-data-dir|remote-debug'
# --remote-debugging-port=9222
# --user-data-dir=/config/profile
# --restore-last-session

Restart. Blank tab. The flag was there; it just had nothing it was willing to restore.

Attempt 2: edit the profile's preferences

Chrome stores "Continue where you left off" in the profile's Preferences JSON as session.restore_on_startup: 1. So I stopped the container and edited it directly, along with the exit_type marker:

import json
p = '/config/profile/Default/Preferences'
d = json.load(open(p))
d.setdefault('session', {})['restore_on_startup'] = 1   # 1 = restore last session
d.setdefault('profile', {})['exit_type'] = 'Normal'
json.dump(d, open(p, 'w'))

This is the classic advice you will find online, and it half-worked: after the next boot, Chrome had kept restore_on_startup: 1 in the file — and still opened a blank tab. What it did not keep was my exit_type: it had already rewritten that to Crashed at startup.

That rewrite turned out to be the first real clue. Chrome treats startup state as a security boundary — the settings a hijacker would most like to change (homepage, search provider, startup pages) are MAC-checked against a separate Secure Preferences file. A value edited on disk does not automatically become a value Chrome obeys.

Attempt 3: a managed policy

The supported way to force browser behaviour is a managed policy, which is authoritative and not subject to that MAC check. On Linux, Chrome reads /etc/opt/chrome/policies/managed/*.json:

{ "RestoreOnStartup": 1 }

Two details before you bother: Chrome ignores a policy file that is writable by the user running the browser, so it needs root:root and mode 644 — and on a Synology share, chmod refuses to run even under sudo for some directories, so verify the mode you already have instead of assuming you changed it.

Mounted into the container, verified present after boot:

docker exec chrome sh -c 'ls -l /etc/opt/chrome/policies/managed/; cat /etc/opt/chrome/policies/managed/hoeleepolicy.json'
# -rw-r--r-- 1 root root 28 hoeleepolicy.json
# { "RestoreOnStartup": 1 }

Restart. Blank tab. Still.

Attempt 4: prove the exit is ungraceful

At this point the question stopped being "which setting do I need" and became "what does this browser actually see when the container stops".

Chrome's restore machinery needs a session it considers interrupted or resumable. It rolls Current Session into Last Session on a clean exit. I asked the CDP endpoint to close the browser gracefully instead of pulling the container out from under it:

# browser-level target from /json/version
ws.send(json.dumps({"id": 1, "method": "Browser.close", "params": {}}))

Chrome exited — and the desktop session relaunched it within seconds. exit_type was back to Crashed immediately.

That is the whole trick, and it explains all three failures: the browser never exits cleanly. When you stop the container, the desktop session is torn down and the browser is killed with it. No clean exit means no Last Session rollover, which means there is nothing for "continue where you left off" — or a RestoreOnStartup policy — to restore. The flag, the preference and the policy were all working correctly and all had nothing to work with.

The data was never missing. The profile's session directory was being written continuously, seconds apart, the whole time:

-rw------- 1 hoelee users 40722 Session_13434833255312492
-rw------- 1 hoelee users 58879 Tabs_13434833757632480
-rw------- 1 hoelee users 79361 Tabs_13434833833646133

The fix: stop configuring, start keeping

If the browser cannot be trusted to remember its own session, the container can remember it instead. Two jobs, both tiny:

On start: wait for the browser, and if it came up with no real page open, re-open the URLs it had last time.
Forever after: every 60 seconds, snapshot the open page URLs.

That is a background script launched from the image's container-init hook. The version I run, trimmed to the useful parts:

tabs_keeper.py — Tab restore & snapshot loop (42 lines)

#!/usr/bin/env python3
"""Restore last session's tabs on start, then snapshot open tabs every 60s."""
import json, os, time, urllib.parse, urllib.request

CDP   = "http://127.0.0.1:9222"          # loopback inside the container
STORE = "/config/tabs-last.txt"          # lives in the persistent profile mount

def open_tabs():
    """http(s) page URLs, or None when the browser isn't answering yet."""
    try:
        with urllib.request.urlopen(CDP + "/json/list", timeout=5) as r:
            tabs = json.load(r)
    except Exception:
        return None
    return [t.get("url", "") for t in tabs
            if t.get("type") == "page" and t.get("url", "").startswith("http")]

saved = [u.strip() for u in open(STORE) if u.strip()] if os.path.exists(STORE) else []

# 1) restore — wait up to 3 minutes for the browser to come up
if saved:
    for _ in range(60):
        if open_tabs() is not None:
            break
        time.sleep(3)
    if open_tabs() == []:                      # nothing but a blank new tab
        for url in saved[:20]:
            req = urllib.request.Request(
                CDP + "/json/new?" + urllib.parse.quote(url, safe=""), method="PUT")
            urllib.request.urlopen(req, timeout=10).read()
            time.sleep(0.7)

# 2) snapshot — write even an empty list, so "I closed everything" is respected
while True:
    time.sleep(60)
    tabs = open_tabs()
    if tabs is None:
        continue
    tmp = STORE + ".tmp"
    open(tmp, "w").write("\n".join(tabs) + ("\n" if tabs else ""))
    os.replace(tmp, STORE)                     # atomic; a kill can't truncate it

Two design choices worth copying:

Restore only when the browser came up empty. If a future Chrome version does restore its own tabs, the keeper stays out of the way instead of duplicating every tab.
The snapshot is atomic and may be empty. Writing through a temp file plus os.replace means a container killed mid-write cannot leave a truncated list, and an empty snapshot is meaningful data: it means you closed everything on purpose, so the next start should stay empty.

The container-init hook that runs it:

#!/bin/sh
# clear stale singleton state (see below)
rm -f /config/profile/Singleton* 2>/dev/null
# long-running, so background it or you block container init
/usr/bin/env python3 /custom-cont-init.d/tabs_keeper.py >> /config/tabs-keeper.log 2>&1 &
exit 0

Landmine 1: a stale `Singleton*` stops Chrome from starting at all

While testing restarts I hit a worse failure than a blank tab: the container came up healthy with no browser in it. No google-chrome process, CDP dead, the relay's healthcheck reporting unhealthy, nothing in the logs beyond the desktop starting.

The cause is the profile's SingletonLock/SingletonSocket/SingletonCookie symlinks. They encode "this profile is owned by PID n on host h" — and inside a container, PIDs are reused across restarts, so the next boot can find its own PID already claiming the profile and exit immediately. Deleting them at container start, before the browser launches, is the fix. It is also why that rm -f sits in the hook above.

Landmine 2: `custom-cont-init.d` only runs executable files — and runs them with bash

The LinuxServer images execute every file in /custom-cont-init.d, but only if it is executable, and each one through /bin/bash. Which means a Python file you drop in there must not be executable, or it gets fed to bash and exits 2:

[custom-init] tabs_keeper.py: executing...
[custom-init] tabs_keeper.py: exited 2

The split that works: a .sh hook at 755 that backgrounds the long-running work, and the .py next to it at 644. And anything that does not return immediately has to be backgrounded, or container init waits on it forever.

The plumbing trap that came with it

The keeper talks to Chrome over CDP on loopback, which is why it needs no ports. If you also want to drive the browser from another machine, there is one more container in the picture, because Chrome ≥ 136 pins its devtools endpoint to 127.0.0.1:9222 and ignores --remote-debugging-address. The standard workaround is a socat sidecar sharing the browser's network namespace:

chrome-relay:
  image: alpine/socat
  network_mode: service:chrome            # must share the browser's netns
  command: socat TCP-LISTEN:9223,fork,reuseaddr TCP:127.0.0.1:9222
  healthcheck:
    test: ["CMD-SHELL", "nc -z 127.0.0.1 9222 || exit 1"]

The port mapping does not go on the relay (it has no namespace of its own) — it goes on the browser service: 9232:9223.

That sidecar has one behaviour you have to internalise: restart the browser and the relay dies with it. The namespace it was living in is rebuilt underneath it, socat keeps running and its sockets stop working, so CDP answers with connection reset while the container still looks fine. Fix the symptoms in this order:

docker restart chrome && docker restart chrome-relay   # order matters

The healthcheck above is what turns a silent failure into a visible unhealthy flag. If you want to drive a container browser yourself, I wrote the client side up separately in Scraping a Bot-Walled Marketplace With a Warm Browser Session.

What I'd do differently

Determine whether the process can exit cleanly before trusting any "restore on startup" mechanism. I spent three attempts configuring a feature whose precondition — an exit the browser notices — was never met. One Browser.close test would have collapsed the whole search.
Don't hand-edit browser preferences. They are MAC-checked against Secure Preferences; a plausible-looking value on disk is not a value Chrome obeys. If you need a switch to be authoritative, use a managed policy — and verify the policy is loaded instead of assuming the file is enough.
Check who is allowed to write the policy. A root-owned 644 file in /etc/opt/chrome/policies/managed is the shape; a user-writable one is ignored silently.
Treat "the browser is running" and "the container is running" as different questions. The stale-singleton failure looked like a healthy container from the outside.
Snapshot state instead of asking software to remember it. Anything with a hard kill in its shutdown path — containers, kiosks, desktop sessions — is a poor place to rely on clean-exit bookkeeping. A 60-second atomic snapshot is boring, and it works.
Watch the empty-snapshot case. Because step 2 writes even an empty list (that is how "I closed everything" gets respected), a browser that dies mid-session and gets relaunched blank by the desktop supervisor will have its empty tab state snapshotted 60 seconds later, erasing the last good file. The cheap guard is to record the browser's webSocketDebuggerUrl from /json/version and only overwrite the snapshot when it is the same instance you started.

The result

Tabs restored 2/2 on both browsers across a real container restart (restored 2/2 tabs in the keeper log), verified over CDP afterwards
Logins, cookies and localStorage confirmed surviving the same restarts — the persistent profile was never the problem
Three stacks (two browsers plus a throwaway experiment) collapsed into one, with the relay's healthcheck making the one restart-order trap visible instead of silent

Want this for your business?

If you want a browser that stays logged in — running on your own hardware, reachable from anywhere, behind SSO, with tab state that survives restarts and a CDP endpoint your scripts can drive — that is the setup I run and maintain.

WhatsApp: +60 12-797 2969 · Email: me@hoelee.com · hoelee.com

Website design and development is my main line of work; self-hosted infrastructure and automation is the other half of it.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

💬 Let's Compare Notes

How do you manage persistent browser user-data-dir volumes and state across container restarts without profile lock corruption? Drop your setup or thoughts in the comments below — I read and reply to every discussion.

Related notes:

Calibrating Intel Quick Sync HEVC: A Cliff at 28, and a Preset That Does Nothing

Hoelee — Tue, 27 Oct 2026 04:26:26 +0000

Originally published on my blog — read it there with the full context, code and updates.

The question I could not answer: which number do I ship?

My transcoding queue decides, per file, which encoder does the work: copy
when the source is already efficient, NVENC on the GPU, x265 on the CPU —
and now Intel Quick Sync. Each tier has two knobs that decide quality, and
every tier except QSV had numbers I had measured on my own library.

QSV had none. And the code refused to guess: a shipping default of
qsv_quality = 0 / qsv_preset = "" means nobody has calibrated this,
and the engine treats that as a hard exclusion — QSV can exist in the code
and never be selected on any machine, including the one that has the
hardware.

So the task was not "turn on QSV". It was "produce measured numbers for two
knobs, so the encoder becomes selectable honestly".

Problem one: the hardware is not in the machine where the app runs

The queue runs on Windows 11 in a VM. That VM has an RTX 3060 passed through
and no Intel graphics of any kind — which is exactly why QSV had never been
measured: the obvious move is to run one test locally and read the numbers
off.

The iGPU is on the host instead (unRaid, i9-13900K), idle and unclaimed:

00:02.0 VGA compatible controller: Intel Raptor Lake-S GT1 [UHD Graphics 770]
  driver: i915            refcount: 0
/dev/dri/card0  /dev/dri/renderD128        (present, world-accessible)
no running container declares a dri device
virsh dumpxml Win11: hostdevs = 01:00.0, 01:00.1 (the 3060 + its HDMI audio)

Two conclusions, both useful. The VM's Error creating a MFX session is
expected and not a bug to chase — the VM can never see that iGPU. And
the only way to measure QSV is on the Linux side, in a container, on the
host.

Measure with the engine's own gate, not with ffmpeg

This is the part that makes the numbers worth anything. I did not write
ffmpeg commands and compute SSIM myself — the calibration runs the engine
that will do the work, with its own verification and its own accept/reject
gate:

python3 library_run.py --work-dir /work --tier qsv --force-tier qsv \
  --qsv-quality Q --qsv-preset P --path <real source> --keep-local --overwrite

--keep-local means encode, verify, run the gate, and then do not land
the file — the verdict is what I am collecting, not a new library. Sources
came in read-only over an existing SMB mount, and I hashed each source
before and after: unchanged.

The image is linuxserver/ffmpeg (ffmpeg 9.0.2, iHD 26.3.5, oneVPL 2.17)
plus python3 — plus numpy, which is not optional. Without it the gate
prints flat-frame scan skipped: No module named 'numpy' and quietly
changes how near-flat frames are judged, which changes verdicts.

-global_quality maps to RateControlMethod: ICQ on hevc_qsv, which is
the knob being calibrated.

The ladder: on a thin source, low ICQ makes the file bigger

First run, on a real 1080p25 source at 951 kbps (0.0184 bits per pixel — a
"thin" file, the kind that is already squeezed):

ICQ	engine verdict	mean SSIM	video = source
18	FAIL size — 176.8%	—	—
20	FAIL size — 141.8%	—	—
22	FAIL size — 107.6%	—	—
24	ok	0.9967	81.1%
26	ok	0.9959	61.0%
28	ok	0.9951	46.4%
30	FAIL quality	0.9942	—
32	FAIL quality	0.9942	—

Three things fall out of that table.

The counterintuitive one: low ICQ does not mean safe. ICQ 18–22 produce
files larger than the source, so the engine's size ceiling (100% of the
source's video bitrate) rejects them. QSV is not alone in this — NVENC at a
low cq does the same thing on thin inputs — but the boundary sits higher
here: the usable range starts at 24, not 18.

Second, there is a cliff between 28 and 30. 30 and 32 fail on quality,
and they fail on the first attempt. That phrase matters, because the
engine retries a failed quality attempt at global_quality - 2 — lower
ICQ, higher quality — and logs the retry. A line reading "ok at quality+2"
is a first-attempt failure, not a pass. I read that line backwards in my
own notes the first time; the rule is to read the first attempt, not the
recovery.

Third, the floor is where the choice gets made: with a 0.995 mean-SSIM
floor, q28 passes with 0.0001 of margin and q26 with 0.0010.

Full files confirm the ladder — and one shortcut survives

Excerpts (180-second lossless slices) are how you rank ten candidates
without ten full encodes. They are only legitimate if they agree with
full-file runs, so I ran the real files end to end:

source	ICQ	preset	video = source	file	SSIM min / mean	encode
254 MB, 2023 s, 872 kbps	26	medium	61.0%	168,435,266 B (66%)	0.9938 / 0.9960	360 s ≈ 5.6× realtime
the same file	28	medium	46.3%	54%	0.9927 / 0.9952	330 s
315 MB, 3280 s, 546 kbps	26	medium	50.4%	60%	0.9965 / 0.9977	600 s ≈ 5.5× realtime

The 180-second excerpt of the first file measured mean 0.9959; the full file
measured 0.9960. That is close enough to trust the shortcut for quality
ranking — and I now have the evidence to say so rather than hope so.

The preset that does nothing

The excerpt runs said preset only changes speed: veryfast 30 s / 61.6%, fast
30 s / 61.0%, medium 60 s / 61.0%, slow and veryslow 60 s. Clean story: pick
veryfast, save 30 seconds per file.

Then the full files disagreed with it. The same source at ICQ 26, one run
with fast and one with medium:

fast:    168,435,266 B   enc 360 s   mean SSIM 0.9960
medium:  168,435,266 B   enc 360 s   mean SSIM 0.9960

Byte-for-byte identical, same encode time, same quality. The 30-vs-60-second
difference in the excerpt layer was per-run fixed overhead, not encoding
speed — and it is the kind of number that would have gone into a config file
and a blog post if I had stopped one step early.

So: excerpts rank quality, never speed. Speed has to be measured on full
files, where the fixed overhead disappears.

What I shipped

qsv_quality = 26, qsv_preset = medium — 61.0% of the source's video
bitrate at mean SSIM 0.9960 (min 0.9938) at roughly 5.6× realtime, and 50.4%
at mean 0.9977 on the thinnest file in the library. q28's margin over the
floor is 0.0002, and the cost of a quality failure is two full encodes plus
a park, so the extra 15 percentage points of savings are not worth the tail
risk.

Two supporting measurements worth recording, because they answer "is it QSV
or is it the file?":

Cross-encoder control. One 720p source with non-uniform cadence scored QSV q24 mean 0.9154, QSV q28 mean 0.9146, and x265 crf26 mean 0.9164 — and the engine's alignment search reported no alignment explains the mismatch. The rejection belongs to the source, not to Quick Sync.
Concurrency. The engine refuses a second encoder on a shared work root: another transcode is already running (pid 7, /work/run-qsv.lock, 3 s old). With separate work roots, two jobs run — and take 201 s where one takes 107 s. Twice the wall clock, zero throughput gain. One QSV job at a time.

Four traps, each of which cost a cycle

Never mount a shared library at /lib. I mounted the SMB share into the container at /lib — which is where the container's own dynamic loader lives. All eight jobs died in under a second with exec /bin/bash: no such file or directory, which reads like a broken image, not a bad mount. /library works.
Synthetic sources lie about ICQ. hevc_qsv's rate control is rejected by the driver for rgb24 input (Selected ratecontrol mode is unsupported) — so a testsrc/mandelbrot self-test concludes the opposite of the truth. Use a real file, or add format=nv12.
Already-HEVC sources cannot be calibrated at all. The gate answers SKIP already HEVC hvc1 and --overwrite does not lift it. The fat 1080p H.264 originals in this library had all been converted long ago; the remaining fat files are HEVC — so that tier has nothing left to measure on. That is a missing input, not a skipped task.
Read the first attempt, not the retry (see the cliff above).

What I would do differently

Measure speed on the artifact you will actually run, not on a slice of it. My one wrong number in this whole exercise came from trusting an excerpt's wall clock.
Make the machine prove it has the hardware before you plan the test. Six minutes of lspci and virsh dumpxml retired the "why does QSV fail in the VM" question that had been open for weeks.
Keep the gate in the loop. Because the engine's own verification judged every run, the tables above are decisions, not opinions — the same gate that will reject a bad file in production rejected my ICQ 30.
Write the honest scope line into the notes. These numbers are Linux iHD 26.3.5 + oneVPL 2.17 on a UHD 770; the production target is the Windows MSDK path on the same silicon, and I cannot claim they agree. What this establishes is that the command runs on real hardware, where the quality↔size curve sits, and how the lock behaves — leaving a re-measurement on Windows+Intel as one table, instead of another afternoon.

The result

QSV went from "present in the code and never selectable" to a calibrated
tier with published numbers, and the safety rule changed with it: the test
that used to pin shipping defaults must mean uncalibrated now pins
shipping defaults must be measured values, while keeping every assertion
that an empty value is a hard exclusion. The doctor output on a machine with
no Intel GPU moved from an informational "waiting for hardware" line to a
warning — which is its own small lesson about what a calibrated encoder
failing self-check should look like, and why hardware the machine does not
have belongs in a summary row rather than a warning.

Need Resilient Hardware Video Pipelines & Automated Quality Gates?

If you manage GPU/iGPU transcoding fleets (NVENC, Quick Sync, VAAPI) or need strict frame-level quality assertions (SSIM/VMAF) to keep defective encodes out of production libraries:

WhatsApp: +60 12-797 2969
Email: me@hoelee.com
Website: hoelee.com

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

💬 Let's Compare Notes

For video pipeline engineers: do you rely on hardware QSV/NVENC for bulk storage transcode or bite the CPU bullet for x265 crf? Drop your setup or thoughts in the comments below — I read and reply to every discussion.

Related notes:

No API for Browser Translation: Adding an English Mode to a Chinese-Only Web App

Hoelee — Mon, 26 Oct 2026 04:26:02 +0000

Originally published on my blog — read it there with the full context, code and updates.

Can I trigger the browser's translate feature from JavaScript? No. That answer cost me an afternoon to accept — but the more interesting discovery came first: my app was already sabotaging the browser's translation, and it did it with a single attribute.

What I was working on

I run a private, invite-only reading platform at book.hoelee.com: a self-hosted reader (Kotlin/Spring backend, Vue single-page app) holding my library, with accounts for invited readers. The interface is Chinese-only. Not "Chinese-first" — only. I grepped every served bundle before believing it: no i18n library, no locale files, no language switcher. Just a Simplified/Traditional character converter and a list of English TTS voice names.

Then I started handing out invite cards to English-speaking readers. They could register, and then they met 书架, 浏览书仓, 确定, 设置 — every control in a language they don't read. That isn't a polish problem. For half the people I invited, the product was unusable.

One constraint shaped everything that follows: I don't modify the app. Its upstream repository is archived and the UI is a minified Vue bundle, so anything I patch inside the image dies the next time I swap the image. Whatever I build has to live one layer out — in the nginx gateway I already run in front of the app.

Finding 1: the app declares the wrong language, and that disables translation

Before writing a single line of code, I looked at what the browser sees:

$ curl -s https://book.hoelee.com/index.html | grep -o '<html[^>]*>'
<html lang="en">

The shell says English. The interface is 100% Chinese.

That matters more than it looks. Chrome, Edge and Safari decide whether to offer translation largely from the page's declared language. A page that declares English while rendering Chinese text doesn't get the "Translate this page?" prompt — the single most useful accessibility feature for my new readers was silently switched off, and right-click → Translate to English is a ritual most people never learn.

The fix is one line, applied to the response on the way out instead of inside the app:

location = /index.html {
    # sub_filter cannot rewrite a gzipped body -> ask upstream for plain HTML
    proxy_set_header Accept-Encoding "";
    sub_filter_once on;
    # the app declares lang="en" while its UI is Chinese -> browsers then never
    # offer Chinese -> English translation. Declare the truth.
    sub_filter '<html lang="en">' '<html lang="zh-CN">';
    proxy_pass http://hectorqin-reader:8080;
}

Verify the same way you found it: curl -s https://book.hoelee.com/index.html | grep -o '<html lang="[^"]*"' → <html lang="zh-CN". The declared language is what the translate prompt keys off, so with the page finally declaring Chinese the offer can fire correctly. (The exact prompt still depends on each browser and its language settings — what I verified on the wire is the attribute itself.) Because the rewrite happens at the proxy, it survives every app update.

Finding 2: there is no browser-translate API

With the page honest again, the obvious next question was whether I could offer translation myself — a button that does what the browser's menu item does. There is no such API. Browser translation is a UI-level feature; a page cannot invoke it, and no web-exposed surface exists to script it.

So I worked through the alternatives and rejected them one by one:

Approach	Why I dropped it
Rewrite the app's JS bundles to swap strings	Couples me to minified internals; the next image breaks it. Also violates the no-modify rule in spirit.
Proxy the site through `translate.goog`	It's a different origin. The app keeps its auth token in `localStorage`, which is per-origin — readers would land logged out, and the service worker/PWA breaks. Every API call would also round-trip through Google.
Tell readers to right-click → Translate	Works for book text, but it's a per-visit, per-browser ritual that does nothing for someone who doesn't know it exists. Not good enough for a product where I control the whole stack.
Fork the app and add i18n properly	Weeks of work on an archived codebase, for an interface I'd have to re-translate on every upstream change.

What's left is the unglamorous one: translate the interface myself, in the browser, with a dictionary — and wrap it in a button that only exists inside the app.

The fix: a dictionary and a button, injected at the gateway

The gateway already serves my front page, so it can serve one more static file and inject a script tag into the app shell only:

location = /gate-en.js {
    root /usr/share/nginx/html;
    add_header Cache-Control "public, max-age=300" always;
}

location = /index.html {
    # ... the lang rewrite above ...
    sub_filter '</head>' '<script src="/gate-en.js" defer></script></head>';
    proxy_pass http://hectorqin-reader:8080;
}

That's the whole deployment: a 40 KB script, loaded only on /index.html, so the button exists inside the app and nowhere else. The script does three things.

1. Harvest the vocabulary from the app's own bundles

The UI strings are already in the minified JavaScript as quoted literals. Extract every one that contains a CJK character and count them:

import re, collections
CJK = re.compile(r'[\u4e00-\u9fff]')
count = collections.Counter()
for bundle in bundles:
    for m in re.finditer(r'"([^"\\\n]{2,40})"', bundle):
        if CJK.search(m.group(1)):
            count[m.group(1)] += 1
print(len(count))   # 840 unique strings, most frequent first

840 unique strings came out: menus (书架, 书源管理), confirmations (确认要删除所选择的书籍吗?), settings labels (段落行距), error messages (本地书籍源文件不存在). The dictionary ended up at 871 entries — the harvest plus everything I found by running the app in English mode and watching what stayed Chinese.

2. Match like the DOM actually is, not like you wish it were

Naive substring replacement mangles Chinese compounds: 书架 sits inside 加入书架, and single characters are worse (页 inside 页面). Three rules fixed it:

Whole-label match first, on whitespace-stripped text. The app stores labels like " 书架 " with padding, so a dictionary key of 设置 has to match " 设置 " in the DOM.
Then longest-first substring replacement, restricted to keys of two or more characters.
One-character keys are whole-label only (章, 页, 条), so they can never corrupt a longer word.

function tr(s) {
  if (!s || !CJK.test(s)) return s;
  var hit = EXACT[s.replace(/[\s\u3000]/g, '')];   // whole label, whitespace-normalised
  if (hit) return keepWS(s, hit);
  var t = s;
  for (var i = 0; i < KEYS.length; i++) {          // KEYS: longest-first, length >= 2
    if (t.indexOf(KEYS[i]) >= 0) t = t.split(KEYS[i]).join(DICT[KEYS[i]]);
  }
  // tidy count phrases: 共58个可用书源 -> "58 usable sources"
  return t.replace(/共\s*(\d+)/g, '$1').replace(/(\d+)\s*个/g, '$1').replace(/(\d)([A-Za-z])/g, '$1 $2');
}

3. Keep up with the framework

It's a Vue app, so the DOM is rewritten constantly — a translation pass that runs once is a translation pass that's half wrong a second later. A MutationObserver with a 150 ms debounce re-runs the walk after any change.

The pass is idempotent: once a label is English it contains no CJK, so nothing is rewritten and there is no write loop. Text nodes plus placeholder, title, aria-label, alt and input value attributes all get translated, which is why tooltips and the search box come along too.

The UI is a small pill in the bottom-right corner: EN when off, 中文 when on, remembered in localStorage, with a "Hide this button" link for readers who prefer their own translator.

What it deliberately does not translate

Book text. The reader renders EPUB content inside an iframe; my walker skips iframes by design, and I would never machine-swap a novel's prose anyway.

Also left alone: book titles and authors, book-source names, and the group names I created myself (梯子, 精品, 正版), plus the app's poem tagline. Those are data, not interface. Translating the chrome is a courtesy; silently rewriting a user's own content is a different and much worse idea.

For book text the browser's own translation is still the right tool — and it now actually gets offered, thanks to Finding 1. The button's popup says exactly that.

Measure it, don't eyeball it

"Screenshots look mostly English" is not a result. I counted CJK-containing text nodes in the live DOM before and after toggling the button:

let cjk = 0, total = 0;                          // walker skips the injected UI + script/style
while ((n = walker.nextNode())) {
  const t = n.nodeValue.trim();
  if (!t) continue;
  total++;
  if (/[\u4e00-\u9fff]/.test(t)) cjk++;
}

Screen	Before	After	Translated
Login / register dialog	80	12	85%
Bookshelf + settings	259	60	77%
Reading page chrome	214	31	88%

Everything still counted as Chinese on those screens is data — titles, authors, source names, my own group names, the tagline — which is exactly what should stay Chinese.

What I'd do differently

Don't rewrite an 871-entry dictionary by hand. I restructured mine once and silently dropped 25 keys. The symptom looked like a logic bug: 设置 stayed Chinese while every sibling label translated correctly. The matcher was fine — the entry was gone. Diff the key sets programmatically before you trust a rewrite:

python -c "print(sorted(set(old_keys) - set(new_keys)))"

Keep automated UI checks short. My headless runner hard-times-out at 60 seconds per call. A six-book loop came back HTTP 408 with an empty body, and my JSON parser reported a parse error rather than a timeout — I debugged the wrong thing for a while.

Learn the SPA's real routes before debugging it. #/reader/1 renders a blank shell and makes you think the app is broken; the actual route is #/reader?bookUrl=<urlencoded>.

Mind your own bounce logic. My gateway sends any reload back to the front page unless a sessionStorage pass flag is set. Switching the interface back to Chinese reloads the page — so the button sets that same flag first, or the reader gets thrown out of the app for changing a language setting.

Result

An English-only reader now opens the platform, taps one button, and reads the interface in English. The app itself is untouched: the whole feature is a 40 KB script served by a sidecar nginx container, so the next image update cannot break it, and the browser's own translation has a page that declares its real language to work with.

The lesson I'm keeping: when the platform won't give you an API, look one layer out. A reverse proxy is a legitimate place to add behaviour you can't add in the app — and it's usually the only place that survives upgrades.

I build and self-host platforms like this one — web apps, private libraries, internal tools for small businesses in Malaysia. If you need something similar: WhatsApp +60 12-797 2969, me@hoelee.com, or hoelee.com.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

💬 Let's Compare Notes

When retrofitting i18n into monolithic legacy SPAs without backend API support, what architectural trade-offs do you make? Drop your setup or thoughts in the comments below — I read and reply to every discussion.

Related notes:

My Backup Was Crawling Because I Was Using Someone Else's API Quota

Hoelee — Sun, 25 Oct 2026 04:25:39 +0000

Originally published on my blog — read it there with the full context, code and updates.

The subscription ended, and I had to move a terabyte

The call came in the least convenient way: the OneDrive account holding my
nightly backups had no subscription any more, and I had a 1.5-year Google
subscription sitting unused. Move the backup, keep the history, don't lose a
night.

The first thing I hit was not a technical problem: I could not remember
how the backup was wired. rclone? Duplicati? Something Synology shipped?
On a three-host estate with 78 compose stacks — the one in
Running Production Infrastructure Solo
— "which tool writes my backups" is a question I should have been able to
answer from memory and could not. So step one was reading the machine.

What was actually there

Two rclone configs, which is the kind of detail that bites: the GUI config
that Backrest (the restic front end) uses, plus a separate stale CLI
copy that a cron job or a human shell had been using at some point. Change
one and you have changed half the system.

The migration itself is boring when done in the right order, and I did it
read-first:

rclone.conf                     append [gdrive-*] + [gdrive-*-crypt]
rclone/rclone.conf (CLI copy)   same two remotes
Portainer stack 177             env → the new crypt remote; config :ro → :rw
restic init                     new repo on Google Drive
Backrest SetConfig              dsm-repo URI → the new location

Before touching the plan URIs I checked one property in the source, because
everything depended on it: a restic repository's identity is
guid = sha256(id) — a stable value derived from the repo id, not from
its URL. So repointing a plan at a new backend cannot silently fork the
history. All six plans came out byte-identical to the originals, and the
repo id/guid were untouched.

Then the seed stopped moving, and the log named a stranger

The new repo seeded to 2.88 GB of roughly 1075 GB — 0.27% — and then slowed
to a crawl. The live log had the answer:

'Read requests per minute' ... reason: RATE_LIMIT_EXCEEDED
consumer: projects/202264815644        <- not my project
quota_limit_value: 840000

202264815644 is rclone's own shared OAuth project. My terabyte seed
was drawing on a per-minute read quota pooled with every rclone user on
earth, which is why it degraded from ~750 KB/s to nothing while my NAS, my
disk and my link were all idle. By the time I measured, the seed was fully
stalled — 0 bytes in 90 seconds, 159 errors, still rateLimitExceeded —
and the six nightly plans had all failed with the same wall:

Fatal: unable to save snapshot: unexpected HTTP response (500)
googleapi: Error 403: Quota exceeded for quota metric 'Queries'
  and limit 'Requests per minute' ...

"Slow" and "blocked" look similar in a progress bar and are completely
different problems. Ninety seconds of sampled zeros is what made the call.

You need your own OAuth client — and the console is unavoidable

There is exactly one real fix: stop using the shared client and bring your
own. That means a Google Cloud project, an OAuth consent screen, and a
credential. Two settings cost real time:

Application type = Desktop app. rclone's redirect is http://127.0.0.1:53682, and that client type permits it without extra configuration.
Publish the app. Leave it in "Testing" and Google expires your refresh token every 7 days — which turns "working" into a silent weekly outage.

I checked whether I could shortcut the console with the CLI that was already
authenticated here. I could not, and the reason is worth knowing: gcloud
was signed in only as a service account in an unrelated project, and
service accounts cannot create consent screens or OAuth client IDs. The
credential has to belong to a project a human owns. It was a dead end, and I
verified that instead of asserting it.

Three traps in the authorization flow

Publishing an unverified app can hard-block a restricted scope. .../auth/drive is restricted, and an app that is published without completing Google's verification review is sometimes blocked outright with no bypass. In my case the bypass existed — the "Google hasn't verified this app" screen, then Advanced → Go to rclone-backup (unsafe) — but that is a coin flip on scope and review state, not a guarantee, and "unsafe" is only a label meaning the app never went through a paid audit.
The callback accepts exactly one request. A reload, or a second tab, hits 127.0.0.1:53682 with no code and kills the attempt. My first try died exactly that way (no code returned), and it looks like a credentials problem. One tab. Don't reload it.
Finish the consent screen on the machine that runs rclone. The redirect goes to localhost, so a phone or another PC cannot complete it — the browser has to be the one on the same host as the process waiting on that port.

Then: read the token, write client_id + client_secret + token into
both configs (the GUI one and the stale CLI copy), and restore the two
flags that matter for a long unattended upload: use_trash = false and
stop_on_upload_limit = true.

The old side: verify by reading, not by asking

Purging the old remote reported an error — itemNotFound — and the space
came back anyway:

Used:   1.012 TiB  ->  3.800 GiB
Free:   ~0         ->  1.007 TiB
DSM folder: present -> gone      Trashed: 0 B

The error was rclone complaining after the work finished. Also worth
recording: the tool's documentation says personal accounts cannot
permanentDelete, which would have meant emptying the recycle bin by hand —
in practice the trash was already empty and roughly 1 TB was reclaimed with
no browser step. Documentation describes the API; the API is not always what
you get.

One more measured thing, because it changed my next decision: pausing the
file-sync container that shared that machine halved the load average — 16.70
→ 8.89 — which is the difference between "the network is slow" and "the box
is busy".

What I would do differently

Own the OAuth client from day one. A shared client id is a queue you are standing in, with no status page and no one to call.
Answer "what writes my backups" from a document, not from memory. Two rclone configs is a normal amount of drift for a system nobody has drawn in a year.
Read the destination, not the exit code. The purge "failed" and freed a terabyte; a successful-looking rename can land nothing. Same lesson as It Reported Success. Nothing Had Changed. — this estate keeps teaching it.
Never start a multi-day upload while the last few percent of your attention is elsewhere. Checking for RATE_LIMIT_EXCEEDED at 0.27% costs minutes; discovering it after two days costs the window.

The result, honestly stated

The plumbing works: the plans live on Google Drive, the repo identity is
unchanged, the seed is now spending my quota instead of a shared one,
and the old side gave back about a terabyte. What it is not, yet, is a
backup: at the time I wrote this, the first seed had uploaded 2.88 GB of
~1075 GB and the new repo had zero snapshots — nothing is restorable
until the first snapshot lands. That is the number I watch, because "the
transfer is running" and "you can restore" are different claims.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

💬 Let's Compare Notes

How do you monitor and isolate third-party rate limits across background backup jobs to prevent silent starvation? Drop your setup or thoughts in the comments below — I read and reply to every discussion.

Related notes:

Docker Reported Exit 0. The Kernel Had Already OOM-Killed It.

Hoelee — Sat, 24 Oct 2026 04:25:54 +0000

Originally published on my blog — read it there with the full context, code and updates.

The notification I expected to be a config mistake

My NAS sent the notification every Docker user learns to skim: container
stopped unexpectedly. This one is the edge proxy in front of my
self-hosted wiki, and it had been up for two weeks. Nobody had deployed
anything that morning and nothing in the compose file had been touched, so
the two available stories were "it crashed" and "something restarted it".

Both were wrong, and the fact that I could not tell from Docker's own report
is the part worth writing down.

Docker's report says nothing happened

State:
  Status:      running
  Running:     true
  ExitCode:    0            <- a clean exit
  OOMKilled:   false        <- not killed by the kernel
  FinishedAt:  2026-10-15T19:39:33.63057879Z
  StartedAt:   2026-10-15T19:39:38.769326672Z   <- restarted 5 s later
RestartCount: 6
RestartPolicy: always

Read that as written and the conclusion is: the process exited cleanly, six
times, and the restart policy gave it a new life each time. A long-running
proxy does not exit with status 0 on its own — and a clean exit that repeats
six times is a contradiction, not an answer. ExitCode and OOMKilled are
exactly the two fields I would have trusted here. On this host, both were
wrong.

The kernel keeps its own record

On Linux, a cgroup memory kill is logged by the kernel even when the
container runtime's bookkeeping does not carry it forward:

[1737797.044856] traefik invoked oom-killer: gfp_mask=0x2400040, order=0, oom_score_adj=0
[1737797.123736] Memory cgroup out of memory: Kill process 615 (traefik) score 970 or sacrifice child
[1737797.132800] Killed process 615 (traefik) total-vm:2525800kB, anon-rss:478480kB, file-rss:0kB

The uptime stamp 1737797 corresponds to 19:39:33 UTC — to the second, the
container's FinishedAt. That is the whole diagnosis, and it took one
dmesg to get it. The container did not exit; it was killed inside its own
memory limit and restarted 5 seconds later.

dmesg holds six such events, and RestartCount was 6. Every other
container in that stack has a restart count of zero and a start time from
weeks ago, so the six kills pin to this one container:

dmesg uptime	local time	anon-rss at kill
1406539	Oct 12, ~07:37	491 MB
1407759	Oct 12, ~07:57	494 MB
1432574	Oct 12, ~14:49	486 MB
1734793	Oct 16, ~02:45	487 MB
1735495	Oct 16, ~02:58	485 MB
1737797	Oct 16, 03:39 ← the notification	478 MB

Every kill happens at 478–494 MB against a limit of 512 MiB (536,870,912
bytes). That is a ceiling being touched, not a runaway.

Two more readings made the mechanism clear. The cgroup's recorded peak — the
counter that survives the restart — was already back at exactly
536,870,912 18 minutes after the restart, i.e. the container touched
its ceiling again in the same cycle without being killed. And the usage
curve oscillated (176 → 308 → 361 MB) instead of climbing monotonically, so
this is allocation churn under load hitting a small cap, not a provable slow
leak. The prime suspect is a geoblock plugin doing an external HTTPS lookup
for every client IP it has not seen before.

A seventh kill landed at 05:33 that morning, before the fix went in.

On this DSM build the truth was only in dmesg plus the cgroup's
memory.max_usage_in_bytes. If you are chasing the same shape on a normal
host, docker inspect -f '{{.State.OOMKilled}}' is the field to read first
— it is supposed to be authoritative. Here it read false five seconds
after the kernel had logged the kill, which is the reason this post exists.

What was actually breaking: a crawler flood

Host memory was never the problem — 64 GB installed, ~39 GB available. One
container's 512 MiB ceiling was.

The driver was traffic, and the numbers came out of the container's own
access log. In the last 50,000 lines:

distinct client IPs      25,976
403 responses            33,636   (geoblock / deny-list doing its job)
429 responses            16,082   (inflight and rate limiting doing their job)
container CPU            118–230%
host load average        26–32

Each restart refilled the heap in 12 to 40 minutes. Nothing in the
configuration was wrong, and neither were the 403s and 429s — they are the
filter rejecting crawlers exactly as designed. The problem is who is
knocking: this container is the public edge for a multi-terabyte Wikipedia
corpus served from a home NAS, behind a country allowlist and a bot filter.
That is an extremely attractive target for a residential-proxy crawler
cloud, and the responses telling it to go away are still work.

The fix, and the sentence that has to come with it

 deploy:
   resources:
     limits:
-      memory: 512M
+      memory: 2g
 restart: always
+memswap_limit: 2g

Verification, because "I raised the limit and it feels better" is not a
verification:

Only that container was recreated. The wiki, the proxy core and the robots container still reported two weeks of uptime — the gap users saw was seconds, not a stack restart.
HostConfig.Memory = 2147483648, MemorySwap = 2147483648, and the cgroup's memory.limit_in_bytes agreed with both; RestartCount went back to 0.
Peak usage after 18 minutes: 210 MB, against a ceiling it had been pinning at 512 MB. By midday it sat at 94 MiB.
The mount list still matched the file, including the plugin directory mount — losing that one had once disabled every router in this stack.
Both plugins were loaded and active (country allowlist, bot filter), and the routes returned 200 once the filter's score window expired.

And what it is not: this is a stopgap. The flood is the driver, so an
order-of-magnitude bigger flood reaches 2 GB too. The real fix is shrinking
the public entry surface, not enlarging the cage around it — and a limit
bump that is not labelled as a stopgap becomes a permanent explanation.

I chose to disable swap for this container rather than let it swap:
memswap_limit equal to the memory limit does exactly that. The host
already had 16 GB of its 40 GB swap in use with committed memory at 92 GB,
and an edge proxy that starts swapping slows the entire NAS — the disk array
is the bottleneck here, so swapping buys a slower failure rather than a
longer life.

The trap that nearly made me report a self-inflicted outage

During verification, my first bare curl probes against the site returned
403, and it looked exactly like a regression I had just caused by
restarting the proxy. It was not: the bot filter had scored my own client
IP (blocked user-agent, score 80) and placed it in a 15-minute score
window, because I had been hammering the host.

The tell was in the same test run — /robots.txt returned 200 and the
internal /cdn path returned 400 at the same moment, i.e. the router was
healthy and only my own requests were being filtered. On a host where you
are also a client, never diagnose with your own bare curl: probe a path the
filter ignores, or read a 200 out of the access log.

What I would do differently

Treat "stopped unexpectedly" plus ExitCode 0 as an instruction to read dmesg, not as evidence of a clean exit. A restart count above zero is a memory investigation until proven otherwise.
Alert on the cgroup peak, not only on restarts. The post-restart reading that was already pinned at the ceiling told me a seventh kill was coming about eighteen minutes before the notification arrived.
Get the "who is hitting me" number early. 25,976 distinct IPs in one log sample is the entire story, and I found it after spending time on the memory curve.
Question the container, not the server. "The server is out of memory" and "one container has a 512 MiB ceiling on a 64 GB machine" need completely different responses, and the second one is invisible unless you look at the limit.

The result

No OOM kill since the change, with a peak at 210 MB after 18 minutes against
a ceiling that used to be touched in every single cycle — and the seven
kills that had been silently restarting the public edge for four days
stopped appearing in dmesg. The crawler flood did not stop, which is why I
recorded the failure mode in my ops notes next to the stack: "container
stopped unexpectedly" plus exit 0 / OOMKilled=false is a misleading
signature, and on this platform the honest evidence lives in the kernel log
and the cgroup's high-water mark.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

💬 Let's Compare Notes

How do you detect silent container crash loops when uptime health checks report green between restarts? Drop your setup or thoughts in the comments below — I read and reply to every discussion.

Related notes:

Why Your CDP Clicks Silently Fail: Smooth Scroll, Minimised Windows, and JS-Dispatched Events

Hoelee — Fri, 23 Oct 2026 04:25:11 +0000

Originally published on my blog — read it there with the full context, code and updates.

If you drive a web app with the Chrome DevTools Protocol, you have met this failure: your script reports clicked @591,361, the element is there, the click landed — and nothing happens. No exception, no console error, no change on the page.

I spent an evening on exactly this while automating a marketplace's seller admin panel, and the three causes I found are worth writing down, because they are all invisible to the code that reports the click.

Cause 1: a stale rect, because the page scrolls smoothly

The obvious click helper looks like this:

r = self.ev("""(()=>{const e=%s; e.scrollIntoView({block:'center'});
  const b=e.getBoundingClientRect();
  return JSON.stringify({x:b.left+b.width/2, y:b.top+b.height/2});})()""")
self.click_xy(json.loads(r)["x"], json.loads(r)["y"])

Scroll into view, read the element's rectangle, click its centre. Correct — unless the page has scroll-behavior: smooth (very common in modern UI kits). Then scrollIntoView() starts an animation, and getBoundingClientRect() in the same tick returns the position the element is about to leave. Your click goes to where the element was, lands on whatever is there, and the log tells you it clicked the coordinates you asked for.

The fix is two lines — turn smooth scrolling off, then re-read the rect:

self.ev("""(()=>{document.documentElement.style.scrollBehavior='auto';
  if(document.body) document.body.style.scrollBehavior='auto'; return 'ok';})()""")
moved = self.ev("""(()=>{const e=%s; if(!e) return 'no';
  e.scrollIntoView({block:'center', behavior:'instant'}); return 'ok';})()""" % sel)
time.sleep(0.35)                      # let layout settle
# then read the rect (and retry a couple of times if the element is not there yet)

Two details matter. behavior:'instant' on scrollIntoView overrides the CSS; and the small wait before reading is what makes the rect trustworthy. Without the retry loop the helper still fails intermittently on elements that render a frame late — intermittent failures are the worst kind, because they make you doubt your own diagnosis.

Cause 2: the window is minimised, and coordinate clicks are dropped

After the fix, my click helper worked in a one-off probe and then never again inside the real script. Same selector, same coordinates, same page. The difference turned out to be visibility:

document.visibilityState → "hidden"
document.hasFocus()      → true

CDP-synthesised mouse events at coordinates are unreliable when the target tab is not the active tab of a visible window — the browser may accept them, throttle them, or route them nowhere. Note that hasFocus() returns true even when the document is hidden, so it is not a useful check; document.hidden/visibilityState is.

Two fixes, and I use both:

def activate(self):                 # bring the tab to the front of its window
    self.cdp("Page.bringToFront")
    time.sleep(0.2)

def click_js(self, sel):            # click by dispatching events from inside the page
    return self.ev("""(()=>{const e=%s; if(!e) return 'no';
      ['mousedown','mouseup','click'].forEach(k=>
        e.dispatchEvent(new MouseEvent(k,{bubbles:true,cancelable:true,view:window})));
      return 'js-clicked';})()""" % sel)

Page.bringToFront fixes the common case. But the JS-dispatched click is the one that never fails: it does not depend on the window being visible, and React's root-level event listener receives it exactly as it receives a real click, because the events bubble to the same root. A date-picker input that opened maybe one time in five with coordinate clicks opens every time this way.

The trade-off to know: dispatching events skips hit-testing, so it will also “click” an element a user could not reach (covered, scrolled off, zero-opacity). That is a feature here — and a trap if you use it to paper over a layout bug — so I keep coordinate clicking for anything visual and reach for click_js when the widget is a custom control.

Cause 3: the page received the click and rejected it

This is the one that wasted the most time, because the symptom is identical to causes 1 and 2: click fires, nothing changes.

Distinguishing them takes one injected listener:

window.__ev = [];
['mousedown','mouseup','click'].forEach(k =>
  document.addEventListener(k, e => window.__ev.push(k + ':' + e.target.tagName), true));

Events recorded → the input path works. The problem is in the app (cause 3), or in how you read the result (see below).
Nothing recorded → causes 1 or 2.

With the listener armed I clicked Confirm and the buffer filled up — so the click was fine, and the app was saying no. The reason was in text I had not thought to look at:

“Please create a new shop welcome voucher after the existing one is expired.”

The form refuses a second voucher of that type. No field-level red text, no toast in the first 500 ms, the URL unchanged. The rejection was real and the click was innocent.

Two more tools for this class of bug:

document.elementFromPoint(x, y)        // what is actually on top at that coordinate?
document.querySelectorAll('button').length   // how many match your selector?

The second one caught a subtle one: the portal leaves a hidden Confirm button in the DOM from the collapsed date-panel, so .pop() returned the invisible duplicate and the click went nowhere visible. Filter by visibility and size:

"[...document.querySelectorAll('button')].filter(b=>/^Confirm$/.test(b.innerText.trim())"
" && b.offsetParent!==null && b.getBoundingClientRect().width>0).pop()"

And the fourth failure I only found by re-reading

There is a class of “failure” that is not one: the write succeeded and your read is stale. I concluded a create had been silently refused because the list page did not show it immediately, clicked again, and later discovered the voucher had existed the whole time — the list just lagged about ten minutes. The lesson generalises: after a write, the only trustworthy verdict is a read-back after a real waiting period, and “no visible change” means unknown, not failed.

The checklist

Turn off smooth scrolling; scroll with behavior:'instant'; wait, then read the rect.
Call Page.bringToFront before clicking; treat document.hidden as the signal, not hasFocus().
For custom widgets, dispatch mousedown/mouseup/click from JS rather than clicking coordinates.
Arm an event listener to prove whether input arrived before you blame the app.
Count your selectors — hidden duplicates are common in component libraries.
After a write, wait and re-read; never re-click a submit you have not verified.

None of this is exotic, but every item is invisible in a log line that says clicked @591,361. If you are building automation against an admin panel you do not control, the click is the least reliable part of the pipeline — and the one most likely to be blamed last.

I build and debug this kind of browser automation for a living — self-hosted services, Docker and Traefik stacks, and the odd scraped-but-bot-walled marketplace. If you have a workflow that keeps breaking because a UI has other ideas, tell me about it: WhatsApp · me@hoelee.com · hoelee.com.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

💬 Let's Compare Notes

When driving headless browsers in production, how do you verify click events actually triggered application side-effects rather than just passing the automation driver's checks? Drop your setup or thoughts in the comments below — I read and reply to every discussion.

Related notes:

Two Ways a JavaScript File Stops Working and Looks Fine

Hoelee — Thu, 22 Oct 2026 04:25:08 +0000

Originally published on my blog — read it there with the full context, code and updates.

The change that did nothing at all

I added a language switcher to a dashboard I run for a media-conversion
queue. Three language packs, a t(key, vars) function, data-i18n
attributes on the chrome, a <select> in the top bar.

After deploying it, the page looked exactly as it had before. Not broken —
unchanged. Same layout, same Chinese labels, no missing panels, no error
card, no console banner. The switcher was simply not there, and neither was
anything else the new code was supposed to do.

That is the shape of bug I want to write down, because the page was not
"mostly working". The script had stopped executing partway through —
twice, for two unrelated one-line reasons — and the page rendered perfectly
anyway.

Why nothing looked broken

Two design decisions I would make again, and both of them hid the failure.

The first is that the default language lives in the markup. The Chinese text
is written inside the elements; data-i18n is only a hook that overrides
it when you switch. That is deliberate — the page needs no fetch to render,
and if a language pack goes missing you get Chinese instead of a row of
top.theme_to_dark keys. It also means a completely dead i18n layer
produces a page that is, visually, correct.

The second is that the buttons are wired with inline handlers
(onclick="rtFoo()"), which after a script abort point at functions that
were never defined. Nothing warns you. The page only reveals the truth when
a human clicks something.

So the symptom was "my change had no effect", which is the least informative
symptom a web page can give you.

Failure one: a parameter that swallowed a function

The file has a theme block and an i18n block. The theme block had this
signature:

function applyTheme(t) {                       // t = the theme
  label.textContent = t('top.theme_to_dark');  // TypeError: t is not a function
}

applyTheme was written before the i18n layer existed, and t was a
perfectly reasonable name for "theme". When the i18n layer arrived, t in
that scope stopped meaning the theme and started meaning the function I
wanted to call — except the parameter shadows it, so the call became
"invoke a string".

A TypeError at the top level of a classic script does not just fail one
line. Everything after it never runs: the language-pack loader, the
data-i18n sweep, the switcher rendering. One wrong letter removed an
entire feature, silently.

The fix is one character — rename the parameter to th — plus a comment
where the next person will look:

function applyTheme(th) {   // NOT `t`: t() is the i18n function in this file

Failure two: a declaration that arrived too late

The second cause was in the same file, and it is a JavaScript rule people
know and still trip over. The theme block runs at load and calls t('…').
The i18n block, further down the file, began with:

let LANG = localStorage.getItem('rt_lang') || 'zh-CN';

let and const are hoisted but not initialised: between the start of
the scope and that line they live in the temporal dead zone, and any access
throws Cannot access 'LANG' before initialization. Because the theme block
sits above the i18n block, the failure was not a race — it was guaranteed
on every single load.

Note the distinction from var, which would have quietly given me
undefined and a working page. The TDZ is the language protecting me from
exactly the bug I was about to write; the only reason it looked like
"nothing happened" is the markup fallback again.

The fix is ordering, and ordering here is a dependency, not a style
choice: if a block calls t(), the i18n declarations have to be above it.
The file had been organised by topic ("theme, then i18n") because that is
how it grew.

Why my checks did not catch either one

I had tests. They checked intent, not effect:

the three language packs exist and parse;
every key used in the markup exists in every pack;
no pack contains a dead key (a typo detector);
the {var} placeholder sets in the three packs match.

All true. All green. None of them executes the page, so none of them can see
a script that aborts on line 12 of 900. Static checks are structurally
unable to catch "the file is valid and the program is dead".

Assert on effects, in a browser

What found both bugs was asking the live page questions that only a running
program can answer:

typeof t === 'function'          // is the entry point still callable?
document.querySelectorAll('[data-i18n]').length   // hooks present
rtCheckHandlers()                // do the inline handlers still resolve?

That last one is worth more than it looks. The page carries 219 inline
onclick attributes naming 217 distinct functions; the check enumerates
them and asserts each name is defined on window. It turns a hand-clicked
smoke test into a two-second automated one, and it is the check that turns
"the page looks right" into a measurable property.

The second tool is cheaper and nastier: a pseudo-language.
?lang=pseudo takes every string in the pack, prefixes it with [~~ and
stretches it 1.6×. Then assert two things — no horizontal overflow at
1440px, and no element in the nav still lacks the [~~ prefix. The first
catches layout that only breaks in a longer language; the second catches
nodes that never got a translation hook at all. It found a real one in the
same run.

Honest scope note: a pseudo-locale does not prove your Malay translation is
good. It proves the plumbing is complete — every string flows through
the mechanism, and the layout survives length. Judgement about the words
stays with a human.

What I would do differently

Load the page in a real browser and assert typeof on the entry points before believing a client-side refactor worked. A dead script is invisible to every static check you own.
Never let a parameter name shadow a function the same file calls. One letter bought me an entire afternoon; the comment is the part that stops it recurring.
Treat declaration order as a dependency, not aesthetics. If a block calls a function or reads a constant, that declaration goes above it — especially in one long script.
Generate the language packs from one table. Then "the keys match" is a test rather than a hope, and drift cannot be merged.
When a change does nothing, suspect the script before the change. "Nothing happened" is the default failure mode of a rendered page, not evidence that your code was inert.

The result

Two one-line causes, both in the same file, both found by browser-side
assertions rather than by reading code again. The dashboard switches
language at runtime now, the default render is byte-for-byte what it was
before the change, and the three packs come from a single generated table —
so a missing key shows up as a red test instead of as a word that quietly
stays Chinese.

The generalisation is the part I keep: a crash tells you where to look. A
silent stop tells you nothing, and looks like success. If a feature can
fail without moving a single pixel, then "the page still looks right" is not
a check.

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

💬 Let's Compare Notes

What is the most deceptive 'silent failure' bug you have tracked down where HTTP returned 200 and console errors stayed quiet? Drop your setup or thoughts in the comments below — I read and reply to every discussion.

Related notes:

DEV Community: Hoelee

The Forward-Auth Gate That Verified Perfectly — and Wasn't Live

Why this matters beyond my one app

The setup: SSO for an app that has no SSO

The trap: I verified the layer I changed

The permission wall on the way to the fix

Optional SSO: gate the UI, leave the API alone

The honest part: this is not single sign-on

Reverting it properly (and the arithmetic that saved ten other apps)

What I'd do differently

The result

Want this for your business?

Your Disk-Full Alert Is Lying: Stop Alerting on Percentages

The three alerts that taught me

Alert on the consequence, not the ratio

Noise is not harmless

Thresholds are personal, shapes are not

The result

Want this for your business?

Why Your Grafana Dashboard Shows No Data (When Prometheus Is Fine)

The searchable question

The actual cause: a variable that filtered on itself

Why my verification missed it

The second trap: $__all is not .*

Not every empty panel is a bug

What I'd do differently

The result

Want this for your business?

Stop Writing Agent Prompts for Deterministic Work

The rule

Where the model does earn its place

The part that actually bit me

The short version

Running Production Infrastructure Solo: 162 Containers Across Three Hosts

Why this matters

The shape of it

What makes three hosts one system

Failure 1: the dashboard was lying to me

Failure 2: one missing config file, three failure modes

How I keep the numbers honest

What I would do differently

The result

Want this for your business?

Why Chrome Forgets Its Tabs in a Container — And How I Fixed It

Why this matters

The setup

Attempt 1: the flag

Attempt 2: edit the profile's preferences

Attempt 3: a managed policy

Attempt 4: prove the exit is ungraceful

The fix: stop configuring, start keeping

Landmine 1: a stale Singleton* stops Chrome from starting at all

Landmine 2: custom-cont-init.d only runs executable files — and runs them with bash

The plumbing trap that came with it

What I'd do differently

The result

Want this for your business?

💬 Let's Compare Notes

Calibrating Intel Quick Sync HEVC: A Cliff at 28, and a Preset That Does Nothing

The question I could not answer: which number do I ship?

Problem one: the hardware is not in the machine where the app runs

Measure with the engine's own gate, not with ffmpeg

The ladder: on a thin source, low ICQ makes the file bigger

Full files confirm the ladder — and one shortcut survives

The preset that does nothing

What I shipped

Four traps, each of which cost a cycle

What I would do differently

The result

Need Resilient Hardware Video Pipelines & Automated Quality Gates?

💬 Let's Compare Notes

No API for Browser Translation: Adding an English Mode to a Chinese-Only Web App

What I was working on

Finding 1: the app declares the wrong language, and that disables translation

Finding 2: there is no browser-translate API

The fix: a dictionary and a button, injected at the gateway

1. Harvest the vocabulary from the app's own bundles

2. Match like the DOM actually is, not like you wish it were

3. Keep up with the framework

What it deliberately does not translate

The second trap: `$__all` is not `.*`

Landmine 1: a stale `Singleton*` stops Chrome from starting at all

Landmine 2: `custom-cont-init.d` only runs executable files — and runs them with bash