DEV Community: Tdude

Most of my robots were me

Tdude — Sun, 01 Nov 2026 23:00:00 +0000

What this is about. I added a counter to my own analytics to find out how
many robots visit my sites, and whether any of them would say who they were.
Nearly every number it gave me was measuring something other than what I
thought.

Why. Because a number you have not grouped by anything is not yet
information. There is a difference between belief and knowing.

5 reads, 62 bots

The counter went live, I posted a new piece on LinkedIn and Facebook, and came
back to "5 reads, 62 bots" a few minutes later. Sixty-two robots on one post
felt like arrival.

It was not. The single biggest source was Facebook, fetching the page over and
over from different addresses to build one preview card. LinkedIn did it twice
more. Sharing a link looks almost exactly like being crawled, and nothing on
my dashboard said so.

The actual AI crawlers had sent one request each. GPTBot, ClaudeBot,
PerplexityBot and Common Crawl had each knocked once and left. That is
discovery, not reading.

So the headline split in two: AI crawlers in one box, link previews in another.
By the end of the day Facebook had fetched that single page from 130 different
addresses, every one of them provably Facebook. I had assumed a preview card
was one request.

21,791 visits, and 20,610 of them were me

The next morning the counter said 21,791.

I was briefly delighted. Then I grouped it, and found 20,709 visits from a
client calling itself curl, hitting the homepage, from three addresses, none
of which had an address at all.

Every one of my app containers has a health check. It runs curl against its
own homepage every fifteen seconds to decide whether the container is still
alive. Seven containers, four probes a minute each, is about forty thousand
requests a day. My bot filter is deliberately broad, it matches the word
curl, and so every single heartbeat was filed as an automated visitor.

Ninety-five percent of my bot traffic was my own server taking its own
pulse.

The fix is one line. Working out which signal to trust took the rest of the
morning. Not the user-agent, because anything can claim to be anything. Not the
path, because real crawlers ask for the homepage too. The health check is
distinguishable because it talks to the app directly and never passes through
the web server in front, so it arrives with no client address at all. No
address means it came from inside the house.

This can break, and I would rather it broke this way round. If the proxy ever
stopped handing me the real client address, I would quietly undercount real
robots instead of confidently counting imaginary ones.

Deleting them mattered more than I expected

Fixing the code only stops the bleeding, so I deleted the 20,787 heartbeats
already recorded.

That sounds like tidying. It was not. The counts are also rolled up into a small
permanent table, so that this October stays comparable with next October: the
raw records are deliberately short-lived, the daily totals are not. Cleaning
only the raw table would have left today's summary looking healthy while the
permanent history went on insisting I had twenty thousand robot visitors on the
day I shipped the counter. Wrong numbers that expire are a nuisance. Wrong
numbers kept forever are a liability.

I nearly made it worse. My first instinct was to delete everything with no
client address, which is the obvious signature. Checking the query before
running it showed that would also have deleted perfectly good records for
Googlebot, Bingbot and ClaudeBot, recorded before I started keeping origins at
all. The predicate had to be narrower.

Pruned on both sides, the real figure for that first day is 914 visits from 26
bot families: far less impressive, and the first one I believe.

I was wrong about asking them who they are

The thing I actually wanted was an email address. If a robot is reading
everything I write, I would quite like to know who to talk to.

Note: I do NOT track human users like this. Please read on!

I assumed this was hopeless. There is a header in the HTTP spec, From:, meant
to carry the email address of whoever is responsible for an automated client,
and my firm expectation was that nobody had used it since about 1996. I said so
out loud, more than once, and logged it mostly as a joke.

Once the heartbeats were out of the way, 66 visits had filled it in.

googlebot(at)googlebot.com
bingbot(at)microsoft.com
gptbot(at)openai.com

Google, Microsoft and OpenAI all announce themselves, unprompted, in a header I
had written off. Two smaller crawlers I did not recognise went further and gave
real abuse-contact addresses. The obfuscated "(at)" is their own, presumably so
the addresses survive being scraped out of server logs by somebody else's robot,
which is a pleasing piece of symmetry.

So the answer to "can I ask a robot who runs it" turns out to be: mostly you do
not have to ask. The well-behaved ones have been telling me all along and I was
not reading it.

What happened, and what else I could have done

Three numbers were wrong in three different ways. The first counted link
previews as readers. The second counted my own health checks as visitors. The
third, the one I never even displayed, was sitting there full of answers because
I had assumed it would be empty.

The fix for the big one was a single condition, and the alternatives were all
worse:

Instead of "no client address"	I would gain	It would cost
Matching the user-agent	Obvious and readable	Anything can claim to be `curl`, and real tools legitimately are
Matching the homepage path	Also obvious	Real crawlers request the homepage constantly
Excluding the container's own network range	Precise today	Breaks the first time the network layout changes, silently
Not counting bots at all	No wrong numbers	No numbers

I went hunting for robots and found a mirror. Before you trust a number, make it
sit still and tell you what it is made of. Mine was 95 percent heartbeat.

Now comes the hard question: are you sure your (third party) tracker is made up
of the same suspicios minds? Are you sure they filter out the noise or do they
sell you snake oil? Give us a shout and we can check
for you!

Next in this series: the most requested page on my server, which is a page that
does not exist, and what I did to the robots asking for it.

Originally published at idunworks.com.

Three holes in a green CI, and a lock for the free plan

Tdude — Fri, 30 Oct 2026 09:55:37 +0000

What this is about. On a site we worked on recently, the test pipeline went green across the board, and we looked into what green actually covered. Three things turned out not to be covered at all. This is how we found and closed each one, on a free GitHub plan.

Why. A green CI tells you that the tests it ran passed. It doesn't tell you which tests it didn't run, or whether anything waits for it before you deploy.

The question

Once CI was green, we wrote the next step down as one line:

go on with test, CI and safety planning. Output a shortlist in order of priority

That gave us ten items, and the top two were about the pipeline itself, not about the code. So we started at the top.

Hole 1: tests that CI never ran

The backend is a handful of small services, and CI runs each one's tests in its own job. A couple of services had plenty of tests and no job at all. Nobody had removed them; they had just never been added to the list.

Adding them was one word each in the workflow file. Then the first run showed what they had been hiding:

A pile of tests errored instead of running. Their database tests looked for a Postgres address in a dedicated variable. One test file skipped politely when it wasn't set; the files that borrowed its fixtures didn't, and crashed on an empty URL.
The async tests failed because pytest-asyncio wasn't installed in CI. They were never broken; they were never run.

The fix was a small conftest.py with one rule: without a test database, skip every test that needs one. In CI, fail instead. The second half matters most. A broken database in CI should turn the run red, not quietly turn the database tests into skips.

def pytest_configure(config):
    global _db_ready
    url = os.environ.get(TEST_DB_ENV, "")
    _db_ready = bool(url) and _database_reachable(url)
    if not _db_ready and os.environ.get("CI") == "true":
        raise pytest.UsageError(f"CI: {TEST_DB_ENV} unreachable at {_redact(url)}")

_redact drops the password from the URL before it reaches a log, because CI logs are read by more people than your secrets file is.

Hole 2: tests that built a different schema than production

In production, the service builds its tables by running its migrations when the container starts. Its tests built them straight from the models with create_all. Two roads to the same place, as long as nobody forgets a migration. Forget one, and the tests pass while production queries fail.

So we added one test: run the real migrations on an empty database, then ask Alembic to compare the result with the models.

diff = compare_metadata(context, Base.metadata)
assert diff == [], "models and migrations differ:\n" + "\n".join(map(str, diff))

It found drift on its first run. Harmless drift, and the database itself needed no change. Still, a test that checks for drift and finds some on day one has earned its place.

Another service had the opposite problem. Its background jobs opened their own database connection from the real connection string, not the test one. In CI that pointed at an empty database, so the jobs failed, logged an error and moved on, and the tests passed. A few tests had in fact been passing because a background job failed silently. Now the jobs use the test database, or stay off unless a test asks for them.

Hole 3: nothing waited for green

The repo is private and on a free plan, and GitHub's branch protection, the thing that stops a red commit from merging, needs a paid plan there. So a commit could go red in CI and still be deployed a minute later.

Every deploy goes through one command, so the lock went there. Before anything is synced or deployed, a small script:

checks that the commit being deployed is the one on the main branch,
finds the CI run for that commit, and waits for it if it's still going,
refuses unless it ended in success.

Docs-only commits don't trigger CI at all, so those get special handling: if every file changed since the last tested commit is documentation, that earlier run counts. Anything else without a run of its own is refused.

There's an escape hatch, ALLOW_UNTESTED=1, which prints a warning and goes ahead. It's for emergencies, not for when CI is slow.

We tested the script against a fake GitHub CLI in a throwaway repo, one case at a time: green, red, cancelled, still running, docs-only, untested code, not pushed, and the override. Then against the real thing, right after a push:

… waiting for Test Suite run on <commit>
✓ CI green on <commit>

The same change fixed the order of pushes in the deploy command, too. It pushed the main repo before the submodule it points to, which is exactly the wrong way round.

What happened, how we fixed it, and how else it could be done

Services missing from CI: added to the job list, plus the one test dependency they needed.
Database tests that errored or skipped: a shared rule that skips without a database and fails in CI.
Schema built two ways: one test that runs the real migrations and compares them with the models.
Background jobs leaking out of the test database: pointed at the test database, or switched off by default.
Red commits deployable: a CI check inside the deploy command, with a loud override.

Other ways to make deploys wait for CI:

Option	Better at	Costs
A check in the deploy command (what we did)	Free, works on any plan, covers the only way you deploy	Only protects that one command; a manual deploy can skip it
Branch protection	Enforced by GitHub for everyone, before merge	Needs a paid plan for private repos on personal accounts
Deploy from CI (CI deploys only after tests pass)	No way around it, and no laptop involved	Server credentials live in CI, and you need to trust and maintain that path
A pre-push hook	Catches it before it even leaves your machine	Runs the tests locally on every push, and is easy to skip

Before you trust a green CI

List your services and compare that with the list CI runs. They drift apart.
Make "no database" a skip on your laptop and a failure in CI.
If production migrates and tests use create_all, add one test that compares the two.
Grep for anything that opens its own database connection; tests should own all of them.
Make your deploy ask CI before it does anything, and make skipping that loud.

Green means a bit more there now than it did this morning. More from the workshop is on the Ship Log.

My most popular page does not exist

Tdude — Thu, 29 Oct 2026 23:00:00 +0000

What this is about. While working out why my new robot counter had such a
flattering opinion of my readership, I read the raw server log and found that
the most requested page on the whole machine was one that has never existed.
This is what I did about it, and the mistake that nearly took all my sites
offline.

Why. Building the trap took an afternoon. Working out what must never be
trapped took longer, and that is the part worth writing down.

Fifteen thousand requests for nothing

/wordpress was the most requested path on the server for my newly deployed
site. More than the homepage. Fifteen thousand requests in a single log file.

I have never run WordPress there (!). I once had a React front end that read
content from WordPress on a different machine entirely, which is not the same
thing and would not leave this trail.

Every one of those requests forged a browser user-agent, rotating through
Chrome, Edge, Safari, Firefox and Opera in almost exactly equal numbers. That is
one scanner with a costume box, not many visitors. It is also the kind of sneaky
little robot my shiny new counter cannot see, because it is pretending to be a
person. Once identified, never doubted. No amount of counting fixes that.

What it can see is that the requests came from a couple of dozen addresses, two
of which accounted for 87 percent of them.

Twenty bytes a second

A tarpit is an old idea (a pit and a load of tar): answer, but answer so
irritatingly slow that the other side wastes real time. Ha! So I gave them
something to download. A file called wordpress_backup.sql, served at twenty
bytes per second. The whole megabyte takes just over fourteen hours.

The sketch I started from bolted on a Python service whose entire job was to
dribble ones and zeros forever. The web server does it alone, sulkily, for
nothing: a finite file served slowly enough is indistinguishable from an
infinite one, so there is no extra process, no extra port and nothing extra to
keep patched. I also dropped a (for me) clever header containing a JavaScript
infinite loop, on the grounds that response headers are not executed by
anything.

Then the web server and I fell out over what "slow" means.

The rate limit throttles the headers too. A client spends eighteen seconds
receiving about 360 bytes of response headers, so it is stuck before it even
learns the filename it is being offered. I spent a while convinced my header
configuration was broken, when in fact it was still in flight. Stained by tar up
to my knees already...

The defaults make a worse trap than small buffers do. Out of the box the
server hands over one four-kilobyte chunk immediately and then goes quiet for
about three and a half minutes, which any scanner with an inactivity timeout
treats as a dead connection and abandons. Shrinking the buffers turns that into
a steady dribble, so the connection stays interesting enough to sit in. Smaller
buffers, stickier trap.

Up to my elbows in tar

Then I widened it to the other things robots ask for. /.env, /.git/config,
the files that hand over credentials when they exist. All of those begin with a
dot, so a rule for dot paths looked obvious.

So does /.well-known/acme-challenge/, which is how my TLS certificates prove
they are mine when they renew.

A rule that trapped every dot path would have tarpitted my own certificate
renewals. Nothing would have broken that day, or that week. It would have
surfaced a month or two later as expired certificates on every site at once, and
I would never have connected it to the "clever" thing I did in October.

There is a second exclusion of the same shape. Search engines speculatively ask
every site for /wp-json/ and /wp-sitemap.xml, just in case WordPress is
hiding there. A blanket rule for anything starting with wp would have put
Googlebot in the tarpit for fourteen hours at a time. I first assumed those
requests were leftovers from some old setup and would fade. They are not. They
are standard crawler behaviour and they will keep arriving forever (or until
Mullenweg goes crazy again), so the exclusion is permanent. The note in the
config now says so. The version where it looks temporary is the version somebody
deletes at the Spring cleanout.

What it has caught

So far the trap has held 91 requests for a combined 25.6 hours. The average
stay is seventeen minutes. Twenty lasted more than ten. The longest single
connection sat there for eight hours and ten minutes.

The funniest part is that it is barely catching WordPress scanners at all. The
top catches are /.env and /.git/config. I built a little black sticky goo
for one kind of robot and it filled up with another. Never thought slow
connections could bring me so much joy :)

What happened, and what else I could have done

A page that does not exist was my most popular page. It now answers, very
slowly, to everyone except the three kinds of visitor that must never be slowed
down: real people, search engines doing their normal speculative sniffing, and
my own certificates renewing themselves.

A tarpit is not the only answer, and it is not obviously the best one:

Instead of a tarpit	I would gain	It would cost
Returning "not found"	Free, no risk at all	The scanner leaves in two milliseconds and is back tomorrow
Blocking the worst networks at the firewall	The cheapest real relief; two addresses were most of the traffic	Those addresses rotate, so it is maintenance forever
Auto-banning after a few probes	Adapts without a hand-maintained list	One more moving part, and a false positive blocks a person
Serving a deliberately huge file at full speed	Costs them bandwidth immediately	Costs me the same bandwidth, which is the wrong trade

The honest caveat is that a tarpit costs me something too. I hold a connection
so that they hold one, and if the scanning came from thousands of addresses
instead of a couple of dozen, that trade would stop favouring me. Hence a hard
ceiling on how many of these connections can exist at once, and a refused one
being closed rather than answered.

And a small confession about measurement, which is becoming a theme. My first
instinct was to keep these requests out of the logs entirely, since fifteen
thousand lines a day is noise. I did exactly that, and then realised I had built
a trap with no way of telling whether it caught anything. They now go to their
own log file, which is how I know about the eight hours.

Hit me in the comments if you think my reasoning is flawed, or if you think bots
should be treated as equals.

Originally published at idunworks.com.

I locked the door my deploy knocked on

Tdude — Wed, 28 Oct 2026 10:04:34 +0000

What this is about. I asked my AI for a full maintenance audit of my CMS, and we spent an afternoon closing what it found. Then the first deploy failed, and one of my own fixes had caused it.

Why. Every fix changes something somebody depends on, and sometimes that somebody is your own tooling. It's better to find out in a rollback than in an outage.

The audit

My brief was short:

i want a full audit with a report at each step.

It came back in parts: UX, marketing, product, and security. I said "let's concentrate on the sec. items first", and we did them in a few rounds. The ones worth telling:

The quiz answered itself. My course pages fetched each quiz with the right answer included, unscrambled, and the page just didn't show it. Anyone opening the browser's network tab with a beautiful or curious mind could get a perfect score. Students now get the questions, and only course editors get the answers.
The browser named its own price. When you bought a course, the checkout trusted the amount the browser sent. Now the server looks up the price itself, and when the payment comes back it checks that what was paid matches what was owed. If it doesn't, nothing gets unlocked and I get a loud log line. In a webshop this is one of the building blocks. I started the courses out of a need for a small course platform, no payment. Easy to forget the nuances.
A door between my services had a wall, but no lock. The route that grants course access after a payment can only be reached from inside my own network. It still trusted anything that reached it. It now wants a shared secret, and it refuses when the secret isn't set, instead of letting everyone in.
Logging out didn't log you out. The button cleared your cookies, but the session behind them stayed valid until it expired. Logout now ends the session on the server too. A deactivated account, or a changed password, ends every session that account has. Easy to miss when you're up to your elbows in code-grease creating a course platform.

Each fix came with a test, and the test suites ran green: about 620 tests across three services. I've seen worse.

The deploy

Another fix from the same evening hid the API's documentation pages in production. FastAPI publishes them at /docs by default, a guided tour of every endpoint you have. Strangers don't need that tour.

The first site went out, and the deploy stopped with:

Health checks timed out after 120s

My deploys are of the blue-green kind: The new version starts next to the old one, a script checks that it's healthy, and only then does traffic move over. The script's idea of "healthy" was asking for /docs and getting a page back. I had just made sure it would never get one again.

So for two minutes the new version stood there, perfectly fine, knocking on a door I had locked an hour earlier. Then the script gave up, the old version kept serving, and nobody visiting the site noticed a thing.

The fix was one word in three scripts: ask /health instead. That route exists precisely to say "I'm alive". Nobody will ever hide it, because hiding it would be pointless. Go there and it will kick your ass to 404.

Then the evening ran long. Halfway through the fleet I said "hold off the rest for tomorrow", and the AI started stopping a deploy that was already halfway out. Yeah, what does "hold off" mean really? I corrected it:

i meant to not start a new deploy. Finish what is started as agreed

Fair. A half-finished deploy is worse than either a finished one or one that never started. The last site went out the next morning, and every site passed the same checks: no docs, logout only by POST, internal routes closed to the public, and a quiz with no answers in it.

What happened, how we fixed it, and how else it could be done

Quiz answers in the response: a separate public schema without the answer field.
Client-set prices: the server prices the order and checks the paid amount before it unlocks anything.
Unlocked internal route: a shared secret, with the route closed if the secret is missing.
Logout that didn't: sessions are revoked on the server at logout, deactivation and password change.
The failed deploy: health checks ask a dedicated /health route, not the docs.

Other ways to handle the docs:

Option	Better at	Costs
Turn them off in production (what we did)	Nothing to find, nothing to maintain	You lose the docs in production, so you read them locally
Keep them behind an admin login	Docs are still there when you need them live	Another protected route to get right
Block them at the web server	One rule covers every service	The app still serves them to anything inside the network
Leave them on	Easy for integrators	A free map of your API for anyone who's curious

Before you lock something

Search your deploy, monitoring and health scripts for the URL you're closing.
Give every service a /health route, and point every probe at it.
Make sure a failed health check rolls back, before you ever need it to.
When someone says "stop", ask if "don't start" or "abort" is what they mean. Or right?

The rollback did exactly what I built it for, on the first night it was needed. More from the workshop is on the Ship Log.

Originally published at idunworks.com.

My translator thought Monday.com was a weekday

Tdude — Mon, 26 Oct 2026 07:38:52 +0000

What this is about. This spring a machine translated my blog into Swedish. This autumn I read the result. Yeah I know, kinda lazy. So much time and so little to do... It translated a product name into a day of the week, and my install commands into Swedish.

Why. Generic AI is good at generic questions. The moment it has to work inside your business, in your voice, with your products and your rules, it falls over, and it looks like an AI problem. It's a context problem. That's good news, because context is something you own. You can has fix!

The Swedish was crap

Most of my older posts have Swedish versions, made in March by the translation service I run next to my CMS. In October I asked my AI to review the old posts and redo the Swedish, because, in my words:

The one I came over is crap.

That turned out to be generous. Some highlights:

Monday.com became Måndag.com. Lovable, a Swedish company, became Lövbar, which means "leafable".
pip install -r requirements.txt became pip installera -r krav.txt. Poetry, the Python tool, became Poesi. Go, the language, became Gå, the verb.
"Execution", as in running a task, became avrättning. That's the kind with a blindfold and an axe.
"Data residency laws" became laws about residence permits.
"$5.55B" became 5,55 dollar. A rounding error of about five billion.

The German was no better. "Digest" became Verdauungstrakt, the digestive tract. Hilarious!

None of this is the model being stupid. Every one of those words was translated correctly, as a word. What the machine didn't know was that Monday.com is a product, that a line in a code block is something you type, and that my post was about servers, not prisons. Language it had plenty of. What it was missing was the context.

Context is more than documents

When people say "give the AI your data" they usually mean documents: the manual, the product catalogue, the FAQ. That's the part everyone has heard of. Look it up before you answer is the whole idea behind what's called RAG: an AI that reads your manual before it opens its mouth. It's a good idea, and I wrote about building one in my Local AI series.

The translations taught me there are other kinds, and they're easier to miss.

What your words mean. "Make" is a verb, unless it's the automation tool. "Execution" is a word, unless you're a developer. A style guide or a glossary of your products, your tools and your no-translate words is context too, and it's usually the cheapest kind to write down.

Which version is current. Several of my Swedish posts were translated from an older English version. The English had moved on and the Swedish was still quoting lines I'd deleted. The AI had read the manual but it was last year's manual! Today, an hour after the new Swedish went in, I edited two English posts in the CMS. The check we'd built refused to write the next Swedish update because the page structure no longer matched. That refusal was the most useful thing it did all day.

Whose corrections count? My translation service keeps the corrections I make by hand, as examples to learn from. A good idea, with two traps in it. First, the obvious way for my AI's script to save its new Swedish was the same door my corrections go through, and everything through that door is filed as "a human fixed this". That would have taught the machine to imitate a machine. The script uses a door that doesn't record anything. Second, some of my own "corrections" from March weren't corrections at all: I'd added lines in Swedish that never existed in English. As training data, that teaches a model to invent.

The same wall in a shop or a course

My translations are a low-stakes place to hit this wall. Here's where it costs money.

A webshop. Ask a generic AI to write a product description and it writes a nice one, for a different brand. It doesn't know that your T-shirts are printed on demand, that you never say "premium", or that returns take fourteen days, not thirty. A support bot without your shipping policy gives a confident, wrong delivery date, and your customer remembers the date, not the bot. Give the same model your catalogue, your policies and ten descriptions you're proud of, and the copy sounds like you.

A course. A student asks a question about lesson four. A generic model answers from the internet, which is fine until it contradicts your lesson. One that reads the course material first answers from what you taught, and can point to the page. The good version also knows which edition of the course the student is on, and that's the stale-version problem again.

In both cases the model is the cheap part. Context is the work: deciding what goes in, keeping it current, and knowing which corrections came from a person.

What happened, and how else it could be done

The machine translations went wrong because the machine had my words and none of my context. The fix was the boring kind:

My AI translated from the live English page, not from a copy.
Code, links and product names were left alone.
A script compares the translation with the live page and refuses it when the structure differs.
The old versions are kept, so I can roll any post back.

Option	Better at	Costs
Better prompt only (tell it the tone)	Free, quick	Doesn't know your products; forgets everything next time
Glossary and style guide	Fixes names and tone everywhere at once	Someone has to write and keep it up to date
Look up your documents first (RAG)	Answers from your own material	Only as current as the documents you feed it
Fine-tune on your corrections	Learns your voice for good	Needs clean data: real human fixes, nothing invented
Run it on your own server	Your catalogue and policies never leave the building	You keep it running, and you pay for the hardware up front

Running it yourself matters more than it sounds. Once the AI reads your price lists, your policies and your customers' questions, that context is the most valuable thing in the system. I'd rather keep it next to the rest of my stack than mail it to someone else's model every time I ask a question.

Before you blame the model, check these things:

Does it know your names, products and no-translate words?
Is it reading the current version of your material?
Can you tell which corrections came from a person?
Is anything checking its output against the source?

The Swedish is fixed. Lovable is Lovable again, and no one gets executed in my Docker posts any more. The old Swedish is still saved, in case I ever want a laugh.

Good luck,//T

Originally published at idunworks.com.