DEV Community: Kingsley Kanu

My monitoring stack costs nothing a month and tells me more than the paid tier did

Kingsley Kanu — Thu, 29 Oct 2026 08:00:00 +0000

The observability layer watching my entire homelab, every host, every container, every virtual machine, the storage arrays, the backups, the reverse proxy, the DNS pair, has a monthly bill of zero. Not a free tier with a retention cliff and a per-host cap that turns into a sales call the moment you grow. Actually zero, running on hardware I already own, with retention and cardinality limited only by disk I already bought.

"Free monitoring" usually means "free until it matters," and this is not that.

What is actually running

The stack is the boring, battle-tested open-source set, each piece doing one job:

A metrics database that scrapes numbers from everything on a schedule and stores the time series. Small agents on every machine that expose those numbers: CPU, memory, disk, network, service states, per box. A dashboard layer that turns the series into graphs and, more importantly, into a single screen I can glance at. A logs system alongside the metrics, with shippers on each host forwarding system and service logs so I can search across the whole estate in one place. An uptime checker hitting my services from the outside the way a user would. And exporters that translate specialized systems, the hypervisor cluster, the backup server, the databases, the DNS servers, into numbers the metrics database understands.

Everything runs in containers on one node. The whole thing is a set of config files and compose definitions I can rebuild from scratch in an afternoon, which I know because I have.

Why it beats the free tiers I used before

Hosted monitoring has a free tier, and the free tier is a shape designed to become a paid tier. It caps the number of hosts, so the day you add the machine that makes your setup interesting is the day you hit the wall. It caps retention to a week or two, so every "when did this start?" question dies just before the answer. It samples aggressively, so the spike that woke you gets averaged into invisibility by morning.

Self-hosted inverts all three. Hosts are free because a host is just another scrape target. Retention is a disk decision, not a pricing decision, and disk is cheap. Resolution is mine to set, so the spike is still there at full detail when I go looking. I am not richer than the hosted vendors; I just moved the cost from a recurring bill to hardware I had already bought for other reasons, and the marginal cost of watching one more machine dropped to nothing.

The honest costs, because there are some

Free of money is not free of everything, and pretending otherwise is how people end up resenting their own stack.

It costs setup time. The first build is a real weekend, learning how the pieces fit, which exporter feeds which panel, why a dashboard is empty because a data source name does not match. The second build, from my own notes, is an afternoon. The difference between the two is entirely the notes.

It costs a small amount of ongoing attention. Components release updates; occasionally one changes a config format and a panel goes blank until I notice. That is what you pay for owning the stack instead of renting it, and it is real, if modest.

And it introduces a genuine trap I had to close deliberately: the monitoring cannot only watch the workloads. It has to watch itself. A metrics database that quietly dies is worse than no monitoring, because now you have false confidence, a green wall that is green because nothing is reporting. So the stack monitors its own components, and an external uptime check watches from outside the node entirely, so that if the whole monitoring host falls over, something not on that host still notices.

What it gives back

For zero recurring cost I get every machine's health at a glance, logs searchable across the whole estate, alerts that reach me when something crosses a line, and history deep enough to answer "was this always like this, or did it start on the fourteenth?" That last capability is the one hosted free tiers quietly take away, and it is the one I use most, because most real problems are not sudden. They are slow, and you only catch slow with retention.

The stack is not free because I found a loophole. It is free because the recurring cost of monitoring was always mostly rent, and rent is optional when you own the building.

Proxmox still flags btrfs as a technology preview, and I guessed wrong about why

Kingsley Kanu — Tue, 27 Oct 2026 08:00:00 +0000

While setting up a node, I reached the storage step and had a small, confident opinion ready. Proxmox has supported btrfs as a root filesystem for years now. btrfs is in the mainline Linux kernel, ships as the default on major distributions, and runs under enormous production workloads elsewhere. So when the installer offered it, I assumed the "technology preview" label attached to it was just conservatism that nobody had gotten around to removing.

Then I read the label instead of dismissing it, went looking for why it was still there, and found that my confident opinion was wrong in an instructive way.

What I assumed

My reasoning felt airtight. The filesystem is mature. It is in the kernel everybody ships. Other platforms trust it in production. Therefore the warning is stale, a leftover from when btrfs was genuinely new, and the sensible move is to pick it and move on.

Every individual fact in that chain is true. The conclusion is still wrong, because I was reasoning about the wrong thing. "Is btrfs mature?" and "is Proxmox's btrfs integration mature?" are different questions, and I had answered the first while believing I had answered the second.

What the label actually means

Proxmox's own developers have been explicit about why the preview label persists, and it has nothing to do with doubting btrfs as a filesystem. It is about the depth of integration with the rest of the platform, compared against the filesystem the platform is genuinely built around, which is ZFS.

The gaps are in the connective tissue. Some of the storage-management features that just work on ZFS, the way the platform handles replication, certain snapshot and backup interactions, the polish of the tooling around them, are simply less complete for btrfs. The filesystem underneath is solid. The platform's assumptions and best-tested paths were built for a different filesystem, and btrfs rides in a lane that has had less traffic.

That is a completely reasonable thing to warn about, and it is invisible if you only evaluate the filesystem in isolation. The label is not saying "this might eat your data." It is saying "you are off the main road, and the main road is better maintained."

The actual lesson, which is not about filesystems

I very nearly made a storage decision for a system that holds real data based on a confident guess I had not checked, about a warning that was trying to tell me something specific. The warning was right there. I almost read past it because I already knew the answer.

That is the failure worth writing down. Being wrong about btrfs cost me nothing; almost skipping the warning because my prior was strong could have cost me a storage layout I would have lived with for years. Strong priors are exactly the ones that stop you reading. The installer was not being conservative; it was being precise, and precision aimed at a person who is sure they already understand looks identical to noise.

I picked ZFS in the end, for reasons that have nothing to do with btrfs being bad and everything to do with wanting the filesystem the platform treats as its home turf, especially for the replication and snapshot workflows my cluster leans on daily. But the decision is not the point. The point is that I made it after reading the warning and understanding it, instead of after assuming I already had.

The next time a mature tool wears a cautious label, the useful question is not "is this label out of date?" It is "what specifically is this label pointing at that I have not looked at yet?" Usually it is pointing at something real, and usually it took someone a bad week to earn the right to warn you.

iLO 3 in 2026: the complete quirks catalog

Kingsley Kanu — Thu, 22 Oct 2026 08:00:00 +0000

My oldest server is an HP ProLiant from the G7 era, which means its lights-out management is iLO 3, whose final firmware shipped in 2020. The chip still does everything a baseboard management controller should: remote console, virtual media, power control, sensor readings. What it no longer does is speak to 2026 software without a fight.

Each fight has a fix. If you run old ProLiant iron, this list is several evenings of your life handed back.

The web UI only speaks TLS 1.0

Every current browser refuses TLS 1.0 by default, so the iLO web page simply does not load, with an error that suggests the device is broken rather than old. It is not broken. In Firefox, about:config, set security.tls.version.min to 1, and the UI appears. I keep one browser profile with that override as the designated old-iron profile, so my daily browser keeps its modern settings.

Do not bother with the embedded System Management Homepage link on the SNMP page, either. It points at a host-side agent that does not exist for modern Linux and never will. Dead surface, ignore it.

SSH requires an archaeology kit

A modern OpenSSH client rejects every algorithm the iLO offers. The connection needs a stanza of legacy allowances in ~/.ssh/config:

KexAlgorithms +diffie-hellman-group1-sha1,diffie-hellman-group14-sha1
HostKeyAlgorithms +ssh-rsa,ssh-dss
PubkeyAcceptedAlgorithms +ssh-rsa,ssh-dss
Ciphers +aes128-cbc,3des-cbc
MACs +hmac-sha1

Scope it to the iLO's host entry only. These algorithms are deprecated for good reasons; the trick is confining the time machine to the one device that needs it.

Key auth: DSA only, and only via a side door

This one cost me a real evening. iLO 3's final firmware accepts exactly one public key type: DSA 1024. RSA 2048, RSA 4096, with comment, without comment, re-wrapped in every plausible format, all return the same unhelpful "invalid input." Generating a DSA key in 2026 requires asking OpenSSH nicely, since it stopped doing so by default years ago.

Then comes uploading it. The web UI's paste box and the official HTTPS upload path both fail from modern machines, because current TLS stacks refuse the iLO's ancient handshake before the upload can even start. The path that works is delightfully sideways: SSH into the iLO with password auth, and tell it to fetch the key file itself over plain HTTP from a little web server you stand up on the same network. The BMC's own HTTP client has no objection to its era's protocols. One fetch, key installed, passwordless ever since.

Changing the IP: the silence after

You can change the iLO's address in-band from the host, no reboot needed. What nobody tells you: iLO 3 ships with gratuitous ARP disabled, so after the change it does not announce its new binding to the network. Everything keeps trying to reach it where it used to be, and the device appears to have died. It has not. Issue a BMC cold reset (from the host: ipmitool mc reset cold), the network stack re-binds and finally speaks up, and the new address works. The gap between "I changed the IP" and "I found the cold reset" is an unpleasant stretch of believing you bricked your only out-of-band path.

The one setting you must never touch

Somewhere in the security pages sits a FIPS mode toggle. On iLO 3, enabling it performs a factory reset as a feature: license key, user accounts, network configuration, identity, time zone, every customization, gone in one click, and the option looks exactly like something a diligent admin would enable while hardening. Do not. On firmware this old, FIPS mode is a self-destruct button wearing a compliance costume. It is the only setting on the device I treat as radioactive, and it has a permanent warning in my runbook.

Worth knowing, not fighting

An Advanced license unlocks virtual media and the full remote console, which is what turns the old BMC from a status page into an actual recovery path; licenses for hardware this old are effectively free on the second-hand market. IPMI over LAN works fine once enabled, and is the cleanest programmatic surface left on the device, with the caveat that its port belongs on a management network you trust, because the protocol's security model predates the modern internet's manners.

Why bother

Every quirk above has the same root cause: the device froze in time and the world kept moving. The fixes are all on our side of the connection, because the iLO's side stopped changing in 2020. That is the actual definition of running end-of-life infrastructure: nothing is broken, but every integration is your job now.

Is it worth it? The server cost less than the drives inside it, the BMC still boots it from an ISO over the network from the couch, and the whole catalog above is one afternoon of configuration once you know it. Knowing it is the hard part. Now you do.

I powered down a node on purpose. Ninety seconds later its workload was serving again.

Kingsley Kanu — Tue, 20 Oct 2026 08:00:00 +0000

There is a category of claim that infrastructure marketing loves: the workload just moves. Node dies, service continues, nobody notices. I wanted to know what that actually looks like on hardware I own, so I picked a production container, powered off the node it was running on, and started a stopwatch.

Ninety seconds later the container was up and serving on the other node. No commands from me after the poweroff. Most of those ninety seconds went somewhere specific, and almost none of the work that made them possible happened that afternoon.

The timeline

At T+0 I issued a clean poweroff on the active node. The container went down with it, and the survivors noticed the silence.

Around T+30, the cluster finished deciding who was in charge. Cluster managers run their own election: the resource manager that decides where high-availability workloads run has a master, and if the master was on the dead node, a new one steps up. Half a minute of that window is bureaucracy, and it is bureaucracy that prevents two nodes both deciding to start the same workload, which would be far worse than waiting.

Around T+90, the container was booted and serving on the surviving node, started from a local copy of its disk that replication had delivered minutes earlier.

The three boring pieces underneath

Each second in that timeline was bought in advance by a specific, unexciting decision.

Replication was already flowing. Every few minutes, an incremental snapshot of the container's disk ships to the partner node. Incremental is the operative word: after the first full send, each cycle moves only changed blocks, which for a small service is seconds of transfer. When the node died, the survivor was holding a disk image at most a few minutes stale. This is the piece people skip, and without it "failover" means "restore from last night's backup," which is a different sport. It stands on the identical-pool-names convention I wrote about last week; replication lands cleanly because the destination looks exactly like the source.

The workload was allowed to fail over to a prepared node only. High-availability groups have a setting that restricts a workload to nodes you list. That reads like a limitation. It is actually the safety: the listed nodes are the ones receiving replication. Without the restriction, the cluster is free to restart the container on a node that has no copy of its disk, and the recovery becomes a hostage negotiation. Restricting placement means the automatic decision is always a correct decision.

The container was small enough to boot fast. Sixty of the ninety seconds are just a Linux container starting. A lean single-purpose container boots in a minute; a bloated one takes five. Failover time is mostly boot time, so every service you keep small is failover you keep fast.

The honest fine print

The replicated copy is minutes stale, so a real crash loses whatever changed inside the replication window. For my workloads, stateless services and things whose state lives in databases elsewhere, that window is acceptable. For a primary database it would not be, and I do not run one this way.

One workaround worth recording: a container originally created on a different storage type cannot be live-migrated onto the replicated ZFS layout. The path that works is backup and restore: take a backup, restore it targeting the right storage, and let replication take over from there. Ten minutes of one-time surgery per guest, and only because those guests predated the storage design.

And the test itself matters more than the setup. Untested failover is a rumor. The first deliberate poweroff is genuinely uncomfortable, which is exactly the argument for doing it on a Tuesday afternoon you chose rather than a night you did not.

Ninety seconds is not luck. It is a naming convention, a replication schedule, a placement restriction, and a small container, each doing its boring job at the same time.

Where my secrets live, and what the repo gets instead

Kingsley Kanu — Thu, 15 Oct 2026 09:00:00 +0000

A repository is a very good place to lose a secret. It is copied to every clone, mirrored to every remote, kept forever, and read by tools that never ask whether a line was meant to be private. So the rule I work to is simple: the repo holds the shape of the configuration, never the values.

What goes where

Secrets live in a password manager I host myself. Nothing else is the source of truth. A database password, an API token or a private key exists in exactly one place, and everything else refers to it.

The repository carries a template instead. Each project has an example environment file with the same variable names as the real one and placeholder values that mean nothing. Anyone reading the repo can see what has to be supplied. Nobody reading it can see what was supplied.

The real values reach a machine at deploy time, through an environment file that is created on the target and never committed. The build and deploy scripts read from the environment. They do not contain a value to fall back on, because a default password in a script is a committed password with extra steps.

Three habits that do the real work

Look at what is staged before every commit. git diff --cached takes seconds, and it catches the file that was only supposed to be in the working tree for a minute.
Make the ignore file say it first. The real environment file is ignored from the first commit of a project, before it exists, so there is never a moment when adding everything picks it up.
Scan automatically as well. A secret scanner running as a pre-commit hook, gitleaks being one example, catches the cases where attention was elsewhere. It is a second net, not a replacement for the first.

When one gets committed anyway

Deleting it in the next commit does not help. The old commit still has it, every clone still has it, and any mirror may already have copied it. Once a secret has been committed it is compromised, and the fix is to rotate it at the source: issue a new one, change what uses it, revoke the old one. Cleaning the history is worth doing afterwards, but it is tidying, not remediation.

That ordering matters because the instinct goes the other way. Removing the file feels like the fix, and rotating the secret feels like admitting something happened. Rotating is the one that actually closes the exposure.

Why bother in a homelab

It is tempting to treat a private repo on my own network as safe enough. I do not, because a habit that only switches on when the stakes are high will not be there on the day they are, and the repo that is private today gets a second remote, a collaborator or a mirror tomorrow. Keeping values out of git from the start costs almost nothing. Retrofitting it later costs a rotation of everything that was ever in there.

The cluster joined fine. Its heartbeat was on the wrong network.

Kingsley Kanu — Thu, 15 Oct 2026 08:00:00 +0000

The day I formed my three-node Proxmox cluster, the create went cleanly, the first join went cleanly, and the cluster reported quorate. Every green light a person could ask for. It took a closer look at the config to notice that one node's cluster heartbeat was running over the wrong network entirely, and that nothing anywhere was going to warn me about it.

The one-flag mistake

Forming a Proxmox cluster is two commands. On the first node:

pvecm create clustername

And on each joining node:

pvecm add <address-of-first-node>

That second command took my instruction about which node to join, and quietly made its own decision about which of the joining node's addresses to register as the cluster link. My nodes are multi-homed: each has an address on the server network, where cluster traffic belongs, and another on a management network that exists for admin access. I pointed pvecm add at the first node's server-network address and assumed the joining node would present its matching interface.

It presented the management interface instead. The flag I did not pass is --link0, which pins which local address the node registers for corosync, the protocol that carries cluster membership and configuration consensus. Without it, the node picks by its own resolution logic, and what it picks is whatever your naming and routing happen to serve up, which on a multi-homed box is a coin toss you did not know was being flipped.

The result: two nodes heartbeating on the server network, one heartbeating from its management address, and a cluster that reports healthy because packets do, in fact, arrive. For now.

Why "it works" was the dangerous part

If the wrong link had failed to connect, I would have caught it in the first minute. Instead everything worked, because the management network happened to route to the others. The problem was latent, and it had teeth:

The management network is not built for cluster traffic. It is the network I take down, re-firewall, and generally treat as a maintenance surface. Corosync is the one protocol that must never flap: if a node loses heartbeat for long enough, the cluster considers it dead, and with high availability enabled, a node that loses quorum will fence itself, a deliberate hard reset to protect shared state. A firewall change on a network I believed was "just admin access" could have rebooted a production node.

That failure would have arrived months later, disconnected from its cause, on an afternoon when I touched something apparently unrelated. The bugs that pass every test and then sit waiting are the worst kind in infrastructure.

The fix: editing the cluster's own consensus file

The corosync config lives in the cluster filesystem, and you fix a wrong ring address by editing it directly, carefully:

cp /etc/pve/corosync.conf /root/corosync.conf.backup
nano /etc/pve/corosync.conf

Inside, each node has a ring0_addr. I corrected the wrong one to the node's server-network address, and, critically, incremented config_version in the totem block. That version bump is how corosync knows this is a newer config to converge on rather than a conflicting one to argue about. Skip it and nodes can reject the change or, worse, split over it.

Then restart corosync on every node, one at a time, watching quorum recover between each:

systemctl restart corosync
pvecm status

Total fix time, once understood: about ten minutes. The cluster never lost quorum during the rolling restart, and the ring has lived on the correct network since.

What I actually learned

The command that joins a node to a cluster takes a flag that pins which network the cluster's nervous system runs on. On any multi-homed node, that flag is not optional politeness; it is the difference between choosing your topology and being assigned one.

So the join I run now, every time, looks like this:

pvecm add <first-node> --link0 <this-node's-cluster-address>

And the check after any join, no matter how green the output: read the config back and confirm every ring0_addr is an address you chose.

grep ring0_addr /etc/pve/corosync.conf

Green lights tell you packets arrived. They do not tell you the packets took the road you intended. On the systems that fence themselves when the road washes out, you check the road.

Every request to my site redirected to itself

Kingsley Kanu — Tue, 13 Oct 2026 09:00:00 +0000

My home cloud sits behind a Cloudflare Tunnel, and the tunnel hands traffic to a Traefik reverse proxy. The day I wired the two together, the page would not load. The browser said it had been redirected too many times.

Nothing was down and every component answered. That made it slow to find: each piece was working, and the loop only existed in the way they were joined.

What the loop looked like

A redirect loop is a conversation where each side keeps giving the same answer. The first thing to establish is who is answering. The cheapest way to find out is to skip the browser and ask the origin directly, with a header that says which site you want:

curl -sI http://<origin>/ -H "Host: <your-hostname>"

The answer was not the page. It was a permanent redirect whose destination was the HTTPS version of the same hostname. The origin was telling the caller to go somewhere that, once it went through the tunnel again, led straight back to the same origin.

Why it looped

Traefik had a redirect from HTTP to HTTPS configured on its port 80 entrypoint. That setting applies at the entrypoint, before any router is matched. So it does not matter what a router would have done with a request: if the request arrives on port 80, it is redirected.

My tunnel route pointed at the origin over plain HTTP. The sequence went like this:

A visitor opens the site over HTTPS. Cloudflare terminates that connection.
Cloudflare sends the request down the tunnel, and cloudflared forwards it to Traefik on port 80, in plain HTTP.
Traefik answers with a redirect to the HTTPS address of the same hostname.
The browser follows it, goes back through Cloudflare, and arrives at step 2 again.

The visitor's connection was secure the whole time. The hop that mattered was the last one, between the tunnel and the proxy, and on that hop the redirect was an instruction to repeat the request forever.

What fixed it

Three changes, none of them clever:

A real certificate for the public hostname on Traefik, issued with the DNS challenge, so the proxy could serve HTTPS for that name.
The tunnel route moved to the HTTPS port, so requests never touch the redirecting entrypoint at all.
One line of resolver config on the cloudflared container. cloudflared checks the origin certificate against the hostname in the route URL. The newer Cloudflare dashboard no longer exposes the origin server name or a skip-verification option on routes, so I made the container resolve the public hostname to the proxy's internal address with an extra_hosts entry. The certificate name then matches the URL, and the traffic stays inside the house. Every public app added to that tunnel later needs the same one line.

What I check first now

Does the origin answer with a redirect? One curl -sI with a Host header, bypassing the browser and the tunnel.
Where is that redirect defined? In Traefik it can sit on the entrypoint or on a router, and the entrypoint wins.
Is the hop into the origin HTTP or HTTPS? A redirect to HTTPS is right for a browser talking to the proxy and wrong for a tunnel that already terminated TLS upstream.

Same configuration, two meanings of "HTTP". The proxy was doing its job, and so was the tunnel. They just disagreed about which end was the front door.

The boring naming decision that unlocks cluster replication

Kingsley Kanu — Tue, 13 Oct 2026 08:00:00 +0000

When I built the storage pools on my cluster nodes, the most consequential decision was not the drive, the layout, or any of the tuning options. It was the name. Both pools are called tank. Not tank1 and tank2, not a name derived from each node's hostname, not anything descriptive. The same word on every node.

Every instinct from general software engineering says this is wrong. Names should be unique. Names should tell you which thing you are looking at. Two different pools on two different machines with the same name feels like a collision waiting to happen.

Those instincts are correct for identifiers that live in one namespace. They are exactly backwards for identifiers that define equivalent slots on interchangeable machines, and a cluster's storage is the second kind.

Why the same name is the whole feature

Proxmox replication and migration move a guest's disk from a pool on one node to a pool on another. The guest's configuration names its storage once: this container's root disk lives on storage tank-local, which maps to pool tank. When the guest moves, or when its replica activates on the other node, the configuration does not change. It says tank, the receiving node has a tank, everything resolves.

Name the pools tank1 and tank2 instead and every movement between nodes becomes a translation problem. The replication has a working source and a mismatched destination. The config that says tank1 lands on a node that only knows tank2. Tools that assume the name resolves identically everywhere, which is most of them, either refuse or need per-move hand-holding. You have taken a single-command operation and given it a lookup table to maintain forever.

The rule generalizes beyond storage. Any time machines are supposed to be interchangeable, the resources that make them interchangeable must be indistinguishable. Same pool name, same dataset structure underneath it, compatible layout, same platform version. The node's own hostname is where individuality belongs; the slots a workload plugs into should be identical everywhere, like electrical sockets.

What it looks like when it works

With both pools named tank and replication scheduled, moving a container between nodes is one command with no storage arguments at all. The failover case is even better: the cluster's HA layer restarts a guest on the surviving node from the replicated copy without consulting anyone about where the disk lives. It knows where the disk lives. It lives on tank, the same as everywhere.

I tested that failover deliberately, powered a node down and watched the workload come back on its partner in about ninety seconds. That test, and the timeline of what happens in those ninety seconds, is next week's story. The point for today is that none of it works without the naming convention underneath.

Renaming later is the expensive path

If you get this wrong at creation, fixing it is not a rename in the friendly sense. A ZFS pool's name is baked into the mounting structure, the hypervisor's storage registrations, every guest config referencing it, and any replication chains already flowing. Renaming means exporting the pool, re-importing under a new name, and re-pointing everything that referenced the old one, with replication state broken and rebuilt from scratch. On a live node that is an afternoon of careful work in the best case.

Thirty seconds of deciding "every pool is called tank" at build time buys out all of that. Thirty seconds against an afternoon of careful work is not a close call.

The pool build itself, drive selection through the option-by-option rationale, is in Tuesday's post if you missed it. Together they are two halves of one idea: build each node's storage identically, and the cluster stops being three computers and starts being one system.

What the Proxmox command line does that the web UI hides

Kingsley Kanu — Thu, 08 Oct 2026 08:00:00 +0000

The Proxmox web interface is genuinely good, which is a problem, because it is good enough that you can run a cluster for a year and never learn that a second, more capable interface sits underneath it. The two command-line tools, one for containers and one for virtual machines, do everything the web UI does and a set of things it does not expose at all. The gap between them is where a lot of "you cannot do that in Proxmox" beliefs actually live.

Several things became possible the day I stopped treating the CLI as a door marked advanced users only.

Batch operations the UI makes you do one click at a time

The web UI is built around one guest at a time: select it, act on it, repeat. When you need to do the same thing to fifteen containers, that is fifteen rounds of the same clicks, and by the tenth you will make a mistake.

The command line takes a list. Restarting or reconfiguring a whole group of guests is a single loop, and a loop does not get bored on the fourteenth item. Anything you would dread doing by hand across many guests, the CLI turns into one line you can read, check, and then run once.

Configuration keys the UI never surfaces

Every guest's real configuration is a plain text file, and the CLI reads and writes it directly. The web UI shows the common subset of settings; it does not show all of them. There are behaviors, boot and startup ordering, specific device and passthrough options, particular flags, that exist as configuration keys but have no checkbox anywhere in the interface.

If you have ever read a forum answer that says "add this line to the guest config" and wondered why the setting was not in the UI, this is why. The setting was always there. The UI simply curates which settings it presents, and the CLI is the uncurated view.

Scripting, which changes what a homelab even is

This is the real unlock. A tool that takes a list and reads and writes plain text is a tool you can put in a script. Once guest management is scriptable, a class of things becomes possible that a point-and-click interface structurally cannot offer: create-from-template flows tuned exactly to your setup, scheduled maintenance actions, reproducible builds where the definition of a guest lives in a file in a git repository rather than in a series of clicks nobody wrote down.

That last shift is the important one. Clicking through the UI produces a running guest and no record of how you made it. A script produces the same guest and is itself the record. The difference shows up the day you have to rebuild, and it is the difference between remembering and re-deriving.

When to use which, honestly

None of this means abandon the web UI. For looking at things, the console, the graphs, the current state of the cluster, the UI is faster and better, and I use it constantly. For a one-off change to a single guest, clicking is genuinely quicker than typing.

The CLI earns its place the moment the work is repetitive, the moment the setting you need is not on screen, and the moment you want the action to be something you can save, review, and run again. Those three moments are exactly where the UI stops helping and, if you do not know the CLI exists, exactly where you conclude the platform cannot do the thing.

It can. It just does not always advertise it. Once you know the command line is there, the list of things you believe Proxmox cannot do gets a lot shorter.

A sub-100-dollar ZFS data tier from a used office PC

Kingsley Kanu — Fri, 18 Sep 2026 10:35:25 +0000

My cluster's newer nodes are small-form-factor office machines, the kind that companies lease for three years and then sell by the pallet. They came with one NVMe boot drive and an empty 2.5 inch SATA bay. Filling that empty bay turned each node into something with a real storage tier, for less than the price of a restaurant dinner.

The recipe is short: a used enterprise SATA SSD in the spare bay, ZFS on top of it, and a handful of pool options that people usually copy from a forum thread without knowing why. The options are the interesting part.

The drive: used enterprise, not new consumer

The drive I put in each node is an Intel S3520 class enterprise SSD. Used, off the second-hand market, for roughly the cost of a new budget consumer drive.

The case for used enterprise over new consumer at the same price is straightforward. Enterprise SATA drives have power-loss protection capacitors, which matter more than any other single feature when the filesystem on top is ZFS and the workload is VM disks. They publish honest endurance ratings, and a lightly used one typically arrives with a few percent of its rated writes consumed. A new consumer drive at the same price has none of the capacitors and a fraction of the endurance.

Before trusting any second-hand drive, read its own account of its history:

smartctl -i -A /dev/sdX | grep -E 'Model|Serial|Reallocated|Power_On_Hours|Wear'

What you want: zero reallocated sectors, wear levelling well under 80 percent used, and a power-on-hours figure consistent with the seller's story. A drive that fails any of those is a drive someone else should own.

The pool: every option earns its place

One drive, one pool. Creation looks like this:

zpool create -o ashift=12 \
  -O compression=lz4 \
  -O atime=off \
  -O xattr=sa \
  -O acltype=posixacl \
  -O dnodesize=auto \
  -O normalization=formD \
  tank /dev/disk/by-id/ata-MODEL_SERIAL

Two things before the option list. First, the pool is built on the /dev/disk/by-id/ path, never /dev/sda. Device letters are assigned by enumeration order and change when you add a disk; by-id paths are welded to the drive's serial number and survive anything short of the drive dying. Second, the pool is named tank on every node in the cluster, identically. That naming choice looks cosmetic. It is what makes replication and migration between nodes work as single commands. It deserves its own write-up, and it gets one on Thursday.

The options, and what each one buys:

ashift=12 tells ZFS the drive writes in 4K blocks. SSDs lie about their sector size for compatibility; taking the lie at face value costs write amplification forever, because ashift is fixed at creation.
compression=lz4 is close to free on any modern CPU and typically returns 1.5 to 2 times effective space on VM workloads, which are full of text, logs, and zero pages. There is no realistic workload where turning it off wins.
atime=off stops the filesystem updating a timestamp on every read. Without it, reading a file costs a write. Nothing on a VM data tier cares about access times.
xattr=sa stores extended attributes inside the inode instead of in hidden directories, one lookup instead of two. Containers touch xattrs constantly.
acltype=posixacl gives containers the ACL semantics they expect. Some distributions inside containers misbehave subtly without it, and subtle misbehaviour is the worst kind.
dnodesize=auto lets metadata-heavy datasets grow their dnodes instead of spilling attributes elsewhere. Pairs with xattr=sa.
normalization=formD makes the filesystem treat Unicode names consistently, so you cannot end up with two files whose names look identical but differ in bytes. You only hit this once, but you remember it.

Then register the pool with the hypervisor:

pvesm add zfspool tank-local --pool tank --content rootdir,images --sparse 1

The storage ID tank-local and the pool name tank are deliberately different names for different layers. --sparse 1 makes volumes thin, so a 32 GB container that holds 4 GB of data consumes 4 GB.

What this tier is for, and what it is not for

With one of these in each node, containers and VMs live on local ZFS with lz4 behind them, snapshots are instant, and scheduled replication can keep a copy of a guest's disk on the other node a few minutes stale. That replication is the foundation the cluster's failover story is built on, and both of the next two posts in this series stand on it.

What a single-drive pool does not give you is redundancy. One drive is one failure domain; ZFS will detect corruption but has no second copy to heal from. My answer to that is nightly backups to a dedicated backup server plus the cross-node replication, which together bound any loss to a small window. A mirror would be better and the bay count says no. So the redundancy lives in the backup schedule instead of in the second bay I do not have.

Total cost per node: one used enterprise SSD, somewhere between 30 and 50 dollars depending on the week. Capability added: snapshots, compression, replication, and a place for guests to live that is not the boot drive. It is the best price-to-capability ratio anywhere in my rack.

The full node build procedure, from bare hardware to cluster member, lives in my runbooks. This drive-and-pool step is section five of it, and it is the section I would keep if I had to throw away the rest.

The one SSH command Windows doesn't ship

Kingsley Kanu — Thu, 03 Sep 2026 08:00:00 +0000

Every time a fresh machine comes up on my network, the first thing I want is my laptop's key in its authorized_keys file. On Linux or macOS that is one command you have known for years:

ssh-copy-id root@192.0.2.20

On Windows, from PowerShell, that command does not exist. Not "is not on your PATH." Does not exist.

ssh-copy-id : The term 'ssh-copy-id' is not recognized as the name of a cmdlet,
function, script file, or operable program.

Windows has shipped a real OpenSSH client since Windows 10, and it is genuinely good. Here is everything in it:

scp.exe   sftp.exe   ssh.exe   ssh-add.exe   ssh-agent.exe
ssh-keygen.exe   ssh-keyscan.exe   ssh-pkcs11-helper.exe   ssh-sk-helper.exe

Nine binaries. No ssh-copy-id. The reason is that ssh-copy-id was never a binary. It is a POSIX shell script that ships alongside OpenSSH, and Microsoft's port compiles the C programs rather than carrying the shell scripts. So the omission is structural rather than an oversight, which also means it is not going to appear in a future update.

Do the same job in one line

The script does something simple: read your public key, connect, create the directory, append the key, set the permissions. You can do all of it inline.

Get-Content "$env:USERPROFILE\.ssh\id_ed25519.pub" | ssh root@192.0.2.20 `
  "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

You type the password once, in your own terminal, and never again for that host.

Two Windows-specific things that trip people up in that line. The path is $env:USERPROFILE\.ssh\, because ~/.ssh is not a thing PowerShell expands the way a POSIX shell does. And the backtick is PowerShell's line continuation character, not a backslash.

If you would rather not pipe text between a Windows program and a remote shell at all, this form avoids the whole class of line-ending and encoding questions by passing the key as an argument:

$key = (Get-Content "$env:USERPROFILE\.ssh\id_ed25519.pub" -Raw).Trim()
ssh root@192.0.2.20 "mkdir -p ~/.ssh && chmod 700 ~/.ssh && printf '%s\n' '$key' >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

This is the one I use, because it is explicit about the trailing newline instead of hoping the pipeline gets it right.

Or use the copy you already have

If you have Git for Windows installed, you already have the real script. Git Bash bundles the MSYS2 build of OpenSSH, which includes the shell scripts the Windows port leaves out:

$ command -v ssh-copy-id
/usr/bin/ssh-copy-id

So open Git Bash instead of PowerShell and ssh-copy-id root@192.0.2.20 works exactly as it does on Linux. Two different SSH clients live on the same machine, one of which has the command and one of which does not, which is a good thing to know before you spend twenty minutes deciding your installation is broken.

When the key lands and login still asks for a password

This is the failure that actually costs time, because everything looks correct. The key is in the file, the file is on the server, and the server still wants a password.

Almost always it is permissions. sshd runs with StrictModes on by default, and it will refuse a key file that anyone other than the owner can write to. That includes the home directory itself, not just .ssh.

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod go-w ~            # the one people miss

sshd will not tell you this on the client side. It just falls through to the next authentication method, which looks identical to the key not being there. The server knows exactly what happened:

tail /var/log/auth.log
Authentication refused: bad ownership or modes for directory /root

Run ssh -v on the client and you will see it offer the key and get nothing back, which at least tells you the problem is on the far end.

The other common cause is having several keys. The client offers them in order, and if the server's MaxAuthTries limit is reached before it gets to the right one, you are refused for having too many keys rather than the wrong one. Pin it explicitly:

Host newnode
    HostName 192.0.2.20
    User root
    IdentityFile ~/.ssh/id_ed25519
    IdentitiesOnly yes

IdentitiesOnly yes is the important line. Without it, the client will still offer everything in your agent regardless of what you specified.

Why I bother with any of this

The point is not saving keystrokes. It is that the password gets typed exactly once, locally, by me, on the machine it belongs to. It never goes into a note, a chat window, a script or a config file. After that first connection the credential for that host is a key on my laptop protected by the operating system, and adding a new machine to my network is a thirty second job rather than a decision about where to write a password down.

That is a small habit that compounds. Every host I have built in the last year got exactly this treatment on its first boot, and I have never had to go looking for one of their passwords, because for most of them I do not remember what it was.

A note on the addresses

Addresses in this post use the range reserved for documentation under RFC 5737.

Resources

Microsoft OpenSSH for Windows documentation - what ships and what does not
sshd_config StrictModes - why permissions silently break key auth
More homelab write-ups: iamkay.eu/blog

I moved a staging environment off the cloud in one evening

Kingsley Kanu — Mon, 31 Aug 2026 20:00:00 +0000

Assembyl is a multi-tenant application I work on. Four services out of one monorepo: an API, an admin interface, a tenant-facing web app, and a public landing page. Until recently the staging environment for all of that ran on managed cloud, with a hosted CI service doing build and test on every push, then building containers and deploying them to a serverless runtime, against a paid database tier.

It worked fine. It also billed every month for an environment nobody touches outside weekday afternoons, while I had a Proxmox cluster in my house with a self-hosted platform layer already running on it, next to my file storage and my git server. The hardware was bought. The electricity was already being spent. Staging was paying rent somewhere else for no reason I could defend.

The complication was that staging had to stay multi-tenant. Every tenant gets their own subdomain. So the job was never "deploy an app." It was "deploy a routing pattern that survives adding the twentieth tenant without twenty pieces of manual work."

What I tried first, and why it failed

The natural structure is the one everybody reaches for:

{tenant}.staging.example.com

One application, one wildcard domain *.staging.example.com, one mental model. I set up the whole path for it: internal DNS, a catch-all route on the reverse proxy, the tunnel configuration, the certificate request. All of it landed cleanly.

Then I tested from outside the network and got a TLS handshake failure.

*.staging.example.com is a two-level-deep wildcard, because the zone apex is example.com. Cloudflare's free Universal SSL covers the apex and one level below it. It does not cover a wildcard two labels deep. That needs Advanced Certificate Manager, which is a paid add-on at roughly ten dollars a month.

I was doing this migration specifically to stop paying a monthly bill. Trading one subscription for a smaller subscription is not a migration, it is a discount. So that was out.

The other option I considered seriously was dropping public access entirely and serving staging over a private network overlay, with the certificate issued through a DNS challenge and no public ingress at all. That works, and it costs nothing. It also means every person who wants to look at staging has to install a VPN client first. For an engineering-only environment that is a fine trade. For something where the point is that a colleague or a prospective customer can open a link in whatever browser they have, it breaks the use case entirely.

The fix was to flatten the namespace

Keep the wildcard one level deep by moving the tenant name into the label rather than in front of it:

{tenant}-as.example.com

The -as suffix acts as the namespace. Same *.example.com certificate that Universal SSL already issues for free, covering every tenant that will ever exist. No paid tier, no VPN requirement, no per-tenant certificate.

Two concessions came with it, and both are worth stating plainly rather than glossing.

Each tenant needs its own DNS record. Wildcard DNS records have to have the wildcard at the start of a label, so *-as.example.com is not a thing you can create. That is a per-tenant API call, which is automatable, but it is not zero.

And tenant names become visible in DNS. Anyone enumerating the zone's records can see who the tenants are. For a staging environment full of test accounts that is acceptable. For production it would not be, and production would need a different pattern.

In exchange: one certificate, one routing rule, and adding a tenant never touches the proxy configuration at all.

One routing rule for every tenant

http:
  routers:
    tenants:
      rule: "HostRegexp(`^[a-z0-9][a-z0-9-]*-as\\.example\\.com$`)"
      entryPoints:
        - websecure
      tls:
        certResolver: letsencrypt
        domains:
          - main: "*.example.com"
      service: tenants-svc
      middlewares:
        - security-headers@file

  services:
    tenants-svc:
      loadBalancer:
        servers:
          - url: "http://192.0.2.80:80"
        passHostHeader: true

Every tenant name matches that regex. Every request lands on the platform proxy, which routes by Host header to the right container. passHostHeader is the load-bearing line: drop it and every tenant arrives at the application looking like the same host, which in a multi-tenant app is a genuinely bad afternoon.

Warm the certificate before anyone arrives

Issuing a wildcard certificate through a DNS challenge takes about thirty seconds. If the first request that triggers issuance is a real user's, that user gets thirty seconds of nothing, or a timeout. So trigger it yourself with a request for a name that does not exist yet:

curl -sk -H "Host: prewarm-as.example.com" \
  https://192.0.2.10/ -o /dev/null -w "HTTP %{http_code}\n"

That matches the regex, which triggers the certificate request, which issues and caches the wildcard. Every subsequent tenant deploy has zero wait. It costs one throwaway request and it is the difference between a launch that looks instant and one that looks broken.

What the migration actually looked like

Four applications on the self-hosted platform layer, each pointing at a different directory of the same monorepo, each with its own domain and its own build. All four went from "create application" to serving in about fifty minutes. The CI moved to a self-hosted runner on the same cluster, which had already replaced the hosted CI service earlier that week.

New recurring cost for the staging environment: nothing. The hardware was already there, already powered, already backed up nightly as part of everything else.

Why this holds up

The free tier of a CDN is genuinely the right scale for a staging environment. Tunnelled ingress plus free certificates handles public access with no port forwarding and no exposed origin. The one-level wildcard limit is only a problem if you insist on fighting it. Flatten the namespace and it stops existing.

A regex-matched catch-all route scales to any number of tenants in one block of configuration. The proxy is the piece you least want to be editing under time pressure, and this pattern means you never edit it after the first time.

Running the platform layer in its own container on the cluster keeps it from becoming a shared fate with everything else. It gets its own backup unit, its own resource ceiling, its own restart. When it needs attention, nothing else is affected.

What I would do differently

Use the flat pattern from the beginning instead of arriving at it after ninety minutes of debugging a TLS failure. The certificate limitation is documented; I just did not go looking until something broke.

Wire the deploy webhook before the first deploy rather than after. I spent the first stretch clicking Deploy by hand, which is fine once and irritating by the fifth time.

Add health check endpoints to the services before deploying rather than afterwards. Two of the four apps sat in an unknown health state for days purely because nothing was answering a probe, and an unknown state is worse than a red one because it is easy to ignore.

The one thing I deliberately have not moved is the production database, which is still on a managed tier. The homelab is not on battery backup yet, and a primary database is the wrong place to find out how your house handles a power cut. Staging can afford to learn that lesson. Production cannot.

A note on the addresses

Domains and addresses in this post are documentation placeholders on reserved ranges (RFC 5737 and RFC 2606). The architecture is exactly what runs; the specific names are stand-ins.

Resources

Coolify documentation - the self-hosted platform layer
Traefik HostRegexp matcher - the catch-all rule
Cloudflare Tunnel - ingress with no open ports
Let's Encrypt rate limits - pre-warming once is well inside them
More homelab write-ups: iamkay.eu/blog