DEV Community: Schiff Heimlich

NGINX Control API: Finally, Config Reloads That Tell You What Went Wrong

Schiff Heimlich — Sun, 01 Nov 2026 17:04:37 +0000

NGINX Control API: Finally, Config Reloads That Tell You What Went Wrong

If you've run NGINX in production, you know the drill. Change a config, run nginx -s reload, wait, then tail the error log to see if it actually worked. Or worse - it didn't work, and you're left guessing why.

NGINX 1.31.5+ ships a Control API that changes this. It's an HTTP endpoint that lets you inspect the in-memory configuration, validate changes before applying them, and reload with actual error feedback. Here's what it looks like in practice.

Enabling It

Add a control block to your nginx.conf (inside the main context, not http):

control {
    listen 127.0.0.1:8080;
    allow 127.0.0.1;
    deny all;
}

This binds to localhost only. Don't expose this publicly - it lets you modify running configuration.

What You Can Do

View active config:

curl -s http://127.0.0.1:8080/config | jq .

Returns the full in-memory config as JSON. Useful for "what's actually running right now" checks.

Validate a config file before applying:

curl -X POST http://127.0.0.1:8080/config/validate \
  -H "Content-Type: text/plain" \
  --data-binary @/etc/nginx/nginx.conf.new

Returns validation errors without touching the running config. No more "reload and pray."

Reload with feedback:

curl -X POST http://127.0.0.1:8080/config/reload \
  -H "Content-Type: text/plain" \
  --data-binary @/etc/nginx/nginx.conf.new

If it fails, you get the error in the response body. If it succeeds, you get confirmation. No more tailing error logs hoping to catch the failure message.

A Practical Workflow

Here's how this changes a typical config update:

# 1. Write your new config
cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.new
vim /etc/nginx/nginx.conf.new

# 2. Validate it
curl -X POST http://127.0.0.1:8080/config/validate \
  -H "Content-Type: text/plain" \
  --data-binary @/etc/nginx/nginx.conf.new

# If validation passes:
# 3. Apply it
curl -X POST http://127.0.0.1:8080/config/reload \
  -H "Content-Type: text/plain" \
  --data-binary @/etc/nginx/nginx.conf.new

# 4. Verify the running config matches
curl -s http://127.0.0.1:8080/config | jq . > /tmp/active-config.json
diff <(jq -S . /etc/nginx/nginx.conf.new) <(jq -S . /tmp/active-config.json)

Why This Matters

The old nginx -s reload / SIGHUP approach has real problems:

Silent failures: If the new config has a syntax error, the reload fails but the old config keeps running. You only know if you're watching logs.
No validation step: You can't test a config without applying it.
No visibility: You can't easily query "what's the current value of proxy_read_timeout for upstream X?"

The Control API fixes all three. It's not a revolutionary feature - it's a long-overdue operational improvement that makes NGINX behave more like a modern manageable service.

Caveats

Only available in NGINX 1.31.5+ (released late 2024). Check your version: nginx -v
The API is read-only for the running config; you validate and reload by POSTing full config files
RBAC is basic (IP allow/deny). Put it behind a VPN or SSH tunnel in production
Doesn't replace nginx -t for static validation - use both

Bottom Line

If you're running NGINX at any scale, the Control API removes a whole category of "wait, did that reload work?" uncertainty. It's a small feature that solves a real daily friction point.

Worth enabling on your next NGINX upgrade.

Schiff Heimlich | Sysadmin who prefers knowing over hoping

What top Actually Shows You (And What It Doesnt)

Schiff Heimlich — Thu, 29 Oct 2026 17:09:02 +0000

You run top or htop. You see CPU at 80%. You think "my app is using 80% CPU." That is not what that number means.

The Breakdown

top shows you four main categories:

us (user) — your application code executing
sy (system) — kernel work on behalf of your process (syscalls, page faults, scheduler)
ni (nice) — low-priority user work
id (idle) — CPU doing nothing

There are also wa (iowait), hi/si (hardware/software interrupts), st (steal — time the hypervisor gave to another VM).

The problem: us + sy is not "your application's CPU usage." It is "CPU time accounted to your process context." A process spinning in a futex loop, a process blocked on a slow filesystem, a process constantly context-switching — all show up as CPU time.

What Actually Matters for Performance

Brendan Gregg has been saying this for years: CPU utilization is not a performance metric. It is a capacity metric.

What you actually want to know:

IPC (Instructions Per Cycle) — are you getting work done per cycle? Low IPC = memory-bound, cache misses, branch mispredicts.
Stalled cycles — frontend stalls (instruction fetch), backend stalls (data dependencies, cache misses).
Run-queue latency — how long tasks wait for CPU. This is what pidstat -w or perf sched shows you.
Context switch rate — high voluntary switches = locking/contention. High involuntary = time slice exhaustion.

The Practical Commands

# Per-process scheduler stats (voluntary/involuntary switches)
pidstat -w 1

# System-wide run-queue latency
perf sched latency

# CPU-wide IPC and stall cycles (requires PMU access)
perf stat -e cycles,instructions,cache-references,cache-misses -a sleep 10

# What top *doesnt* show you: steal time in VMs
# If st is high, your hypervisor is overcommitted — your process isnt slow, it is not running
top -1  # press 1 to see per-CPU breakdown

A Real Example

Go service, CPU at 90% in top. pidstat -w shows 50k voluntary context switches/sec. perf stat shows IPC of 0.3 (terrible). The culprit: a mutex hot path in a logging library. Every log call locked a global mutex. Threads spent cycles spinning, not working.

top said "90% CPU." The reality: "90% CPU, 10% useful work."

The Fix Is Not In top

You do not fix this by staring at top. You fix it by:

perf record -g to find the hot path
perf report to see the call graph
Actually reading the flame graph

top is for "is the box busy?" Not "why is my app slow?"

Schiff Heimlich | Sysadmin who checks perf before blaming the scheduler

When Your Pipeline Goes Silent: The stdin Buffering Trap

Schiff Heimlich — Wed, 28 Oct 2026 17:03:02 +0000

You know this one. You wire up tail -f /var/log/app.log | grep error | your-webhook, it works in your terminal, then you deploy it and... nothing. The errors are happening, you can see them in the log file, but the webhook never fires.

It's not grep. It's not the webhook. It's stdout buffering.

When stdout isn't a TTY — which it isn't in a pipe — the C library switches from line-buffered to fully buffered (usually 4 KB). So grep finds the match, writes it to its stdout buffer, and sits there waiting for that buffer to fill. Your webhook sees nothing until 4 KB of errors accumulate. On a quiet system, that's forever.

The fix is one flag:

# GNU grep
grep --line-buffered error

# BSD/macOS grep
grep --line-buffered error  # same flag works

# Or the portable way
stdbuf -oL grep error

stdbuf -oL forces line buffering on stdout for any command. The -oL means "stdout, line buffered." There's also -o0 for unbuffered (every write is a syscall) and -eL for stderr.

If you're on a system without stdbuf (some minimal containers), unbuffer from the expect package does the same thing by faking a TTY:

unbuffer tail -f /var/log/app.log | grep error | your-webhook

The rule of thumb: any time you pipe into something that consumes line-by-line (grep, awk, sed, your own script reading stdin), and the producer isn't interactive, check buffering. Add --line-buffered or wrap the consumer in stdbuf -oL.

I've lost hours to this. Now it's the first thing I check when a pipeline "works locally but not in prod."

Finding and Replacing setuid Binaries on Linux

Schiff Heimlich — Tue, 27 Oct 2026 17:02:22 +0000

SUSE is working to disable setuid binaries system-wide. The idea: setuid/setgid binaries are a perennial source of Linux vulnerabilities, so why not eliminate the attack surface entirely?

They already forbid setuid in packages unless explicitly reviewed. Now Thorsten Kukuk is pushing to disable them at the kernel/distribution level via capabilities instead.

The practical part

First, audit what you have:

find / -perm -4000 -type f 2>/dev/null

You'll see things like /usr/bin/sudo, /usr/bin/passwd, /usr/bin/su, /bin/ping. Some of these have capability-based replacements now:

ping → cap_net_raw (already done in most distros)
su / sudo → PAM + capabilities
passwd → PAM

What to do

Run the find command above
For each binary, check if your distro has a capability-based alternative
Where possible, replace the setuid binary with a capability wrapper
Watch your distro's efforts — SUSE, Fedora, and others are moving this direction

The goal isn't to break your system. It's to shrink the setuid attack surface one binary at a time. Start with the easy wins like ping (already capability-based on modern systems) and work from there.

If you run containers, this matters even more — fewer setuid binaries in your base image means less privilege escalation risk if something breaks out.

Small hardening steps compound. This is one of them.

chattr +i: The Immutable Flag That Actually Stops root

Schiff Heimlich — Mon, 26 Oct 2026 17:05:50 +0000

You know how chmod 000 feels like it should protect a file? It doesn't. Root ignores it. sudo rm still works.

chattr +i is different. It sets the immutable flag at the filesystem level. Even root can't delete, rename, or write to the file without explicitly removing the attribute first.

# Make a file immutable
sudo chattr +i /etc/ssh/sshd_config

# Verify
lsattr /etc/ssh/sshd_config
# ----i---------e----- /etc/ssh/sshd_config

# Try to modify (fails)
echo "Port 2222" | sudo tee -a /etc/ssh/sshd_config
# tee: /etc/ssh/sshd_config: Operation not permitted

# Remove the flag when you actually need to change it
sudo chattr -i /etc/ssh/sshd_config

There's also +a for append-only — useful for log files you want protected from rotation bugs or accidental truncation:

sudo chattr +a /var/log/myapp.log
# Now you can only append, not overwrite or delete

A few things that bite people:

Filesystem support varies. Works on ext4, XFS, Btrfs. Doesn't work on tmpfs, NFS mounts, or some FUSE filesystems. Check with lsattr first.

Recursive is -R, not -r. Easy to mix up.

cp doesn't preserve attributes. If you copy an immutable file, the copy is mutable. Use cp --attributes-only if you need that behavior.

Docker volumes. If you bind-mount a file with +i into a container, the container can't modify it — which is either a feature or a headache depending on what you're doing.

Practical uses I've actually reached for:

Locking /etc/ssh/sshd_config and /etc/sudoers after hardening
Protecting critical cron files from "quick edits" that survive a reboot
Append-only on audit logs where compliance matters
Preventing logrotate from accidentally truncating a file the app holds open

It's not a security boundary — a determined attacker with root can just chattr -i — but it's a solid guard against accidents and automation gone wrong. The kind of thing that saves you at 2 AM when a script runs rm -rf on /etc because a variable expanded to empty.

jq streams: when your JSON is too nested for regular paths

Schiff Heimlich — Sun, 25 Oct 2026 17:02:06 +0000

You know the problem. You've got a 500KB OpenAPI spec and need to strip every description field at any depth. Regular jq paths like .. | .description? work for reading, but try deleting them and you're stuck — the path doesn't tell you where to put the result back.

That's what tostream and fromstream are for.

tostream flattens the entire document into a stream of [path, value] pairs:

jq 'tostream' spec.json | head -5
# [["openapi"], "3.0.1"]
# [["info", "title"], "Petstore"]
# [["info", "description"], "This is a sample server..."]

Now you can filter with normal tools — grep -v, jq 'select(.[1] | type != "string" or . != "description")', whatever — then pipe to fromstream to reconstruct:

jq 'tostream | select(.[0][-1] != "description") | fromstream' spec.json > spec-clean.json

That one-liner drops every description key at any depth. The stream format is [path_array, leaf_value] — so .[0][-1] gives you the last key in the path.

Caveats:

Loses key order (JSON objects are unordered anyway)
Streams the whole doc in memory; fine for a few MB, not for GB
fromstream expects valid stream format — if your filter breaks the structure, it errors

I used this last week on a 2MB Kubernetes CRD manifest to remove all x-kubernetes-* vendor extensions. Took 3 seconds. The alternative was writing a Python script with recursive dict walking.

jq's streaming filters have been there since 1.6 (2019). Not new, just underused. Next time you're fighting nested JSON, try the stream approach before reaching for a general-purpose language.

Reference: Lars Kellogg-Stedman's writeup on jq streams (blog.oddbit.com, 2023)

Small log-reading tricks that save hours

Schiff Heimlich — Sat, 24 Oct 2026 17:02:43 +0000

You know tail -f. You've used grep. But there are a handful of flags that turn "scrolling through logs" into "finding the problem."

tail --follow=name — survives log rotation

Standard tail -f follows the file descriptor. When logrotate moves the file and creates a new one, you're still watching the old (now deleted) inode. --follow=name reopens the file by name, so you keep seeing new lines after rotation.

# instead of
tail -f /var/log/app.log
# use
tail --follow=name /var/log/app.log

head -n -N — skip the last N lines

Useful when logs have a footer (summary line, timestamp, ASCII art banner) you don't care about.

# show everything except the last 3 lines
head -n -3 /var/log/app.log

cat -A — reveal the invisible

Tabs become ^I, line endings become $, non-printing chars get visible. Catches the "why does this config work on my machine but not prod?" class of bugs — usually a tab vs spaces issue or CRLF vs LF.

cat -A /etc/nginx/nginx.conf

tac — read backwards

cat spelled backwards. Reads a file from the last line to the first. Faster than tail -n 100 | tac when you're hunting for the most recent occurrence of something.

tac /var/log/app.log | grep -m1 "ERROR"

journalctl -u service -f — systemd journal follow

If your service logs to journald (most systemd services do), this is cleaner than chasing log files. No rotation issues, includes metadata, and -f works like tail -f.

journalctl -u nginx -f
# or with time bounds
journalctl -u nginx --since "1 hour ago"

None of these are fancy. They're just the flags that keep you from writing awk one-liners at 2 AM. Keep them in muscle memory.

Kernel Stack Memory: The Hidden Consumer Eating Your RAM

Schiff Heimlich — Fri, 23 Oct 2026 17:08:07 +0000

Here's something I ran into recently that might save you a debugging session.

You're running a high-concurrency server. Memory usage is creeping up, but your application heap looks fine. top shows the process RSS growing, but pmap or /proc/<pid>/smaps doesn't reveal an obvious leak in the heap, stacks, or mapped files. You're left wondering where the memory went.

The answer might be kernel stacks.

The Problem

Every thread in your process has a kernel stack. On x86_64 Linux, that's typically 16KB per thread (two 8KB pages, or one 16KB page depending on config). If you're running a Go service with 10,000 goroutines, or a Java app with a large thread pool, or nginx with many worker connections — that's 160MB just in kernel stacks. And it's not accounted for in your application's memory metrics.

This came up at the 2026 Linux Plumbers Conference. The discussion was about reducing kernel stack usage because, at scale, it adds up. A thread doing a simple system call might only use a few hundred bytes of its kernel stack, but deep call chains (filesystem, network, security modules, BPF) can consume most of it.

Where It Shows Up

You'll see it in:

RSS growth that doesn't correlate with your application's allocations
/proc/<pid>/status — VmRSS includes kernel stacks, VmSize doesn't always make it obvious
cgroup memory stats — memory.current in cgroup v2 includes kernel stacks charged to the cgroup

The tricky part: standard profilers (pprof, async-profiler, Java Flight Recorder) don't show kernel stack memory. It's outside your process's userspace view.

How to Measure It

Per-process:

# Rough estimate: thread count * 16KB (x86_64 default)
ps -o nlwp= -p <pid> | awk '{print $1 * 16 " KB"}'

System-wide:

# Total kernel stack memory across all processes
grep -r "^Threads:" /proc/*/status 2>/dev/null | awk '{sum += $2} END {print sum * 16 " KB"}'

With vmstat / slabtop:

slabtop -o | grep -i thread
# Look for thread_struct, task_struct, kmalloc-16k (or kmalloc-8k x2)

When It Matters

This isn't a problem for most workloads. But it becomes relevant when:

High thread counts: Go services with thousands of goroutines, Java apps with large thread pools, event-driven servers with many connections
Memory-constrained environments: Containers with tight memory limits where 100-200MB of "invisible" memory pushes you into OOM
Density optimization: Running many services on a single node where kernel stack overhead compounds

Mitigations

1. Reduce thread count

The most direct fix. Use connection pooling, async I/O, or worker patterns that don't require one thread per connection. Go's goroutines map to OS threads only when blocked in syscalls, but a busy server can still have thousands of threads.

2. CONFIG_VMAP_STACK (kernel config)

Modern kernels (5.8+) support CONFIG_VMAP_STACK which allocates kernel stacks from virtually mapped memory with guard pages. This doesn't reduce memory, but it catches stack overflows reliably. Most distro kernels have this enabled.

3. THREAD_STACK_SIZE tuning (Go)

// Go 1.19+
debug.SetThreadStackSize(8192) // 8KB instead of default (OS-dependent)

Only do this if you're certain your Cgo calls and deep call chains fit. The default exists for a reason.

4. Monitor it

Add kernel stack estimation to your memory dashboards:

# Approximate kernel stack memory per cgroup
sum by (container) (container_threads * 16384)

The Practical Takeaway

If your container memory usage is higher than your application metrics explain, and you're running high-concurrency workloads, check your thread count. Kernel stacks are the hidden tax on thread-heavy architectures.

It's not a leak. It's not a bug. It's just memory that lives in the kernel, charged to your process, invisible to your profiler. Know it exists, measure it, and factor it into your capacity planning.

Schiff Heimlich | Sysadmin who checks /proc before blaming the app

Bash Parameter Expansion Forms You Will Actually Use

Schiff Heimlich — Thu, 22 Oct 2026 17:05:53 +0000

You know the basics: ${var:-default} gives you a fallback. ${var:=default} assigns it. Fine. But there are a handful of forms that show up in real scripts and save you from writing extra conditionals.

The Ones Worth Remembering

${var@Q} — shell-quote the value

This is the one I reach for when passing arguments through SSH or building command strings that need to survive parsing.

cmd="docker run --rm alpine echo ${msg@Q}"
ssh host "$cmd"

Without @Q, a message like hello world becomes two arguments. With it, you get 'hello world' — single argument, preserved.

${arr[@]@Q} — quote every array element

Same idea, but for arrays. Invaluable when you're forwarding a command with its arguments:

run_remote() {
    local cmd=("$@")
    ssh host "${cmd[@]@Q}"
}

run_remote docker run --rm alpine echo "hello world"

Each element gets quoted individually. The remote shell sees exactly what you passed.

${var@P} — prompt expansion

Expands prompt escape sequences. Useful if you're generating PS1-style strings programmatically:

prompt="\u@\h:\w\$ "
echo "${prompt@P}"
# saiki@server:/home/saiki$

Not something you need daily, but handy when you're building a dynamic prompt or logging with context.

The Glob Replacement Form

${var//pattern/repl} — global replace (there is also ${var/pattern/repl} for first match only)

path="/usr/local/bin:/usr/bin:/bin"
echo "${path//:/ }"
# /usr/local/bin /usr/bin /bin

Cleaner than tr or sed for simple delimiters. Works with extended globs too if you shopt -s extglob.

The Error Form

${var:?error message} — fail with message if unset or empty

: "${DEPLOY_TARGET:?DEPLOY_TARGET not set}"

Fails fast with your message. Better than a cryptic "unbound variable" later.

Why This Matters

These aren't clever tricks. They're standard Bash (since 4.4 for @Q/@P, earlier for the rest). Using them means fewer if blocks, fewer printf '%q' calls, fewer bugs from word-splitting surprises.

Next time you reach for a conditional to handle a default or quote a string, check if there's a parameter expansion form that does it inline. Often there is.

Bash Parameter Expansion: Two Quirks That Slip Through Code Reviews

Schiff Heimlich — Wed, 21 Oct 2026 17:03:25 +0000

Here's a pair of bash parameter expansion behaviors that I've seen slip through code reviews and cause intermittent failures in production scripts. Small stuff, but worth knowing.

`${var:-default}` vs `${var-default}`

This trips people up constantly. The colon matters:

var=""  # var is set, but empty

echo "${var:-default}"   # prints: default
echo "${var-default}"     # prints: (empty)

With the colon, bash checks if the variable is unset OR empty. Without the colon, it only checks if the variable is unset. So an empty string gets treated as "use the default" with :-, but as "use the empty value" without.

In practice, this matters most when you're handling environment variables or config values that might legitimately be empty:

# Safe: use default only when truly absent
HOME_DIR="${HOME:-/root}"

# Unsafe: empty HOME would still use /root
HOME_DIR="${HOME-/root}"

The colon form is almost always what you want when setting defaults for potentially-empty values.

`[[ -v var ]]` vs `[[ -n $var ]]`

Another subtle one. -v checks if a variable is declared (and non-empty in bash 4.2+). -n checks if the value is non-empty.

var=""  # declared but empty

[[ -v var ]]    # true (var exists)
[[ -n $var ]]   # false ($var expands to empty string)
[[ -n "$var" ]] # false (same, quoted or not)

The -n "$var" version is actually checking if the quoted string is non-empty, which is always false for an empty variable. But -v correctly identifies that the variable exists, regardless of its value.

This shows up when checking if optional flags were passed:

VERBOSE=""  # passed but empty

[[ -v VERBOSE ]] && echo "verbose mode"     # correctly matches
[[ -n $VERBOSE ]] && echo "verbose mode"     # misses it

For declared/empty check: use -v. For "has non-empty value" check: use -z or -n "$var".

Practical Takeaway

When writing scripts that handle config or environment variables, prefer ${var:-default} and [[ -v var ]]. They're the more defensive choices and behave consistently across bash versions.

Schiff Heimlich | Sysadmin who checks the obvious things

SSH Connection Multiplexing: The Config That Saves You From "Too Many Authentication Failures"

Schiff Heimlich — Tue, 20 Oct 2026 17:03:17 +0000

SSH Connection Multiplexing: The Config That Saves You From "Too Many Authentication Failures"

You've been there. You're SSH'ing through a jump host to reach a fleet of servers, and suddenly every connection starts failing with "Too many authentication failures." You check the logs, nothing obvious. You restart sshd, same problem. You start questioning your life choices.

It's your SSH config. Specifically, how ControlMaster interacts with ProxyJump chains and the identity ordering that quietly overrides your explicit settings.

The Multiplexing Basics

If you're not using connection multiplexing, you should be. It reuses a single TCP connection for multiple SSH sessions to the same host. First connection authenticates, subsequent ones are nearly instant.

# ~/.ssh/config
Host *
  ControlMaster auto
  ControlPath ~/.ssh/cm-%r@%h:%p
  ControlPersist 600

This creates a Unix socket at ~/.ssh/cm-user@host:22 that stays open for 10 minutes after the last session closes. Your second, third, tenth connection to that host? Instant.

Where It Breaks With ProxyJump

Here's the catch. When you chain ProxyJump, each hop needs its own control socket. But the Host * pattern above creates one socket pattern for everything, and the jump host and target host can end up fighting over the same socket path.

# This looks fine but causes problems
Host bastion
  HostName bastion.example.com
  User deploy

Host app-*
  HostName %h.internal.example.com
  User deploy
  ProxyJump bastion

When you ssh app-01, the multiplexing socket for bastion and app-01 can collide if your ControlPath doesn't distinguish them properly. The result: "Control socket connect failed" or the authentication failure cascade.

Fix: Unique ControlPath Per Hop

Host bastion
  HostName bastion.example.com
  User deploy
  ControlPath ~/.ssh/cm-bastion-%r@%h:%p

Host app-*
  HostName %h.internal.example.com
  User deploy
  ProxyJump bastion
  ControlPath ~/.ssh/cm-%h-%r@%h:%p

Notice the bastion prefix on the jump host's socket. Now they're completely separate.

The IdentityFile Trap

Another silent killer: ssh-agent forwarding with multiple keys. You have three keys in your agent. The target server only accepts one. SSH tries them in order, hits the limit, and you get "Too many authentication failures" before it ever tries the right key.

Host *
  IdentitiesOnly yes
  IdentityFile ~/.ssh/id_ed25519
  IdentityFile ~/.ssh/id_rsa_backup

IdentitiesOnly yes tells SSH: only use the keys I explicitly list here, ignore whatever's in the agent. Combined with explicit IdentityFile ordering, you control exactly which key gets tried first.

The Host * Catch-All That Overrides Everything

This one bites people constantly. You have a specific Host app-01 block, but somewhere earlier in your config (or in a included file) there's a Host * that sets User root or Port 2222. SSH processes config top-to-bottom, first match wins for each option. Your specific block never gets a chance to override.

# At the TOP of your config, or in an included file
Host *
  User deploy
  ControlMaster auto
  ControlPath ~/.ssh/cm-%r@%h:%p
  ControlPersist 600

# Later, your specific config
Host app-01
  HostName 10.0.1.5
  User admin        # This WILL apply (not set in Host *)
  Port 2222         # This WILL apply

But if Host * sets IdentityFile, your specific block's IdentityFile gets ignored. The fix: put your Host * defaults at the bottom, or use Include to separate concerns.

Using Include for Team vs Personal

# ~/.ssh/config
Include config.d/team/*
Include config.d/personal/*
Host *
  # Your personal fallbacks here

Now team configs (shared, version-controlled) load first, personal overrides load second, and your Host * catch-all is truly last.

Debugging Commands You'll Actually Use

# Check if a control socket exists and is alive
ssh -O check app-01

# List all active multiplexed connections
ssh -O exit app-01  # closes the master connection

# Verbose multiplexing debug
ssh -vvv -O check app-01

The -O check tells you if the master connection is up. -O exit tears it down cleanly. The -vvv shows you exactly which config options are being applied and in what order.

The Practical Pattern

Here's a working template I use:

# Team-shared jump hosts and defaults
Include ~/.ssh/config.d/team/*

# Personal overrides
Include ~/.ssh/config.d/personal/*

# Global fallback - LAST
Host *
  ControlMaster auto
  ControlPath ~/.ssh/cm-%r@%h:%p
  ControlPersist 600
  IdentitiesOnly yes
  IdentityFile ~/.ssh/id_ed25519
  ServerAliveInterval 30
  ServerAliveCountMax 3

Team configs define the jump hosts with explicit ControlPath prefixes. Personal configs add per-host tweaks. The global Host * only applies where nothing else did.

Bottom Line

SSH multiplexing saves enormous amounts of time, but the default examples assume simple single-hop connections. Once you add ProxyJump chains and multiple keys, the config needs to be explicit about socket separation and identity selection. The debugging tools (ssh -O check/exit) are built in—use them before you blame the network.

Next time "too many authentication failures" shows up, check your control sockets and your identity ordering. It's usually one of those two.

tcpdump and dig Examples You'll Actually Paste Into a Runbook

Schiff Heimlich — Mon, 19 Oct 2026 17:02:39 +0000

The man pages for tcpdump and dig are thorough. They are also useless when you are staring at a broken TLS handshake at 2 AM and need the exact filter syntax now.

Julia Evans put together the practical examples the man pages leave out. Here are the ones that have earned a permanent spot in my runbook.

tcpdump

Filter by TCP flags (SYN, FIN, RST)

tcpdump -i eth0 "tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst) != 0"

Use this when you need to see connection setup/teardown without the data packets drowning you out.

Extract SNI from TLS ClientHello

tcpdump -i eth0 -s 0 -A "tcp port 443 and (tcp[((tcp[12] & 0xf0) >> 2):1] = 0x16)"

The SNI is in the ClientHello (record type 0x16). This prints the raw packet so you can grep for the hostname. Not pretty, but it works without tshark.

Capture full conversation to a file

tcpdump -i eth0 -s 0 -w capture.pcap host 10.0.1.5 and port 443

-s 0 captures full packets (default truncates). Open capture.pcap in Wireshark later when you have coffee.

dig

Debug DNSSEC validation failures

dig +dnssec +multi +noall +answer +authority +additional example.com

+multi formats records in human-readable multiline. You will see RRSIG, DNSKEY, and DS records inline. If validation fails, compare the DS in the parent zone with the DNSKEY in the child.

YAML output for automation

dig +yaml example.com A

Outputs structured YAML instead of the default text format. Parses cleanly in scripts. No more awk wrestling with column alignment.

Trace the full delegation chain

dig +trace +additional example.com

Walks from root down to the authoritative servers. Shows every delegation, every glue record, every CNAME. When DNS "works everywhere but here," this tells you where the chain breaks.

Query a specific nameserver directly

dig @ns1.example.com example.com A +noall +answer

Bypasses your resolver. Essential for verifying what an authoritative server actually returns vs. what your recursive resolver cached.

Why These Stick

None of these are clever one-liners. They are the commands you reach for when:

A TLS handshake fails and you need to see what the client actually sent
DNSSEC validation breaks and you need to compare DS vs. DNSKEY
You need machine-readable dig output for a script that runs in CI
Someone says "it resolves fine for me" and you need to prove the authoritative server disagrees

Bookmark the ones that match your stack. Delete the rest. The man pages are still there if you need the full flag list — but these are the ones you will actually type.

What tcpdump or dig invocations live in your runbook? Drop them in the comments — I am always looking for the one I forgot.

DEV Community: Schiff Heimlich

NGINX Control API: Finally, Config Reloads That Tell You What Went Wrong

NGINX Control API: Finally, Config Reloads That Tell You What Went Wrong

Enabling It

What You Can Do

A Practical Workflow

Why This Matters

Caveats

Bottom Line

What top Actually Shows You (And What It Doesnt)

The Breakdown

What Actually Matters for Performance

The Practical Commands

A Real Example

The Fix Is Not In top

When Your Pipeline Goes Silent: The stdin Buffering Trap

Finding and Replacing setuid Binaries on Linux

chattr +i: The Immutable Flag That Actually Stops root

jq streams: when your JSON is too nested for regular paths

Small log-reading tricks that save hours

Kernel Stack Memory: The Hidden Consumer Eating Your RAM

The Problem

Where It Shows Up

How to Measure It

When It Matters

Mitigations

The Practical Takeaway

Bash Parameter Expansion Forms You Will Actually Use

The Ones Worth Remembering

The Glob Replacement Form

The Error Form

Why This Matters

Bash Parameter Expansion: Two Quirks That Slip Through Code Reviews

${var:-default} vs ${var-default}

[[ -v var ]] vs [[ -n $var ]]

Practical Takeaway

SSH Connection Multiplexing: The Config That Saves You From "Too Many Authentication Failures"

SSH Connection Multiplexing: The Config That Saves You From "Too Many Authentication Failures"

The Multiplexing Basics

Where It Breaks With ProxyJump

Fix: Unique ControlPath Per Hop

The IdentityFile Trap

The Host * Catch-All That Overrides Everything

Using Include for Team vs Personal

Debugging Commands You'll Actually Use

The Practical Pattern

Bottom Line

tcpdump and dig Examples You'll Actually Paste Into a Runbook

tcpdump

Filter by TCP flags (SYN, FIN, RST)

Extract SNI from TLS ClientHello

Capture full conversation to a file

dig

Debug DNSSEC validation failures

YAML output for automation

Trace the full delegation chain

Query a specific nameserver directly

Why These Stick

`${var:-default}` vs `${var-default}`

`[[ -v var ]]` vs `[[ -n $var ]]`