DEV Community: speed engineer

Nagle vs. Delayed ACK: The 40ms Stall Hiding in Your TCP Stack

speed engineer — Mon, 02 Nov 2026 06:37:34 +0000

A service I debugged a while back had a strange latency histogram: a tight cluster at 2ms, dead air, then a second cluster sitting almost exactly at 40ms. Not a long tail — a second peak, like the request either finished fast or got teleported 40 milliseconds into the future. No GC pause showed up in the logs. No lock contention. No downstream timeout. The thread was just sitting in a recv() call, waiting for bytes that the other side had, by every appearance, already sent.

The problem

The client was doing something extremely common: writing a small header in one write() call and the body in a second write() call, on the same socket, in the same request. Maybe 20 bytes, then 200 bytes. Nothing about that looks wrong — plenty of serialization code is written exactly this way, two writes instead of one buffer. And most of the time it was fine. Then, under certain request shapes, every single request on a connection would eat a flat 40ms penalty, like a tax that only applied when the planets aligned.

Why it happens

Two optimizations, each sensible on its own, ruin each other when combined.

The sender's side has Nagle's algorithm, which exists to stop chatty applications from sending a flood of tiny TCP segments. The rule is simple: if you have unacknowledged data already in flight, don't send a new small segment — buffer it and wait for the pending ACK first. Reasonable. It keeps a dumb line-by-line telnet client from generating one packet per keystroke.

The receiver's side has delayed ACK, which exists to avoid acknowledging every single packet separately. Instead of ACKing immediately, the receiver waits a short window — on Linux, the delayed-ACK timer defaults to 40ms — hoping either more data arrives to ACK in bulk, or the receiving application sends a response it can piggyback the ACK onto.

Put those two together on the same connection and you get a deadlock, not a race:

Client sends the small header. Nagle lets this one through (nothing in flight yet).
Server receives it, has nothing to say back yet, and delays the ACK — waiting to see if more data or a response shows up to batch it with.
Client's second write() (the body) now has unacknowledged data outstanding (the header), so Nagle holds it, waiting for the ACK of the header.
Nobody sends anything. The server is waiting to batch an ACK. The client is waiting for that ACK before it'll send more.

Nothing is actually broken — both sides are doing exactly what their algorithm says to do. The deadlock only breaks when the delayed-ACK timer finally fires and flushes the ACK on its own, which is precisely why the stall is a suspiciously flat, constant number: it's not network latency, not scheduling jitter, not a retry backoff curve. It's a timer value. On Linux that's commonly 40ms; other stacks and configurations use different defaults (historically up to 200-500ms on some systems), which is why "weirdly consistent latency spike that's suspiciously round" is the tell to look for.

This is why it's so easy to miss in isolation: a single write() call per request never triggers it. A request/response protocol that always sends one full message per flush never triggers it. It specifically needs split writes on the sending side and "I'll respond in a moment" behavior on the receiving side — which describes an uncomfortable amount of hand-rolled RPC and proxy code.

What to do about it

The fix that actually addresses the cause: stop doing split writes for things that belong together. Buffer the header and body into one write (or use writev/scatter-gather I/O) so there's only ever one segment per logical message. This removes the setup for the deadlock entirely, on both sides, without touching socket options.

The fix that's more commonly reached for: disable Nagle on the sender.

int flag = 1;
setsockopt(sock, IPPROTO_TCP, TCP_NODELAY, &flag, sizeof(flag));

sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

This is safe for request/response protocols where you control the write pattern, but it's a blunt instrument — it means every small write becomes its own packet, which can matter if you're chatty at high connection counts. The receiver-side equivalent, TCP_QUICKACK on Linux, disables delayed ACK but has to be re-armed after every recv() call since the kernel resets it — which is why most people fix this from the sender side instead.

Most modern HTTP clients, gRPC implementations, and database drivers already set TCP_NODELAY by default, which is why you mostly don't see this anymore in off-the-shelf stacks. It resurfaces in exactly three places: hand-rolled binary protocols, language runtimes or socket libraries where someone changed the default, and anything that proxies traffic through a layer that creates a fresh socket and doesn't carry the option forward.

Key takeaways

A flat, suspiciously round latency spike (40ms is the classic Linux signature) is a strong hint you're looking at a timer firing, not real network variance — go check delayed ACK before you chase anything else.
Nagle (sender) and delayed ACK (receiver) are each reasonable in isolation and can deadlock each other when a request is split across multiple small writes with a "respond in a moment" receiver.
The root-cause fix is coalescing small writes into one segment; TCP_NODELAY is the common workaround and is safe for most request/response code, but understand what you're disabling.
Check this before blaming the network, the GC, or "flaky infra" for latency spikes that are oddly consistent in magnitude.

False Sharing: Why Your Lock-Free Counter Is Slower Than a Mutex

speed engineer — Sun, 25 Oct 2026 04:40:31 +0000

A team I was pairing with had "fixed" a hot path by replacing a mutex-protected counter with four separate atomic counters — one per worker thread — summed up on read. No lock contention, right? Throughput dropped by roughly 6x under load, worse than the mutex version it replaced.

Nothing in the code was wrong. The four atomic counters were correctly implemented, lock-free, and uncontended at the logical level. The CPU didn't see it that way.

The problem

Nothing in the code was wrong, and yet the "lock-free" rewrite was dramatically slower than the lock it replaced. That contradiction is the whole point of this post.

Why it happens

Modern CPUs don't move memory in single bytes or even single words — they move it in fixed-size cache lines, almost always 64 bytes on current x86 and ARM hardware. When four 8-byte counters sit next to each other in an array or a struct, all four fit inside one or two cache lines.

Here's the part that bites people: cache coherence protocols (MESI and its variants) track ownership of a cache line, not of the individual variable inside it. When core 0 writes to counter[0], it has to take exclusive ownership of that cache line — even though cores 1-3 only care about counter[1], counter[2], and counter[3]. The moment core 1 writes to counter[1], it has to invalidate core 0's copy and pull the line into its own cache. Then core 2 does the same. Then core 0 writes again and has to re-fetch.

The four threads never touch each other's data. They are still fighting over the same 64 bytes of silicon, every single write, because the coherence protocol can't see "variable" — it only sees "cache line." This is false sharing: contention introduced by physical memory layout, invisible at the source-code level, and it gets worse the more cores you throw at the problem, which is exactly backwards from what you want out of a "fix."

The slowdown is not subtle. Depending on core topology and how many sockets are involved, published benchmarks (and the ones I've personally run) put false-sharing penalties anywhere from 4x to 20x+ versus the padded version — because every write now costs a cross-core cache-line transfer (tens to low hundreds of nanoseconds) instead of a local L1 hit (~1 ns).

What to do about it

The fix is almost insultingly simple once you know what you're looking for: pad or align the hot fields so each one owns its own cache line.

// Before: all four counters likely share 1-2 cache lines
struct Counters {
    long count[4];
};

// After: each counter gets its own 64-byte line
struct PaddedCounter {
    long count;
    char pad[64 - sizeof(long)];
};
struct PaddedCounter counters[4];

Most languages give you a primitive for this instead of hand-rolled padding:

Java: @jdk.internal.vm.annotation.Contended (or the older sun.misc.Contended)
C++: alignas(64) on the struct or field
Go: the runtime pads several internal sync primitives for this exact reason; for your own hot structs, add explicit _ [N]byte padding fields
Rust: #[repr(align(64))]

The second, less obvious trap: this isn't limited to arrays of counters. It happens any time a frequently-written field sits in the same struct as a frequently-read-by-other-threads field. A lastUpdated timestamp written on every request, sitting right next to a config pointer that every thread reads — same bug, different shape. Before reaching for sharding or lock-free structures as a performance fix, check whether you've actually solved contention or just relocated it one layer down, into the cache-coherence fabric.

Key takeaways

Cache coherence tracks ownership at the cache-line level (typically 64 bytes), not the variable level — "no shared variables" doesn't mean "no contention."
False sharing turns logically-independent, lock-free writes into real cross-core traffic, and it scales worse with more cores, not better.
The fix is padding/alignment to give hot fields their own cache line — a one-line change once you know to look for it.
Profile with hardware counters (perf c2c on Linux is purpose-built for this) before assuming a concurrency bug is a logic bug. Sometimes the bug is in the memory layout, not the algorithm.

Exponential Backoff Isn't Enough: The Retry Storm Hiding in Your Recovery Path

speed engineer — Sun, 18 Oct 2026 06:28:48 +0000

The problem

A dependency blips for 400ms — a deploy, a GC pause, a brief network partition, doesn't matter which. Every client with an in-flight request fails at roughly the same instant. They all do the responsible thing: back off exponentially and retry. And yet the service doesn't recover cleanly. It gets hit by a second, sharper outage a few seconds later, caused entirely by the retries that were supposed to be polite about coming back.

I've watched this pattern take down a freshly-recovered service harder than the original blip did — not once, but as a repeating pattern, every few seconds, for the better part of a minute, while everyone stared at dashboards wondering why a "fixed" service kept re-breaking.

Why it happens

Exponential backoff fixes one problem and leaves another completely untouched. The problem it fixes: a client that fails shouldn't hammer you immediately and repeatedly — waiting longer between attempts reduces total load. The problem it doesn't fix: synchronization. If 2,000 clients all fail at t=0, and every one of them runs the identical formula delay = min(cap, base * 2^attempt), every one of them computes the identical delay. They don't spread out. They retry in lockstep.

Walk through it with base=1s, cap=32s, 2,000 clients that all fail at t=0:

Attempt 1: all 2,000 retry at t=1s, simultaneously. The service, mid-recovery, can't absorb a 2,000-request spike. It fails again.
Attempt 2: all 2,000 back off to 2s and retry at t=3s, simultaneously. Same result.
Attempt 3: 4s delay, retry at t=7s. Attempt 4: 8s delay, retry at t=15s. Attempt 5: 16s delay, retry at t=31s.

Five attempts later, you've had five synchronized full-fleet spikes, each one arriving exactly when the service was most fragile — right after the previous spike knocked it back down. The growing delay buys you less frequent storms, not smaller ones. Every single storm is still the full 2,000 requests, because nothing in the formula ever desynchronizes the clients from each other. Pure exponential backoff makes the disease chronic instead of acute.

What to do about it

The fix is jitter — randomizing the delay instead of computing it deterministically — but which kind of jitter matters more than people assume:

No jitter (what's above): everyone collides, forever, on a slowly growing clock.
Equal jitter (delay = cap/2 + random(0, cap/2)): spreads clients across half the window. Better, but still guarantees a minimum synchronized floor every round — half the fleet is still packed into the back half of the window.
Full jitter (delay = random(0, cap)): each client's retry lands uniformly anywhere in the window, including near zero. With 2,000 clients spread across attempt 3's 4-second window instead of landing on one instant, you get roughly 500 requests/second of smooth, absorbable load instead of 2,000 at once — and the spread widens geometrically with every attempt, so the tail trickles rather than detonates.
Decorrelated jitter (delay = min(cap, random(base, previous_delay * 3))): approximates full jitter's spread without needing to track attempt count, which is convenient in stateless retry wrappers.

Full jitter is the one to default to unless you have a specific reason not to — it minimizes both total requests sent and the time until the last straggler succeeds, because it never re-synchronizes clients that failed together. The implementation cost is one extra random() call; there's no excuse for skipping it once you know the failure mode exists.

One more lever worth pairing with jitter: cap the maximum number of retry attempts and fail fast past it, so a client that still can't get through after, say, 5–6 attempts gives up instead of circling forever at the cap value, quietly contributing to a permanent low-grade storm.

Key takeaways

Exponential backoff alone does not break synchronization between clients that failed at the same time — it just lowers the frequency of full-strength collisions, not their size.
"Jitter" is not one thing. No jitter collides every time; equal jitter still guarantees a synchronized floor; full jitter is the one that actually desynchronizes the fleet.
Default to full jitter (random(0, min(cap, base * 2^attempt))) unless you've measured a reason to deviate.
If your service has ever had a second, sharper outage right after recovering from the first one, check your clients' retry logic before you look anywhere else — it's a textbook symptom.

Coordinated Omission: Why Your Load Test's p99 Is Lying to You

speed engineer — Sun, 11 Oct 2026 06:56:57 +0000

The problem

Your load test says p99 latency is 12ms. You ship it. Three weeks later, real users are complaining about multi-second hangs, and your dashboards — fed by the same percentile math — say everything's fine. This isn't a monitoring gap. It's a measurement bias baked into almost every load-testing tool you've ever used, and it has a name: coordinated omission.

Here's the setup that causes it. Most load generators work in a closed loop: send a request, wait for the response, record the latency, send the next request. That's how ab, naive wrk scripts, and a lot of homegrown load test harnesses work by default. It feels correct — you're measuring exactly how long each request took.

Why it happens

The problem is what happens during a stall. Say your service has a GC pause, a lock contention spike, or a downstream timeout that freezes request processing for 1 second. In a closed-loop generator targeting 100 req/s, during that 1-second stall you issue exactly one request — the one that's already in flight. It comes back slow, and you record one bad sample: 1000ms.

But that's not what your real traffic does. Real clients don't politely wait their turn. If your service normally handles 100 req/s, a 1-second stall means roughly 100 requests should have arrived and are now queued up behind the one you're measuring. Your closed-loop tool never issued them — it was still blocked waiting on the first one. Those 99 "missing" requests, each of which would have experienced close to a full second of delay, simply never get sampled. They're omitted from your percentile calculation, and the omission is coordinated with the stall itself — it happens exactly when the result would've been ugliest.

The term comes from Gil Tene (Azul Systems, creator of HdrHistogram), originally describing how JVM pause measurement tools lied about GC impact. It generalizes to any closed-loop benchmark of any system: HTTP APIs, databases, queues, LLM inference endpoints, all of it.

The numbers get dramatic fast. Correcting a closed-loop measurement for omission — recomputing what the actual arrival-rate-based percentiles would have been — routinely turns a reported p99 in the single-digit milliseconds into a real p99 north of several hundred milliseconds to a full second. The pain isn't a rounding error. It's an order-of-magnitude lie, concentrated exactly in the tail you care about.

What to do about it

Three concrete fixes, in order of effort:

Use an open-loop (or corrected) load generator. Tools like wrk2, k6 (with arrival-rate executors), Gatling, and Locust's constant-arrival-rate shape issue requests on a fixed schedule regardless of whether the previous one has returned. That matches how production traffic actually behaves — users don't queue politely behind your slow request.
Record "intended" vs. "actual" send time, and correct for it. If you can't switch tools, at minimum log when a request should have been sent versus when it was sent, and compute corrected percentiles. HdrHistogram has built-in support for this via its coordinated-omission correction methods.
Treat any closed-loop percentile number as a lower bound, never a ceiling. If your load test says p99 is 12ms, the honest statement is "p99 is at least 12ms under ideal, non-stalled conditions" — not "p99 is 12ms." That reframing alone changes how much you should trust a green load test before a launch.

This matters more, not less, in the agentic/LLM era: inference endpoints have wildly variable per-request latency (cache hits vs. cold starts, short vs. long generations), which means stalls are frequent and your closed-loop load test is actively hiding your worst-case behavior right when you need to see it.

Key takeaways

Closed-loop load generators under-sample exactly the requests that would show your worst latency — the omission is coordinated with the stall, not random.
A reported p99 from a closed-loop tool is a floor, not a real number. The true tail can be an order of magnitude worse.
Switch to open-loop / constant-arrival-rate tools (wrk2, k6 arrival-rate, Gatling, Locust) or correct for omission with HdrHistogram.
This bites hardest on systems with high latency variance — which includes almost every LLM-backed service shipping today.

The Utilization Cliff: Why 70% CPU Is Fine and 90% Falls Over

speed engineer — Sun, 04 Oct 2026 05:36:13 +0000

The problem

A service sits at 70% CPU utilization for months. p99 latency is a boring, flat 40ms. Someone decides 70% feels too close to the edge, "let's alert at 80% so we have room to react." Three weeks later the alert fires almost every afternoon during normal traffic peaks, pages people at 2am for nothing, and everyone starts snoozing it. Then one day utilization crosses 92% during a traffic bump that's maybe 15% above normal — not a spike, not an outage, just a slightly busier Tuesday — and p99 latency goes from 40ms to 800ms in about three minutes. Nobody changed anything. The system didn't get "a bit worse." It fell off a cliff.

This isn't a story about a bad on-call culture or a noisy alert. It's a story about treating utilization as a linear dial when it behaves nothing like one.

Why it happens

Model any single resource — a CPU core, a connection pool, a database — as a queue with one server and random arrivals (the standard M/M/1 queueing model). The expected time a request spends waiting in that queue, relative to how long it takes to actually get served, is:

wait multiplier = ρ / (1 − ρ)

where ρ (rho) is utilization, between 0 and 1.

That formula is a hyperbola, not a line. Here's what it actually produces:

Utilization (ρ)	Wait multiplier
50%	1.0x
60%	1.5x
70%	2.33x
80%	4.0x
90%	9.0x
95%	19.0x
99%	99.0x

Going from 70% to 90% utilization — a 20-point jump that sounds like "a bit more load" — multiplies queueing delay by roughly 4x (2.33x → 9x). Going from 90% to 95% does almost as much damage again. The denominator (1 − ρ) is heading to zero, and dividing by a number approaching zero is the entire story.

This is also why the damage looks instantaneous. Request arrivals aren't a smooth, even stream — they're bursty, roughly Poisson-distributed. Near saturation, a small burst has nowhere to drain to before the next one lands, so the queue compounds instead of recovering. Below ~70% utilization there's enough slack to absorb bursts between requests. Above ~90%, there isn't, so the same burst that used to be invisible now stacks on top of an already-full queue.

CPU percentage is just a visible proxy for ρ. The real quantity that matters is headroom: 1 − ρ. A dashboard showing "92% CPU" is really telling you your headroom shrank from 30% to 8% — a number that explains the multiplier a lot better than "92%" does on its own.

What to do about it

First, stop alerting on a flat utilization percentage. It mixes up two very different signals: "I'm busy" and "I'm about to fall off the curve." Alert on the thing you actually care about instead — p99 latency, queue depth, or request wait time directly — and treat utilization as a leading indicator you watch on a dashboard, not a page trigger.

Second, if you do keep a utilization-based alert (plenty of teams reasonably want an early-warning signal before latency visibly degrades), pick the threshold from the curve, not from a feeling. For latency-sensitive paths, the curve is already steep by 70% and brutal by 85%. "80% feels conservative" is backwards — 80% is already in the zone where a 10-point move in either direction swings delay by 2-3x.

Third, size your autoscaling buffer off the same math. If your scale-out trigger is slower than the time it takes ρ to cross from 80% to 95% under real traffic variance, the new capacity will land after the multiplier has already done its damage, not before.

A ten-line simulation makes the curve concrete:

def wait_multiplier(rho):
    return rho / (1 - rho)

for rho in [0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.99]:
    print(f"{int(rho*100)}%  ->  {wait_multiplier(rho):.2f}x")

50%  ->  1.00x
60%  ->  1.50x
70%  ->  2.33x
80%  ->  4.00x
90%  ->  9.00x
95%  ->  19.00x
99%  ->  99.00x

Run it once, and "80% feels safe" stops sounding reasonable.

Key takeaways

Queueing delay scales as ρ/(1−ρ), a hyperbola — not linearly with utilization.
Moving from 70% to 90% utilization isn't "20% more load," it's roughly 4x more queueing delay.
The number that predicts pain is headroom (1−ρ), not the utilization percentage itself.
Alert on latency and queue depth directly where you can; if you alert on utilization, pick the threshold from the curve, not a gut feeling.
Autoscaling has to react faster than ρ crosses the steep part of the curve, or the new capacity arrives after the damage is done.

CAP Theorem Covers 0.01% of Your Database's Life. PACELC Covers the Other 99.99%.

speed engineer — Sun, 20 Sep 2026 03:39:51 +0000

The problem

Every distributed systems interview eventually gets to CAP theorem: during a network partition, you can have consistency or availability, not both. Engineers love quoting this. It sounds rigorous, it name-drops a real theorem, and it explains an outage story well.

Here's what it doesn't explain: why your "strongly consistent" database is slower than your "eventually consistent" one on a totally healthy Tuesday afternoon, with zero partitions in sight.

CAP only makes a claim about the P — the rare window when the network is actually split. For a well-run multi-region system, that might be minutes per year. The other 99.99% of the time, you're still making a consistency trade-off, on every single request, and CAP has nothing to say about it. The theorem that does is PACELC, and almost nobody outside distributed-systems papers has heard of it.

Why it happens

PACELC, coined by Daniel Abadi in 2010, says: if Partitioned, choose Availability or Consistency (that's CAP); Else, choose Latency or Consistency.

That "Else" is the part that actually governs your day-to-day production behavior. Strong consistency isn't free even when the network is fine — it costs coordination, and coordination costs round trips.

Take a quorum-based system with N=3 replicas, W=2, R=2 (Dynamo-style, or Cassandra with QUORUM consistency). To satisfy a strongly-consistent read after a write, you need overlapping quorums, which means a write has to get acknowledged by 2 of 3 nodes before it returns, and a read has to check 2 of 3 nodes and reconcile versions before it returns. If those replicas are same-AZ, that's maybe 1-2ms of extra round-trip. If they're cross-region — which is exactly when you'd want the durability guarantee most — each of those round trips is 40-150ms depending on the region pair.

Compare that to reading from the nearest single replica with no quorum check: sub-5ms, no coordination, no waiting on the slowest of N acks. Same data. Same hardware. The only difference is how much agreement you demanded before answering, and that demand is a tax you pay on every request, forever, not just during a partition.

This is why Spanner (optimizes for the C, pays the L, uses TrueTime and atomic clocks to shrink the tax as much as physics allows), DynamoDB in eventual-read mode (optimizes for L, accepts stale reads), and Cassandra with tunable consistency (lets you dial the trade-off per query) all make genuinely different, defensible choices — they're not disagreeing about CAP, they're landing in different spots on the E-L-C axis, which is a decision you make thousands of times a day, not once during an incident.

What to do about it

Stop treating "consistency" as a single on/off setting picked once at database-selection time. Instead:

Identify which specific reads actually need read-your-writes or linearizable guarantees (payment status, inventory decrement, leader election) versus which ones are fine stale by a few hundred milliseconds (a feed, a dashboard, a "last seen" timestamp).
For the first group, pay the latency tax deliberately and measure it — know your P99 quorum round-trip, don't discover it in an incident.
For the second group, actually use the weaker consistency mode your database offers. Most teams default every query to the strongest available mode "to be safe" and then wonder why a read-heavy service has a P99 three times higher than it needs.
If you're using a system with tunable consistency (Cassandra, ScyllaDB, CockroachDB's follower reads), treat the consistency level as a per-query parameter, not a cluster-wide constant.
When evaluating a new datastore, ask what it does on the "Else" branch specifically — vendors will happily tell you their partition behavior, but the everyday latency-consistency trade-off is usually buried three pages into the architecture docs.

Key takeaways

CAP theorem describes rare-event behavior (during a partition); it says nothing about normal operation.
PACELC's "Else" clause — latency vs. consistency with no partition present — governs your system's behavior essentially all the time.
Strong consistency costs real, measurable round-trip latency because it requires coordination (quorums, consensus), not because of anything partition-related.
Different databases making different CAP choices often aren't disagreeing about partitions at all — they're landing in different spots on the everyday latency/consistency trade-off.
Treat consistency level as a per-query decision informed by what that specific read or write actually needs, not a single global setting.

The ID Went Negative. The Sharding Router Didn't Notice For Six Days.

speed engineer — Thu, 17 Sep 2026 18:11:17 +0000

The problem

A payments-adjacent service I worked on used a plain AtomicInteger-backed auto-increment ID for a high-write event table. It had been running fine for two years. Then a backfill migration kicked write throughput from ~300/sec to ~4,200/sec for three weeks straight, and the counter did something nobody had modeled: it hit 2,147,483,647 and wrapped to -2,147,483,648.

No crash. No exception. No log line. The service kept issuing IDs — they just went negative.

Six days later, a nightly reconciliation job flagged that roughly 0.006% of events existed in the source stream but not in the sharded store. Not zero — worse than zero. A small, silent, growing hole.

Why it happens

The routing layer picked a shard with shard = id % NUM_SHARDS. That line had been correct for two years because id had always been positive. Once IDs went negative, the formula didn't stop working — it started returning different wrong answers depending on which service evaluated it, because languages don't agree on what negative modulo means.

Python uses floored division: the result always takes the sign of the divisor. -7 % 16 in Python is 9 — still a valid, in-range shard index.

Java, C, C++, C#, JavaScript, and Go (and Rust, by default) use truncated division: the result takes the sign of the dividend. -7 % 16 in Java is -7. Not in range. Not a valid shard.

Our ID generator was a Java service. Our routing layer was Go. Both computed id % 16 on the exact same ID and both were "correct" by their own language's rules — and both disagreed with the one thing that mattered, which was "which of our 16 real shards owns this record."

The Go router held shard clients in a map[int]*ShardClient. A negative key it had never seen didn't panic — Go just handed back the zero value for a missing map key, which in this codebase was a *ShardClient with a no-op Write() stubbed in for testing and never removed. Every write to a negative shard index silently succeeded from the caller's point of view and went nowhere.

That's the actual failure: not the overflow, and not even the sign mismatch — the fact that an impossible routing key was handled by quietly doing nothing instead of failing loudly.

What to do about it

Stop using 32-bit signed IDs for anything unbounded. At 4,200 inserts/sec, int32 gives you about six days of headroom before wraparound. int64 gives you roughly 68 years at the same rate. This is a five-minute schema change that eliminates the entire bug class.

Normalize modulo sign explicitly, or avoid modulo entirely. If you must support mixed languages or can't rule out negative inputs, use the floor-mod pattern: ((id % n) + n) % n. Better: if your shard count is a power of two, use id & (n - 1) instead of id % n. Bitwise AND operates on the two's-complement bit pattern directly, so it's immune to the sign question altogether — and it's faster than a division-based modulo to boot.

Make "unknown routing key" a hard failure. A map lookup that silently returns a zero-value client is a landmine. Write() on an unrecognized shard should panic, alert, or reject — never no-op. The cost of a loud failure is a page. The cost of a quiet one is a six-day-old, slowly growing gap in your data that a human has to notice on their own.

Instrument the counter, not just the outage. We added an alert at 80% of int32 max on every unbounded counter in the system, well before any of them are near the edge. Overflow should be a scheduled maintenance ticket, not an incident.

Key takeaways

Signed integer overflow doesn't crash — it wraps and keeps producing plausible-looking, wrong answers.
The identical id % n expression can be correct in one language and silently wrong in another, because languages disagree on the sign of negative modulo.
For power-of-two shard counts, id & (n - 1) is both faster and sign-safe — prefer it over modulo.
An unrecognized routing key should fail loudly. A silent no-op is how a bug becomes a six-day data gap instead of a two-minute page.

Amdahl's Law Predicts a Plateau. It Doesn't Explain Why We Went Slower.

speed engineer — Sun, 13 Sep 2026 08:37:22 +0000

The problem

We had a write-heavy service backed by a cache cluster, and throughput wasn't keeping up with load. The fix looked obvious: add more nodes. We went from 8 to 16 — throughput went up, as expected. Feeling good, we kept going, 16 to 24.

Throughput went down.

Not "leveled off." Down. Fewer completed requests per second with more hardware running than we had at 16 nodes. The first instinct on the call was "we shipped a regression." We hadn't. What we'd hit is a well-documented, 40-year-old-plus piece of math that most engineers never see until it bites them: retrograde scaling.

Why it happens

Everyone's heard of Amdahl's Law: speedup from parallelism is capped by the fraction of work that's inherently serial. The formula is:

Speedup(N) = 1 / (s + (1 - s) / N)

Where s is the serial fraction. As N grows, speedup approaches 1/s and flattens out. That's the ceiling everyone budgets for. It predicts a plateau.

It cannot predict a decline. If your only cost is "some work can't be parallelized," adding more workers never makes you slower — it just stops helping. So why did our throughput actively drop?

Because there's a second cost Amdahl's Law doesn't model: what nodes pay to stay consistent with each other. Dr. Neil Gunther formalized this as the Universal Scalability Law (USL), which extends Amdahl's Law with a coherency (or "crosstalk") term:

C(N) = N / (1 + α(N - 1) + βN(N - 1))

α is the same contention/serialization cost as Amdahl's s. β is new: the cost of keeping nodes coherent — cache invalidation broadcasts, distributed lock coordination, gossip protocols, leader-election chatter, quorum round trips. Critically, that coordination cost tends to scale closer to O(N²) than O(N), because every node potentially has to talk to every other node.

When β is exactly zero, USL collapses back into Amdahl's Law and you get a plateau. When β > 0, the function has a maximum, at:

N* = sqrt((1 - α) / β)

Past N*, each additional node adds more coordination overhead than it adds useful work, and total throughput falls. That's retrograde scaling, and it's exactly what we watched happen on that call.

In our case, the culprit was cache invalidation: every write triggered an invalidation broadcast to every other node in the cluster to keep reads consistent. At 8 nodes, that's 8×7=56 messages per write cycle. At 24 nodes, it's 24×23=552 — a 10x jump in coordination traffic for a 3x increase in nodes. The useful work per node grew linearly; the coordination tax grew quadratically. Past a certain point, the tax bill exceeded the income.

What to do about it

Load-test at more than two points. Two data points (before/after one scaling change) can only show you a line. You need at least three node counts to see curvature — and curvature is the whole signal for retrograde scaling.

Fit α and β instead of guessing. With three or more (N, throughput) pairs, you can fit USL's parameters with basic nonlinear regression (Gunther has published spreadsheet and R/Python tooling for this). You don't need a PhD-level setup — three good load-test runs and a regression call will tell you your predicted peak node count before you buy or provision anything past it.

Go looking for the O(N²) term specifically. In practice, β is almost always one of: cache invalidation fan-out, distributed lock/mutex contention, consensus protocol round trips (Raft/Paxos elections and log replication), or gossip-based membership protocols. If any of these exist in your system and scale with cluster size, you have a β term whether or not you've measured it yet.

Bound the coherence domain, not just the node count. If β is unavoidable (some form of consistency requires it), the fix usually isn't "add fewer nodes" — it's sharding, so that any given coordination domain stays small even as your total fleet grows. Twenty-four nodes split into six coherence domains of four each can dramatically outperform twenty-four nodes all coordinating with each other.

Key takeaways

Amdahl's Law bounds speedup from serialized work. It predicts a ceiling — never a decline.
The Universal Scalability Law adds a coherency term (β) for the cost of nodes staying consistent with each other, and that term is why throughput can actively fall as you scale out.
A peak node count, N* = sqrt((1 - α) / β), exists whenever β > 0 — and you can estimate it from as few as three load-test data points, before it costs you an incident.
If a system gets slower after you scale it out, look for O(N²) cross-node chatter first: cache invalidation, distributed locks, consensus, gossip. That's β made concrete.

The Ledger Was Off By One Cent. By Month-End It Was $340.

speed engineer — Thu, 10 Sep 2026 14:24:07 +0000

The problem

A nightly job reconciled every transaction against the daily balance and flagged anything that didn't tie out to the cent. For months it was silent. Then one Tuesday it flagged a $0.01 discrepancy on an account with about 40,000 line items. Nobody paged over a penny â€” the ticket sat in the backlog for a week.

By the following Monday, the same account was off by $2.14. Two weeks after that, $340.07. The discrepancy wasn't random noise that would cancel out â€” it only ever grew in one direction, and it grew faster the more transactions the account processed that month. That's the detail that turned "round to the nearest cent and move on" into an actual incident.

The balance was computed by summing a Python list of float transaction amounts â€” sum(amount for amount in transactions) â€” then rounding the total to two decimal places for display. Every individual amount, printed on its own, looked exactly right: 19.99, 4.50, 102.33. The bug wasn't in any single number. It was in adding them together, tens of thousands of times, in whatever order the query happened to return them.

Why it happens

float in Python (and in virtually every language) is IEEE-754 double precision: a fixed 53 bits of mantissa. Most decimal amounts â€” 0.10, 19.99 â€” have no exact binary representation, the same way 1/3 has no exact finite decimal representation. 0.10 in a double is actually stored as 0.1000000000000000055511151231257827021181583404541015625. That error is roughly 5.5e-18 per value â€” utterly invisible when you print one number, because printing rounds it back to something that looks clean.

The part most people miss: floating-point addition is not associative. (a + b) + c does not always equal a + (b + c) once rounding is involved, because each intermediate addition rounds to the nearest representable double, and which values you round first changes the final result. Summing 40,000 of these tiny errors doesn't average out to zero the way independent random noise would â€” the rounding direction correlates with the sign and magnitude of the running total, so on this workload the errors compounded in the same direction almost every time. Change the table scan order, add an index, reorder the query â€” any of it could push the running total onto a different, still-wrong, path.

This is exactly why the discrepancy only grew: it wasn't measurement noise, it was systematic error from a data type that was never designed to represent money exactly in the first place.

What to do about it

Never use float/double for currency amounts you intend to sum, compare, or reconcile. Two solid options:

Integer minor units. Store and sum everything in cents (or the smallest currency unit) as an integer â€” 1999 instead of 19.99. Integer addition has no rounding error, period. Convert to a display string only at the edge, right before rendering.

# Instead of this:
total = sum([19.99, 4.50, 102.33])  # accumulates binary rounding error

# Do this:
total_cents = sum([1999, 450, 10233])  # exact integer arithmetic
display = f"${total_cents / 100:.2f}"  # convert only for display

decimal.Decimal, if you need fractional-cent precision (tax calculations, currency conversion) or can't restructure storage to integers:

from decimal import Decimal
total = sum((Decimal("19.99"), Decimal("4.50"), Decimal("102.33")))

Decimal is exact for base-10 values because it stores digits, not binary fractions â€” but it's slower, and it's easy to accidentally reintroduce the bug by constructing a Decimal from a float (Decimal(19.99) inherits the float's existing error; Decimal("19.99") built from the string does not).

If you're stuck with floats for now â€” legacy schema, a third-party API you don't control â€” at minimum switch your summation to Kahan summation, which tracks and compensates for the running rounding error at each step. It doesn't fix the root representation problem, but it caps the error growth instead of letting it compound.

Whatever you pick, add a reconciliation test that sums a large, realistic transaction set and asserts the total matches a value computed an independent, exact way (integer cents, say) â€” not just a handful of clean round numbers in a unit test, which is exactly the kind of input that hides this bug.

Key takeaways

float/double cannot represent most decimal fractions exactly â€” the error per value is tiny, but real.
Floating-point addition is not associative; summing many floats compounds rounding error in ways that don't cancel out, especially under a consistent processing order.
Never sum currency as float. Use integer minor units (cents) or decimal.Decimal, and construct Decimal from strings, not from floats.
A bug invisible at the single-value level can still be a systemic, growing problem in aggregate â€” test reconciliation logic against large, realistic datasets, not clean round numbers.

Two Random Choices Beat One Careful One: The Load Balancer Mental Model Nobody Teaches

speed engineer — Sun, 06 Sep 2026 03:42:02 +0000

The problem

Picture a fleet of 100 identical workers behind a load balancer. You send requests uniformly at random — each request goes to a random worker, independent of the others. It sounds fair. It isn't.

If you send 100 requests to 100 workers this way, the most loaded worker doesn't get 1 request. On average, it gets around log(100) / log(log(100)) — roughly 4 to 5 requests — while plenty of workers sit idle. Send 10,000 requests to 10,000 workers and the busiest one gets over 9, not ~1. This isn't a bug in your random number generator. It's math, and it's called the "balls into bins" problem.

Most engineers have felt the symptom — a "perfectly balanced" random or round-robin-ish LB that still produces one hot node, one node pegged at 90% CPU while its siblings idle at 20% — without ever learning the mechanism. So they reach for the wrong fix: bigger instances, more replicas, a mysterious "just restart it" ritual. None of that touches the actual cause.

Why it happens

Uniform random assignment doesn't spread load evenly — it clusters, the same way random points in a room form clumps and empty patches instead of a neat grid. The formal result: if you throw n balls into n bins independently and uniformly at random, the maximum bin load is, with high probability, Θ(log n / log log n).

That's not linear in n, but it's not constant either — it grows, slowly but unboundedly, as your fleet scales. The more workers you add expecting things to smooth out, the more that log(n) term keeps producing a stubborn outlier. This is exactly why "just add more instances" often makes the imbalance more visible in absolute terms even as it helps in relative terms — the tail keeps growing with the fleet.

What to do about it

The fix has a name — the power of two choices (Azar, Broder, Karlin, Upfal, 1994; popularized for systems by Michael Mitzenmacher) — and it's absurdly cheap for how much it buys you.

Instead of picking one random worker, pick two random workers and send the request to whichever currently has less load. That's the entire algorithm. The result: the maximum load drops from Θ(log n / log log n) to Θ(log log n / log log log n) — an exponential improvement in the exponent. At n = 10,000, that's the difference between a worst node carrying ~9x the average and one carrying ~2-3x.

You've probably already used this without naming it:

Envoy's P2C (power-of-two-choices) load balancer is a built-in policy, not something you write yourself.
HAProxy's leastconn and AWS ALB's least-outstanding-requests algorithm are cousins of the same idea — they just skip the "pick two" sampling step and check global state directly, which works at small scale but gets expensive to coordinate globally at large scale (hence P2C's popularity: it needs no central coordinator).
Consistent hashing ring hot spots — the classic complaint that one shard runs hot even with a "good" hash function — are the same balls-into-bins effect. The standard mitigation, virtual nodes (100-200 vnodes per physical node), works by turning one ball into many smaller, independently-placed balls, which flattens the same log(n) tail.

The one gotcha that bites people who implement this from scratch: power of two choices needs reasonably fresh load signal. If your "current load" metric is stale — cached for 30 seconds, propagated through a slow gossip protocol — every requester samples the same stale "least loaded" node and stampedes it. You've now built a synchronized herd instead of a load balancer. The fix is either querying live local queue depth at decision time (what Envoy does) or adding jitter/randomization to which two nodes get sampled, so staleness doesn't correlate across requesters.

Key takeaways

Uniform random load balancing is not fair load balancing — balls-into-bins guarantees a growing max-load outlier as your fleet scales, Θ(log n / log log n).
Sampling two random candidates and picking the lesser-loaded one collapses that outlier to Θ(log log n / log log log n) — two lookups instead of one, no central coordinator required.
This is already built into Envoy, and it's the theoretical justification behind leastconn-style balancers you've probably deployed without reading the paper behind them.
The failure mode of power-of-two-choices itself is stale load data causing correlated stampedes — check freshness before you trust the "least loaded" signal.

The One Number That Actually Moves Your Latency (And Why Your Team Keeps Optimizing the Wrong Thing)

speed engineer — Sun, 30 Aug 2026 09:12:27 +0000

The problem

A team I worked with once burned a full sprint shaving a service from 9ms down to 6ms. Clean win, nice PR, everyone felt good. Total request latency at p99: unchanged. Not "improved slightly." Unchanged, down to the millisecond.

That service was never the problem. It was just the easiest one to fix — small codebase, one owner, an obvious N+1 query to kill. Meanwhile a lock-heavy write in a shared Postgres table, three hops downstream, was eating 280ms on the same request path, and nobody had touched it in months because it was owned by a different team and looked scary.

This happens constantly, and it has a name that most engineers know from manufacturing and never apply to their own systems: Theory of Constraints. Eli Goldratt's version is blunt — a chain is only as strong as its weakest link, and reinforcing any other link does nothing for the chain's strength. Applied to a request path: your system has exactly one bottleneck at any given moment, and improving anything that isn't the bottleneck is not "a smaller win." It's a rounding error that shows up in your commit history and nowhere else.

Why it happens

Two things make this trap easy to fall into.

First, the bottleneck is usually the least pleasant thing to fix. It's often owned by someone else, wrapped in a lock or a queue you don't fully understand, or requires a schema change instead of a code change. The 9ms service is pleasant. The 280ms lock contention is not. Teams under sprint pressure gravitate toward pleasant.

Second, most latency dashboards show you averages or per-service breakdowns, not the serial chain a single request actually walks through. If service A is 9ms and service B is 280ms but they're graphed on separate panels with separate y-axes, they look like two roughly-equal-sized problems. They are not. One of them is 97% of your controllable latency and the other is noise.

What to do about it

Goldratt's original framework has five steps, and they map onto engineering almost without translation:

Identify the constraint. Don't guess — trace one real request end-to-end (a flame graph, distributed trace, or even manual timestamps at each hop) and rank stages by wall-clock time, not by whose code it is or how ugly it looks.
Exploit the constraint. Before you architect anything new, squeeze the bottleneck itself: can that lock be shortened, that query indexed, that call made async, without touching anything else?
Subordinate everything else to it. This is the step teams skip. If service A finishes in 9ms and immediately has to wait on service B's 280ms lock, optimizing A to 3ms buys you exactly nothing — A was never the pacing item. Stop spending story points there until the constraint moves.
Elevate the constraint. If step 2 isn't enough, this is where you actually add capacity — a read replica, a cache in front of the hot table, breaking the lock's critical section apart, splitting the write path.
Repeat. Once you fix the constraint, a new one appears somewhere else in the chain. This isn't a one-time exercise; it's a loop.

The practical version of step 1, if you don't have distributed tracing yet: pick your ten slowest requests from the last day, and for each one, log the wall-clock time spent in every downstream call. Sum by destination, not by your own service boundary. The bottleneck is almost never where the on-call rotation assumes it is — it's usually invisible precisely because nobody's dashboard is shaped like the actual request path.

Key takeaways

A system has one bottleneck at a time; everything else you optimize is a rounding error on the metric that matters.
"Easy to fix" and "worth fixing" are unrelated — the bottleneck is often the ugly, shared, poorly-owned piece nobody wants to touch.
Before adding capacity anywhere, trace a real request end-to-end and rank stages by actual wall-clock time, not by service ownership.
Subordinate step 3 is the one teams skip: stop improving non-bottleneck stages, even when it feels like progress.
Fixing the constraint doesn't end the exercise — it just reveals the next one.

The TCP Checksum Passed. The Data Was Corrupted Anyway.

speed engineer — Sat, 29 Aug 2026 13:58:05 +0000

The problem

"It's fine, it's over TCP" is one of the more expensive sentences in engineering. Teams treat TCP's checksum as a data-integrity guarantee. It isn't one, and the gap between "checksum passed" and "data is correct" is exactly where silent corruption gets through.

Why it happens

TCP's checksum is a 16-bit one's-complement sum over the segment — a design from an era when the threat model was electrical noise on a cable, not a buggy NIC driver or a router with corrupted line-card memory. A 16-bit sum has a small, fixed number of possible values. Specific corruption patterns — certain multi-bit flips, certain byte-swaps — land on the same sum as the clean data and pass straight through. This isn't theoretical: Jonathan Stone and Craig Partridge's measurement study of real production traffic ("When the CRC and TCP Checksum Disagree," SIGCOMM 2000) found that a small but persistent fraction of segments arrive with corrupted payloads and checksums that say everything is fine.

Modern hardware narrows the window further. Most NICs compute the TCP checksum in hardware, before your OS's network stack — let alone your application — ever touches the bytes. That's great for throughput and bad for the mental model of "the kernel checked this for me." Anything that corrupts memory between the NIC's checksum step and your application reading the buffer is invisible to TCP, full stop.

This is the same failure shape as the storage side of this problem (the deep-dive on that is linked below): a RAID array faithfully mirroring corrupted bytes across every disk because nobody told it to verify content, only to survive a drive failure. The layer you're trusting was never designed to certify the thing you actually care about.

What to do about it

Treat TCP as "probably intact," not "guaranteed intact," anywhere correctness actually matters — financial records, replicated state, anything you'd hate to silently corrupt. The fix is the same principle as filesystem-level checksums: push verification to the two endpoints that know what "correct" means, not the layers in between.

A minimal version of this, independent of whatever transport you're on:

func sendChecked(w io.Writer, payload []byte) error {
    sum := crc32.Checksum(payload, crc32.MakeTable(crc32.Castagnoli))
    header := make([]byte, 8)
    binary.LittleEndian.PutUint32(header[0:4], uint32(len(payload)))
    binary.LittleEndian.PutUint32(header[4:8], sum)
    if _, err := w.Write(append(header, payload...)); err != nil {
        return err
    }
    return nil
}

func recvChecked(r io.Reader) ([]byte, error) {
    header := make([]byte, 8)
    if _, err := io.ReadFull(r, header); err != nil {
        return nil, err
    }
    length := binary.LittleEndian.Uint32(header[0:4])
    want := binary.LittleEndian.Uint32(header[4:8])
    payload := make([]byte, length)
    if _, err := io.ReadFull(r, payload); err != nil {
        return nil, err
    }
    got := crc32.Checksum(payload, crc32.MakeTable(crc32.Castagnoli))
    if got != want {
        return nil, fmt.Errorf("payload checksum mismatch: want %x got %x", want, got)
    }
    return payload, nil
}

A few things follow from that:

This is why gRPC computes its own message-level checksums on top of HTTP/2 on top of TCP. The protocol authors didn't trust the bottom layer to be the last line of defense, because it was never designed to be one.
For anything replicated or cached, checksum the object once at rest and re-verify on read — the same discipline ZFS and Btrfs apply at the filesystem layer. Corruption introduced anywhere in the path between two verifications gets caught at the next one.
"It's on TLS, so it's covered" has the identical gap. TLS's integrity check does verify what crossed the TLS boundary — but only that boundary. Corruption in application buffers before encryption, or after decryption, or introduced by a proxy that terminates and re-encrypts, is outside what TLS ever promised to catch.
Pick your algorithm for the job: CRC32C (hardware-accelerated on most modern CPUs) if you want speed, SHA-256 or BLAKE3 if you want cryptographic strength against deliberate tampering, not just accidental corruption.

Key takeaways

TCP's checksum is a coarse, best-effort noise filter, not an integrity guarantee — and real measurement studies confirm corrupted-but-checksum-valid segments do occur in production networks.
Hardware checksum offloading shrinks the window TCP actually protects, since neither the OS nor the application ever inspects the raw wire bits.
TLS's integrity guarantee has the same shape of gap: it protects its own boundary, not your application buffers on either side of it.
The fix is end-to-end verification at the layer that actually knows what "correct" means for your data — not any transport underneath it. It's the same lesson filesystem-level checksums (ZFS, Btrfs) teach at a different layer of the stack: never let an intermediate hop stand in for verification it was never designed to do.

Full deep-dive on the storage side of this — corruption sources most teams never audit, and what ZFS/Btrfs do differently from ext4/XFS — on Medium.