DEV Community: Packet Path

RoCEv2 Explained: PFC, ECN & DCQCN for Lossless AI Fabrics

Packet Path — Fri, 30 Oct 2026 16:05:20 +0000

Originally published at Packet Path.

Why does an AI cluster care whether Ethernet drops packets?

Imagine a warehouse where forklifts drive loads straight into the right shelf slot in another building — no front desk, no paperwork, nobody checking them in. That is RDMA (Remote Direct Memory Access): one machine's network card puts data directly into another machine's memory. It is extremely fast precisely because almost nothing in software touches the data on the way.

The catch is built into that design. Software checks are what normally notice a lost packet and quietly ask for it again. An RDMA card expects the network to deliver — when a packet does drop, recovery is slow and coarse: a transfer can stall while a whole window of data is sent again, and a training job with thousands of GPUs all waiting on each other feels one machine's stall everywhere. Ordinary Ethernet makes no promises: when a switch queue fills, it drops. That is fine for web traffic. It is poison for RDMA.

In plain English, RoCEv2 is RDMA riding inside ordinary routed Ethernet packets (UDP destination port 4791, so any IP fabric can carry it). Because the Ethernet underneath still wants to drop packets under pressure, a RoCEv2 fabric adds three helpers to fake a network that never drops: PFC pauses senders instead of dropping, ECN warns senders early, and DCQCN is the rulebook that turns those warnings into smooth speed changes.

Helper 1: PFC — the emergency brake

PFC (Priority Flow Control, IEEE 802.1Qbb) changes what a full queue does. Instead of dropping the next RoCE packet, the switch sends a pause frame to the device directly upstream: stop sending this one traffic priority — the lossless one carrying RDMA — while other priorities keep flowing. When the queue drains, a resume lets traffic start again. Nothing drops, so RDMA never notices.

Two details make or break it:

Pausing is not instant. Packets already on the wire keep arriving after the pause is sent, so every port needs headroom buffer — reserved space to catch those in-flight packets. Longer cables put more bits in flight, so longer links need more headroom; run out of headroom and you drop packets on the one class that must never drop.
PFC works hop by hop. A congested receiver pauses its leaf, the leaf's buffers fill and it pauses the spine, and congestion can climb the fabric tree — helpful for a brief burst, dangerous when it lasts.

Note the price: the sender fully stops. It does not slow down.

Helper 2: ECN — the early warning

Pausing is a blunt tool: a sender is either full speed or fully stopped. ECN (Explicit Congestion Notification) is the gentle tool. When a switch queue passes a marking threshold — starting to build, nowhere near full — the switch sets a mark in the IP header of passing RoCE packets instead of dropping or pausing anything. The mark rides the packet to the receiver, whose card answers with a small CNP (Congestion Notification Packet): the path is getting busy, ease off.

Think of the difference like traffic control on a highway. PFC is a roadblock: everything in one lane stops. ECN is the brake-light wave and the variable speed-limit sign: everyone slows a little, early, and the roadblock never becomes necessary. A healthy RoCE fabric lives almost entirely on ECN signals. PFC exists for the moments ECN could not prevent — a sudden burst arriving faster than feedback can travel.

Helper 3: DCQCN — deciding how much to slow down

A warning is useless without a rule for reacting to it. DCQCN (Data Center Quantized Congestion Control) is that rule, running in the network cards. When CNPs arrive, the sender cuts its rate — by an amount that grows if marks keep coming. When marks stop, the sender cautiously probes upward again, in small steps, until it finds the rate the fabric can actually sustain.

Thousands of senders doing this at once is what keeps an AI fabric busy but not clogged: queues stay short, PFC stays silent, and collective operations — all GPUs swapping results at the same moment, the normal rhythm of training — finish on time.

The tuning tension: ECN must bite before PFC fires

Here is the one idea to keep if you keep nothing else. Both mechanisms watch the same queue. ECN marking is set by a queue threshold; PFC pause is set by a higher buffer threshold (plus that reserved headroom). For the system to work, the order on the way up must be:

queue starts building → ECN marks → CNPs fly → DCQCN slows senders → queue drains → PFC never fires.

Get the order wrong and the fabric misbehaves in predictable ways. ECN thresholds set too high (or PFC thresholds too low) mean pauses fire before feedback can help: senders lurch between full speed and full stop, queues elsewhere sit empty, and training throughput sags. ECN thresholds set too low mean senders are told to slow down for queues that would have drained on their own: the fabric runs permanently below capacity.

And because every vendor exposes these knobs differently — queue thresholds in bytes or cells, marking probabilities, CNP rates, DCQCN timers — values copied from another site's runbook are a starting point to validate under load, never a guarantee.

When lossless goes wrong: storms and deadlock

PFC's hop-by-hop kindness has two famous failure modes:

A PFC storm is a device that pauses and never resumes — one sick card can freeze a priority across every switch it can reach, which is why fabrics run PFC watchdogs that intervene when a port stays paused abnormally long.
PFC deadlock is subtler: imagine a ring of switches, every buffer full, each switch paused waiting for the next one downstream to resume — and the next one is waiting on it. Nobody can drain, so nobody resumes. It needs a cycle in buffer dependencies, which is why lossless classes are kept few (usually one) and topologies and routing are designed to avoid those cycles.

One more everyday hazard: incast. AI collectives make dozens of senders transmit to one receiver at the same instant by design. That receiver's queue is where your ECN thresholds, headroom, and DCQCN settings stop being theory — size and tune for the collective pattern you actually run, and spread flows with good hashing.

Five mistakes that break RoCE fabrics

Turning PFC on for everything. Every lossless priority multiplies headroom use and backpressure spread. Keep lossless to the RDMA class; let ordinary traffic stay lossy — TCP handles drops fine, that is what it was built for.
A marking mismatch somewhere in the path. The DSCP-to-priority mapping, ECN marking, and PFC class must agree on the NICs and every switch hop. One hop that maps RoCE to a different priority silently turns that link lossy — and RDMA finds out first.
ECN thresholds above the PFC trigger. That inverts the intended order: the fabric pauses before it ever warns. Check the effective queue depths, not just the numbers typed into two different config screens.
Ignoring cable length in headroom. Headroom must cover the bits in flight while a pause travels and takes effect. Swap in longer cables or faster links without revisiting headroom and the lossless class starts dropping.
Tuning on a quiet fabric. All three mechanisms only act under congestion. Validate with a real collective or incast pattern at production scale — a fabric that looks perfect at 10% load has told you almost nothing.

Try it yourself

Build the fabric intuition this runs on with free tools: size the spine-leaf fabric itself with the data-center capacity planner, watch how flow hashing spreads — or fails to spread — collective traffic in the ECMP hash polarization visualizer, and check the MTU side of a jumbo-frame RoCE design with the MTU/MSS tunnel overhead calculator.

Key takeaways

RoCEv2 is RDMA over routed UDP/IP (destination port 4791): the network card moves data straight into another machine's memory, bypassing the CPU and the kernel — which is why it is fast, and why it assumes packets will not drop.
PFC is the emergency brake: a congested switch pauses one traffic priority at a time instead of dropping its packets. It prevents loss, but a pause can spread congestion upstream and, in a loop of full buffers, deadlock the fabric.
ECN is the early warning: switches mark packets when a queue starts building, the receiver answers with a CNP, and DCQCN cuts the sender's rate smoothly — so traffic slows down before PFC ever has to fire.
The whole tuning game is ordering: ECN marking thresholds must bite below the queue depth that triggers PFC.

Further reading

Why AI Clusters Have Two Networks — the front-end vs back-end split; RoCEv2 lives on the back-end fabric.
ECMP Explained — how fabrics spread flows across links, and the polarization trap that concentrates them instead.

Packet Path is a free site for network engineers — 28 browser tools and plain-English guides, built with AI help on the build and drafting. The original version of this article is on the site: RoCEv2 Explained: PFC, ECN & DCQCN for Lossless AI Fabrics.

ECMP Hash Polarization: Why Two Identical Links Carry 88% and 12%

Packet Path — Fri, 23 Oct 2026 21:24:45 +0000

Originally published at Packet Path.

One best path is a wasted fabric

Picture a modern spine-leaf data center: every leaf switch connects to every spine. Between any two racks there aren't one or two paths — there are dozens, all the same length, all the same cost. Now ask the router how it forwards. Classic routing picks exactly one best path and programs it into the forwarding table. The other thirty-one links become expensive decoration. You've built a 32-lane highway and you're driving every car down a single lane.

Equal-cost multi-path routing — ECMP — is the fix. When the routing table holds several equal-cost routes to the same destination, the router installs all of them and spreads traffic across every link. In hyperscale fabrics ECMP isn't an optimization; it is the forwarding model. And understanding its failure modes is the difference between a fabric that hums and one that mysteriously congests at 40% utilization.

What ECMP actually does

It starts in the control plane. OSPF, IS-IS, or BGP computes routes; when several routes to a prefix tie — same OSPF cost, or BGP multipath policy — the RIB keeps all of them instead of one winner. The forwarding plane programs an ECMP group (vendors call it a next-hop group) into the FIB: one destination, N next-hops, N egress interfaces.

Each arriving packet is assigned to one member of the group. The assignment happens in hardware, per packet, at line rate — and it must be consistent: every packet of a flow has to take the same path, or TCP sees reordering, assumes loss, and collapses its congestion window. So the switch doesn't round-robin packets. It hashes flows.

The hash: five fields in, one link out

The classic ECMP hash input is the 5-tuple: source IP, destination IP, protocol, source port, destination port. Platforms vary — some mix in the ingress port or VLAN ID. The hash output picks the egress member, conceptually member = hash(fields) mod N.

Two properties carry the whole design. Determinism: the same 5-tuple always selects the same member, so flows never reorder. Distribution: across many flows, each member should receive roughly 1/N of them.

Burn this distinction in: ECMP balances flows, not bytes. Ten thousand small flows spread beautifully. Three elephant flows can land on the same member and saturate it while sibling links idle. "Even hash distribution" and "even link utilization" are not the same thing — and confusing them is the root of half of all ECMP troubleshooting sessions.

Hash polarization: the silent killer

Now the failure mode that burns experienced engineers. The hash is deterministic — and on many platforms it is the same deterministic function, with the same seed, on every switch. Follow one flow across three hops: the leaf hashes its 5-tuple and picks member 2; the spine hashes the identical 5-tuple with the identical function and picks its member 2; the next leaf does it again. Not just this flow — every flow with a similar 5-tuple rides the same relative link at every tier. Traffic polarizes onto a handful of links instead of spreading across the fabric. You get hot spots, drops, and retransmissions while most of the fabric sits idle — and nothing in any routing table looks wrong.

Polarization is vicious because every device is behaving correctly in isolation. The bug is emergent: identical hash behavior everywhere. It presents as persistent, unexplained congestion on a few links in an otherwise healthy fabric, and it survives reboots, because determinism is the point.

The rule: Polarization is a property of the fabric, not of any single device. If every switch hashes identically, your 32-way ECMP is effectively 1-way for correlated flows.

Breaking the polarization

Every fix introduces per-device variation into the hash:

Hash seed or offset per device. Most vendors expose a configurable hash seed. Different seed, different member selection for the same flow, polarization broken. This is the first knob to reach for, and the one most often left at default.
Vary the hash inputs by tier. Hash on different field combinations at different layers — include the ingress port at the leaf, exclude it at the spine. Same flow, different inputs, different path.
Resilient hashing. Modern implementations remap only the flows hashed to a member that disappears, instead of rehashing the entire table on every link flap. It matters during maintenance windows and brownouts.

See it with your own eyes: the free ECMP Hash Polarization Visualizer simulates the hash hop by hop across a fabric. Toggle identical versus seeded hash functions and watch flows collapse onto the same links — or spread the way you intended.

ECMP in the wild: protocols and overlays

In the underlay, OSPF and IS-IS give you ECMP almost for free: equal-cost paths are installed by default up to the platform's max-paths limit — check it, because 8, 16, 32, and 64 are all common ceilings, and silently exceeding one is a classic. BGP needs explicit multipath configuration, and at hyperscale the interesting knobs are multipath relax and Add-Paths for backup diversity.

Overlays add a wrinkle that bites everyone once: VXLAN. The underlay ECMP hash sees only the outer headers — outer source and destination IP, outer UDP ports. If the encapsulation stamps the same outer source port on every inner flow, the underlay sees one giant flow and pins it to a single link. The fix is UDP source-port entropy (RFC 7348): the VTEP hashes the inner 5-tuple into the outer source port, restoring per-flow distribution to the underlay. If your VXLAN fabric shows one hot spine link, check entropy before you check anything else.

Troubleshooting lopsided links

When link utilization is uneven and the routing table is innocent, work this list:

Count flows per member, not just bytes. A member carrying 3 elephant flows at 90% while others carry 3,000 mice at 10% is a distribution problem, not a capacity problem.
Check max-paths. If the platform installed 8 members but 16 equal paths exist, you're blind to half your fabric.
Compare hash seeds across tiers. Identical seeds on leaf and spine is polarization waiting to happen — or already happening.
Verify VXLAN entropy. One outer source port for all inner flows is the overlay version of the same disease.
Look for asymmetric hashing on bundled links. LAG and ECMP hashing interact; a flow pinned by ECMP to one member can be pinned again inside a port-channel.

Where this fits

ECMP is the forwarding half of the hyperscaler story; BGP is the control-plane half. Read them together: BGP: How the Internet Actually Finds a Path covers how those equal-cost routes get computed in the first place.

Further reading:

ECMP Hash Polarization Visualizer — watch polarization happen hop by hop, then fix it with hash seeds.
TCP Internals Every Network Engineer Should Know — why reordering from inconsistent paths destroys throughput.
Packet Path — more free tools and plain-English guides for network engineers.