DEV Community: InstaWebhook

Memory-Mapped Webhook Buffers: Using `mmap` and Ring Buffers to Cut Kernel-User Crossings at 500k RPS

InstaWebhook — Mon, 02 Nov 2026 02:21:37 +0000

Webhooks connect payment gateways, CI/CD pipelines and SaaS platforms. At scale, the service that receives them has to accept a request, verify it, acknowledge it quickly and hand it off for processing, hundreds of thousands of times per second.

This article looks at what actually costs CPU time at 500,000 requests per second. It shows where a shared-memory ring buffer built on mmap helps, how to write one correctly, and how modern io_uring features fit around it. It also covers where the approach does not help, because those limits matter as much as the gains.

1. Where the time goes at 500k RPS

500,000 RPS leaves a very small time budget per request. A single core handling all of that traffic would have 2 µs per request. Spread across 32 cores, each request gets about 64 µs of core time, and that has to cover TCP, TLS (if you terminate it), HTTP parsing, signature verification and the handoff to workers. A 1 KB payload at this rate is about 512 MB/s, or roughly 4 Gbit/s of body data before protocol overhead, so a 25 GbE NIC is not the limit. CPU time per request is.

The traditional receive path

A typical blocking or epoll-based server handles an incoming webhook like this:

The NIC writes the packet into a kernel-owned receive ring by DMA.
An interrupt (or a polled NAPI cycle) triggers softirq processing, and the kernel runs the packet through IP and TCP.
The payload lands in the socket’s receive queue (sk_buff structures in kernel memory).
The application calls recv/read, which switches into the kernel.
The kernel copies the data into the user-space buffer (copy_to_user) and returns to user mode.

+-------------------------------------------------------------+
| USER SPACE                                                  |
|   Application buffer                                        |
|        ^                                                    |
|        | copy_to_user()  (copy + syscall entry/exit)        |
+--------|----------------------------------------------------+
| KERNEL SPACE                                                |
|   Socket receive queue (sk_buff)                            |
|        ^                                                    |
|        | TCP/IP stack processing                            |
+--------|----------------------------------------------------+
| HARDWARE                                                    |
|   NIC receive ring (DMA)                                    |
+-------------------------------------------------------------+

A system call is not a context switch

The terminology matters here. A system call is a privilege-mode transition within the same thread. A context switch is the scheduler replacing one thread or process with another, which usually means saving and restoring state and possibly changing address spaces.

Both cost something, but they are different amounts. In one widely cited 2010 measurement on older Intel hardware, the cheapest system call (gettid) cost roughly 52 to 105 ns depending on the CPU. The same benchmark measured process context switches at about 1.3 to 1.9 µs, and about 3 µs when futex calls were included. A University of Rochester study put the direct cost of a context switch at about 3.4 µs on 2000s-era hardware. It also found that the indirect cost of refilling caches ranged from several microseconds to hundreds of microseconds, depending on working-set size. More recent measurements report a direct kernel-level context switch of around 0.62 µs. These numbers vary with CPU, kernel, mitigations and measurement method, so treat them as orders of magnitude rather than constants. Always measure your own hardware.

The practical conclusion is that per-request system calls, copies, and scheduler wake-ups add up. At 500k RPS they can take a meaningful share of your CPU budget. The size of that share depends on your workload, so profile with perf before assuming a number.

2. What `mmap` and ring buffers actually buy you

mmap() maps a file or shared-memory object into a process’s address space. If two processes map the same object with MAP_SHARED, they can exchange data through plain loads and stores. The kernel is only involved when the mapping is created and when pages fault in, not on every message.

That makes it a good fit for handing parsed webhooks from an ingestion process to worker processes. It does not remove the kernel from the network receive path, because TCP still runs in the kernel unless you bypass the stack entirely (see section 4).

+-----------------------------------------------------------------------+
| SHARED MEMORY REGION (mmap of a shm object)                           |
|                                                                       |
|   +------------+------------+------------+------------+               |
|   | Slot 0     | Slot 1     | Slot 2     | Slot 3     | ...           |
|   | (Payload)  | (Payload)  | (Payload)  | (Empty)    |               |
|   +------------+------------+------------+------------+               |
|         ^                                     ^                       |
|    [Head: consumer reads]             [Tail: producer writes]         |
+-----------------------------------------------------------------------+

A ring buffer is a good structure for this handoff because:

No allocation on the hot path. Slots are preallocated, so enqueue and dequeue are a few arithmetic and atomic operations in constant time.
Cache-friendly layout. Slots are contiguous and sequentially accessed, which suits hardware prefetchers. Aligning the head and tail counters to separate cache lines also avoids false sharing.
No locks for the single-producer, single-consumer (SPSC) case. One side only writes the tail, the other only writes the head, so acquire/release atomics are enough. Multiple producers or consumers on one ring need additional machinery (section 3).

If ingestion and workers are threads in one process, an ordinary in-process ring buffer gives you the same benefits without mmap. Use shared memory when you need process isolation (a worker crash must not take down the listener), independent deploys, or the buffer to survive an ingestion restart.

3. A correct SPSC shared-memory ring

Here is a compact C++ implementation for one producer process and one consumer process. It differs from naive versions in a few ways that matter:

It uses a power-of-two capacity, so the index is a bitmask rather than a modulo.
It uses monotonically increasing 64-bit counters, so full and empty are unambiguous and every slot is usable.
It stores each payload’s length in its slot.
It separates “create” from “attach”, so two processes never race to initialize the header.

#include <atomic>
#include <cstdint>
#include <cstring>
#include <stdexcept>
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>

constexpr size_t kCacheLine = 64;

struct RingHeader {
    alignas(kCacheLine) std::atomic<uint64_t> head;  // written by consumer
    alignas(kCacheLine) std::atomic<uint64_t> tail;  // written by producer
    alignas(kCacheLine) uint32_t capacity;           // power of two
    uint32_t slot_size;                              // includes 4-byte length prefix
};

static_assert(std::atomic<uint64_t>::is_always_lock_free,
              "atomics in shared memory must be lock-free");

class ShmRing {
public:
    // Producer side creates; consumer side attaches.
    static ShmRing create(const char* name, uint32_t capacity, uint32_t slot_size) {
        if (capacity == 0 || (capacity & (capacity - 1)) != 0)
            throw std::invalid_argument("capacity must be a power of two");
        size_t bytes = sizeof(RingHeader) + size_t(capacity) * slot_size;
        int fd = shm_open(name, O_CREAT | O_EXCL | O_RDWR, 0600);
        if (fd < 0) throw std::runtime_error("shm_open failed");
        if (ftruncate(fd, bytes) != 0) { close(fd); throw std::runtime_error("ftruncate failed"); }
        ShmRing r(fd, bytes);
        r.h_->capacity = capacity;
        r.h_->slot_size = slot_size;
        r.h_->head.store(0, std::memory_order_relaxed);
        r.h_->tail.store(0, std::memory_order_release);
        return r;
    }

    static ShmRing attach(const char* name) {
        int fd = shm_open(name, O_RDWR, 0600);
        if (fd < 0) throw std::runtime_error("shm_open failed");
        // Map the header first to learn the full size, then remap.
        void* hdr = mmap(nullptr, sizeof(RingHeader), PROT_READ, MAP_SHARED, fd, 0);
        if (hdr == MAP_FAILED) { close(fd); throw std::runtime_error("mmap failed"); }
        auto* ph = static_cast<RingHeader*>(hdr);
        size_t bytes = sizeof(RingHeader) + size_t(ph->capacity) * ph->slot_size;
        munmap(hdr, sizeof(RingHeader));
        return ShmRing(fd, bytes);
    }

    bool push(const void* data, uint32_t len) {            // producer only
        if (len > h_->slot_size - sizeof(uint32_t)) return false;
        uint64_t tail = h_->tail.load(std::memory_order_relaxed);
        uint64_t head = h_->head.load(std::memory_order_acquire);
        if (tail - head == h_->capacity) return false;     // full: apply backpressure
        char* slot = slots_ + (tail & (h_->capacity - 1)) * h_->slot_size;
        std::memcpy(slot, &len, sizeof(len));
        std::memcpy(slot + sizeof(len), data, len);
        h_->tail.store(tail + 1, std::memory_order_release);
        return true;
    }

    bool pop(void* dst, uint32_t dst_cap, uint32_t& out_len) {  // consumer only
        uint64_t head = h_->head.load(std::memory_order_relaxed);
        uint64_t tail = h_->tail.load(std::memory_order_acquire);
        if (head == tail) return false;                    // empty
        const char* slot = slots_ + (head & (h_->capacity - 1)) * h_->slot_size;
        uint32_t len;
        std::memcpy(&len, slot, sizeof(len));
        if (len > dst_cap) return false;
        std::memcpy(dst, slot + sizeof(len), len);
        out_len = len;
        h_->head.store(head + 1, std::memory_order_release);
        return true;
    }

private:
    ShmRing(int fd, size_t bytes) : bytes_(bytes) {
        void* m = mmap(nullptr, bytes, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
        close(fd);  // the mapping stays valid after the descriptor is closed
        if (m == MAP_FAILED) throw std::runtime_error("mmap failed");
        h_ = static_cast<RingHeader*>(m);
        slots_ = static_cast<char*>(m) + sizeof(RingHeader);
    }
    RingHeader* h_;
    char* slots_;
    size_t bytes_;
};

A few things to note about this design:

It is SPSC. The producer only reads head and writes tail, and the consumer does the reverse. The release store on one side pairs with the acquire load on the other, so a consumer that sees the new tail also sees the slot contents. A fetch_add on the tail does not make this multi-producer safe, because a producer could claim a slot before finishing its write. To get multiple producers, use one ring per producer (simple and fast), or use a per-slot sequence-number scheme such as Dmitry Vyukov’s bounded MPMC queue.
Use one ring per worker, or per ingestion thread, as the scaling strategy. The ingestion layer can shard by connection or hash, and each worker drains its own ring. This keeps every ring SPSC and avoids cross-core contention.
Slot size is a trade-off. Fixed slots make indexing trivial but waste space for small payloads. If your webhooks vary widely in size, consider variable-length records with a length header and wrap-around handling, or a second tier for oversized payloads.
The ring only moves bytes. Authentication, parsing and idempotency live in the code on either side of it.

4. Pairing the ring with `io_uring` for network ingress

io_uring is the Linux async I/O interface merged in kernel 5.1. It uses a submission queue and a completion queue shared between user space and the kernel, so many operations can be queued and reaped with few or no system calls.

Several features matter for a webhook receiver, and each needs a minimum kernel version:

Batching. One io_uring_enter() call can submit many queued operations and wait for completions. In ordinary mode you still enter the kernel, but once per batch rather than once per operation.
SQPOLL. With IORING_SETUP_SQPOLL, a kernel thread polls the submission queue, so the application can submit and reap I/O without issuing system calls in the steady state. The cost is that the polling thread consumes a CPU core. Unprivileged use of SQPOLL has been allowed since kernel 5.12.
Multishot receive and provided buffer rings. Multishot recv (kernel 6.0 and later) lets a single submission produce a completion for every arrival, with the kernel picking buffers from a pre-registered pool. This removes the need to re-arm a receive after every packet. Receive bundles, which fill several buffers per operation, arrived in kernel 6.10.
Zero-copy receive (zcrx). Kernel 6.15 added io_uring zero-copy receive, which lets the NIC deliver payload data directly into application memory. This is the feature that actually removes copy_to_user from the receive path. It has strict requirements: the NIC must support hardware header/data split, you must configure flow steering and RSS so that only the intended flows hit a zero-copy receive queue, and the ring must be created with IORING_SETUP_SINGLE_ISSUER, IORING_SETUP_DEFER_TASKRUN and IORING_SETUP_CQE32. The NIC configuration is done outside io_uring. In a demonstration described by the pull request coverage, a 200G link was saturated from a single CPU core.

Without zcrx, a normal io_uring socket receive still copies data from kernel buffers into your buffers. What you save is system-call and wake-up overhead, not the copy itself. An accurate description of the pipeline is:

+-------------------------------------------------------------------+
| NETWORK: kernel TCP/IP stack, io_uring multishot recv             |
|          (zcrx on supported NICs and kernels >= 6.15)             |
+-------------------------------------------------------------------+
                                  │
                                  ▼
+-------------------------------------------------------------------+
| INGESTION THREADS: HTTP parsing, auth, size limits                |
+-------------------------------------------------------------------+
                                  │
                                  ▼  (memcpy into shared slot, no syscall)
+-------------------------------------------------------------------+
| SHARED MEMORY: one SPSC ring per worker                           |
+-------------------------------------------------------------------+
        │                         │                         │
        ▼                         ▼                         ▼
+---------------+         +---------------+         +---------------+
| Worker 1      |         | Worker 2      |         | Worker 3      |
+---------------+         +---------------+         +---------------+

Workers can verify HMAC-SHA256 signatures, apply business logic and dispatch downstream. The ingestion layer should acknowledge the sender only after the payload is safely enqueued (and, if you need durability, persisted), so a slow downstream never stretches your response time.

Full kernel bypass is a different tool. DPDK and AF_XDP move packet handling into user space. DPDK uses poll-mode drivers that poll NIC queues from user space, and its AF_XDP driver redirects packets from a NIC queue into a user-space memory buffer. Both bypass the kernel network stack, which means you must supply your own TCP, and TLS if needed. That is a large step up in complexity, and for a typical HTTP webhook receiver it is usually unnecessary. io_uring keeps the kernel stack and its tooling (ss, tcpdump, firewalling, congestion control).

5. Production hardening

Backpressure and durability

When consumers fall behind, the ring fills, and push returns false. Decide in advance what happens:

Shed load. For low-value telemetry, answer with 429 Too Many Requests (or 503 with a Retry-After) and let the sender retry. Dropping the oldest entries is a deliberate data-loss policy, so only use it where loss is acceptable.
Persist before acknowledging. For payments or other critical events, a shared-memory ring is not durable. A shm_open object lives in memory (on Linux typically on tmpfs), so a machine reboot loses it. Write to an append-only log or a durable queue before returning 2xx, and make processing idempotent, because senders retry and at-least-once delivery means duplicates.

Be careful with file-backed `mmap`

A MAP_SHARED mapping of a regular file lets the kernel write dirty pages back to storage on its own schedule, and it can evict pages under memory pressure. The 2022 CIDR paper “Are You Sure You Want to Use MMAP in Your Database Management System?” lists the resulting hazards. Page faults can happen at any memory access and stall the thread. Evicting pages requires TLB shootdowns, which are inter-processor interrupts that can hurt multi-core scaling. I/O errors surface as SIGBUS signals rather than return codes. The paper is about database buffer pools, but the same mechanisms apply to a file-backed ingest buffer. For a hot ring, prefer an anonymous or shm_open-backed region, pre-fault it at startup (for example with MAP_POPULATE or by touching every page), and use mlock or huge pages if you need to keep it resident. If you need a durable log, write it with explicit I/O so you control when data is flushed and how errors are handled.

NUMA and CPU affinity

On multi-socket machines, memory access cost depends on which socket owns the memory. Keep the NIC’s interrupt or polling cores, the ingestion threads and the shared-memory region on the same NUMA node where possible, and pin threads to cores with pthread_setaffinity_np or taskset. This reduces scheduler migrations and cross-socket cache traffic. Verify the effect with perf rather than assuming it.

Check the rest of the path

Shared memory only fixes the handoff. At this rate you should also look at TLS (many deployments terminate it at a load balancer so the webhook service sees plain HTTP), connection reuse, HTTP parsing cost and signature verification, which is a real CPU cost at 500k verifications per second.

6. What measurements actually show

You will see articles quote tidy tables claiming a 5x throughput gain and a 10x p99 improvement for an mmap/io_uring design. Treat any such table with suspicion unless the benchmark setup is published, because results depend on hardware, kernel version, payload size, connection count and what the application does per request.

Published, independent measurements are more modest:

A University of Edinburgh MSc project comparing io_uring with epoll in HTTP server benchmarks found io_uring ahead by up to 33% in throughput.
A single-threaded TCP echo benchmark with 1024-byte payloads measured about 176k requests/s for io_uring against 169k for epoll with one connection, and nearly identical results (about 421k) at 256 connections. At 512 connections epoll was ahead.
A Netty benchmark on Linux 6.6 found io_uring with half as many worker threads delivered about 3% higher throughput than epoll, with over twice the per-thread efficiency and far fewer CPU migrations.
The maintainers of uWebSockets have reported that io_uring can lose to epoll for larger messages when it uses more memory and falls out of CPU cache, and that localhost benchmarks can mislead.

The gains are real, but they are usually tens of percent, not an order of magnitude, and they depend heavily on the workload. For reference, public HTTP framework benchmarks (such as HttpArena) show well-tuned epoll and io_uring servers exceeding a million simple requests per second on large multi-core machines. So 500k RPS is reachable without exotic techniques when the per-request work is small. The real constraint is how much work each webhook needs.

To evaluate this design for your own workload:

Start with a baseline: a well-tuned epoll server with in-process queues, under realistic payloads and connection counts.
Profile with perf and flame graphs to see how much time is actually spent in system calls, copies, scheduling and TLS.
Change one thing at a time (batching or multishot receive, then SQPOLL, then shared-memory handoff) and measure throughput, CPU per request and p99/p999 latency.
Test on separate client and server machines over a real network, because loopback results can mislead.

Summary

Reaching 500k RPS on a webhook ingestion service means spending a few microseconds of CPU per request, so syscalls, copies, wake-ups and lock contention all matter. A shared-memory SPSC ring buffer gives you a clean, allocation-free, syscall-free handoff between ingestion and worker processes. io_uring reduces the cost of the network side through batching, multishot receive, optional SQPOLL, and zero-copy receive on kernel 6.15+ with supporting NICs. Neither technique removes the kernel TCP stack, and independent benchmarks suggest incremental rather than dramatic gains over a well-tuned epoll server. The reliable path is to measure your baseline, adopt these techniques where profiling shows a bottleneck, and design for backpressure and durability from the start.

References

Linux 6.15 io_uring zero-copy receive: https://www.phoronix.com/news/Linux-6.15-IO_uring
Kernel documentation for io_uring zero-copy Rx: https://kernel.org/doc/html/v6.15/networking/iou-zcrx.html
Evolution of io_uring Zero-Copy Receive (netdev 0x1a): https://netdevconf.info/0x1A/docs/netdev-0x1a-paper41-talk-paper.pdf
io_uring_setup(2) and SQPOLL: https://docs.rs/crate/axboe-liburing/latest/source/liburing-rs/docs/io_uring_setup.md
io_uring_prep_recv(3) (multishot, buffer rings, bundles): https://man7.org/linux/man-pages/man3/io_uring_prep_recv.3.html
io_uring SQPOLL overview: https://man.archlinux.org/man/extra/liburing/io_uring_sqpoll.7.en
Quantifying the Cost of Context Switch (Li, Ding, Shen): https://www.cs.rochester.edu/~kshen/papers/expcs2007.pdf
Context-switch measurements: https://web.eecs.umich.edu/%7Echshibo/files/microkiller_report.pdf
System call and context switch measurements (2010): https://git.captnemo.in/nemo/hn-classics/src/branch/master/_stories/2010/13930305.md
Are You Sure You Want to Use MMAP in Your Database Management System? (CIDR 2022): https://www.pdl.cmu.edu/ftp/Database/p13-crotty.pdf
DPDK AF_XDP poll mode driver: https://doc.dpdk.org/guides-22.07/nics/af_xdp.html
io_uring vs epoll HTTP server study (University of Edinburgh): https://project-archive.inf.ed.ac.uk/msc/20236779/msc_proj.pdf
io_uring and epoll echo server benchmark: https://github.com/joakimthun/io-uring-echo-server
Netty io_uring vs epoll benchmark: https://www.besthub.dev/articles/does-netty-s-io-uring-make-the-2-cpu-thread-rule-obsolete-3358ba3ca620
uWebSockets io_uring discussion: https://git.tdem.in/uNetworking/uWebSockets/issues/1603

Originally published at https://instawebhook.com/blog/memory-mapped-webhook-buffers-using-mmap-and-ring-buffers-to-cut-kernel-user-cro

Building a Resilient Webhook Ingress Proxy in Go: Handling Slow Consumers Without Thread Starvation

InstaWebhook — Sun, 01 Nov 2026 02:17:23 +0000

Payment gateways, CI/CD systems, e-commerce platforms and chat tools all push events to your servers as HTTP POST requests. Accepting those requests is cheap. The hard part is what happens when something behind your endpoint slows down: a database under lock contention, a downstream microservice in a garbage-collection spiral, or a third-party API having a bad afternoon.

A naive receiver that does the downstream work inside the request path, or spawns a goroutine for every event, degrades in a predictable way. Memory grows, sockets pile up, latency climbs, and eventually the process stops accepting new webhooks.

A note on terminology: Go multiplexes goroutines over a small set of OS threads, so the failure here is not classic thread-pool starvation. It is exhaustion of memory, sockets and file descriptors, plus unbounded queueing delay. This article uses “thread starvation” loosely, as most people searching for this problem do. The fix is the same: decouple ingestion from delivery and put hard limits on everything in between.

By the end you will have a small, complete Go service that verifies signatures, acknowledges quickly, buffers into a bounded queue, delivers through a fixed worker pool with a tuned HTTP transport, sheds load explicitly, and shuts down without losing queued work. The code was compiled with Go 1.22 and smoke-tested locally.

1. The Anatomy of the Webhook Ingress Problem

Every webhook provider gives you a deadline and a retry policy, and they differ a lot:

Provider	Response deadline	Automatic retries
GitHub	2xx within 10 seconds	None; failed deliveries must be redelivered manually or via the API
Shopify	2xx within 5 seconds	Up to 8 retries over about 4 hours; persistent failures can remove the subscription
Slack Events API	2xx within 3 seconds	Up to 3 retries with exponential backoff
Stripe	No specific timeout published; return 2xx before any complex logic	Live mode: up to 3 days with exponential backoff. Sandbox: 3 retries over a few hours

Three practical conclusions follow:

There is no universal “200–500 ms” rule. The tightest documented deadline among these providers is Slack’s 3 seconds. Acknowledge as fast as you can, but design for the strictest sender you integrate with.
Slow responses cause duplicates. If your handler misses a deadline, the provider counts the delivery as failed even if you eventually process it. Providers that retry will send the event again.
Dropping an event is not equally recoverable everywhere. If you shed load with a 503, Stripe, Shopify and Slack will retry. GitHub will not, so you must redeliver missed events yourself once you recover.

The architecture that satisfies all of this looks like this:

[ Third-Party Senders ]
         │  HTTP POST (webhooks at scale)
         ▼
┌─────────────────────────────────────────┐
│         Go Webhook Ingress Proxy        │
│  - Bounded body read                    │
│  - HMAC signature verification          │
│  - Fast validation, then 202 Accepted   │
└─────────────────────────────────────────┘
         │ non-blocking enqueue (full → 503 + Retry-After)
         ▼
[ Bounded in-memory queue ]  ──(optional)──►  [ Durable broker ]
         │
         ▼
[ Fixed worker pool ] ──► tuned HTTP transport ──► [ Downstream service ]

The Naive Anti-Pattern: Unbounded Goroutines

A common trap is spawning a goroutine per request to handle downstream delivery:

// ANTI-PATTERN: DO NOT USE IN PRODUCTION
func WebhookHandler(w http.ResponseWriter, r *http.Request) {
    payload := parseBody(r)

    go func() {
        if err := sendToSlowDownstream(payload); err != nil { // can block for a long time
            // handle error
        }
    }()

    w.WriteHeader(http.StatusOK)
}

Goroutines are cheap, not free. A goroutine starts with a 2 KB stack, and since Go 1.19 the runtime adapts the initial stack size to the program’s observed average stack usage. That is why the old “2–8 KB” rule of thumb is only approximate.

The real cost is everything each goroutine holds. Little’s Law says average concurrency equals arrival rate multiplied by time in the system. At 15,000 events per second:

With a 50 ms downstream, you have about 750 deliveries in flight.
When the downstream slows to 5 seconds, you have about 75,000.

The stacks alone are roughly 150 MB at 2 KB each. On top of that, each goroutine pins its payload and typically an open connection. The failure modes:

Memory growth. Stacks, payload copies and per-connection buffers accumulate with no upper bound.
GC pressure. The garbage collector has to scan every live goroutine’s stack, so tens of thousands of parked goroutines make each cycle more expensive.
Socket and file-descriptor exhaustion. Each in-flight call to a slow downstream holds a TCP connection, and every open connection consumes a file descriptor, which the OS caps per process (ulimit -n on Linux). Once you hit the cap, new inbound connections fail too.
Silent data loss. The handler returned 200 before the event was durable. If the process dies, those events are gone and the provider believes they were delivered.

Go’s HTTP server already runs each connection in its own goroutine, so the handler is exactly where you must not block on downstream work.

2. Architecture Blueprint: Decoupling Ingress and Egress

A resilient proxy splits an event’s lifecycle into three phases:

Ingress and validation. Read a size-limited body, verify the HMAC signature, check that the payload is well-formed, and return 202 Accepted.
Buffer. Enqueue into a bounded structure. A buffered Go channel is the simplest option, but it lives in memory only. A channel cannot be “backed by persistent storage”; durability requires writing to something external (see section 11).
Dispatch. A fixed pool of workers pulls from the buffer and delivers through a shared, tuned HTTP client, with timeouts, bounded retries and (optionally) a circuit breaker.

Every stage has a hard limit: body size, queue length, worker count, connection count, request timeout. When the downstream is slow, pressure propagates back to the one place designed to absorb it, the queue, and then to an explicit “no” at the front door.

3. Sizing the Pool: Little’s Law Again

Before writing code, decide the numbers:

Delivery capacity ≈ workers ÷ average delivery latency. With 100 workers and a 100 ms downstream, you can deliver about 1,000 events per second. If latency rises to 1 second, capacity falls to about 100 per second.
Queue runway ≈ queue size ÷ (arrival rate − delivery rate). If events arrive at 3,000 per second against a capacity of 1,000, a 5,000-slot queue fills in about 2.5 seconds.

Two things follow. A queue is a shock absorber for bursts, not a fix for sustained overload. And the queue size should be chosen from how much delay and memory you can tolerate, not set as large as possible. A very large queue only converts overload into long delays and eventually an out-of-memory crash.

4. Deep Dive 1: A Bounded Worker Pool

You will often see a two-level pattern in tutorials: a dispatcher, plus a chan chan Job that workers register their private channels into. It is unnecessary in Go. Goroutines blocked on receiving from a single channel are queued by the runtime, and each job goes to exactly one waiting worker. A single shared channel is simpler and easier to shut down correctly:

func (p *WorkerPool) Start(workers int) {
    for i := 0; i < workers; i++ {
        p.wg.Add(1)
        go func() {
            defer p.wg.Done()
            for job := range p.jobs { // exits when the channel is closed and drained
                p.deliver(job)
            }
        }()
    }
}

The worker count is the concurrency limit toward the downstream. Whatever the downstream does, you never have more than workers deliveries in flight.

5. Deep Dive 2: Non-Blocking Enqueue and Backpressure

When arrivals outpace delivery, the handler must never block waiting for space. A select with a default case makes the enqueue non-blocking:

func (p *WorkerPool) Enqueue(job Job) bool {
    select {
    case p.jobs <- job:
        return true
    default:
        return false // queue full: shed load
    }
}

When Enqueue returns false, the handler answers 503 Service Unavailable (or 429 Too Many Requests) with a Retry-After header. Providers are not obliged to honor Retry-After, but providers that retry will use their own backoff schedule and try again. Choose your overflow policy deliberately:

Shed load. Return 503 and rely on sender retries. This works for Stripe, Shopify and Slack, but for GitHub you need your own redelivery process.
Spill to a durable queue. Write overflow to Redis Streams, RabbitMQ or Kafka instead of rejecting.
Drop selectively. For low-value events (heartbeats, noisy notifications), dropping may be acceptable. Make that an explicit per-event-type decision.

6. Deep Dive 3: Connection Pooling in Go’s HTTP Client

Go’s http.DefaultTransport is reasonable for general use, but it is tuned for modest concurrency: MaxIdleConns is 100, IdleConnTimeout is 90 seconds, and DefaultMaxIdleConnsPerHost is only 2. A webhook proxy that sends many concurrent requests to a single downstream host will keep at most 2 idle connections for it. Every other connection is closed after use and re-dialed, with a new TCP handshake and, for HTTPS, a new TLS handshake. Under load, that churn also produces many sockets in TIME_WAIT.

Two clarifications on common advice:

Creating a new http.Client is not itself the problem. A client with a nil Transport shares DefaultTransport. The problem is creating a new http.Transport per request, which defeats connection reuse entirely. Create one transport and share it.
To reuse a connection, the response body must be read to EOF and closed. Skipping the drain makes Go close the connection instead of returning it to the pool.

A transport tuned for a proxy delivering to one primary downstream:

transport := &http.Transport{
    Proxy: http.ProxyFromEnvironment,
    DialContext: (&net.Dialer{
        Timeout:   5 * time.Second,
        KeepAlive: 30 * time.Second,
    }).DialContext,
    ForceAttemptHTTP2:     true,
    MaxConnsPerHost:       100, // hard cap on connections to a single downstream
    MaxIdleConns:          100,
    MaxIdleConnsPerHost:   100, // the default is 2
    IdleConnTimeout:       90 * time.Second,
    TLSHandshakeTimeout:   5 * time.Second,
    ResponseHeaderTimeout: 8 * time.Second,
    ExpectContinueTimeout: 1 * time.Second,
}
client := &http.Client{Transport: transport, Timeout: 10 * time.Second}

Why these values matter for slow consumers:

MaxConnsPerHost and MaxIdleConnsPerHost should track the worker count. There is no point keeping 500 idle connections to a host when only 100 workers can use them.
Timeouts bound how long a worker can be held hostage. Without Client.Timeout and ResponseHeaderTimeout, a single hung downstream can occupy every worker indefinitely.
ForceAttemptHTTP2 is needed here. Go disables automatic HTTP/2 on a custom transport that sets DialContext or a TLS config unless you ask for it. With HTTP/2, requests are multiplexed over few connections, so the per-host idle settings matter less for HTTPS targets.

7. A Complete, Runnable Example

The program below combines everything: size-limited reads, HMAC verification, a non-blocking bounded queue, a worker pool with bounded retries and jitter, a tuned transport, and a graceful shutdown that drains the queue. It requires Go 1.22 or newer (method-based ServeMux patterns, math/rand/v2).

package main

import (
    "bytes"
    "context"
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "log"
    "math/rand/v2"
    "net"
    "net/http"
    "os"
    "os/signal"
    "strings"
    "sync"
    "syscall"
    "time"
)

const (
    maxWorkers   = 100
    queueSize    = 5000
    maxBodyBytes = 1 << 20 // 1 MiB
    maxAttempts  = 4
)

// Job is one accepted webhook waiting for delivery.
type Job struct {
    EventID string
    Body    []byte // the original bytes, forwarded untouched
}

// WorkerPool delivers jobs with a fixed number of goroutines.
type WorkerPool struct {
    ctx    context.Context // cancelled to abort in-flight work on a hard shutdown
    jobs   chan Job
    wg     sync.WaitGroup
    client *http.Client
    target string
}

func NewWorkerPool(ctx context.Context, client *http.Client, target string, queueSize int) *WorkerPool {
    return &WorkerPool{
        ctx:    ctx,
        jobs:   make(chan Job, queueSize),
        client: client,
        target: target,
    }
}

func (p *WorkerPool) Start(workers int) {
    for i := 0; i < workers; i++ {
        p.wg.Add(1)
        go func() {
            defer p.wg.Done()
            for job := range p.jobs { // exits when the channel is closed and drained
                p.deliver(job)
            }
        }()
    }
}

// Enqueue never blocks: it returns false when the queue is full.
func (p *WorkerPool) Enqueue(job Job) bool {
    select {
    case p.jobs <- job:
        return true
    default:
        return false
    }
}

// Drain stops accepting work and waits for queued jobs to finish or ctx to expire.
// Call it only after the HTTP server has stopped, so nothing can send on a closed channel.
func (p *WorkerPool) Drain(ctx context.Context) error {
    close(p.jobs)
    done := make(chan struct{})
    go func() { p.wg.Wait(); close(done) }()
    select {
    case <-done:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

func (p *WorkerPool) deliver(job Job) {
    backoff := 500 * time.Millisecond
    for attempt := 1; attempt <= maxAttempts; attempt++ {
        if p.ctx.Err() != nil {
            log.Printf("event %s abandoned: shutting down", job.EventID)
            return
        }
        retry, err := p.post(job)
        if err == nil {
            return
        }
        if !retry || attempt == maxAttempts {
            // In production: write to a dead-letter queue instead of only logging.
            log.Printf("event %s failed after %d attempt(s): %v", job.EventID, attempt, err)
            return
        }
        sleep := backoff + rand.N(backoff/2) // jitter avoids synchronized retries
        select {
        case <-time.After(sleep):
        case <-p.ctx.Done():
            return
        }
        backoff *= 2
    }
}

func (p *WorkerPool) post(job Job) (retry bool, err error) {
    req, err := http.NewRequestWithContext(p.ctx, http.MethodPost, p.target, bytes.NewReader(job.Body))
    if err != nil {
        return false, err
    }
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("X-Event-ID", job.EventID)

    resp, err := p.client.Do(req)
    if err != nil {
        return true, err // timeouts and connection errors are retryable
    }
    defer resp.Body.Close()
    // Read the body to EOF (bounded) so the connection can be reused.
    _, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 64<<10))

    switch {
    case resp.StatusCode >= 200 && resp.StatusCode < 300:
        return false, nil
    case resp.StatusCode == http.StatusTooManyRequests,
        resp.StatusCode == http.StatusRequestTimeout,
        resp.StatusCode >= 500:
        return true, fmt.Errorf("downstream returned %d", resp.StatusCode)
    default:
        return false, fmt.Errorf("downstream rejected event with %d", resp.StatusCode)
    }
}

func validSignature(secret, body []byte, header string) bool {
    sig, ok := strings.CutPrefix(header, "sha256=")
    if !ok {
        return false
    }
    got, err := hex.DecodeString(sig)
    if err != nil {
        return false
    }
    mac := hmac.New(sha256.New, secret)
    mac.Write(body)
    return hmac.Equal(got, mac.Sum(nil)) // constant-time comparison
}

func webhookHandler(pool *WorkerPool, secret []byte) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        r.Body = http.MaxBytesReader(w, r.Body, maxBodyBytes)
        body, err := io.ReadAll(r.Body)
        if err != nil {
            var tooBig *http.MaxBytesError
            if errors.As(err, &tooBig) {
                http.Error(w, "payload too large", http.StatusRequestEntityTooLarge)
                return
            }
            http.Error(w, "bad request", http.StatusBadRequest)
            return
        }

        // Authenticate before doing anything else with the payload.
        if !validSignature(secret, body, r.Header.Get("X-Hub-Signature-256")) {
            http.Error(w, "invalid signature", http.StatusUnauthorized)
            return
        }

        var evt struct {
            EventID string `json:"event_id"`
        }
        if err := json.Unmarshal(body, &evt); err != nil || evt.EventID == "" {
            http.Error(w, "invalid payload", http.StatusBadRequest)
            return
        }

        if !pool.Enqueue(Job{EventID: evt.EventID, Body: body}) {
            w.Header().Set("Retry-After", "5")
            http.Error(w, "overloaded, retry later", http.StatusServiceUnavailable)
            return
        }

        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(http.StatusAccepted)
        _, _ = w.Write([]byte(`{"status":"accepted"}`))
    }
}

func main() {
    secret := os.Getenv("WEBHOOK_SECRET")
    target := os.Getenv("DOWNSTREAM_URL")
    if secret == "" || target == "" {
        log.Fatal("WEBHOOK_SECRET and DOWNSTREAM_URL must be set")
    }

    transport := &http.Transport{
        Proxy: http.ProxyFromEnvironment,
        DialContext: (&net.Dialer{
            Timeout:   5 * time.Second,
            KeepAlive: 30 * time.Second,
        }).DialContext,
        ForceAttemptHTTP2:     true,
        MaxConnsPerHost:       maxWorkers, // hard cap on connections to one downstream
        MaxIdleConns:          maxWorkers,
        MaxIdleConnsPerHost:   maxWorkers, // the default is only 2
        IdleConnTimeout:       90 * time.Second,
        TLSHandshakeTimeout:   5 * time.Second,
        ResponseHeaderTimeout: 8 * time.Second,
        ExpectContinueTimeout: 1 * time.Second,
    }
    client := &http.Client{Transport: transport, Timeout: 10 * time.Second}

    workCtx, cancelWork := context.WithCancel(context.Background())
    defer cancelWork()
    pool := NewWorkerPool(workCtx, client, target, queueSize)
    pool.Start(maxWorkers)

    mux := http.NewServeMux()
    mux.HandleFunc("POST /webhook", webhookHandler(pool, []byte(secret)))
    mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
    })

    srv := &http.Server{
        Addr:              ":8080",
        Handler:           mux,
        ReadHeaderTimeout: 5 * time.Second,
        ReadTimeout:       10 * time.Second,
        WriteTimeout:      10 * time.Second,
        IdleTimeout:       60 * time.Second,
    }

    sigCtx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    serverErr := make(chan error, 1)
    go func() { serverErr <- srv.ListenAndServe() }()
    log.Println("webhook ingress proxy listening on :8080")

    select {
    case err := <-serverErr:
        log.Fatalf("HTTP server error: %v", err)
    case <-sigCtx.Done():
        log.Println("shutdown signal received")
    }
    stop()

    shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    // 1. Stop accepting requests and wait for in-flight handlers to return.
    if err := srv.Shutdown(shutdownCtx); err != nil {
        log.Printf("HTTP shutdown: %v", err)
    }
    // 2. Let workers finish what is already queued, up to the deadline.
    if err := pool.Drain(shutdownCtx); err != nil {
        log.Printf("drain timed out with %d event(s) still queued", len(pool.jobs))
    }
    cancelWork() // abort anything still in flight
    log.Println("stopped")
}

Design notes on the code

http.MaxBytesReader instead of io.LimitReader. LimitReader silently truncates at the limit, so an oversized payload turns into confusing invalid JSON. MaxBytesReader returns a typed error, so the handler can answer 413.
Signature verification uses the raw bytes. Re-serializing JSON before checking can change the bytes and break the signature. The check uses hmac.Equal for constant-time comparison, and runs before JSON parsing and before the queue, so unauthenticated traffic never consumes queue capacity. The proxy also forwards the original bytes untouched.
Shutdown ordering matters. After Shutdown is called, ListenAndServe returns ErrServerClosed immediately, and the Go documentation warns that the program must not exit before Shutdown itself returns. A common bug is to call Shutdown in a goroutine and let main return as soon as ListenAndServe does. Here main waits for Shutdown, then closes the queue (safe, because no handler can still be enqueueing), then waits for workers.
Server timeouts are set. ReadHeaderTimeout, ReadTimeout, WriteTimeout and IdleTimeout protect the ingress side from slow senders, the mirror image of the slow-consumer problem.
Method routing. Since Go 1.22, ServeMux accepts patterns like "POST /webhook" and automatically replies 405 Method Not Allowed with an Allow header for other methods.

Try it against a slow consumer

Start any stub downstream that sleeps before replying, point DOWNSTREAM_URL at it, and send a signed request:

BODY='{"event_id":"evt_1","data":{}}'
SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$WEBHOOK_SECRET" | sed 's/^.* //')
curl -i -X POST localhost:8080/webhook \
  -H "X-Hub-Signature-256: sha256=$SIG" \
  -d "$BODY"

You should see 202 Accepted within milliseconds regardless of how slow the stub is. A request with a bad signature returns 401, and a GET returns 405. To see load shedding, make the stub sleep for several seconds and fire more than maxWorkers + queueSize events quickly. The surplus gets 503 with Retry-After, while the process’s goroutine count stays flat.

8. Observability: Know When You Are Absorbing Pain

Building the proxy is half the job. These signals tell you whether a slow consumer is hurting you:

Ingestion rate and response codes. Track 202, 401, 413 and 503 separately. A rising 503 rate means the queue is overflowing.
Queue depth (len(jobs)) and oldest-event age. Depth near capacity means your workers cannot keep up. Age is often more useful than depth because it is the delay your users actually experience.
In-flight deliveries against the worker count. If every worker is permanently busy, you are at capacity.
Downstream latency and error rate (p95/p99 of outgoing calls), plus retry counts and dead-lettered events.
Goroutine count via runtime.NumGoroutine(). In this design it should be roughly constant: workers plus per-connection server goroutines. A steady climb indicates a leak. Go 1.27 (released August 2026) also makes the goroutineleak profile in runtime/pprof generally available, after it was an experiment in Go 1.26.

9. Idempotency: Assume Duplicates

Retries mean duplicates, so your downstream must tolerate seeing an event more than once. Stripe’s documentation says events can be delivered more than once and are not guaranteed to arrive in order. Slack documents retry headers (X-Slack-Retry-Num, X-Slack-Retry-Reason). GitHub reuses the same X-GitHub-Delivery GUID when you request a redelivery.

Practical rules:

Deduplicate on the provider’s event ID, with a TTL longer than the provider’s retry window.
Authenticate first, then deduplicate. GitHub’s delivery header is not itself a signed event ID, so don’t trust identifiers until the signature has been verified.
Don’t assume ordering. Design handlers so that a late or repeated event cannot overwrite newer state.
The proxy above forwards an X-Event-ID header so the downstream can implement idempotency cheaply.

10. Security Notes

Use the right header. GitHub’s SHA-256 signature is in X-Hub-Signature-256. Don’t rely on the older SHA-1 X-Hub-Signature.
Reject stale requests where the provider supports it. Some providers include a timestamp in the signed material, which lets you reject replays. Check your provider’s documentation.
Consider IP allow-lists as an extra layer. GitHub publishes its delivery IP ranges through its /meta endpoint and notes that they change occasionally, so refresh the list periodically. Treat it as defense in depth, not a replacement for signatures.
Terminate TLS properly and require HTTPS. Both GitHub and Stripe expect HTTPS endpoints.

11. Advanced Enhancements

Durability. An in-memory queue loses everything on a crash or restart, and a 202 only promises that the event reached memory. For high-stakes events such as payments, write to a durable broker (Redis Streams, RabbitMQ, Kafka) before acknowledging, and let the workers consume from there. A common compromise is the in-memory queue as a fast path with a durable spillover when it fills.

Per-destination isolation. One shared pool means one slow destination can occupy every worker and starve healthy ones (head-of-line blocking). If you fan out to several downstreams, give each its own queue and concurrency limit.

Circuit breaking. When a downstream is failing, retrying only adds load. The sony/gobreaker library (use the generic v2 module) wraps calls in a breaker that fails fast once failures cross a threshold:

import "github.com/sony/gobreaker/v2"

cb := gobreaker.NewCircuitBreaker[int](gobreaker.Settings{
    Name:    "downstream",
    Timeout: 30 * time.Second, // how long the breaker stays open before probing
    ReadyToTrip: func(c gobreaker.Counts) bool {
        return c.Requests >= 20 &&
            float64(c.TotalFailures)/float64(c.Requests) >= 0.5
    },
})

status, err := cb.Execute(func() (int, error) {
    // perform the POST and return the status code
})
if errors.Is(err, gobreaker.ErrOpenState) {
    // don't burn a retry attempt; requeue with a delay or pause consumption
}

Decide what workers do while the breaker is open. Failing fast without pausing consumption simply moves events from the queue into your retry or dead-letter path faster.

Rate limiting. Cap per-sender or per-tenant ingress and per-destination egress so one noisy source cannot consume the shared queue.

12. Conclusion

A webhook proxy is resilient when every resource is bounded and every limit has an explicit failure behavior:

Bounded body size protects memory at the edge.
A fixed worker pool caps concurrency toward the downstream, regardless of how slow it gets.
A bounded, non-blocking queue absorbs bursts and turns sustained overload into a fast, explicit 503.
A shared, tuned http.Transport with timeouts, connection caps and body draining prevents connection churn and hung workers.
Ordered graceful shutdown drains accepted work instead of dropping it.
Idempotent consumers and durable queues cover duplicates and crashes.

None of this makes failures impossible. It makes them bounded, visible and recoverable, which is what you want when someone else’s system is slow and your ingress tier still has to stay up.

Sources and Further Reading

Originally published at https://instawebhook.com/blog/building-a-resilient-webhook-ingress-proxy-in-go-handling-slow-consumers-without

High-Performance Ingress & Network Architecture: Building Resilience at Scale

InstaWebhook — Sat, 31 Oct 2026 02:15:30 +0000

Webhook ingress gateways have to absorb bursty traffic from third-party senders and stay responsive when some of those connections are slow or hostile. This article covers three techniques for hardening such a gateway: using HTTP/3 and QUIC where they fit, defending against slow-connection attacks (Slowloris and Slow-Read), and using memory-mapped (mmap) ring buffers to absorb bursts. It also states where each technique stops helping, because each one is often oversold.

The article is aimed at teams who run their own ingress layer, whether they receive webhooks from providers such as Stripe or GitHub, or ingest events from their own devices and agents.

1. Know the Contract Before Tuning the Network

Network tuning only matters relative to what the sender expects from you. Two documented examples:

Stripe expects a webhook endpoint to use HTTPS, accept POST requests with a JSON event payload, and quickly return a 2xx status. In live mode, Stripe attempts delivery of an event for up to three days with exponential backoff. Events created in a sandbox are retried three times over a few hours.
GitHub expects a 2xx response within 10 seconds. If your server takes longer, GitHub terminates the connection and counts the delivery as failed. GitHub’s guidance is to enqueue the payload and process it in the background. GitHub does not automatically redeliver failed deliveries, so recovering from an outage means redelivering missed webhooks yourself once the server is back.

Deadlines and retry behavior vary by provider, so check each sender’s documentation rather than assuming a general standard. Two consequences follow for gateway design:

The gateway’s main job is to acknowledge fast and persist durably. Everything else (parsing, business logic, fan-out) belongs behind a queue.
Your own endpoint must tolerate duplicates. Retries and redeliveries mean the same event can arrive more than once, so deduplicate on an event or delivery ID.

2. HTTP/3 and QUIC: What They Fix and What They Don’t

Where TCP-based HTTP hurts

Handshake cost. A new HTTPS connection over TCP needs a TCP handshake plus a TLS handshake before the first request byte, which is roughly two round trips with TLS 1.3 and more with TLS 1.2. QUIC integrates TLS 1.3 into its own handshake, so a new connection completes in one round trip. Resumed connections can send application data in the first flight (0-RTT).
Transport-level head-of-line (HoL) blocking. TCP delivers a byte stream in order. When a segment is lost, data that arrived after it waits in the receive buffer until the retransmission arrives. HTTP/2 multiplexes many streams over one TCP connection, so one lost packet can stall every stream on that connection. HTTP/1.1 sidesteps this by using many connections, at the cost of more handshakes and sockets.

TCP head-of-line blocking (HTTP/2 over one TCP connection):
Packet 1 [OK] -> Packet 2 [LOST] -> Packet 3 [HELD] -> Packet 4 [HELD]
                 Delivery to all streams stalls until Packet 2 is retransmitted

What QUIC changes

HTTP/3 (RFC 9114) runs over QUIC (RFC 9000), a UDP-based transport with TLS 1.3 built in. QUIC streams are independent at the transport layer, so loss on one stream does not block delivery on another.

QUIC independent streams:
Stream A: frame [OK]   -> delivered immediately
Stream B: frame [LOST] -> retransmitted, only Stream B waits
Stream C: frame [OK]   -> delivered immediately

Two caveats matter for webhook gateways:

0-RTT data is replayable. An attacker who captures a 0-RTT flight can replay it, so 0-RTT should only carry idempotent requests. The quic-go library makes this explicit: its client allows 0-RTT only for GET requests and notes that 0-RTT provides no replay protection. A webhook is a POST that triggers side effects, so for webhook endpoints leave 0-RTT off or make processing strictly idempotent.
Plain HTTP/3 already gives you stream independence. You do not need datagrams to get the head-of-line benefit. Ordinary HTTP/3 request streams are reliable and independent.

Where HTTP/3 Datagrams fit

QUIC DATAGRAM frames (RFC 9221) carry unreliable data. They are not retransmitted if lost, but unlike raw UDP they remain subject to the connection’s congestion control. HTTP/3 Datagrams (RFC 9297) map them onto HTTP requests. They are associated with a request stream, and in practice they are used with extended CONNECT, as in WebTransport and CONNECT-UDP (RFC 9298).

That has direct consequences for webhook design:

Standard webhook providers do not send datagrams. They send HTTPS POST requests, and the documentation of the providers cited above specifies exactly that. Datagrams are relevant only when you control both ends, for example telemetry agents, IoT devices, or location trackers that you operate.
Datagrams suit data where the newest value supersedes older ones. For a status heartbeat or a position update, dropping a stale value is better than delaying fresh ones behind a retransmission. They are a poor fit for anything that must not be lost, such as payment events.
Datagrams are size-limited. A datagram must fit in a single QUIC packet, so payloads are limited to roughly the path MTU minus protocol overhead. Larger payloads need ordinary streams.

Server sketch in Go (`quic-go`)

The quic-go HTTP/3 package supports HTTP Datagrams (RFC 9297) on both client and server, and the HTTP/3 API changed substantially across releases. For example, Settingser was removed in v0.43.0, and Settings.EnableDatagram was renamed EnableDatagrams. At the time of writing, pkg.go.dev lists v0.61.0 as the latest release. Treat the following as a sketch and verify the types against the version you pin.

package main

import (
    "log"
    "net/http"

    "github.com/quic-go/quic-go"
    "github.com/quic-go/quic-go/http3"
)

// Bounded hand-off to the persistence layer. When it is full, datagrams
// are dropped, which is acceptable for supersedable telemetry.
var ingest = make(chan []byte, 4096)

func datagramHandler(w http.ResponseWriter, r *http.Request) {
    // HTTP Datagrams ride on an HTTP request stream, typically extended CONNECT.
    if r.Method != http.MethodConnect {
        http.Error(w, "CONNECT required", http.StatusMethodNotAllowed)
        return
    }
    w.WriteHeader(http.StatusOK)

    // The ResponseWriter implements http3.HTTPStreamer, which gives access
    // to the underlying request stream and its datagram methods.
    str := w.(http3.HTTPStreamer).HTTPStream()
    defer str.Close()

    for {
        data, err := str.ReceiveDatagram(r.Context())
        if err != nil {
            return // connection closed or request canceled
        }
        select {
        case ingest <- data:
        default: // queue full: drop rather than spawn unbounded goroutines
        }
    }
}

func main() {
    server := http3.Server{
        Addr:            ":4433",
        EnableDatagrams: true,
        QUICConfig:      &quic.Config{EnableDatagrams: true},
        Handler:         http.HandlerFunc(datagramHandler),
    }
    log.Fatal(server.ListenAndServeTLS("cert.pem", "key.pem"))
}

Note the bounded channel: spawning a goroutine per datagram, as many examples do, lets an attacker or a traffic spike create unbounded memory growth.

QUIC also has its own amplification and address-validation concerns, so the usual UDP-facing protections (rate limiting, retry tokens, and a fronting proxy or CDN that terminates QUIC) apply to any public HTTP/3 listener.

3. Preventing Slowloris and Slow-Read Attacks

Webhook endpoints are public by necessity, which makes them targets for application-layer “low-and-slow” attacks that exhaust connection capacity rather than bandwidth.

The attack patterns

Slowloris (popularized by Robert Hansen in 2009). The attacker opens many connections and sends partial HTTP request headers, adding a header line every few seconds to keep each connection alive. If the server dedicates a thread, process, or fixed connection slot to each connection, the pool fills and legitimate requests are refused.
Slow request body. The attacker sends valid headers, then trickles the body (including chunked bodies) at a few bytes per interval.
Slow-Read. The attacker sends a valid request but advertises a very small TCP receive window, so the server can only transmit its response in tiny segments. Response buffers and connection state stay allocated for a long time.

Slowloris lifecycle:
Client --[ POST /webhook HTTP/1.1 ]--------> Server allocates connection state
Client --[ ...wait... one more header ]-----> connection stays open
Client --[ ...wait... one more header ]-----> connection stays open
Result: connection slots exhausted, legitimate senders refused

Event-driven servers such as NGINX handle idle connections cheaply, so thread-per-connection servers are the most exposed. Even so, every server has a finite number of connections and file descriptors, so timeouts and limits still matter.

Key principle: most timeouts measure gaps, not totals

A common mistake is assuming client_header_timeout-style settings bound the total time to receive a request. In NGINX, client_header_timeout, client_body_timeout, and send_timeout apply to the interval between two successive read or write operations, not to the whole transfer. An attacker who sends one byte every few seconds can therefore stay under a gap-based timeout indefinitely. Effective defense combines several layers:

gap timeouts (so completely idle connections are dropped quickly),
caps on header and body size,
per-IP or per-client connection limits,
an absolute deadline for receiving the request head, where the server supports one,
and a front proxy or CDN that absorbs the slow clients.

NGINX has no directive for a minimum transfer rate. If you need rate-based eviction, you must implement it in your own server code or rely on a CDN or WAF feature.

NGINX example

http {
    # Timeouts between successive reads/writes, not whole-request limits
    client_header_timeout 5s;
    client_body_timeout   10s;
    send_timeout          10s;

    # Drop timed-out connections immediately instead of lingering
    reset_timedout_connection on;

    # Bound request size; webhooks are usually small
    client_max_body_size 2m;

    # Per-IP concurrent connection cap
    limit_conn_zone $binary_remote_addr zone=per_ip:10m;

    server {
        listen 443 ssl;
        http2 on;   # the "http2" parameter of listen is deprecated since 1.25.1
        server_name ingress.example.com;

        limit_conn per_ip 100;

        location /api/v1/webhooks {
            # NGINX buffers the request body by default, so a slow client
            # does not hold an upstream connection while it trickles data.
            proxy_pass http://internal_webhook_cluster;
        }
    }
}

Two practical notes. First, the http2 directive was introduced in NGINX 1.25.1, which is why listen 443 ssl http2; is now deprecated. Second, per-IP limits can punish legitimate senders: providers often deliver from a small set of shared IP ranges, so give published sender ranges (such as those GitHub exposes through its /meta endpoint) higher limits or exempt them, and apply the strictest limits to everyone else.

Go `net/http` example

srv := &http.Server{
    Addr:              ":8443",
    ReadHeaderTimeout: 4 * time.Second,  // absolute deadline for request headers
    ReadTimeout:       15 * time.Second, // absolute deadline for the whole request
    WriteTimeout:      15 * time.Second,
    IdleTimeout:       60 * time.Second,
    MaxHeaderBytes:    16 << 10,
}

// In the handler, also cap the body size:
// r.Body = http.MaxBytesReader(w, r.Body, 2<<20)

Go’s server timeouts are absolute, per-request deadlines, which is stricter than NGINX’s gap-based timeouts. The trade-off is that a fixed ReadTimeout can cut off a large but legitimate upload on a slow link. If that matters, use http.ResponseController to extend the read deadline as data arrives.

Rust (Actix Web) example

use actix_web::{web, App, HttpResponse, HttpServer};
use std::time::Duration;

async fn handle_webhook(_body: web::Bytes) -> HttpResponse {
    // Persist or enqueue the payload, then acknowledge quickly.
    HttpResponse::Ok().finish()
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    HttpServer::new(|| App::new().service(web::resource("/webhook").to(handle_webhook)))
        // Time allowed for the client to send the full request head.
        // Slow clients receive 408 Request Timeout. Default: 5 seconds.
        .client_request_timeout(Duration::from_secs(4))
        // Time allowed for connection shutdown. Default: 1 second.
        .client_disconnect_timeout(Duration::from_secs(1))
        .bind(("0.0.0.0", 8080))?
        .run()
        .await
}

client_request_timeout covers only reading the request head, and client_disconnect_timeout governs connection shutdown; neither is a minimum-throughput control. Enforce body size limits and body-read deadlines separately.

Related protocol-level attacks

HTTP/2 introduced its own resource-exhaustion vector: the “Rapid Reset” attack (CVE-2023-44487), disclosed in October 2023, abuses stream creation and immediate cancellation to overload servers. Keep your HTTP/2 implementation and proxies patched and configure concurrent-stream limits. Slow-connection defenses and HTTP/2 hardening are separate concerns.

4. Surviving Traffic Bursts: Durable Buffering with `mmap`

The problem

Bursts of webhooks happen when an upstream system emits many events at once, such as a batch job, a retry storm after your own outage, or a bulk operation. If the ingress process holds every in-flight payload in heap memory (Go slices, Rust Vec<u8>), memory grows with the burst. In a container, exceeding the memory limit gets the process killed by the kernel’s OOM killer and in-flight data is lost.

The most reliable defense is architectural, not a buffer trick:

Cap request body sizes and concurrent in-flight requests.
Apply backpressure: return 429 or 503 with Retry-After when capacity is exhausted. Providers that retry, such as Stripe, will redeliver later.
Acknowledge only after the payload is durably stored, so an acknowledged event is never lost.

What an `mmap` ring buffer gives you

mmap maps a file into the process’s address space. Writes go to kernel page-cache pages that the kernel writes back to disk in the background. A fixed-size file used as a circular buffer gives you a bounded, disk-backed staging area with these properties:

Bounded, preallocated capacity. The buffer cannot grow past the file size. When it is full you reject new work explicitly instead of crashing.
Survives process crashes. Pages written through a MAP_SHARED mapping belong to the kernel’s page cache, so data written before the process dies is not lost when the process is killed. It is not durable against power loss or a kernel crash unless you flush with msync or fsync. Each flush costs latency, so choose a flush policy deliberately.
Less heap pressure. Payload bytes live in file-backed pages rather than the Go or Rust heap, which relieves garbage-collector and allocator pressure.

What it does not give you

It is not “zero OOM.” On Linux, page cache and file-backed memory are charged to the container’s memory cgroup. Kubernetes’ working-set metric, which is what the OOM decision tracks, includes active file-backed memory. The kernel can reclaim clean pages and write back dirty ones, but under heavy write load into a tight limit, dirty pages can cause memory-cgroup OOM kills. Test your configuration under load, tune dirty-page writeback, and size limits with headroom.
It is not zero-copy. Copying the payload from the network buffer into the mapping is still a memory copy. The savings are in allocation and GC behavior, not in copy count.
It needs a defined recovery protocol. After a crash, the consumer must be able to find valid records. Production designs add record checksums or sequence numbers and flush ordering rules.

A correct single-producer, single-consumer ring in Go

Head and tail are monotonically increasing byte counters, with the position in the buffer computed modulo capacity. Records are length-prefixed and may wrap around the end of the region, which the copy helpers handle. This version supports exactly one producer goroutine and one consumer goroutine; with multiple producers, serialize writes with a mutex or give each producer its own ring.

package ringbuf

import (
    "encoding/binary"
    "errors"
    "fmt"
    "os"
    "sync/atomic"
    "syscall"
    "unsafe"
)

const (
    headerSize = 64 // head at offset 0, tail at offset 8, rest reserved
    lenPrefix  = 8
)

var (
    ErrFull  = errors.New("ring buffer full")
    ErrEmpty = errors.New("ring buffer empty")
)

type Ring struct {
    file   *os.File
    mem    []byte // whole mapping, including header
    region []byte // data area
    head   *uint64
    tail   *uint64
}

// Open maps a file-backed ring of the given total size. Reopening an
// existing file of the same size preserves its contents and positions.
func Open(path string, size int) (*Ring, error) {
    f, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0o600)
    if err != nil {
        return nil, err
    }
    if err := f.Truncate(int64(size)); err != nil {
        f.Close()
        return nil, err
    }
    mem, err := syscall.Mmap(int(f.Fd()), 0, size,
        syscall.PROT_READ|syscall.PROT_WRITE, syscall.MAP_SHARED)
    if err != nil {
        f.Close()
        return nil, fmt.Errorf("mmap failed: %w", err)
    }
    return &Ring{
        file:   f,
        mem:    mem,
        region: mem[headerSize:],
        head:   (*uint64)(unsafe.Pointer(&mem[0])),
        tail:   (*uint64)(unsafe.Pointer(&mem[8])),
    }, nil
}

// Write appends one record. It returns ErrFull instead of overwriting
// unread data, so the caller can apply backpressure (for example HTTP 503).
func (r *Ring) Write(p []byte) error {
    need := uint64(lenPrefix + len(p))
    capacity := uint64(len(r.region))
    tail := atomic.LoadUint64(r.tail)
    head := atomic.LoadUint64(r.head)
    if tail-head+need > capacity {
        return ErrFull
    }
    var hdr [lenPrefix]byte
    binary.LittleEndian.PutUint64(hdr[:], uint64(len(p)))
    r.copyIn(tail, hdr[:])
    r.copyIn(tail+lenPrefix, p)
    atomic.StoreUint64(r.tail, tail+need) // publish only after data is written
    return nil
}

// Read removes and returns the oldest record.
func (r *Ring) Read() ([]byte, error) {
    head := atomic.LoadUint64(r.head)
    tail := atomic.LoadUint64(r.tail)
    if head == tail {
        return nil, ErrEmpty
    }
    var hdr [lenPrefix]byte
    r.copyOut(head, hdr[:])
    n := binary.LittleEndian.Uint64(hdr[:])
    out := make([]byte, n)
    r.copyOut(head+lenPrefix, out)
    atomic.StoreUint64(r.head, head+lenPrefix+n)
    return out, nil
}

func (r *Ring) copyIn(pos uint64, src []byte) {
    off := pos % uint64(len(r.region))
    n := copy(r.region[off:], src)
    copy(r.region, src[n:]) // wrap around
}

func (r *Ring) copyOut(pos uint64, dst []byte) {
    off := pos % uint64(len(r.region))
    n := copy(dst, r.region[off:])
    copy(dst[n:], r.region) // wrap around
}

// Sync flushes dirty pages to disk. Call it on your chosen durability schedule.
func (r *Ring) Sync() error {
    _, _, errno := syscall.Syscall(syscall.SYS_MSYNC,
        uintptr(unsafe.Pointer(&r.mem[0])), uintptr(len(r.mem)), uintptr(syscall.MS_SYNC))
    if errno != 0 {
        return errno
    }
    return nil
}

func (r *Ring) Close() error {
    _ = syscall.Munmap(r.mem)
    return r.file.Close()
}

This sketch uses Linux-specific system calls and omits checksums, corruption recovery, and multi-producer support. Treat it as a starting point rather than production code; mature alternatives such as a write-ahead log library or an embedded queue are often a better choice than writing your own.

Choosing a buffering strategy

I deliberately omit latency figures for the options below, because they depend heavily on hardware, payload size, and configuration. Benchmark on your own infrastructure.

Strategy	Memory behavior	Durability	Main risk
In-process heap queue	Grows with the burst	Lost on process crash	OOM kill under bursts
Remote queue or broker (Redis, Kafka, RabbitMQ)	Offloaded to a separate system	Depends on broker configuration	Network hop and an extra dependency on the ingest path
`mmap`-backed ring on local disk	Bounded file size; page cache counts toward cgroup limits	Survives process crash; power loss needs `msync`/`fsync`	Dirty-page pressure and a custom recovery protocol

5. Putting It Together

A defensible layered design looks like this:

Public senders
      │
      ▼
┌──────────────────────────────────────────────────────┐
│ Edge layer: CDN / reverse proxy (NGINX, Envoy, etc.) │
│  • Header/body gap timeouts, size caps, conn limits  │
│  • Terminates TLS (and HTTP/2, optionally HTTP/3)    │
│  • Buffers slow clients away from your application   │
└───────────────────────────┬──────────────────────────┘
                            ▼
┌──────────────────────────────────────────────────────┐
│ Ingress worker (Go / Rust)                           │
│  • Verify signature, enforce body limit              │
│  • Append to durable buffer; ack only after success  │
│  • Return 503 + Retry-After when the buffer is full  │
└───────────────────────────┬──────────────────────────┘
                            ▼
┌──────────────────────────────────────────────────────┐
│ Durable buffer (mmap ring, WAL, or message broker)   │
└───────────────────────────┬──────────────────────────┘
                            ▼
              Async consumers: dedupe, process, route

The priorities, in order:

Acknowledge fast, process later. Providers publish deadlines (10 seconds for GitHub, for example), and handlers that do real work inline miss them and trigger retries.
Bound everything. Connection counts, header and body sizes, queue depth, and buffer capacity all need explicit limits, with backpressure when they are reached.
Use HTTP/3 where you control the clients and it measurably helps. For third-party webhook providers, expect HTTPS POST over HTTP/1.1 or HTTP/2 and design for that. Reserve datagrams for loss-tolerant, supersedable telemetry from your own clients.
Make duplicates harmless. Retries and redelivery are part of every provider’s delivery model, so idempotent processing is mandatory.
Load-test the failure modes. Simulate slow clients, bursts at several times your normal peak, and a process kill mid-write, and watch the container’s memory working set, not just the process RSS.

Deterministic Reordering of Concurrent Webhooks: Handling Out-of-Order State Transitions with CRDTs

InstaWebhook — Fri, 30 Oct 2026 02:13:23 +0000

Introduction: The Multi-Region Webhook Delivery Dilemma

Webhooks are the default way for SaaS providers to push state changes to your systems. Once you accept them through several regional ingress points, you inherit a hard problem: the order in which events arrive is not the order in which they happened.

Providers say this openly. Stripe states that it does not guarantee events arrive in the order they were generated, and that the same event can be delivered more than once. In live mode it retries failed deliveries with exponential backoff for up to three days. Its own example is a subscription creation that can emit customer.subscription.created, invoice.created, invoice.paid and charge.created, with no promise about their arrival order. Shopify and GitHub give similar warnings. GitHub says deliveries can arrive in a different order than the events occurred. It also does not automatically redeliver failed deliveries, so you must redeliver them manually. Analyses of dozens of webhook platforms find the same pattern: most providers disclaim ordering without supplying the metadata a consumer needs to rebuild it.

Deploying ingress in several regions adds more reordering. Different network paths, queue workers that finish jobs in a different order than they started them, and retries all interleave events for the same resource.

Retries also mean you cannot get exactly-once delivery. If a sender never retries, it loses data when a delivery fails. If it does retry, it will occasionally duplicate events that had already arrived. The practical target is therefore at-least-once delivery plus an idempotent, order-insensitive consumer.

This article shows one way to build such a consumer. It combines Conflict-Free Replicated Data Types (CRDTs) with logical timestamps and a deterministic tie-breaker, so that every replica converges to the same state no matter how events are interleaved. It also covers when this design is overkill.

1. The Anatomy of Out-of-Order Webhook Failures

HTTP itself gives you no delivery guarantee. Reliability comes from the provider’s retry logic, and that logic is the source of both duplicates and reordering.

Common Failure Modes in Multi-Region Ingress

The ghost update. An updated event arrives before the created event it depends on. A handler that does a strict update fails. An upsert that fills in missing fields with defaults or nulls creates a half-populated record.
State reversal. An older event is retried or delayed and arrives after a newer one has been committed. If the handler applies events in arrival order, it overwrites newer state with older state.
Concurrent mutation divergence. Two regions ingest different events for the same resource at nearly the same time and apply them in different orders. Without a merge rule that gives the same result regardless of order, the replicas diverge permanently.

Provider emits:  E1 (created, t=1)   E2 (updated, t=2)

                 E1 ── slow route / retry ──────────► Region B ── applies 2nd
                 E2 ── fast route ───────► Region A ── applies 1st

Region A sees: E2, E1     Region B sees: E1, E2
Arrival-order handling → A and B end in different states

A common first fix is Last-Write-Wins (LWW) on the payload’s wall-clock timestamp. That fix is only as good as the clocks behind it. If the timestamps come from machines with skewed clocks, or you compare them against your own server’s Date.now(), “latest” is not reliable. Plain LWW also silently discards one side of a true concurrent write. LWW is not inherently broken, though. As shown below, an LWW register with logical timestamps and a deterministic tie-breaker is itself a valid CRDT.

2. Foundations: CRDTs and Strong Eventual Consistency

CRDTs were formalized by Shapiro, Preguiça, Baquero and Zawirski in 2011. They defined Strong Eventual Consistency (SEC) and gave sufficient conditions under which replicas of a data type are guaranteed to converge without rollback or coordination. SEC requires three things:

Eventual delivery. An update applied at one correct replica is eventually applied at all correct replicas.
Termination. Every update execution finishes.
Strong convergence. Replicas that have applied the same set of updates are in equivalent states, regardless of order.

State-based CRDTs (CvRDTs)

A state-based CRDT models its state as a join-semilattice. This is a partially ordered set in which every pair of states has a least upper bound, and the merge function computes that bound (⊔). The merge must satisfy three laws:

Commutative: A ⊔ B = B ⊔ A, so arrival order does not matter.
Associative: (A ⊔ B) ⊔ C = A ⊔ (B ⊔ C), so batching does not matter.
Idempotent: A ⊔ A = A, so duplicate deliveries are harmless.

Updates must also be monotonic: a state may only move up the lattice. This is why a plain counter cannot support decrement in a naive state-based design, since merging by max would ignore it.

Operation-based CRDTs (CmRDTs)

Operation-based CRDTs send operations instead of state. They are compact on the wire, but they push guarantees onto the transport: each operation must be delivered exactly once, and in causal order where the data type requires it.

That is the wrong fit for webhooks, which are at-least-once and unordered, and whose operation history you do not control. Webhook handlers should therefore treat each payload as a state fragment and merge it with a state-based join. State-based merges tolerate duplicate and out-of-order delivery by construction.

Logical Clocks

To order events without trusting physical clocks, use logical ones:

Lamport timestamps (Lamport, 1978) guarantee that if event a causally precedes b, then L(a) < L(b). The converse does not hold, so they cannot tell you whether two events were concurrent.
Vector clocks (Fidge and Mattern, 1988) do capture causality, so concurrent events can be detected. They cost one counter per participant.
Provider-supplied versions are the best source of all when they exist: a monotonic sequence number per resource, an updated_at or version field on the object, or an explicit parent event reference.

If the provider supplies none of these, anything you stamp at the edge reflects arrival order at your edge, not the provider’s order. Edge stamping can still give you a consistent internal order for deduplication and replay. It cannot recover the provider’s true event order.

Deterministic Tie-Breaking

When two events carry the same logical timestamp, compare a fixed sequence of keys: (logicalTimestamp, actorId, eventId). Every replica applies the same comparison, so every replica picks the same winner. This works only if that tuple is unique per event. That is why the example below includes eventId as the final key.

3. Architecture of a Deterministic Reordering Layer

+--------------------------------------------------------------+
|                   Multi-Region Edge Ingress                  |
|   verify signature · return 2xx fast · persist raw event     |
+--------------------------------------------------------------+
                              │
                              ▼
+--------------------------------------------------------------+
|             Dedup + Metadata Extraction                      |
|   eventId · resourceId · logical timestamp / version · actor |
+--------------------------------------------------------------+
                              │
                              ▼
+--------------------------------------------------------------+
|                  CRDT State Resolution                       |
|   event → state fragment → join (⊔) with stored state        |
+--------------------------------------------------------------+
                              │
                              ▼
+--------------------------------------------------------------+
|        Visibility Gate (dependencies) + Reconciliation       |
|   hold side effects until `created` seen; TTL → API fetch    |
+--------------------------------------------------------------+
                              │
                              ▼
+--------------------------------------------------------------+
|                    Persisted State Storage                   |
+--------------------------------------------------------------+

Ingestion. Verify the signature, acknowledge with a 2xx quickly, and process asynchronously. Providers enforce delivery timeouts. For example, Primer documents a 10-second timeout. A slow synchronous handler therefore causes retries, which cause duplicates.
Deduplication. Store processed event IDs with a unique constraint. Stripe events carry a stable id for this purpose. Deduplication is not the same as domain idempotency, though. A provider can emit two distinct events that describe the same business transition. The idempotent join below handles that case as well.
Join. Convert the event into a state fragment and merge it into the stored state. Because the merge is commutative, associative and idempotent, arrival order is irrelevant to the final state.
Visibility gate. The join converges even when updated arrives before created. What you still need to hold back are side effects and foreign-key-dependent writes. Keep the merged state internal until the creation event has been seen. If it never arrives within a time limit, fall back to fetching the resource from the provider’s API.

4. Reference Implementation (TypeScript)

This sketch models a resource as a map of per-field LWW registers, plus a grow-only created flag and a delete tombstone. It is not a drop-in production system.

interface WebhookEvent {
  eventId: string;
  resourceId: string;
  eventType: 'CREATED' | 'UPDATED' | 'DELETED';
  logicalTimestamp: number;   // provider version/sequence, or a Lamport value
  actorId: string;            // issuing node/region, used for tie-breaking
  data: Record<string, unknown>;
}

interface Stamp { ts: number; actor: string; eventId: string }
interface Field { value: unknown; stamp: Stamp }

interface ResourceState {
  fields: Map<string, Field>; // per-field last-writer-wins registers
  created: boolean;           // grow-only flag: once true, stays true
  deleted?: Stamp;            // tombstone (delete-wins policy)
}

// Total order on stamps: (ts, actor, eventId). Unique per event.
const cmp = (a: Stamp, b: Stamp): number =>
  a.ts !== b.ts ? a.ts - b.ts
  : a.actor !== b.actor ? (a.actor < b.actor ? -1 : 1)
  : a.eventId < b.eventId ? -1 : a.eventId > b.eventId ? 1 : 0;

const EMPTY: ResourceState = { fields: new Map(), created: false };

// The join (⊔): commutative, associative, idempotent.
function merge(a: ResourceState, b: ResourceState): ResourceState {
  const fields = new Map(a.fields);
  for (const [key, fb] of b.fields) {
    const fa = fields.get(key);
    if (!fa || cmp(fb.stamp, fa.stamp) > 0) fields.set(key, fb);
  }
  let deleted = a.deleted;
  if (b.deleted && (!deleted || cmp(b.deleted, deleted) > 0)) deleted = b.deleted;
  return { fields, created: a.created || b.created, deleted };
}

function toFragment(e: WebhookEvent): ResourceState {
  const stamp: Stamp = { ts: e.logicalTimestamp, actor: e.actorId, eventId: e.eventId };
  const fields = new Map<string, Field>();
  for (const [k, v] of Object.entries(e.data)) fields.set(k, { value: v, stamp });
  return {
    fields,
    created: e.eventType === 'CREATED',
    deleted: e.eventType === 'DELETED' ? stamp : undefined,
  };
}

class WebhookReorderingEngine {
  private states = new Map<string, ResourceState>();
  private waitingSince = new Map<string, number>(); // local TTL bookkeeping only

  ingest(e: WebhookEvent, now: number = Date.now()): void {
    const merged = merge(this.states.get(e.resourceId) ?? EMPTY, toFragment(e));
    this.states.set(e.resourceId, merged);

    if (merged.created) this.waitingSince.delete(e.resourceId);
    else if (!this.waitingSince.has(e.resourceId)) this.waitingSince.set(e.resourceId, now);
  }

  // Visible only after `created` is known and while not tombstoned.
  read(resourceId: string): Record<string, unknown> | undefined {
    const s = this.states.get(resourceId);
    if (!s || !s.created || s.deleted) return undefined;
    return Object.fromEntries([...s.fields].map(([k, f]) => [k, f.value]));
  }

  // Resources stuck without a `created` event: reconcile via the provider's API.
  expiredWaiting(now: number, ttlMs: number): string[] {
    return [...this.waitingSince]
      .filter(([, since]) => now - since > ttlMs)
      .map(([id]) => id);
  }
}

A few design notes:

Date.now() appears only in the local TTL bookkeeping. It never influences event ordering.
The version number is not stored in the state. A counter incremented on every applied event would differ across replicas that drop or reorder events differently, which breaks convergence.
Merging per field means the result can combine fields from different events into a state the sender never actually had. If an invariant spans several fields, keep the whole entity in one register keyed by the provider’s version.
Delete-wins is a policy choice. A later “recreate” would need a new incarnation ID in the key.
State lives in memory here. In production it belongs in a durable store, and the merge must run atomically per resource, for example inside a transaction or compare-and-set.

5. Production Resilience Patterns

A. Unbounded buffers and missing events

If a created event is lost, for instance because the provider’s retries were exhausted, resources waiting for it accumulate. Bound the wait with a TTL, move the stuck events to a dead-letter queue, and reconcile by fetching the resource’s current state from the provider’s API. Choose the TTL against the provider’s behavior. Stripe retries over days, so a short TTL means you will usually reconcile by fetching rather than wait. Stripe’s List Events API only returns events from the last 30 days, so a lost event can only be backfilled within that window. Treat that as a hard limit when you plan recovery.

B. Clock drift and sequence sources

Do not order by physical Date.now() across regions. Prefer, in this order:

A version or sequence that the provider supplies.
A Lamport timestamp that your consumers maintain.
An edge-assigned counter per resource. If you use one, note that Redis Cluster does not provide strong consistency. Its replication is asynchronous, so it can lose acknowledged writes during failovers. The WAIT command reduces that risk but does not eliminate it. A counter that must never go backwards needs a store with consensus, such as Raft-based systems like etcd.

Remember that an edge-assigned counter records arrival order at your edge, not the provider’s order.

C. Tombstone garbage collection

You cannot drop a tombstone arbitrarily. If it is pruned while a delayed created or updated event is still in flight, that event can resurrect the deleted resource. The CRDT literature handles this with causal stability: a tombstone is safe to remove only once every replica is known to have seen the delete. For webhooks, a practical rule is to keep tombstones longer than the provider’s maximum retry window plus your own replay window. Stripe retries for up to three days. Adyen queues events for up to 30 days. Hookdeck documents both. A fixed 72-hour window would therefore be too short for some providers, so make it configurable per provider.

D. When to skip the CRDT

If the provider has an API that returns an authoritative current object, the simplest approach is often to treat the webhook as a notification. You fetch the object by ID and overwrite your local copy from that response. Many engineers recommend this for Stripe subscription, customer and invoice objects, at the cost of one extra API call per event. The CRDT approach is worth its complexity when:

the payload is the only source of truth,
the API is rate limited or slow,
you need replicas in several regions to converge without calling the provider on each event.

Summary: Approaches Compared

Strategy	Out-of-order delivery	Concurrent multi-region writes	Duplicates
Apply in arrival order	Overwrites newer data with older data	Replicas diverge	Unsafe without dedup
LWW on physical timestamps	Correct only if clocks are reliable	Subject to clock skew; one concurrent write is silently lost	Safe if writes are idempotent
LWW/CRDT with logical stamps + tie-breaker	Converges regardless of arrival order	Deterministic winner at every replica	Idempotent join, so safe
Fetch current state on notification	Not affected by order	Not affected by order	Safe, but costs an API call per event

Conclusion

Webhook ordering is not something to rely on. Major providers document that it is not guaranteed, and retries guarantee that duplicates will happen. A robust consumer acknowledges quickly, deduplicates on a stable event ID, and merges each event into state with a function that is commutative, associative and idempotent. A fixed tie-breaker keeps every replica’s choice identical, and logical clocks or provider versions replace unreliable wall-clock time. Where the provider offers an authoritative API, combine that approach with reconciliation by fetch. Together these give you convergence in every region, even when the network interleaves events differently at each one.

References

Shapiro, Preguiça, Baquero, Zawirski, “A Comprehensive Study of Convergent and Commutative Replicated Data Types,” INRIA RR-7506, 2011: https://inria.hal.science/inria-00555588/document
Shapiro et al., “Conflict-free Replicated Data Types,” SSS 2011: https://www.cs.tufts.edu/~nr/cs257/archive/marc-shapiro/CRDTs_SSS-2011.pdf
Lamport, “Time, Clocks, and the Ordering of Events in a Distributed System,” CACM, 1978
Fidge, “Timestamps in Message-Passing Systems That Preserve the Partial Ordering,” 1988; Mattern, “Virtual Time and Global States of Distributed Systems,” 1988
Stripe webhook behavior (retries, ordering, duplicates): https://dev.to/shannonllc/stripe-webhooks-retries-ordering-duplicates-timeouts-and-ips-46f0
Hookdeck, “Common Mistakes with Outbound Webhooks”: https://hookdeck.com/outpost/guides/common-outbound-webhook-mistakes.md
Hookdeck, “At-Least-Once vs. Exactly-Once Webhook Delivery Guarantees”: https://hookdeck.com/webhooks/guides/webhook-delivery-guarantees.md
Provider ordering summary (Stripe, Shopify, GitHub): https://www.vincentschmalbach.com/why-webhook-events-arrive-out-of-order/
Primer webhook documentation: https://primer.io/docs/api-reference/get-started/configure-webhooks
Redis Cluster consistency guarantees: https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/

Originally published at https://instawebhook.com/blog/deterministic-reordering-of-concurrent-webhooks-handling-out-of-order-state-tran

High-Velocity Webhook Engineering: Low-Level Ingress, Dynamic Flow Control, and Multi-Tenant Isolation

InstaWebhook — Thu, 29 Oct 2026 02:07:54 +0000

Webhook traffic is bursty by nature. A payment provider, a Git host, or a SaaS platform can fire a large batch of events at your endpoint at once. When a router fans those events out to many downstream customers, three problems show up: the cost of network I/O at the edge, one slow customer stalling everyone else, and noisy senders consuming shared capacity.

This guide covers three building blocks of a resilient multi-tenant webhook router:

Efficient ingress with Linux io_uring: multishot receive and provided buffer rings, plus the kernel’s separate zero-copy receive feature.
Dynamic flow control: bounded per-tenant queues, credit-style backpressure, and load shedding to stop “slow consumer” cascades.
Per-tenant rate limiting: an atomic token bucket in Redis (Lua), with notes on Redis Cluster.

One caveat applies throughout. Many webhook workloads are limited by TLS termination, HTTP parsing, signature verification, and downstream I/O rather than by recv() calls. Profile before you adopt anything in section 1.

1. Ingress with `io_uring`: Multishot Receive and Provided Buffers

Where the standard epoll model spends its time

Most high-performance Linux network services use readiness-based event loops. Tokio, for example, runs on epoll through the mio library. Epoll tells you a socket is ready, and your code then issues the actual system call. Reading one chunk of a webhook request looks like this:

The NIC DMA-copies the packet into kernel memory, and the kernel’s network stack processes it and queues the payload on the socket.
epoll_wait() returns, telling userspace the socket is readable.
Userspace calls read() or recv(), which crosses the user/kernel boundary.
The kernel copies the payload from the socket buffer into the application’s buffer.
The application parses and processes the bytes.

Each step is cheap. At very high request rates, though, per-call overhead and the kernel-to-user copy become measurable. Gains from io_uring vary widely with workload, kernel version, mitigations, and batching, so benchmark your own stack instead of relying on generic figures.

What `io_uring` changes

io_uring is a Linux asynchronous I/O interface built around two ring buffers shared between the kernel and the application: a submission queue (SQ) and a completion queue (CQ). Instead of waiting for readiness and then making a call, the application submits operations and later reaps completions. Many operations can be submitted and completed with a single io_uring_enter() call.

Two features matter most for webhook ingestion:

Multishot receive. The kernel’s io_uring_prep_recv documentation describes a multishot variant in which one receive request repeatedly posts completions as data arrives, instead of needing a new submission per receive. Multishot receive is available since kernel 6.0.
Provided buffer rings. The application registers a pool of buffers with the kernel, and each completion consumes one buffer from the pool. The completion reports which buffer was used. The application hands buffers back to the ring when it has finished with them. This allows memory to be allocated per-pool rather than per-connection.

A correction on “zero-copy”

Multishot receive with provided buffers is not zero-copy. The kernel still receives packets into its own memory, runs the TCP stack, and copies the payload into your provided buffer. What you save is submission overhead and per-connection buffer pinning.

True zero-copy receive is a separate feature, io_uring zero-copy Rx (zcrx), added in Linux 6.15. The kernel documentation says it removes the kernel-to-user copy on the receive path, so payload data lands directly in userspace memory. Unlike full kernel-bypass stacks such as DPDK, packet headers are still processed by the kernel TCP stack. It also has strict hardware requirements:

The NIC must support header/data split, so headers go to kernel memory and payloads go to userspace memory.
Flow steering is needed to direct only the intended flows to the zero-copy queues.
RSS is needed to steer all other traffic away from those queues.

The kernel API does not configure the NIC for you; you have to set it up yourself. For a typical webhook gateway behind a load balancer on cloud VMs, zcrx may simply be unavailable. Multishot receive with provided buffers is the more practical starting point.

How the data flows

 Userspace                                   Kernel
 ┌──────────────────────┐   shared rings    ┌──────────────────────────┐
 │ Submission queue (SQ)│ ───────────────►  │ Executes multishot recv  │
 │ Completion queue (CQ)│ ◄───────────────  │ on the socket            │
 │ Provided buffer ring │ ◄── payload ───── │ Copies payload from the  │
 └──────────────────────┘    copied into    │ socket buffer into a     │
                             a free buffer  │ provided buffer          │
                                            └────────────▲─────────────┘
                                                         │
                                        NIC → kernel network stack (TCP)

A Rust sketch

Rust has several io_uring options. The io-uring crate from the tokio-rs project is a low-level binding, and several runtimes (for example tokio-uring, monoio, and glommio) build on or implement io_uring. Tokio’s main runtime itself uses epoll.

The sketch below submits a multishot receive and handles completions. It is simplified: it omits buffer-ring registration and error handling, and the exact API differs between crate versions, so check the docs for the version you use. Buffer rings are registered through the Submitter, for example with register_buf_ring_with_flags, and require careful unsafe memory management.

use io_uring::{cqueue, opcode, types, IoUring};
use std::{io, os::fd::RawFd};

const BUF_GROUP: u16 = 1; // buffer group id of a registered provided buffer ring

fn submit_multishot_recv(ring: &mut IoUring, conn_fd: RawFd) -> io::Result<()> {
    // One submission; the kernel posts a completion each time data arrives.
    let sqe = opcode::RecvMulti::new(types::Fd(conn_fd), BUF_GROUP)
        .build()
        .user_data(conn_fd as u64);

    unsafe {
        ring.submission()
            .push(&sqe)
            .map_err(|_| io::Error::other("submission queue full"))?;
    }
    ring.submit()?;
    Ok(())
}

fn reap_completions(ring: &mut IoUring) {
    for cqe in ring.completion() {
        let bytes = cqe.result(); // number of bytes received, or a negative errno
        let buf_id = cqueue::buffer_select(cqe.flags()); // which provided buffer was filled
        let multishot_still_armed = cqueue::more(cqe.flags());

        // 1. If bytes > 0 and buf_id is Some, read the payload from that buffer,
        //    then return the buffer to the ring.
        // 2. If the multishot request is no longer armed (no MORE flag),
        //    re-submit it for this connection.
        let _ = (bytes, buf_id, multishot_still_armed);
    }
}

Security and operational caveats

io_uring has a large kernel attack surface. In June 2023, Google’s security team reported that 60% of the Linux kernel exploits submitted to its bug bounty program in 2022 involved io_uring. As a result, Google disabled it for Android apps, on ChromeOS, and on its servers. Docker also removed it from its default seccomp profile, which means io_uring calls are blocked by default in many container environments. Check your runtime’s seccomp policy before assuming it will work in production. Administrators can also restrict it system-wide using the kernel.io_uring_disabled sysctl. For a service that handles untrusted internet input, such as a webhook gateway, weigh the throughput gains against that exposure.

2. Preventing the Slow Consumer Cascade

Head-of-line blocking in a multi-tenant router

A webhook router accepts events and forwards them to many customer endpoints. If one customer’s endpoint slows or goes offline and the router does not isolate that customer, the failure spreads:

Shared workers block while trying to deliver to the slow tenant.
Healthy tenants’ events queue behind them.
Threads, connections, and memory run out, and the failure becomes platform-wide.

Incoming events
   │
   ├─► Tenant A queue (full) ──► Endpoint A (slow or failing)  → circuit opens, A's events shed or parked
   │
   └─► Tenant B queue (healthy) ─► Endpoint B (fast)           → unaffected

The core rule is that a delivery problem for one tenant must never consume a resource that another tenant needs.

Bounded queues as credits

Credit-based flow control is an established idea in networking. It appears in technologies such as Fibre Channel, InfiniBand, and PCIe, where a sender may transmit only while it holds credits granted by the receiver. The same idea works at the application layer.

Give each tenant a fixed budget of in-flight events:

Accepting an event consumes one credit from that tenant’s budget.
A credit is returned only when the event leaves the queue, either after successful delivery or after it is moved to durable storage.
When a tenant has no credits left, the ingress path does not block. It sheds load or spills to durable storage.

A bounded Tokio channel is a natural fit for this. Its capacity is the credit budget, and try_send (or try_reserve) fails immediately instead of waiting.

What “shedding” means depends on who is sending

How you reject an event matters because the upstream provider’s retry behavior varies. Stripe, for instance, documents that in live mode it retries delivery for up to three days with exponential backoff, and recommends returning a 2xx quickly before doing heavy work. If your router returns 429 or 503 to such a sender, the event will come back later. Other senders may retry only briefly or not at all. The safer pattern is:

Verify the signature and persist the event durably first, then return 2xx quickly.
Deliver downstream asynchronously from a per-tenant queue.
When a tenant’s in-memory queue is full, spill to a disk-backed or broker-backed queue (RocksDB, Kafka, SQS, or similar) rather than dropping.
Reserve 429 or 503 with a Retry-After header for cases where durable storage itself is saturated, and only for providers you know will retry.

Per-tenant isolation in Rust

The sketch below gives each tenant a bounded channel and its own consumer task. It uses a read-write lock so lookups for different tenants do not serialize on a single mutex.

use std::{collections::HashMap, sync::Arc, time::Duration};
use tokio::sync::{mpsc, RwLock};

#[derive(Debug)]
pub struct WebhookPayload {
    pub event_id: String,
    pub body: Vec<u8>,
}

#[derive(Debug)]
pub enum IngestError {
    UnknownTenant,
    TenantSaturated, // out of credits: spill to durable storage or shed
    TenantClosed,
}

pub struct RouterManager {
    tenants: RwLock<HashMap<String, mpsc::Sender<WebhookPayload>>>,
}

impl RouterManager {
    pub fn new() -> Arc<Self> {
        Arc::new(Self { tenants: RwLock::new(HashMap::new()) })
    }

    pub async fn register_tenant(&self, tenant_id: String, capacity: usize) {
        let (tx, mut rx) = mpsc::channel::<WebhookPayload>(capacity);
        let id = tenant_id.clone();

        // One isolated consumer per tenant: a slow endpoint only stalls its own queue.
        tokio::spawn(async move {
            while let Some(event) = rx.recv().await {
                // Bound every delivery attempt so a hung endpoint cannot block forever.
                let _ = tokio::time::timeout(
                    Duration::from_secs(10),
                    deliver(&id, &event),
                )
                .await;
                // On timeout or error: schedule a retry, and trip a per-tenant circuit breaker.
            }
        });

        self.tenants.write().await.insert(tenant_id, tx);
    }

    pub async fn ingest(&self, tenant_id: &str, event: WebhookPayload) -> Result<(), IngestError> {
        let tenants = self.tenants.read().await;
        let tx = tenants.get(tenant_id).ok_or(IngestError::UnknownTenant)?;

        match tx.try_send(event) {
            Ok(()) => Ok(()),
            Err(mpsc::error::TrySendError::Full(_)) => Err(IngestError::TenantSaturated),
            Err(mpsc::error::TrySendError::Closed(_)) => Err(IngestError::TenantClosed),
        }
    }
}

async fn deliver(_tenant_id: &str, _event: &WebhookPayload) {
    // HTTP POST to the customer's endpoint with signing, timeouts, and retry scheduling.
}

Two design notes. First, a bounded channel gives you fairness in memory, but you still need per-tenant concurrency limits on outbound connections, otherwise one tenant with thousands of queued events can monopolize the shared HTTP client pool. Second, a circuit breaker per endpoint, which stops attempts after repeated failures and probes occasionally for recovery, prevents wasted work against a dead endpoint.

3. Per-Tenant Rate Limiting with a Redis Token Bucket

Why rate limit

Failures and bugs can create floods without any malicious intent. A customer’s endpoint that returns errors triggers retries, and a buggy integration can loop and resend events. Per-tenant limits keep one sender’s problem from consuming shared capacity.

Atomic token bucket in Lua

A token bucket stores the number of tokens and the time of the last refill. A read-modify-write sequence done through separate GET and SET commands is racy when several gateway nodes run it concurrently. A Redis Lua script runs atomically as a single command, so no other command executes in the middle of it.

The script below improves on the common version in three ways:

It reads time from Redis with TIME instead of trusting each gateway’s clock, which avoids skew between nodes. Reading time in scripts works with current Redis replication behavior, where scripts replicate by their effects.
It uses HSET, because HMSET has been deprecated since Redis 4.0.
It returns floats as strings, because Redis converts Lua numbers in replies to integers and truncates them.

-- KEYS[1]: bucket key, e.g. "{tenant_xyz}:rate_limit"
-- ARGV[1]: bucket capacity (maximum burst)
-- ARGV[2]: refill rate in tokens per second
-- ARGV[3]: tokens requested for this call (usually 1)

local key         = KEYS[1]
local capacity    = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local requested   = tonumber(ARGV[3])

local t   = redis.call("TIME")
local now = tonumber(t[1]) + tonumber(t[2]) / 1000000

local bucket = redis.call("HMGET", key, "tokens", "ts")
local tokens = tonumber(bucket[1])
local ts     = tonumber(bucket[2])

if tokens == nil or ts == nil then
  tokens = capacity
else
  local elapsed = math.max(0, now - ts)
  tokens = math.min(capacity, tokens + elapsed * refill_rate)
end

local allowed = 0
local retry_after = 0
if tokens >= requested then
  tokens = tokens - requested
  allowed = 1
else
  retry_after = (requested - tokens) / refill_rate
end

redis.call("HSET", key, "tokens", tokens, "ts", now)
-- Expire idle buckets once they would have fully refilled anyway.
redis.call("PEXPIRE", key, math.ceil(capacity / refill_rate * 1000))

return { allowed, tostring(tokens), tostring(retry_after) }

On a denied request, use retry_after to populate a Retry-After header.

Redis 7.0 also introduced Redis Functions, which are invoked with FCALL. Unlike EVAL scripts, functions are persisted with the dataset and replicated to replicas, so you do not need to resend script source from every client. Consider them if you run Redis 7 or later.

Redis Cluster considerations

Declare every key in KEYS. In Redis Cluster, all keys a script touches must be passed in KEYS and must hash to the same slot. Accessing undeclared keys inside a script breaks cluster routing.
Use hash tags when a script touches multiple keys. The script above uses one key per tenant, so it works as is. If you extend it to also read, say, a per-tenant configuration key, name both with the same tag (for example {tenant_xyz}:bucket and {tenant_xyz}:config) so they land in the same slot.
Expect hot slots. Each tenant’s bucket lives on one node, so a single very busy tenant is bounded by what that node can serve. Spread tenants across slots and monitor slot-level load.
Limit round trips. Redis executes commands on a main thread per node, so every limiter call costs capacity there. For very high-volume tenants, let each gateway keep a local bucket and lease tokens from Redis in batches (for example, take 100 tokens at a time and refill when they run low). This reduces Redis calls and latency at the cost of precision: the global limit becomes approximate by up to the total size of outstanding leases. Choose the batch size according to how strict the limit must be. The reduction in Redis traffic depends on your batch size and traffic pattern, so measure it.

Summary

Layer	Common baseline	Alternative discussed here	What you gain, and what to watch
Network ingress	epoll plus a `recv()` call per read	`io_uring` multishot receive with provided buffer rings; zcrx on supported NICs and kernels (6.15+)	Fewer syscalls and simpler buffer management. Multishot is not zero-copy, and the security exposure of `io_uring` needs review.
Flow control	Shared worker pool, unbounded queues, blocking writes	Bounded per-tenant queues with credit-style limits, timeouts, circuit breakers, and durable spill	One tenant’s slowness stays contained; durable persistence avoids data loss when shedding.
Rate limiting	Non-atomic `GET`/`INCR`/`SET` sequences	Atomic Lua token bucket using Redis `TIME`, with hash-tagged keys and optional local token leasing	No race conditions and cluster-safe routing, traded against slot hot spots and approximate global limits when leasing.

Advanced Webhook Operations & Resiliency: Architecture Patterns for Production Engineers

InstaWebhook — Wed, 28 Oct 2026 02:05:19 +0000

Webhooks are the main way SaaS platforms, payment processors and internal microservices tell other systems that something happened. At a few thousand events a day, a loop that POSTs to a URL and retries on failure is enough. At hundreds of millions of events a day, the same loop turns small outages into large ones. Retries pile up, healthy customers get starved by unhealthy ones, and the infrastructure sitting in front of your consumers starts rejecting requests for reasons your application never sees.

This guide covers four areas of production webhook operations:

Adaptive backoff combined with per-endpoint circuit breakers.
Preventing retry storms and cascading failures.
Zero-downtime migration of delivery infrastructure and payload schemas.
Header bloat and the request-size limits of reverse proxies and load balancers.

The numbers in the examples (timeouts, thresholds, cooldowns) are illustrative starting points. Tune them against your own traffic.

1. Adaptive Backoff with Circuit Breakers

Plain exponential backoff (delay = base * 2^attempt) treats every failure the same way. Failures differ in what they say about the consumer, and the dispatcher should react to each differently:

HTTP 429 (Too Many Requests): the consumer, or something in front of it, is shedding load on purpose. Retrying on your normal schedule ignores that signal and can get your delivery IPs throttled or blocked. If the response carries a Retry-After header, honor it. RFC 9110 defines the value as either a number of seconds or an HTTP date.
HTTP 5xx: the consumer is crashing, deploying, or out of database connections. Fast retries add load to a system that is trying to recover.
Timeouts and connection resets: the consumer is slow, overloaded or unreachable. Every slow attempt also holds one of your own workers and sockets, so keep client timeouts short.

                +----------------------------------------+
                |           Incoming Event Queue         |
                +----------------------------------------+
                                    |
                                    v
                +----------------------------------------+
                |     Circuit Breaker State Inspection   |
                +----------------------------------------+
                         /                      \
                [CLOSED / HALF-OPEN probe]    [OPEN]
                       /                          \
                      v                            v
     +----------------------------+     +--------------------------+
     |  Attempt webhook delivery  |     | Skip the network call,   |
     +----------------------------+     | schedule for later retry |
        /          |            \       +--------------------------+
 [2xx Success] [429 Rate Limit] [5xx / Timeout]
      |             |                 |
      v             v                 v
 Reset failure  Honor Retry-After  Increment failure
 counter, close or scaled backoff  counter, trip if
 circuit                           threshold is met

Retry Policy by Response Type

Response / failure	Likely cause	Suggested action
429 Too Many Requests	Consumer or its gateway is rate limiting	Honor `Retry-After` (seconds or HTTP date) and pause delivery to that endpoint. If the header is missing, use a steeper backoff multiplier than for 5xx.
502 / 503 / 504	Gateway or app server down, or mid-deployment	Exponential backoff with full jitter.
500	Bug or unhandled exception in the consumer’s handler	Slower backoff and a lower attempt cap. Retrying rarely fixes a code bug, so don’t let these events clog shared queues.
408, connection or read timeout	Thread pool exhaustion, slow queries, network problems	Short client timeouts (a few seconds) so workers are freed quickly. Route retries to a slower queue.
400, 401, 403, 404	Bad request, expired credentials, misconfigured or removed endpoint	Usually not worth retrying quickly. Dead-letter after a small number of attempts, or retry slowly, because 401/403/404 can appear transiently during consumer deployments.
410 Gone	The consumer has explicitly removed the endpoint	Stop delivering and disable the endpoint.

Providers differ on this. Stripe, for example, treats any non-2xx response as a failed delivery and retries in live mode with exponential backoff for up to three days (Stripe docs). Other platforms stop retrying on most 4xx responses but still retry 408 and 429. Whichever policy you pick, document it for your consumers.

Why Full Jitter

If many events fail at the same moment and all use the same backoff formula, they all retry at the same moments too. Jitter spreads those retries out. In full jitter, described in the AWS Architecture Blog’s Exponential Backoff and Jitter, the delay is a random value between 0 and the capped exponential ceiling:

sleep = random_between(0, min(cap, base * 2^attempt))

Full jitter can occasionally return a very short delay. If that matters, use a variant that keeps a minimum floor (often called “equal jitter”), or add a minimum delay yourself.

Distributed Circuit Breakers for Outbound Webhooks

When one consumer endpoint fails for a long time, retrying millions of events against it wastes worker time and sockets, and it slows delivery to healthy consumers that share the worker pool. A circuit breaker keyed per endpoint (or per consumer domain) stops that. Keep its state in a fast shared store such as Redis, so every dispatcher instance sees the same state.

Closed (normal): events are dispatched normally, and failures are counted.
Open (tripped): when failures cross a threshold (for example, 10 failures within a window, or an error rate above a set percentage), the breaker opens. Events for that endpoint skip the network and go straight to a delayed retry queue.
Half-open (probing): after a cooldown, one probe request is let through.
- If the probe succeeds, the breaker closes and normal delivery resumes.
- If it fails, the breaker re-opens, ideally with a longer cooldown.

Count only failures that indicate an unhealthy consumer (timeouts, connection errors, 5xx, and arguably 429). A 400 or 401 usually indicates a configuration problem and shouldn’t trip a breaker on its own.

Python Sketch: Adaptive Delay, Jitter and a Redis-Backed Breaker

This is a simplified sketch. The Redis operations are not atomic as written. In production, move the read-modify-write logic into a Lua script or use a library that provides atomic breaker state.

import math
import random
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from typing import Mapping, Optional


def parse_retry_after(value: str) -> Optional[float]:
    """Retry-After is either delay-seconds or an HTTP-date (RFC 9110)."""
    try:
        return max(0.0, float(value))
    except ValueError:
        pass
    try:
        when = parsedate_to_datetime(value)
        if when.tzinfo is None:
            when = when.replace(tzinfo=timezone.utc)
        return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
    except (TypeError, ValueError):
        return None


class AdaptiveWebhookDispatcher:
    def __init__(self, redis_client):
        self.redis = redis_client          # redis-py client, decode_responses=True
        self.base_delay = 2.0              # seconds
        self.max_delay = 3600.0            # cap retry delay at 1 hour
        self.failure_threshold = 10        # failures per window before tripping
        self.failure_window = 300          # fixed 5-minute counting window
        self.cooldown_period = 120         # first OPEN period, seconds
        self.max_cooldown = 3600           # cap on escalated cooldowns
        self.probe_ttl = 30                # how long a half-open probe slot is held

    # ---- retry delay -------------------------------------------------
    def retry_delay(self, attempt: int, status: Optional[int],
                    headers: Mapping[str, str]) -> float:
        h = {k.lower(): v for k, v in headers.items()}

        if status == 429 and "retry-after" in h:
            requested = parse_retry_after(h["retry-after"])
            if requested is not None:
                requested = min(requested, self.max_delay)
                # Never retry earlier than asked; add a little jitter on top.
                return requested + random.uniform(0, requested * 0.1)

        # Steeper growth for 429 without Retry-After, standard growth otherwise.
        multiplier = 3 if status == 429 else 2
        ceiling = min(self.max_delay, self.base_delay * math.pow(multiplier, attempt))
        return random.uniform(0, ceiling)  # full jitter

    # ---- circuit breaker --------------------------------------------
    def _k(self, endpoint: str, name: str) -> str:
        return f"cb:{endpoint}:{name}"

    def get_state(self, endpoint: str) -> str:
        if self.redis.exists(self._k(endpoint, "open")):
            return "OPEN"
        if self.redis.exists(self._k(endpoint, "tripped")):
            return "HALF_OPEN"          # cooldown expired, not yet proven healthy
        return "CLOSED"

    def allow_request(self, endpoint: str) -> bool:
        state = self.get_state(endpoint)
        if state == "CLOSED":
            return True
        if state == "OPEN":
            return False
        # HALF_OPEN: allow exactly one probe at a time
        return bool(self.redis.set(self._k(endpoint, "probe"), 1,
                                   nx=True, ex=self.probe_ttl))

    def record_failure(self, endpoint: str) -> None:
        if self.get_state(endpoint) == "HALF_OPEN":
            self._trip(endpoint)        # probe failed: re-open with longer cooldown
            return
        key = self._k(endpoint, "failures")
        count = self.redis.incr(key)
        if count == 1:
            self.redis.expire(key, self.failure_window)  # fixed window, set once
        if count >= self.failure_threshold:
            self._trip(endpoint)

    def _trip(self, endpoint: str) -> None:
        trips = self.redis.incr(self._k(endpoint, "trips"))
        cooldown = min(self.cooldown_period * 2 ** (trips - 1), self.max_cooldown)
        pipe = self.redis.pipeline()
        pipe.setex(self._k(endpoint, "open"), int(cooldown), "1")
        pipe.set(self._k(endpoint, "tripped"), 1)
        pipe.delete(self._k(endpoint, "failures"), self._k(endpoint, "probe"))
        pipe.execute()

    def record_success(self, endpoint: str) -> None:
        self.redis.delete(*(self._k(endpoint, n) for n in
                            ("failures", "open", "tripped", "trips", "probe")))

2. Webhook Storms: Detecting and Preventing Cascading Failures

A webhook storm (or retry storm) happens when an outage creates a large backlog of failed deliveries, and the retries of that backlog, added to normal live traffic, overload the consumer as it recovers and knock it over again.

          [ Consumer outage or bad deployment ]
                          |
                          v
             Many deliveries fail at once
                          |
                          v
          Failed events enter retry pipelines
                          |
             Retries scheduled at identical
             offsets (no jitter): T+30s, T+60s ...
                          |
                          v
        Consumer recovers, receives a retry spike
        on top of live traffic, and fails again
                          |
                          v
              Repeated or cascading outage

Common Causes of Amplification

Synchronized retries. Without jitter, every event that failed in the same outage retries on the same schedule, so the consumer sees repeated spikes instead of a smooth ramp.
Head-of-line blocking. If fresh events share a queue with thousands of failing retries, they wait behind them. Healthy consumers are delayed because one consumer is down.
Naive re-enqueueing. Pushing a failed message straight back to the front of the main queue creates a tight failure loop. Failed work should go through a delayed path instead.
No per-consumer limits. Without concurrency and rate caps per endpoint, a backlog drains as fast as your workers allow, which may be far faster than the consumer can absorb.

Architecture to Contain Retry Storms

Separate real-time delivery from retries, and put limits in front of every consumer:

                 +-------------------------+
                 |     Primary event       |
                 |        ingress          |
                 +-------------------------+
                             |
                             v
                 +-------------------------+
                 | Tier 1: ingestion queue |
                 | (Kafka, Pulsar, etc.)   |
                 +-------------------------+
                             |
                             v
                 +-------------------------+
                 |  Real-time worker pool  |
                 |  (per-endpoint limits)  |
                 +-------------------------+
                      /               \
               [2xx Success]       [Failure]
                    |                  |
                    v                  v
           +----------------+  +---------------------------+
           |   Completed    |  | Tier 2: delayed retry     |
           +----------------+  | queues (short/medium/long)|
                               +---------------------------+
                                          |
                                          | attempts exhausted or
                                          | non-retriable error
                                          v
                               +---------------------------+
                               | Dead Letter Queue (DLQ)   |
                               | inspection and replay     |
                               +---------------------------+

Design Principles

Two tiers of queues.
- Tier 1 handles first-attempt deliveries with short timeouts, so slow consumers can’t hold workers for long.
- Tier 2 holds retries, split into delay buckets (for example 1 minute, 15 minutes, 1 hour). Retry traffic never competes with first attempts for the same workers.
- Implementation note: Kafka has no built-in per-message delay, so delay buckets are usually implemented as separate topics or a scheduler that re-publishes when a message is due. Pulsar supports delayed message delivery natively.
Per-endpoint concurrency and rate limits.
- Cap in-flight requests per endpoint (a semaphore or lease in Redis) so a deep backlog cannot open hundreds of simultaneous connections to one consumer.
- Add a per-endpoint token bucket for request rate. When a consumer returns 429, reduce its rate and recover gradually.
Circuit breakers (Section 1) so persistently failing endpoints stop consuming worker capacity.
Bounded retries and a dead letter queue.
- Stop retrying after a fixed budget, for example a maximum number of attempts or a maximum age (24 hours in your policy, 3 days like Stripe’s live-mode window, or whatever you document).
- Move exhausted or non-retriable events to a DLQ.
- Offer replay, via API or dashboard, so consumers can re-ingest events after they fix the problem. Replays should also be rate limited.
Visibility. Track retry queue depth, age of the oldest retry, per-endpoint failure rate and breaker state. A growing retry backlog is an early warning of a storm.

3. Zero-Downtime Migration of Webhook Delivery

Moving webhook delivery to a new region, a new gateway or a new delivery engine, or introducing a new payload version, must not drop events, send duplicates or break the contract consumers rely on.

Dual-Run with Shadow Delivery

Run the new pipeline alongside the old one, but let only one of them deliver to real customer endpoints.

                 +----------------------------------+
                 |       Upstream event publisher   |
                 +----------------------------------+
                                  |
                                  v
                 +----------------------------------+
                 |          Routing layer           |
                 +----------------------------------+
                      /                        \
                     v                          v
   +-------------------------------+  +-------------------------------+
   | Legacy engine                 |  | New engine                    |
   | ACTIVE: delivers to customers |  | SHADOW: delivers to test sink |
   +-------------------------------+  +-------------------------------+
                     |                          |
                     v                          v
        Customer webhook endpoint       Validation sink and logs

Phase 1: Shadow Mirroring and Validation

Publish each event to both engines. The new engine builds payloads, signs them and applies its routing logic, but sends the HTTP request to an internal validation sink instead of the customer URL. Compare outputs with the legacy engine: payload bytes, signatures, headers, latency and error behavior. Because the shadow engine never reaches customers, it can’t prove that customer endpoints are reachable from the new network. Check reachability separately, for example by testing egress IPs and DNS from the new region.

Phase 2: Payload Versioning

When the payload schema changes, for example a timestamp format change or a restructured object, avoid forcing every consumer to upgrade at once:

Pin a version per endpoint. Each webhook endpoint is configured with a payload version, and the dispatcher renders that version. Stripe takes this approach: events are rendered according to the API version configured for the endpoint.
Prefer additive changes. Adding fields is usually backward compatible. Renaming fields, changing types or changing units is not.
Include a version field in the payload and in your documentation, and publish a deprecation timeline.
Support old versions for a defined period and track which endpoints still use them before turning a version off.

Avoid sending both the old and new payload shapes in a single body. It makes payloads larger, ambiguous for consumers and harder to deprecate.

{
  "id": "evt_987654321",
  "version": "2026-10-01",
  "type": "payment.succeeded",
  "data": {
    "account_id": "acc_123",
    "amount_in_cents": 4999,
    "currency": "USD"
  }
}

Phase 3: Idempotency and Standard Headers

Failover, replays and migrations all raise the chance that a consumer receives the same event twice. Give consumers a stable ID to deduplicate on, and sign requests in a way libraries already understand. The Standard Webhooks specification defines a widely adopted set of headers:

webhook-id: unique per message and stable across all delivery attempts, so consumers can use it as a deduplication key. Any unique string works (it doesn’t have to be a UUID).
webhook-timestamp: Unix timestamp in seconds, set when that attempt is sent (retries carry a new timestamp). Consumers use it to reject stale or replayed requests.
webhook-signature: one or more space-separated signatures, each in the form v1,<base64>, where the value is a Base64-encoded HMAC-SHA256 over {webhook-id}.{webhook-timestamp}.{raw body}.

Because the header holds a list of signatures, you can rotate signing secrets by sending signatures from both the old and new secret during the transition. Stripe uses a similar scheme with its own format: the Stripe-Signature header carries t=<timestamp> and v1=<hex HMAC-SHA256 of "timestamp.payload">.

Consumers must verify the signature against the raw, unparsed request body. A framework that parses and re-serializes the JSON first will produce a different byte sequence and the signature will fail.

Phase 4: Canary Cutover

Move traffic to the new engine gradually, by tenant. Assign each tenant to exactly one active engine using a stable hash of the tenant ID, so that no tenant is delivered to by both engines at once. Raise the share in steps (for example 1%, 10%, 50%, 100%), watching delivery success, latency, signature-verification failures reported by consumers, and retry volume. Keep the routing switch simple enough to roll back immediately.

4. Header Bloat and Proxy Limits

Webhook requests often pass through several hops: a CDN or WAF, a load balancer, an API gateway, and an ingress proxy, before reaching the application. Each hop enforces its own limits on request line and header size. Requests that exceed them are rejected before they reach your consumer’s application code, so the consumer’s application logs show nothing. The evidence is in the proxy or load balancer access logs.

Common sources of large headers:

Large cookies and JWT bearer tokens (a JWT with many claims can be several kilobytes on its own).
X-Forwarded-For and Via chains that grow with each proxy hop.
Multiple custom signature or key-identifier headers.
tracestate and baggage headers from distributed tracing. Note that traceparent itself is small and fixed-length (55 characters per the W3C Trace Context spec), so the tracing headers that grow are tracestate and baggage.

Incoming request
  |
  +-- Headers:
  |    |-- Authorization: Bearer <large JWT>
  |    |-- traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
  |    |-- tracestate / baggage: <vendor and application entries>
  |    |-- Cookie: <many cookies>
  |    |-- X-Forwarded-For: 203.0.113.7, 10.0.0.1, 172.16.0.1 ...
  |    +-- Custom signature / metadata headers
  v
+---------------------------------------------------------+
| Reverse proxy or load balancer                          |
| Header size limit exceeded                              |
+---------------------------------------------------------+
  |
  v
Request rejected at the proxy (400 / 431 / other), never reaching the app

Default Limits of Common Proxies

Defaults vary by version and configuration. Verify against the version you run.

Proxy	Default limits	Typical failure	Tuning
NGINX	`client_header_buffer_size` 1k; `large_client_header_buffers` 4 × 8k. The request line and each individual header line must fit in one 8k buffer, and all headers together must fit in the allocated buffers (32k at defaults).	`414` if the request line is too long; `400 Bad Request` (“Request Header Or Cookie Too Large”) for oversized headers	`large_client_header_buffers 4 16k;`
Envoy	`max_request_headers_kb`: 60 KiB; `max_headers_count`: 100	`431 Request Header Fields Too Large`	Raise `max_request_headers_kb` in the `HttpConnectionManager` (older versions cap it at 96; recent versions allow much higher values)
Apache HTTPD	`LimitRequestLine` 8190 bytes; `LimitRequestFieldSize` 8190 bytes per header; `LimitRequestFields` 100 headers	Request rejected	`LimitRequestFieldSize 16384`, `LimitRequestFields 200`
AWS Application Load Balancer	Request line 16 K; single header 16 K; entire request header 64 K	Request rejected	Hard limits that cannot be changed. Reduce header size.

Two notes on status codes. 431 is the status defined by RFC 6585 for oversized headers. 413 refers to the request body being too large, so a 413 from a webhook endpoint points to payload size, not header size. A 502 can also appear when an upstream closes the connection because of its own limits, so check the logs of every hop.

Also remember that raising the limit on your edge proxy only helps if every proxy and application server behind it accepts the larger size too. An Envoy that admits 128 KB headers in front of a backend that rejects anything above 8 KB just moves the failure one hop inward.

Mitigations

1. Keep Headers Small

Keep authentication to a single compact credential. Where you control token contents, avoid stuffing large claim sets into a bearer token.
Limit tracestate and baggage to what you actually use, and drop them at the edge when sending to third-party consumers.
Don’t add per-attempt or per-hop debug headers to outbound webhooks unless they are needed.
If you sign with several secrets during a rotation, use the standard single webhook-signature header with space-separated signatures, rather than one header per version.

# Anti-pattern: one header per signature version plus extras
X-Webhook-Signature-V1: <signature>
X-Webhook-Signature-V2: <signature>
X-Webhook-Key-Identifier: key_prod_9921381

# Better: one header carrying all active signatures
webhook-signature: v1,<base64-signature-A> v1,<base64-signature-B>

2. Don’t Move Metadata into a Signed Body

It is tempting to push tracing or delivery metadata into the JSON body to save header space. Be careful with this for outbound webhooks. The body is what gets signed, so anything in it that changes per attempt (such as a delivery-attempt counter) changes the signature and makes it harder for consumers to deduplicate by content. Keep the event payload immutable across retries, and carry per-attempt data only where it is needed, such as the timestamp in the signed headers.

3. Tune NGINX Ingress

For a Kubernetes cluster running the NGINX-based ingress controller, the relevant ConfigMap keys are client-header-buffer-size (default 1k) and large-client-header-buffers (default 4 8k):

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  client-header-buffer-size: "4k"
  large-client-header-buffers: "4 16k"   # up to 64k total, 16k per header line

Two cautions:

The http2-max-field-size and http2-max-header-size keys are deprecated in favor of large-client-header-buffers, so don’t rely on them.
The community Ingress NGINX project was retired by the Kubernetes project in March 2026, meaning it no longer receives maintenance or security fixes. The Kubernetes Steering and Security Response Committees urged users to plan a migration to Gateway API or another supported controller. If you still run it, treat the migration as a priority. The F5/NGINX-maintained “NGINX Ingress Controller” is a different project.

4. Raise Envoy’s Header Limit Where Needed

In Envoy, set max_request_headers_kb on the HttpConnectionManager. The configuration below is a minimal, complete listener, including the router filter that an HTTP connection manager requires:

static_resources:
  listeners:
  - name: ingress_webhook_listener
    address:
      socket_address: { address: 0.0.0.0, port_value: 443 }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          max_request_headers_kb: 96     # default is 60
          route_config:
            name: local_route
            virtual_hosts:
            - name: webhook_services
              domains: ["webhooks.example.com"]
              routes:
              - match: { prefix: "/v1/ingest" }
                route: { cluster: webhook_backend_cluster }
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

A production listener would also need TLS configuration (transport_socket) and a defined webhook_backend_cluster, which are omitted here for brevity. Older Envoy releases cap max_request_headers_kb at 96. Newer ones accept larger values, but raising it far above what you actually need weakens the protection the limit provides.

Production Checklist

Error classification: Do 429, 5xx, timeouts, and 4xx responses each follow their own retry policy? Does the dispatcher honor Retry-After in both its formats?
Jitter: Do all backoff schedules use jitter?
Circuit breakers: Are persistently failing endpoints isolated automatically, with half-open probes and escalating cooldowns?
Queue separation: Are first-attempt deliveries decoupled from retry traffic?
Per-endpoint limits: Are concurrency and request rate capped per consumer?
Bounded retries and DLQ: Is there a retry budget, a dead letter queue and a replay path?
Idempotency: Is webhook-id (or equivalent) stable across retries, and are consumers told to deduplicate on it?
Signatures: Is the signed content documented (id, timestamp, raw body), and are timestamps checked on the consumer side?
Versioning: Is the payload version pinned per endpoint, with a published deprecation policy?
Migration safety: Does each tenant have exactly one active delivery engine during cutover, with an instant rollback?
Header budget: Are header sizes measured, and are every hop’s limits (CDN, WAF, load balancer, ingress, app server) known and consistent?
Observability: Do you monitor retry backlog age, per-endpoint failure rates, breaker state and proxy-level rejections?

References

RFC 9110, HTTP Semantics (Retry-After): https://www.rfc-editor.org/rfc/rfc9110
RFC 6585, Additional HTTP Status Codes (429, 431): https://www.rfc-editor.org/rfc/rfc6585
AWS Architecture Blog, “Exponential Backoff And Jitter”: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
Standard Webhooks specification: https://www.standardwebhooks.com/
Stripe documentation, receiving webhook events and retry behavior: https://stripe.com/docs/webhooks
W3C Trace Context: https://www.w3.org/TR/trace-context/
AWS, Quotas for your Application Load Balancers: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-limits.html
NGINX, large_client_header_buffers: https://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers
Envoy, HttpConnectionManager (max_request_headers_kb): https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto
Apache HTTP Server, core directives (LimitRequestFieldSize, LimitRequestFields, LimitRequestLine): https://httpd.apache.org/docs/2.4/mod/core.html
Kubernetes blog, “Ingress NGINX Retirement: What You Need to Know”: https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/
Kubernetes blog, “Ingress NGINX: Statement from the Kubernetes Steering and Security Response Committees”: https://www.kubernetes.io/blog/2026/01/29/ingress-nginx-statement/

Originally published at https://instawebhook.com/blog/advanced-webhook-operations-resiliency-architecture-patterns-for-production-engi

Enterprise Webhook Architecture at Scale: High-Throughput Ingress, Multi-Region CRDTs, and Adaptive Backpressure

InstaWebhook — Tue, 27 Oct 2026 02:05:03 +0000

Webhook traffic is bursty and outside your control. Senders fire when something happens, such as a flash sale, a settlement batch or a fleet of devices reconnecting, and they all do it at once. This article covers three layers that together keep an ingestion tier alive under that load: an in-memory ingress buffer, active-active multi-region replication, and adaptive load shedding. For each layer it also covers the trade-offs, which matter as much as the mechanisms.

A note on numbers: “500,000 requests per second” is used here as a design target, not a benchmark. The real ceiling depends on payload size, TLS termination, core count and NIC bandwidth, so measure on your own hardware.

Part 1: RAM-Disk Ingress Buffers — Using Linux `tmpfs` and Shared Memory to Absorb Webhook Bursts

Why senders force this design

Major webhook senders expect a fast acknowledgment and do the waiting for you. Stripe’s documentation tells receivers to return a 2xx immediately and defer complex logic. In live mode it retries failed deliveries for up to three days with exponential backoff. GitHub expects a 2xx within 10 seconds and otherwise terminates the connection and records a failure. It also does not automatically redeliver failed deliveries, so a missed delivery stays missed until someone redelivers it.

The consequence is that the hot path should do as little as possible before acknowledging. That means validating the signature, appending the payload to a buffer and returning. Writing each payload straight into a relational database, or into a broker configured for synchronous disk persistence, puts storage latency inside the sender’s timeout window. During a spike this produces timeouts, and the timeouts produce retries that add to the spike.

How `tmpfs` behaves

tmpfs keeps its contents in the kernel’s page cache and, if swap is enabled, can push unneeded pages out to swap. According to the kernel documentation, a few details matter for an ingress buffer:

The default size limit is half of physical RAM, and nr_inodes caps the inode count.
Swap is enabled by default. The noswap mount option disables it, which you want here, because a buffer that silently spills to swap gives up the latency profile you built it for.
/dev/shm is normally a tmpfs mount, so POSIX shared memory (shm_open) and files under a dedicated tmpfs mount draw on the same memory pool.

# 64 GiB buffer, no swap, owner-only access
mount -t tmpfs -o size=64G,nr_inodes=2M,mode=0700,noswap tmpfs /mnt/ingress-buffer

Memory in tmpfs is real RAM that other processes can no longer use, so alert on df usage and size the mount against your worst-case burst.

Sizing arithmetic (illustrative). Assume an average payload of 4 KiB at 500,000 RPS. That is roughly 1.9 GiB/s of inbound payload data, so a 64 GiB buffer absorbs about 33 seconds of sustained peak if nothing drains. Two consequences follow:

The buffer buys you time but is not unlimited. The drain path must be sized to catch up after the burst.
Do not write one file per webhook. At 500,000 RPS, a 2M-inode limit would be exhausted in about four seconds. Append payloads to large segment files instead, for example 64–256 MiB each, with a small length-prefixed record header.

Ring buffers and `io_uring`

The pipeline looks like this:

Client TLS / HTTP ──► Edge proxy (Rust or Go; epoll or io_uring)
                            │  validate signature, enforce size limits
                            ▼
                 Append-only segment files on tmpfs
                            │
                            ▼
               Drain workers (user space, batched)
                            │
                            ▼
           Local NVMe log ► Kafka / object storage

A webhook endpoint needs TCP and TLS termination, so the data path goes through the normal kernel network stack. XDP and eBPF are useful earlier in the path, for dropping obviously bad traffic at the NIC before it costs you CPU. They don’t replace the HTTP server.

Within the proxy, lock-free single-producer/single-consumer rings (one per worker thread) are a common way to hand records to a writer without contention. io_uring can then batch the file writes.

io_uring has an IORING_SETUP_SQPOLL mode in which the kernel creates a thread that polls the submission queue. The man page says this lets an application issue I/O without context switching into the kernel, with two caveats worth knowing:

If the polling thread is idle for longer than sq_thread_idle milliseconds, it goes to sleep and sets a flag. The application must then call io_uring_enter(2) to wake it. SQPOLL reduces system calls under sustained load, but it does not eliminate them.
Unprivileged SQPOLL requires a newer kernel (5.13, per the man page). The polling thread also occupies a CPU, so budget cores for it. The io_uring_sqpoll(7) page notes that it only affects submissions, so completion handling still costs CPU.

Some hardened or containerized environments restrict io_uring, so confirm it is permitted on your hosts before building around it.

An illustrative sketch with the Rust io-uring crate follows. Check the API against the crate version you pin:

use io_uring::{opcode, types, IoUring};
use std::{fs::OpenOptions, os::unix::io::AsRawFd};

fn main() -> std::io::Result<()> {
    // SQPOLL: a kernel thread polls the submission queue and sleeps after 1000 ms idle.
    let mut ring = IoUring::builder().setup_sqpoll(1000).build(256)?;

    let file = OpenOptions::new()
        .create(true)
        .write(true)
        .open("/mnt/ingress-buffer/segment-000001.log")?;

    let payload = b"{\"event\":\"example\"}\n";
    let write = opcode::Write::new(
        types::Fd(file.as_raw_fd()),
        payload.as_ptr(),
        payload.len() as _,
    )
    .offset(0)
    .build()
    .user_data(1);

    // Safety: `payload` and `file` outlive the completion below.
    unsafe { ring.submission().push(&write).expect("submission queue full") };
    ring.submit_and_wait(1)?;

    let cqe = ring.completion().next().expect("completion missing");
    assert!(cqe.result() >= 0, "write failed: {}", cqe.result());
    Ok(())
}

The durability trade-off

A buffer in RAM does not survive a power failure or kernel panic, and fsync() on tmpfs buys nothing because there is no backing store. Acknowledging with 202 Accepted after a tmpfs write therefore creates a window in which the sender believes the event is safe but your platform could lose it. Because senders only retry on failure, a lost event that was acknowledged is not retried.

RFC 9110 defines 202 as acceptance for processing, not completion. That fits the pattern, but it doesn’t make the data durable. There are three common ways to narrow the window:

Replicate before acknowledging. Copy the record into a second host’s memory in another failure domain and acknowledge after both writes succeed. This costs a network round trip inside a region, typically on the order of a millisecond between availability zones, according to ThousandEyes measurements.
Group-commit to local NVMe. Acknowledge after a batched, sequential append to a local log. This is slower than RAM but far cheaper than per-request database writes.
Accept the window deliberately for low-value event classes, and reconcile against the sender’s API or redelivery tools after an incident.

Pick one per event class and document it.

Draining to persistent storage

Drain workers run in user space. They read sealed segments from tmpfs, batch them and write them onward. A few practical points:

Object storage. S3 multipart parts must be 5 MiB to 5 GiB, except the last part, with at most 10,000 parts per upload. Use chunks of 8–16 MiB or larger. Chunks below 5 MiB are rejected for non-final parts.
Direct I/O. O_DIRECT is relevant when writing to a local disk log, not when uploading over HTTP. tmpfs itself gained O_DIRECT support in Linux 6.6; on older kernels, opening a tmpfs file with it fails. The drain side can simply use ordinary reads or mmap.
Idempotency. Draining can crash midway and replay a segment. Give every record a stable ID, such as the sender’s delivery ID (GitHub supplies X-GitHub-Delivery) or a hash, so downstream consumers can deduplicate.

Part 2: Active-Active Cross-Region Webhook Ingress — Convergence with CRDTs and Vector Clocks

Why active-active

Consider a provider serving customers from AWS us-east-1 (N. Virginia), eu-central-1 (Frankfurt) and ap-southeast-1 (Singapore). Each region should accept webhooks locally, so that a regional outage or a slow inter-region link doesn’t block ingestion.

To steer clients to a nearby region, there are two options:

Anycast. The same IP is announced from multiple locations and network routing (BGP) delivers each client to a nearby one. Services such as AWS Global Accelerator work this way.
DNS-based routing (latency or geolocation policies). This is a separate mechanism from anycast. Don’t conflate them: anycast operates at the network layer, not through DNS.

A webhook sender such as Stripe will use whichever endpoint you give it. If that is a single hostname, steering happens on your side.

Why not synchronous consensus across regions

Quorum protocols such as Paxos or Raft need at least one round trip to a majority of replicas per decision. Inside a region that is around a millisecond. Across continents it is limited by distance and routing, and typically ranges from tens to hundreds of milliseconds. AWS publishes measured inter-region numbers through Network Manager Infrastructure Performance. Putting that on the acknowledgment path eats into the sender’s timeout budget and ties availability to the slowest link. The alternative is to accept writes locally and replicate asynchronously, and to design the data model so that replicas converge.

Terminology: conflicts, not split-brain

“Split-brain” usually describes two nodes both believing they are the sole leader. In an active-active design, every region is supposed to accept writes, so the real problem is concurrent, possibly conflicting updates to the same entity, plus out-of-order delivery. Two tools address this: vector clocks, which detect concurrency, and CRDTs, which merge it deterministically.

Vector clocks

Each entity carries a vector clock, a map from region ID to a counter. A region increments its own entry when it records an event and takes the element-wise maximum when it receives remote state.

Region A: {A:1, B:0, C:0}  ← event 1
Region B: {A:0, B:1, C:0}  ← event 2 (neither saw the other: concurrent)
After exchange: merged clock {A:1, B:1, C:0}

For vector clocks V₁ and V₂ over N regions:

V₁ ≤ V₂ if V₁[i] ≤ V₂[i] for every region i.
V₁ causally precedes V₂ if V₁ ≤ V₂ and V₁[j] < V₂[j] for at least one j.
If neither V₁ ≤ V₂ nor V₂ ≤ V₁, the events are concurrent, and the merge rule decides the outcome.

Vector clocks grow with the number of participants. That is fine for three or five regions but not for per-client identifiers. Keep them per region, not per sender.

Many systems pair this with hybrid logical clocks, which combine physical time with a logical counter, so that timestamps stay close to wall-clock time yet never go backwards. They are useful as a tiebreaker or for ordering display, but wall clocks alone are not a safe basis for correctness.

CRDTs

Shapiro, Preguiça, Baquero and Zawirski’s 2011 study formalized two families of conflict-free replicated data types. Both guarantee that replicas converge without foreground coordination:

State-based (CvRDT): replicas exchange full or delta state and merge it with an associative, commutative, idempotent function.
Operation-based (CmRDT): replicas exchange operations that commute, which requires reliable, exactly-once delivery of those operations.

Useful building blocks for a webhook platform include:

PN-Counters for approximate global metrics such as delivery attempts or events ingested. They converge, but they cannot enforce a strict global limit, because that would require coordination. Treat a PN-Counter-based rate limit as approximate.
Observed-Remove Sets (OR-Sets) for subscription or endpoint registries, where concurrent adds and removes must converge predictably. An add concurrent with a remove of the same element wins.
LWW registers for single-value fields where “latest write wins” is acceptable.

Here is a last-writer-wins register that uses the vector clock for causality and falls back to a timestamp only for truly concurrent writes. The original idea of merging clocks but then deciding on the timestamp alone can silently discard a causally newer update when clocks are skewed, so the causal check comes first:

use std::collections::HashMap;

#[derive(Clone)]
pub struct LwwRegister<T> {
    pub value: T,
    pub clock: HashMap<String, u64>, // vector clock: region -> counter
    pub hlc_micros: u64,             // hybrid logical timestamp
    pub replica_id: String,          // deterministic tiebreaker
}

enum Causality { Before, After, Equal, Concurrent }

fn compare(a: &HashMap<String, u64>, b: &HashMap<String, u64>) -> Causality {
    let (mut a_less, mut b_less) = (false, false);
    for k in a.keys().chain(b.keys()) {
        let x = a.get(k).copied().unwrap_or(0);
        let y = b.get(k).copied().unwrap_or(0);
        if x < y { a_less = true } else if x > y { b_less = true }
    }
    match (a_less, b_less) {
        (false, false) => Causality::Equal,
        (true, false) => Causality::Before,
        (false, true) => Causality::After,
        (true, true) => Causality::Concurrent,
    }
}

impl<T: Clone> LwwRegister<T> {
    pub fn merge(&mut self, remote: &Self) {
        let remote_wins = match compare(&self.clock, &remote.clock) {
            Causality::Before => true,                      // remote is causally newer
            Causality::After | Causality::Equal => false,   // local is newer or identical
            Causality::Concurrent => {
                // Timestamp first, replica ID as a deterministic tiebreaker
                (remote.hlc_micros, &remote.replica_id) > (self.hlc_micros, &self.replica_id)
            }
        };

        let mut merged = self.clock.clone();
        for (node, &r) in &remote.clock {
            let e = merged.entry(node.clone()).or_insert(0);
            *e = (*e).max(r);
        }

        if remote_wins {
            self.value = remote.value.clone();
            self.hlc_micros = remote.hlc_micros;
            self.replica_id = remote.replica_id.clone();
        }
        self.clock = merged;
    }
}

Two cautions apply. First, with a concurrent update, LWW keeps one value and discards the other. That is acceptable for something like “current endpoint URL” but not for data where both writes matter. For those, use a multi-value register or a domain-specific merge. Second, because senders may deliver the same event more than once and don’t guarantee ordering, receivers should be idempotent regardless. Stripe’s documentation says as much, and GitHub exposes a per-delivery ID header for deduplication.

Part 3: Adaptive Backpressure — Shedding Load Gracefully Before Downstream Saturation Hits 100%

The failure mode

Even with a RAM buffer and multiple regions, downstream consumers can saturate: a database exhausts connections, a partner endpoint slows down, or workers fall behind. Fixed rate limits struggle here because the right limit depends on how much the downstream can handle at that moment. The goal is to start slowing intake before the queue is full, and to slow the least important traffic first.

Little’s Law

For a stable system in steady state, Little’s Law states:

L = λ × W

where L is the average number of items in the system, λ is the average arrival rate, and W is the average time an item spends in the system.

The practical use is as a sizing and sanity check. If downstream slows and W rises while λ stays constant, L must rise. More precisely, once the arrival rate exceeds the service rate, the steady-state assumption no longer holds and the queue grows until it hits a limit, which for an unbounded in-memory queue means running out of memory. Queueing delay also climbs sharply as utilization approaches 100%, so waiting until the queue is nearly full is too late. Shedding reduces λ to something the downstream can sustain. A bounded queue turns an eventual crash into an explicit, controlled rejection.

Saturation signal and dynamic token bucket

Define queue saturation as:

S = current queue depth / maximum queue capacity

A token bucket with a fixed refill rate cannot react to S. Instead, scale the refill rate down as S rises between a low and high watermark. For example, with S_low = 0.75 and S_high = 0.90:

R = R_max × (1 − clamp((S − S_low) / (S_high − S_low), 0, 1))

Below S_low, requests get the full rate. Above S_high, low-priority traffic is refused entirely. Many systems use AIMD (additive increase, multiplicative decrease) on the admitted rate instead, borrowing from TCP congestion control. Either way, feed the controller a smoothed signal such as queue depth plus drain rate, so it doesn’t oscillate.

Incoming webhook ──► classify priority ──► token bucket (rate scaled by S)
                                              │
                          ┌───────────────────┴──────────────────┐
                      tokens available                       no tokens
                          │                                      │
                  append to buffer ──► 202                 429 + Retry-After

HTTP semantics: `429` and `Retry-After`

RFC 6585 defines 429 Too Many Requests. A Retry-After header may accompany it, either as a number of seconds or as an HTTP date. Add jitter so that all clients don’t retry at the same instant. AWS’s “Exponential Backoff And Jitter” article is a good reference on why. A formula consistent with the thresholds below:

delay = min(MaxBackoff, BaseDelay × 2^(k × p)) × jitter

where p = clamp((S − S_low) / (1 − S_low), 0, 1), k is a steepness constant and jitter is a random factor, for example between 0.5 and 1.0.

package backpressure

import (
    "math"
    "math/rand"
    "strconv"
)

const (
    lowWatermark  = 0.75
    baseDelaySecs = 5.0
    maxDelaySecs  = 120.0
    steepness     = 4.0
)

// RetryAfter returns a Retry-After value in seconds, or ok=false when no
// throttling applies.
func RetryAfter(saturation float64) (value string, ok bool) {
    if saturation <= lowWatermark {
        return "", false
    }
    p := math.Min((saturation-lowWatermark)/(1-lowWatermark), 1.0)
    delay := math.Min(baseDelaySecs*math.Pow(2, steepness*p), maxDelaySecs)
    delay *= 0.5 + 0.5*rand.Float64() // jitter between 50% and 100% of delay
    return strconv.Itoa(int(math.Ceil(delay))), true
}

With these constants, the pre-jitter delay is 5 s at S = 0.75, about 26 s at S = 0.90 and 80 s at S = 1.0.

Not every sender will cooperate

Shedding with 429 only helps if the sender retries. The differences between senders are large:

Stripe retries failed deliveries in live mode for up to three days with exponential backoff, so a 429 or 503 costs a delay, not data.
GitHub does not automatically redeliver failed deliveries, so a shed GitHub webhook is lost until someone redelivers it manually or via the API.

So classify by what the sender will do, not only by business importance. Events whose senders won’t retry should be the last to be shed, and should preferably go to the durable buffer or a spill queue. Where you must drop them, log enough to reconcile later. A 503 Service Unavailable with Retry-After is also a reasonable shed response for tiers that are unavailable rather than rate-limited.

Shedding states

State	Queue saturation (S)	Ingress action	Response
Normal	S ≤ 0.75	Accept all traffic into the buffer	`202 Accepted`
Degraded	0.75 < S ≤ 0.90	Shed low-priority events (for example, analytics pings) with a growing delay	`429` with `Retry-After` of roughly 5–26 s
Tripped	S > 0.90	Reject all non-critical events; keep accepting critical ones until the hard limit	`429` with `Retry-After` of roughly 26–80 s; `503` once the hard limit is hit

These are starting points, not universal constants. Tune watermarks against measured drain rates and your senders’ retry behavior.

Export the state machine and its inputs (queue depth, drain rate, admitted vs. rejected counts per priority, current Retry-After) as Prometheus or OpenTelemetry metrics. Alert on transitions, not only on levels, because a system that flaps between Normal and Degraded is a tuning problem you want to see.

Putting it together

The three layers address different failure modes:

RAM-backed ingress keeps acknowledgment latency low during bursts, in exchange for a deliberate durability decision (replicate, group-commit or accept the window) and careful sizing of tmpfs, inodes and swap.
Active-active regions with vector clocks and CRDTs keep ingestion available across regions without cross-continent consensus on the hot path. The cost is eventual consistency, and conflict semantics you must choose per data type.
Adaptive backpressure keeps downstream failures from becoming memory exhaustion, provided the shedding policy accounts for which senders retry.

No design makes webhook ingestion lossless under all failures. What it can do is make the trade-offs explicit and measurable, so that overload degrades the least valuable traffic first and the data you’ve promised to keep survives.

Originally published at https://instawebhook.com/blog/enterprise-webhook-architecture-at-scale-high-throughput-ingress-multi-region-cr

Zero-Trust Webhook Ingress: Enforcing Cryptographic Sender Identity with DNSSEC, DANE, and Verified mTLS

InstaWebhook — Mon, 26 Oct 2026 02:03:58 +0000

Webhooks are how many B2B SaaS integrations learn that something happened: a payment settled, a resource was provisioned, an order shipped. Each webhook endpoint is also a public HTTPS listener that accepts state-changing requests from systems you don’t control.

Most webhook security still lives at the application layer: shared HMAC secrets, static tokens, or IP allowlists. These are workable, and for most providers they are all that’s offered. They also have well-known limits. This guide looks at how to push sender authentication down to the transport layer with mutual TLS (mTLS) enforced at the edge, how DNSSEC and DANE-style TLSA records can serve as a tamper-evident place to publish sender key pins, and how to degrade gracefully for senders that can’t support any of this.

One caveat up front. mTLS for webhooks is real and supported by some providers. Using DNSSEC-signed TLSA records to pin client certificates is not a standardized practice. It is a design pattern built from standardized parts, and this article is explicit about where the standards stop and the engineering begins.

1. Why Shared Secrets and IP Allowlists Fall Short

How HMAC webhook signing works

In the common model, the provider and consumer share a secret. For each event, the provider computes an HMAC (typically HMAC-SHA256) over the raw request body and sends it in a header:

POST /v1/webhooks/orders HTTP/1.1
Host: api.example.com
X-Hub-Signature-256: sha256=a3f5c9e2b1...
Content-Type: application/json

{"event": "order.fulfilled", "order_id": "99214"}

GitHub, for example, signs webhook deliveries with X-Hub-Signature-256, the SHA-256 successor to the older SHA-1 X-Hub-Signature header. Stripe uses a Stripe-Signature header that carries a timestamp and one or more v1 HMAC-SHA256 signatures computed over the string {timestamp}.{body}.

Done correctly, this gives you payload integrity and proof that the sender held the secret. The limitations are structural:

Symmetric secrets must be distributed and rotated. Both sides hold the same key, so anyone who can read it can forge events, and the consumer can forge events “from” the provider. Rotation without dropping deliveries needs an overlap window in which two secrets are accepted. Stripe’s header format supports multiple signatures for this reason.
Replay protection depends on the scheme. A signature over only the body can be replayed indefinitely. Stripe signs a timestamp together with the body, and its official libraries reject events outside a default tolerance of five minutes. GitHub’s signature covers only the payload, so receivers need other controls, such as deduplicating on the delivery ID. Timestamp checks in turn require reasonably synchronized clocks, and clock drift is a common cause of spurious signature failures.
The signature identifies a secret, not a network or cryptographic identity. It says “someone with this key sent this,” nothing more. It is checked after your edge, TLS terminator, and HTTP parser have already accepted and processed the request.

Why IP allowlists are a weak identity signal

Many providers publish source IP ranges. GitHub exposes its hook delivery ranges through its meta API, and Stripe publishes lists in its documentation. Allowlisting them is reasonable defense in depth, but a poor primary control:

Lists change. Providers add and remove egress ranges, and consumers have to track the changes or drop legitimate traffic.
IP says nothing about which tenant sent the request. An allowlisted range identifies the provider’s platform at best. It cannot tell you which customer or workload on that platform originated the request.
Shared egress widens the trust boundary. Anything that can send traffic from the allowlisted range inherits the trust you granted.

Spoofing the source address of a full TCP/TLS exchange is hard in practice, so the real weakness is coarse granularity rather than easy forgery. Use allowlists to narrow exposure, not to establish identity.

2. mTLS: Authenticating the Sender During the Handshake

With mutual TLS, the sender presents an X.509 client certificate during the TLS handshake, and the receiver verifies it before any HTTP request is processed. Compared with HMAC headers:

The sender proves possession of a private key that never leaves it, rather than a secret both parties know.
Authentication happens before the request body is accepted, so unauthenticated senders are rejected at the handshake or edge.
Identity is a certificate (subject, SAN, fingerprint), which maps cleanly to a partner record.

This is already offered by some webhook senders. Truv documents mTLS for webhook delivery, including delivery using a client-supplied certificate. Google’s Dialogflow supports mTLS when calling a customer’s webhook server: Dialogflow presents a client certificate, and the webhook server verifies it. Support is still the exception, so plan on a mix of sender capabilities (see Section 5).

A significant change: public CAs and client authentication

Many teams used to run mTLS with certificates from public CAs, relying on certificates that carried both the serverAuth and clientAuth Extended Key Usages. That is ending. Driven by Chrome Root Program policy, public CAs are removing the clientAuth EKU from newly issued certificates through 2026. Let’s Encrypt, for instance, stopped including it in certificates issued from mid-May 2026 (with an extended deadline for some users of its TLS-client profile), and other large CAs followed similar schedules. Failures tend to appear quietly when a certificate renews and no longer carries clientAuth.

The practical consequence for webhook ingress is that you should not design around public-CA client certificates. Use a private CA (yours or the partner’s), or pin individual partner certificates or public keys directly.

3. Where DNSSEC and DANE Fit (and Where They Don’t)

What the standards define

DANE (DNS-Based Authentication of Named Entities) is defined in RFC 6698, with operational guidance in RFC 7671. It lets a domain publish TLSA records in DNS that bind a TLS server’s certificate or public key to a service name, and it depends on DNSSEC to make those records tamper-evident. DNSSEC signs DNS data using DNSKEY and RRSIG records and links zones through DS records, forming a chain of trust from the DNS root zone down through the TLD (such as .com) to the domain.

A TLSA record is published at a name built from the port, protocol, and server hostname, for example _443._tcp.webhooks.example.com, and has four fields:

_443._tcp.webhooks.example.com. IN TLSA 3 1 1 (
    a1b2c3d4e5f67890123456789abcdef0123456789abcdef0123456789abcdef0 )

Certificate usage 3 (DANE-EE): the end-entity certificate or key itself is the trust anchor. No public CA is needed, so self-signed or privately issued certificates work.
Selector 1: match the SubjectPublicKeyInfo (SPKI) rather than the whole certificate, so the certificate can be re-issued without changing the record as long as the key pair stays the same.
Matching type 1: compare a SHA-256 digest.
Association data: the digest itself.

RFC 7671 recommends publishing records for both the current key and the next key ahead of a rollover, so key changes don’t cause validation failures while DNS caches expire.

DANE is widely deployed for server authentication in SMTP, defined in RFC 7672. For HTTPS, major browsers do not support it. This is why most web traffic still relies on the public CA ecosystem.

The gap for webhooks

Standard DANE authenticates a server by looking up TLSA records at a name derived from the server’s hostname. In webhook delivery, the sender is the TLS client. A client sends no SNI naming itself, so there is no standardized way for a receiver to derive a DNS name from the handshake and check the client’s certificate against it. Any scheme that uses TLSA records for client identity is therefore a custom convention, not an interoperable standard.

A workable version of the pattern looks like this:

At onboarding, each partner gives you a DNS name they control (for example webhook-client.partner.example) and agrees to publish its client-key pin there as a TLSA-format record, in a DNSSEC-signed zone.
A scheduled sync job, not the request path, queries that name using a DNSSEC-validating resolver, rejects results that do not validate, and writes the resulting fingerprints into your edge key-value store.
The edge worker checks incoming client certificates against that store.

This gives you a partner-controlled place to rotate keys without a manual exchange for every rotation, plus tamper-evidence from DNSSEC. Know its costs before committing:

Partner adoption is limited. DNSSEC signing is far from universal. Verisign reported a signing rate of about 4.3% for second-level domains in .com and 5.3% in .net in 2023, and a Q3 2025 report on DNSSEC validation put the global average validation rate at roughly 35%. Many partners will not be able to publish signed records.
You are trusting a validator. Unless you run DNSSEC validation yourself, your trust rests on the resolver that performs it and on the path to it.
Operational risk shifts to DNS. A broken DS record or expired signatures can invalidate a partner’s pin set. Decide in advance whether the sync job fails closed (keep last-known-good pins for a bounded period) or open.
It is non-standard. Document the convention, version it, and expect each partner to need explanation.

For many organizations, a simpler out-of-band registry of partner certificate fingerprints, populated at onboarding and updated through an authenticated portal or API, provides most of the benefit with less moving machinery. DNSSEC-published pins are best treated as an optional enhancement for partners who can support them.

4. Enforcing mTLS at the Edge

Terminating mTLS at an edge platform lets you reject unauthenticated senders before traffic reaches your origin, and lets you update trusted CAs and pinned certificates without redeploying application servers.

On Cloudflare, for example, mTLS can be enabled per hostname and enforced with API Shield. Client certificate details are exposed to Workers in request.cf.tlsClientAuth, with fields such as certPresented, certVerified, certRevoked, certFingerprintSHA256, certSubjectDN, and certSerial. The edge can also forward the certificate to the origin, either through Cloudflare-specific headers (Cf-Client-Cert-Sha256, Cf-Client-Cert-Der-Base64) or through headers following RFC 9440, which standardizes Client-Cert and Client-Cert-Chain headers for conveying client certificates from a TLS-terminating proxy to a backend. Other CDNs and gateways offer comparable features, but their field names and capabilities differ, so check your provider’s documentation.

Worker pattern

The worker below is a sketch of the pattern, written for Cloudflare Workers. Field value formats should be checked against the current documentation before use.

// Edge gatekeeper: accept webhooks only from verified, pinned partner certificates.
// env.PARTNERS is a KV namespace mapping "fp:<sha256-fingerprint>" -> partner record.

const TRUST_HEADERS = ["x-verified-sender-id", "x-sender-cert-fingerprint", "x-ingress-auth-method"];

export default {
  async fetch(request, env) {
    if (request.method !== "POST") {
      return new Response("Method Not Allowed", { status: 405, headers: { Allow: "POST" } });
    }

    const auth = request.cf?.tlsClientAuth;

    // 1. A certificate must have been presented and validated by the edge
    //    against the CA(s) you configured. Verify the exact value formats
    //    for these fields in your platform's documentation.
    if (!auth || auth.certPresented !== "1") {
      return new Response("Unauthorized", { status: 401 });
    }
    if (auth.certVerified !== "SUCCESS" || auth.certRevoked === "1") {
      return new Response("Forbidden", { status: 403 });
    }

    // 2. Pin: the certificate must match a known partner.
    const fingerprint = String(auth.certFingerprintSHA256 || "").toLowerCase();
    const partner = await env.PARTNERS.get(`fp:${fingerprint}`, "json");
    if (!partner) {
      return new Response("Forbidden", { status: 403 });
    }

    // 3. Strip any client-supplied copies of our trust headers, then set our own.
    const headers = new Headers(request.headers);
    for (const name of TRUST_HEADERS) headers.delete(name);
    headers.set("X-Verified-Sender-ID", partner.id);
    headers.set("X-Sender-Cert-Fingerprint", fingerprint);
    headers.set("X-Ingress-Auth-Method", "mtls-pinned");

    return fetch(new Request(request, { headers }));
  },
};

Points worth noting in this design:

The edge already checks validity. Chain, expiry, and revocation status are evaluated by the platform when it verifies the client certificate, so the worker reads the result instead of re-parsing dates. Revocation behavior depends on the platform and how you configured it, so confirm what your provider checks and how quickly revocations take effect.
Pinning adds a second control. Chain validation says “a CA you trust issued this.” The fingerprint lookup says “this is a specific partner we onboarded.” If you use a private CA scoped to a single partner, the fingerprint check can be relaxed to a subject or SAN check.
Strip inbound trust headers. Clients can send their own Client-Cert-style or custom headers. Cloudflare’s documentation explicitly warns about this. Remove any such headers before setting your own, as the worker does.
Protect the origin path. The origin must accept traffic only from the edge (for example, via authenticated origin pulls, a private tunnel, or network restrictions). Otherwise anyone who reaches the origin directly can forge the assertion headers. For higher assurance, have the edge sign the identity headers or pass a short-lived token the origin verifies.
Application-layer checks still matter. mTLS authenticates the sender’s connection. It does not make a payload idempotent or protect against a compromised partner. Keep schema validation, idempotency keys, and replay detection.

What edge verification does and doesn’t buy you

Moving authentication to the edge means unauthenticated traffic is dropped before it consumes origin resources, and trusted-certificate changes become a configuration update at the edge instead of an origin redeploy. Claims that it removes all cryptographic load from the origin or “eliminates” attack surface overreach. The origin still terminates its own TLS (or trusts an edge tunnel), parses requests, and handles business logic.

5. Tiered Fallback for Senders That Can’t Do mTLS

Most webhook senders will not support client certificates, let alone DNSSEC-published pins. A realistic ingress layer treats trust as tiers and routes accordingly:

Tier	Mechanism	What it establishes	Handling
1: Pinned mTLS	mTLS with a pinned partner certificate or key (optionally DNSSEC-published)	Sender proves possession of a private key tied to an onboarded partner	Normal processing; strongest identity signal
2: Signed with replay protection	HMAC signature over timestamp and body, per-partner secrets, optional provider IP ranges	Sender knew a shared secret; replay limited to the tolerance window	Normal processing with signature, timestamp, and dedupe checks
3: Legacy	Static token or HMAC without timestamp	Weakest; vulnerable to leaked secrets and replay	Isolated queue, tighter rate limits, tracked for migration

Implementation notes for the fallback paths:

Route by what the sender can do, not by what it claims. Decide the tier from the verified handshake result and the partner registry, not from request headers the sender controls.
Enforce replay windows where the scheme allows. If the signature covers a timestamp, reject stale events. Stripe’s default tolerance is five minutes, and tighter windows are possible if clocks are well synchronized. Where the signature has no timestamp, deduplicate on the sender’s delivery or event ID.
Use IP allowlists as an additional filter only. Combine provider-published ranges with signature checks, and refresh the lists automatically since they change.
Emit telemetry per tier. Log which tier accepted each event and which partners remain on Tier 3, so migration progress is measurable and you can raise it with vendors.

Conclusion

Static shared secrets and IP allowlists remain the baseline in the webhook ecosystem, and they work when combined with timestamped signatures, secret rotation, and replay protection. Where a partner can support it, mTLS at the edge, with certificates pinned per partner and a private CA instead of a public one, moves sender authentication into the handshake and rejects unauthenticated traffic before it reaches your application.

DNSSEC and TLSA records can add a tamper-evident, partner-controlled way to publish and rotate those pins. This is a custom pattern, not a standard, and DNSSEC adoption among partners is limited, so treat it as an optional layer on top of a straightforward partner registry. Around all of it, keep a tiered fallback so that senders who cannot meet the highest bar are still accepted under compensating controls instead of being cut off.

References

RFC 6698, The DNS-Based Authentication of Named Entities (DANE) Transport Layer Security (TLS) Protocol: TLSA: https://www.rfc-editor.org/rfc/rfc6698
RFC 7671, The DANE Protocol: Updates and Operational Guidance: https://www.rfc-editor.org/rfc/rfc7671
RFC 7672, SMTP Security via Opportunistic DANE TLS: https://www.rfc-editor.org/rfc/rfc7672
RFC 9440, Client-Cert HTTP Header Field: https://www.rfc-editor.org/rfc/rfc9440
Cloudflare Docs, mTLS with Workers: https://developers.cloudflare.com/learning-paths/mtls/mtls-workers
Cloudflare Docs, Enable mTLS: https://developers.cloudflare.com/ssl/client-certificates/enable-mtls/
Truv Docs, Security and mTLS: https://docs.truv.com/api-reference/security
Google Cloud, Dialogflow CX mutual TLS authentication: https://docs.cloud.google.com/dialogflow/cx/docs/concept/mtls
Stripe Docs, Webhooks: https://docs.stripe.com/webhooks

Originally published at https://instawebhook.com/blog/zero-trust-webhook-ingress-enforcing-cryptographic-sender-identity-with-dnssec-d

Zero-Allocation Webhook Parsers: Optimizing Go and Rust Ingress Buffers with SIMD

InstaWebhook — Sun, 25 Oct 2026 04:24:19 +0000

Webhook ingestion is a common bottleneck at the edge of event-driven systems. Payment processors, source-control platforms and IoT fleets can all send bursts of small JSON events to a single endpoint. Handlers built on general-purpose frameworks typically spend much of their CPU time on allocation and on building object trees for payloads that only need to be routed.

This article covers where that cost comes from, what SIMD-accelerated JSON parsing can and cannot do about it, and how to structure a Go or Rust ingress gateway around it. It also covers the limits of “zero allocation”, which is better treated as a target than as a guarantee.

1. Where Webhook Ingestion Spends Its Time

Most webhook endpoints are designed like ordinary REST endpoints. At a few hundred requests per second, that is fine. Under a burst, the per-request overhead adds up.

A typical handler does the following:

Reads the body into a freshly allocated buffer. In Go, the standard net/http server allocates a Request and header map for every request, and io.ReadAll grows a slice as it reads.
Copies bytes into strings. Converting []byte to string in Go copies the data in the general case, although the compiler can avoid the copy in a few narrow cases such as map lookups and comparisons. Rust’s String::from_utf8 or to_owned does the same.
Builds a full document tree. Unmarshaling into map[string]interface{} or serde_json::Value allocates for every object, array and string, even when the handler only needs one field.
Verifies the signature and routes. Webhook signatures cover the exact bytes the sender transmitted, so the HMAC must be computed over the raw body. Re-serialized JSON will not verify.

As a back-of-the-envelope figure, 50,000 requests per second at an average of 2 KB each is about 100 MB/s of payload. That is a modest amount of network traffic. The cost comes from the per-request allocations and tree building that sit on top of it.

What this means for Go and for Rust

Go’s collector is non-generational and concurrent (tri-color mark and sweep). It does not stop the world for long. It has two brief stop-the-world phases per cycle, which the runtime keeps in the sub-millisecond range. The main costs of a high allocation rate are:

CPU spent on background marking, which uses roughly 25% of CPU during a cycle.
Mark assists, where goroutines that allocate quickly are conscripted into GC work.
More frequent cycles as the heap grows and shrinks.

For latency-sensitive services, even sub-millisecond pauses can be worth monitoring. A sustained p99 GC pause above roughly 10 ms is a sign that the collector is struggling with heap size.

Rust has no garbage collector, so the issue there is different. Every String, Vec and HashMap is a call into the allocator. Under high concurrency that means allocator contention, poor cache locality and per-request latency that depends on allocator state. The remedies are the same in spirit: borrow from the input buffer instead of copying, and reuse buffers instead of allocating new ones.

2. What SIMD Brings to JSON Parsing

SIMD (Single Instruction, Multiple Data) lets one instruction operate on a whole vector register. The common register widths are:

Instruction set	Register width	Typical platform
ARM NEON	128 bits (16 bytes)	aarch64
AVX2	256 bits (32 bytes)	x86-64 since about 2013
AVX-512	512 bits (64 bytes)	Some recent x86-64 server CPUs

The reference design is simdjson, by Geoff Langdale and Daniel Lemire. Their paper, Parsing Gigabytes of JSON per Second, describes it as the first standard-compliant JSON parser to process gigabytes per second on a single core using commodity processors. It can use a quarter or fewer instructions than a state-of-the-art conventional parser such as RapidJSON.

The design has two stages:

Stage 1 builds a structural index. It processes the input in 64-byte blocks. It uses vector comparisons to find quotes, backslashes and structural characters ({ } [ ] : ,) and produces bitmasks. This step also works out which bytes are inside strings and validates UTF-8.
Stage 2 walks the index and builds a “tape”. This is a flat array describing the document structure, not a tree of heap objects.

A few of the claims commonly made about SIMD parsing need correcting:

It does not process a block “in a single CPU cycle”. It uses a small number of vector instructions per block, with no data-dependent branches in stage 1.
It is not tied to AVX-512. simdjson’s reference implementation targets several instruction sets. The Rust port simd-json supports AVX2, SSE4.2, NEON and WebAssembly SIMD128. simdjson-go requires AVX2 and CLMUL, and uses AVX-512 only to speed up stage 1 where available.
SIMD speeds up parsing and validation. It does not remove the cost of reading from the socket, TLS, HTTP framing or signature verification.

For webhooks, the biggest win is often not raw parsing speed. A tape or on-demand parser lets you read one field, such as an event type, without constructing a full object model of a payload you are only going to forward.

3. Go: `simdjson-go`

github.com/minio/simdjson-go is a pure-Go port of simdjson, with no cgo. According to its README:

It runs at roughly 40–60% of the speed of the C++ simdjson on average.
It is about 10× faster than encoding/json, in the project’s benchmarks.
It is a validating parser, so malformed input is rejected rather than silently accepted.
It needs a CPU with both AVX2 and CLMUL. That means Intel Haswell (2013) or later, or AMD Ryzen/EPYC (2017) or later.

Two practical cautions apply before adopting it:

Check maintenance status. The latest tagged release on pkg.go.dev, v0.4.5, was published in March 2023. At least one fork (DataDog’s) has been archived. Review the repository’s current activity before committing to it in a production gateway.
The 10× figure is a benchmark result, not a promise. It depends on payload shape and on how much of the document you read. Measure with your own payloads.

Reusing parser state

simdjson.Parse accepts a previously used *ParsedJson to reuse, which avoids reallocating the tape and string buffers on each request. The sketch below shows the shape of a handler that reads a body into a pooled buffer, verifies the signature, and then extracts the event type. It is illustrative. Check the method names against the version of the package you pin.

package ingress

import (
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "io"
    "net/http"
    "sync"

    "github.com/minio/simdjson-go"
)

const maxBody = 1 << 20 // 1 MiB; reject anything larger

type scratch struct {
    body []byte
    pj   *simdjson.ParsedJson // reused between requests
}

var pool = sync.Pool{
    New: func() any { return &scratch{body: make([]byte, 0, 64<<10)} },
}

func handle(secret []byte) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        if !simdjson.SupportedCPU() {
            http.Error(w, "unsupported CPU", http.StatusInternalServerError)
            return
        }

        s := pool.Get().(*scratch)
        defer pool.Put(s)

        // Read into the pooled buffer, bounded by maxBody.
        buf := bytes.NewBuffer(s.body[:0])
        if _, err := io.Copy(buf, io.LimitReader(r.Body, maxBody+1)); err != nil || buf.Len() > maxBody {
            http.Error(w, "bad body", http.StatusRequestEntityTooLarge)
            return
        }
        s.body = buf.Bytes()

        // 1. Verify the signature over the raw bytes, before parsing.
        if !validSignature(secret, s.body, r.Header.Get("X-Hub-Signature-256")) {
            http.Error(w, "invalid signature", http.StatusUnauthorized)
            return
        }

        // 2. Parse, reusing the previous ParsedJson where possible.
        pj, err := simdjson.Parse(s.body, s.pj)
        if err != nil {
            http.Error(w, "invalid JSON", http.StatusBadRequest)
            return
        }
        s.pj = pj

        // 3. Walk the tape to read only the fields needed for routing,
        //    then enqueue the raw body and return 2xx quickly.
        w.WriteHeader(http.StatusAccepted)
    }
}

func validSignature(secret, body []byte, header string) bool {
    const prefix = "sha256="
    if len(header) <= len(prefix) {
        return false
    }
    got, err := hex.DecodeString(header[len(prefix):])
    if err != nil {
        return false
    }
    mac := hmac.New(sha256.New, secret)
    mac.Write(body)
    return hmac.Equal(got, mac.Sum(nil))
}

Add "bytes" to the imports if you use this pattern. A few notes on what the sketch does and does not achieve:

It reduces allocations on the body-reading and parsing paths. It is not allocation-free end to end. net/http still allocates per request, and hmac.New allocates unless you also pool the MAC. Servers that avoid the standard library’s per-request allocations, such as fasthttp, trade API convenience for lower overhead.
Verify with go test -bench . -benchmem and look at allocs/op. Treat that number, not the design intent, as the measure of success.
If GC cost remains a concern, GOGC and GOMEMLIMIT (available since Go 1.19) are separate levers. GOMEMLIMIT sets a soft memory limit, and combining it with a tuned GOGC can reduce collection frequency. It trades memory for CPU.

4. Rust: Borrowing from the Input Buffer

Two crates matter here, and they are easy to confuse:

simd-json is a Rust port of simdjson with serde compatibility. It supports AVX2, SSE4.2, NEON and WASM SIMD128, with a native fallback. It offers borrowed and owned DOM value types and a lower-overhead tape API.
simdjson-rust provides bindings to the C++ simdjson library. As of this writing, the latest version listed on docs.rs is a 0.4.0-alpha pre-release.

For production use, simd-json is the more established choice.

An important constraint follows from how simd-json works: it parses in place and modifies the input buffer. Its documentation states that the input must be &mut [u8] because strings are unescaped in place. This affects design in two ways:

Parsed strings borrow from your buffer, so the borrow checker ties their lifetime to it. That is where the zero-copy benefit comes from.
You must verify the HMAC before parsing. After parsing, the buffer no longer holds the bytes the sender signed.

use hmac::{Hmac, Mac};
use sha2::Sha256;
use simd_json::prelude::*;

type HmacSha256 = Hmac<Sha256>;

pub struct Ingress {
    secret: Vec<u8>,
    buf: Vec<u8>, // reused across requests
}

impl Ingress {
    pub fn new(secret: &[u8], capacity: usize) -> Self {
        Self { secret: secret.to_vec(), buf: Vec::with_capacity(capacity) }
    }

    /// Verifies the signature, then reads the event type.
    /// Returns an owned `String` for the type only to keep the example
    /// simple; a hot path would route on a borrowed `&str` instead.
    pub fn route(&mut self, body: &[u8], sig_hex: &str) -> Result<String, &'static str> {
        // 1. Verify over the raw bytes, before the parser mutates anything.
        let sig = hex::decode(sig_hex.trim_start_matches("sha256="))
            .map_err(|_| "bad signature encoding")?;
        let mut mac = HmacSha256::new_from_slice(&self.secret).map_err(|_| "bad key")?;
        mac.update(body);
        mac.verify_slice(&sig).map_err(|_| "signature mismatch")?; // constant-time

        // 2. Copy into the reusable buffer; simd-json will modify it.
        self.buf.clear();
        self.buf.extend_from_slice(body);

        // 3. Parse in place and read one field.
        let value = simd_json::to_borrowed_value(&mut self.buf)
            .map_err(|_| "invalid JSON")?;
        value
            .get("type")
            .and_then(|t| t.as_str())
            .map(str::to_owned)
            .ok_or("missing event type")
    }
}

Again, this is a sketch. Pin crate versions and check the API against them. Some caveats:

BorrowedValue is a DOM. It borrows string data from the buffer, but it still allocates for objects and arrays. For a routing-only workload, the tape API reduces that overhead further. Benchmark both against your payloads.
The copy into self.buf is deliberate. If your network layer already gives you an owned, mutable buffer, you can verify and parse in place and skip it.
Build configuration matters. SIMD crates select instruction sets at compile time or runtime. Check how your target and build flags interact with the crate’s feature detection, and test on the actual production CPU architecture.
Heap allocation is not eliminated. Allocating nothing at all in a Rust async server also requires choices in the HTTP stack. Count allocations with a profiler or a counting allocator rather than assuming.

5. Performance: What Is Actually Known

Be skeptical of any single headline multiplier for “SIMD vs standard”, including those in earlier drafts of this article. Throughput and latency depend on payload size, document structure, how much of the document you read, CPU generation and the rest of the stack.

What the primary sources support:

The simdjson authors report that it processes gigabytes of JSON per second on a single core. An early README figure on one benchmark file (twitter.json, Skylake) was about 2.2 GB/s, against about 0.10 GB/s for nlohmann/json. That is a different language and a different CPU generation from your service.
simdjson-go reports roughly 10× the speed of encoding/json and 40–60% of the C++ simdjson’s speed.
The simdjson paper reports using a quarter or fewer of the instructions of a state-of-the-art conventional parser.

Parsing is only one stage of the request. If your handler spends most of its time in TLS, the kernel or downstream I/O, a faster parser will not change overall throughput much. Profile first, and measure P99 and P99.9 latency under a realistic burst profile.

6. Gateway Architecture

A parser is only one component. The sender-facing requirements from the major webhook providers shape the rest of the design:

GitHub expects a 2xx response within 10 seconds, or it terminates the connection and treats the delivery as failed. Its documentation recommends putting payloads on a queue and processing them asynchronously. It signs deliveries with HMAC-SHA256 in X-Hub-Signature-256, recommends a constant-time comparison, and sets X-GitHub-Event and X-GitHub-Delivery headers for routing and replay protection. Missed deliveries should be redelivered once your server is back.
Stripe signs the raw request body with a timestamp in the Stripe-Signature header (an HMAC-SHA256 over the timestamp, a dot, and the body). In live mode it retries failed deliveries with exponential backoff for up to three days. A retry is not a new event, so handlers should deduplicate on the event ID.

These behaviors suggest the following design:

Read the raw body with a size limit. Reject oversized requests early.
Verify the signature first. Use a constant-time comparison. This discards forged traffic before any parsing cost.
Parse minimally. Extract only what routing needs, such as event type and ID.
Persist, then acknowledge. Write the raw body to a durable queue or log, return 2xx, and do the real work in workers. Acknowledging before the event is durably stored risks losing it.
Deduplicate. Use X-GitHub-Delivery or the Stripe event ID as an idempotency key.
Apply backpressure deliberately. A 429 or 5xx response only helps if the sender retries. Stripe retries non-2xx responses, but GitHub treats a failed delivery as failed until someone redelivers it. For senders without automatic retries, shedding load means losing events unless you reconcile later.

For network I/O, Go’s runtime uses the operating system’s readiness-based poller (epoll on Linux), so goroutine-per-connection code is already multiplexed. Rust services typically use an async runtime such as Tokio. io_uring-based runtimes exist in the Rust ecosystem, but whether they help depends on the workload and kernel version. Benchmark before adopting one.

Summary

SIMD-accelerated parsers such as simdjson-go and simd-json can cut parsing cost substantially, especially when you only need a few fields. Combined with pooled buffers and borrowed data, they reduce allocation pressure in both Go and Rust. A few points hold regardless of language:

“Zero allocation” is a goal to verify with -benchmem, allocation counters and profiles, not something a library name guarantees.
Verify HMAC signatures on the raw bytes before parsing, especially with simd-json, which mutates its input.
Go’s GC is mostly concurrent, with short pauses. The cost of heavy allocation shows up as CPU overhead and mark assists as much as pause time.
Check CPU requirements and library maintenance before building on a parser. simdjson-go requires AVX2 and CLMUL, and its last tagged release is from 2023.
Durability and idempotency matter more than parser speed for reliability: persist before acknowledging, and deduplicate on delivery IDs.

References

Langdale, G. and Lemire, D., Parsing Gigabytes of JSON per Second (arXiv:1902.08318; VLDB Journal, 2019).
minio/simdjson-go README and pkg.go.dev listing (performance, AVX2/CLMUL requirement, v0.4.5).
simd-lite/simd-json README (supported instruction sets, in-place parsing, borrowed values).
simdjson-rust on docs.rs (bindings to C++ simdjson, pre-release status).
Go runtime documentation and GC guide (go.dev/doc/gc-guide; GOGC, GOMEMLIMIT, concurrent mark and sweep).
GitHub Docs: Validating webhook deliveries and Best practices for using webhooks.
Stripe documentation on webhook signatures and retry behavior.

Originally published at https://instawebhook.com/blog/zero-allocation-webhook-parsers-optimizing-go-and-rust-ingress-buffers-with-simd

Unicast vs. Anycast for Webhook Ingress: Cutting Handshake Latency for Global Senders

InstaWebhook — Sat, 24 Oct 2026 02:01:01 +0000

Webhook publishers such as Stripe, GitHub, Shopify and Slack deliver event notifications to endpoints all over the world. Each delivery has to resolve DNS, open a TCP connection, complete a TLS handshake and send an HTTP request before your server sees a single byte of payload. For a receiver far from the sender, those handshake round trips are the largest part of the network cost.

Publishers also give you very little time to respond. GitHub expects a 2xx response within 10 seconds and terminates the connection otherwise. Shopify has a one-second connection timeout and a five-second timeout for the whole request, and Slack’s Events API expects a response within three seconds. Stripe retries failed deliveries for up to three days with exponential backoff in live mode.

This article covers what each handshake step costs, how BGP Anycast shortens the path, and where TCP Fast Open (TFO) and TLS 1.3 0-RTT help or hurt. It also corrects some commonly repeated claims about all three.

1. The anatomy of webhook ingress latency

A fresh delivery from a publisher to your endpoint goes through these steps:

DNS lookup → TCP handshake → TLS handshake → HTTP POST → response

Step	Cost in round trips (RTTs) to your endpoint
DNS resolution	Depends on the sender’s resolver and cache, not on your endpoint’s distance
TCP three-way handshake	1 RTT before the client can send data
TLS 1.2 full handshake	2 RTTs
TLS 1.3 full handshake	1 RTT (RFC 8446)
HTTP request and first response byte	1 RTT, plus your processing time

Take a sender in Dublin and an endpoint in Virginia, where the round trip is typically somewhere around 70–80 ms. Use 75 ms as an illustrative figure and measure your own paths.

TLS 1.3: TCP (75 ms) plus TLS (75 ms) is 150 ms of setup. The first response byte arrives about 225 ms after the sender starts connecting.
TLS 1.2: setup is 225 ms and the first response byte arrives at about 300 ms.

These figures exclude DNS and server processing time.

Handshake latency is real, but it is usually not what causes webhook timeouts. A few hundred milliseconds of setup is small against timeouts of 3 to 10 seconds. Timeouts and retry storms are mostly caused by slow application handlers, which section 7 addresses. Shorter handshakes help most with tight limits such as Shopify’s one-second connection timeout, and with the volume of cold connections you absorb during bursts.

Connection reuse also matters. Shopify’s delivery system uses HTTP Keep-Alive to reuse connections to the same host. Where a sender reuses connections, the handshake cost is paid only on cold connections, not on every event.

2. Unicast vs. Anycast

Unicast

With unicast, the endpoint’s IP address maps to one location, such as a single region or load balancer cluster. Every sender, wherever it is, sends packets across the Internet to that location. A sender 50 km away and one 8,000 km away both end up at the same place.

BGP Anycast

With anycast, several sites (points of presence, or PoPs) announce the same IP prefix over BGP. Each network along the way picks a path to the prefix according to its own routing policy. That decision usually depends first on local preference, then on AS-path length. The result is that senders generally reach a topologically close PoP. A few clarifications:

“Nearest” means topologically nearest, not geographically nearest. Routing policy and peering decide the path, so a sender can land on a PoP that is not the closest one by distance.
Prefix length: the common rule of thumb is that routes more specific than a /24 (IPv4) or /48 (IPv6) should not be expected to work globally, because many networks filter them. An anycast service therefore announces at least a /24 (IPv4) or /48 (IPv6), not a /32. You need your own address space and AS number for this, or you can use a provider that already operates an anycast network.
Stability: a USC/ISI study of anycast DNS services found that anycast works well for the vast majority of vantage points. About 1% saw frequent shifts between anycast sites. The study found TCP-level flipping much rarer than UDP flipping, in roughly 0.15% of vantage point and service combinations, though it does occur. If a route shifts mid-connection, the TCP connection breaks. Webhook deliveries are short-lived HTTP requests, so exposure is small, but senders should retry. Stripe, GitHub and Shopify all retry or support redelivery, as does your own fallback logic.

Anycast IPs vs. anycast DNS

The title of this article’s original draft conflates two things:

Anycast DNS means your authoritative nameservers use anycast. This speeds up name resolution.
Anycast IP ingress means the webhook endpoint’s own address is anycast. This shortens the TCP and TLS round trips.

You can use either or both. A third option is DNS-based steering, such as latency-based or geo-aware DNS, where each region has its own unicast IP. It is simpler to operate than BGP and has no flip risk, but it depends on the sender’s resolver location and on DNS caching. Managed products such as CDNs and global accelerators give you anycast without running BGP yourself.

3. Withdrawing routes when a PoP is unhealthy

An anycast PoP that keeps announcing its prefix while its local proxy is dead will blackhole the traffic that BGP sends it. Each PoP therefore needs to withdraw its route when it is unhealthy. ExaBGP is one option. It reads route announcements from a helper process, so a health-check loop can drive announcements directly.

#!/bin/bash
# /opt/anycast/announce.sh — emits ExaBGP text-API commands on stdout
PREFIX="198.51.100.0/24"   # documentation prefix; use your own
state="withdrawn"

while true; do
  if curl -fsS --max-time 2 http://127.0.0.1:8080/healthz >/dev/null; then
    if [ "$state" != "announced" ]; then
      echo "announce route $PREFIX next-hop self"
      state="announced"
    fi
  else
    if [ "$state" != "withdrawn" ]; then
      echo "withdraw route $PREFIX next-hop self"
      state="withdrawn"
    fi
  fi
  sleep 2
done

Two operational notes. Add hysteresis so a flapping health check does not cause route flaps. And make sure a shared failure, such as a bad deploy, cannot withdraw every PoP at once. BGP convergence takes time, so withdrawal is not instantaneous.

4. TCP Fast Open (TFO)

TFO (RFC 7413) lets a client send data in the SYN packet on repeat connections, saving one round trip. The first connection gets a cookie from the server. Later connections present the cookie and carry data in the SYN. The RFC is an experimental one, and the practical caveats are significant:

The sender must opt in. The server can only enable its side. If the publisher’s HTTP client does not use TFO, you gain nothing. None of the provider documentation reviewed for this article mentions TFO support, so do not assume it.
Middleboxes. Some network devices drop TCP packets carrying options they do not understand. Operators report connections that appear to die when TFO is used, so clients commonly fall back to a regular handshake.
Duplicates. Data in a SYN can occasionally be delivered more than once, so the application must tolerate duplicate requests, which idempotent webhook handling already requires.

Enabling TFO on a Linux ingress node

# /etc/sysctl.d/99-tfo-ingress.conf
# Bitmask: 1 = client, 2 = server, 3 = both
net.ipv4.tcp_fastopen = 3

NGINX also needs TFO enabled on the listen socket:

listen 443 ssl fastopen=256;
listen [::]:443 ssl fastopen=256;

To check whether senders actually use it, read the kernel counters. ss does not show a TFO marker.

nstat -az TcpExtTCPFastOpenPassive TcpExtTCPFastOpenPassiveFail \
          TcpExtTCPFastOpenListenOverflow TcpExtTCPFastOpenCookieReqd

A non-zero TCPFastOpenPassive count means clients are completing fast-open connections to you. If it stays at zero, your senders are not using TFO.

5. TLS 1.3 and 0-RTT resumption

TLS 1.3 completes a full handshake in one round trip. With a session ticket from an earlier connection, a client can also send application data in its first flight. This is 0-RTT early data (RFC 8446, section 2.3).

Early data has two limitations:

It is replayable. An attacker who captures the encrypted early data can resend it, and the server cannot cryptographically tell the copy from the original. The data also lacks forward secrecy.
Major CDNs treat POST as unsafe for early data. Cloudflare does not enable 0-RTT by default and always rejects requests that are obviously not idempotent, such as POST or PUT.

Webhook deliveries are POST requests, so 0-RTT is the riskiest optimisation here. It is only appropriate if your handler is strictly idempotent. RFC 8470 defines the Early-Data: 1 header, which a proxy adds to forwarded early-data requests, and the 425 Too Early status code that a server returns to refuse them. The original draft’s advice to accept 0-RTT for “idempotent methods (POST ...)” was wrong, because POST is not idempotent by definition.

NGINX configuration

NGINX supports ssl_early_data with OpenSSL 1.1.1 or later, and the NGINX documentation warns that early data requests are subject to replay attacks and should be handled at the application layer using $ssl_early_data.

server {
    listen 443 ssl fastopen=256;
    listen [::]:443 ssl fastopen=256;
    http2 on;                          # nginx 1.25.1+; older versions use "listen ... http2"
    server_name webhooks.example.com;

    ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

    # Keep TLS 1.2: Stripe documents TLS 1.2 or higher as its requirement,
    # and a TLS 1.3-only endpoint would shut out clients that stop at 1.2.
    ssl_protocols TLSv1.2 TLSv1.3;

    # Off by default. Enable only on a dedicated server block whose
    # handlers are idempotent, and always forward the signal downstream.
    ssl_early_data on;

    location /webhook {
        proxy_set_header Early-Data $ssl_early_data;
        proxy_pass http://webhook_backend;
    }
}

The original draft set ssl_protocols TLSv1.3 only. That is a risk for a public webhook endpoint, because any sender that cannot negotiate TLS 1.3 will fail to deliver. Enabling both versions and testing each publisher’s delivery is safer.

In an anycast fleet, session resumption only works if any PoP can decrypt a ticket issued by another. This means sharing session ticket keys across PoPs and rotating them regularly.

Replay defence

If you accept early data on webhook endpoints:

Deduplicate by event ID. Use Stripe’s evt_… ID or GitHub’s X-GitHub-Delivery header with an atomic insert such as Redis SET key 1 NX EX <ttl>. GitHub says redelivery keeps the same X-GitHub-Delivery value. Stripe retries for up to three days, so the TTL should be at least that (259,200 seconds) for Stripe events.
Use a shared dedupe store. In an anycast fleet a replayed request can land on a different PoP, so a per-PoP cache does not stop a cross-PoP replay.
Verify signatures. This establishes authenticity, but a replay of a valid signed request still has a valid signature. Signature timestamps limit the replay window but do not remove it.
Return 425 Too Early from handlers that cannot guarantee idempotency when Early-Data: 1 is present.

6. Putting the numbers together

The table shows time to first response byte, excluding DNS and server processing time. The 15 ms anycast RTT is an illustrative assumption, not a measurement.

Stack	RTT	Calculation	Time to first response byte
Unicast, TLS 1.2	75 ms	TCP 1 + TLS 2 + request 1 = 4 RTT	~300 ms
Unicast, TLS 1.3	75 ms	TCP 1 + TLS 1 + request 1 = 3 RTT	~225 ms
Anycast, TLS 1.3	15 ms	3 RTT	~45 ms
Anycast + TFO, TLS 1.3	15 ms	2 RTT (ClientHello rides in the SYN)	~30 ms
Anycast + TFO + TLS 1.3 0-RTT	15 ms	1 RTT (request in first flight)	~15 ms
Unicast + TFO + TLS 1.3 0-RTT	75 ms	1 RTT	~75 ms

The last two rows show that the techniques are complementary. Anycast shrinks the RTT itself, while TFO and 0-RTT reduce how many RTTs you pay. Even the best case is one RTT, not zero. The earlier claim of “~0 ms connection overhead” overlooks that the request must still reach the server and the response must come back.

These gains also require sender cooperation. TFO and 0-RTT only apply if the publisher’s client uses them, and TFO and 0-RTT early data are both replayable. Anycast needs no sender support, so it is the most dependable of the three.

7. Acknowledge at the edge, process later

Fast ingress only helps if the application behind it is also fast. If the PoP must wait for a distant database before replying, you have moved the delay rather than removed it. Provider documentation says the same: GitHub and Shopify both recommend queuing payloads and responding quickly, with processing done asynchronously.

A resilient design looks like this:

Validate and acknowledge at the PoP. Verify the signature against the raw request body, such as the Stripe-Signature header, then durably enqueue the event and return a 2xx. Return the exact status the sender requires, because Shopify requires 200 while GitHub accepts any 2xx.
Buffer in a replicated queue. Use something like Apache Kafka or Amazon Kinesis, or a managed edge queue such as Cloudflare Queues, so events are persisted before you acknowledge them.
Keep PoP-to-origin connections warm. The PoP can hold persistent connections to the origin, so the long-haul handshake is paid once and not once per webhook.
Process idempotently. All three publishers can deliver an event more than once. Shopify explicitly notes that duplicate deliveries can occur after network timeouts or retries.
Reconcile. If you cannot afford to miss events, periodically compare against the publisher’s API. Shopify recommends this, and Stripe lets you query missed events.

8. Measure before and after

You can measure handshake costs from any sender-like location with curl:

curl -so /dev/null --tlsv1.3 \
  -w 'dns=%{time_namelookup}s tcp=%{time_connect}s tls=%{time_appconnect}s ttfb=%{time_starttransfer}s\n' \
  https://webhooks.example.com/healthz

Run it from several regions against both your unicast and anycast addresses. Add --tcp-fastopen to test TFO from a client that supports it. Then compare the results with your ingress logs and the kernel counters from section 4 to see which senders benefit.

Conclusion

For globally distributed webhook senders, anycast gives the largest and most dependable latency improvement, because it shortens every round trip and needs no cooperation from the sender. Pair it with route withdrawal on health-check failure, TLS 1.3 alongside 1.2 compatibility, and an edge that acknowledges quickly and queues the work. Add TFO where your senders support it. Enable TLS 1.3 0-RTT only on endpoints whose handlers are strictly idempotent, with a shared deduplication store, and treat it as a niche optimisation rather than a default for POST webhooks.

Originally published at https://instawebhook.com/blog/unicast-vs-anycast-for-webhook-ingress-cutting-handshake-latency-for-global-send

Anycast Routing for Global Webhook Ingress: Minimizing Latency and Packet Loss for Worldwide Senders

InstaWebhook — Fri, 23 Oct 2026 02:30:46 +0000

Modern SaaS ecosystems rely heavily on webhooks. When a payment succeeds on Stripe, an order is created on Shopify, or a push lands on GitHub, the platform sends an asynchronous HTTP POST request to an endpoint you control. The sender usually expects a quick 2xx response, and it treats anything else, including a slow response, as a failed delivery.

Scaling webhook ingestion globally is therefore a problem of physics and transport reliability. If your endpoint lives in a single region, every sender outside that region pays for distance on every connection. This guide explains how BGP Anycast shortens that distance, where it helps, and where it doesn’t.

+-----------------------------------------------------------------------------------+
|                         UNICAST INGRESS (illustrative)                            |
|                                                                                   |
|  Sender (Tokyo) ---[ TCP + TLS handshake over a long path ]---> Origin (Virginia) |
+-----------------------------------------------------------------------------------+

+-----------------------------------------------------------------------------------+
|                         ANYCAST EDGE INGRESS (illustrative)                       |
|                                                                                   |
|  Sender (Tokyo) ---[ short handshake ]---> Nearby PoP                             |
|                                               |                                   |
|                                  [ warm connections / private backbone ]          |
|                                               v                                   |
|                                          Origin / Queue                           |
+-----------------------------------------------------------------------------------+

What Webhook Senders Expect

Before designing ingress, it helps to know what the biggest providers require. These figures come from their public documentation:

GitHub expects a 2xx response within 10 seconds. A slower response is treated as a failed delivery, and you can redeliver failed webhooks later. GitHub lowered this limit from 30 seconds in 2017.
Shopify fails a delivery if your app doesn’t respond within five seconds. It retries failed deliveries up to eight times over about four hours, and it removes the subscription if failures persist.
Stripe recommends returning a 2xx quickly, before any complex logic that could cause a timeout. In live mode it retries delivery for up to three days with exponential backoff. In sandbox it retries three times over a few hours. Stripe doesn’t publish an exact timeout value, so don’t design around a specific number.

Every one of these providers tells you to acknowledge first and process later. Anycast ingress applies that advice at the network layer.

The Problem with Unicast Ingress for Global Webhooks

1. Handshake Latency Scales with Distance

A unicast IP routes traffic to one location. If your ingestion service runs in a single region such as us-east-1 and the sender dispatches from Tokyo, every new connection pays the full round-trip time (RTT) several times before any payload is processed.

Before the request can be sent, a new HTTPS connection needs:

One RTT for the TCP handshake.
One RTT for the TLS 1.3 handshake. TLS 1.2 needs two.

Client (Tokyo) <------------------ ~160 ms RTT (illustrative) ------------------> Origin (Virginia)

1. TCP SYN --------------------------------------------------------------->
2. <---------------------------------------------------------- TCP SYN-ACK
3. TCP ACK + TLS ClientHello ---------------------------------------------->
4. <----------------------------------- TLS ServerHello, Certificate, Finished
5. TLS Finished + HTTP POST /webhook --------------------------------------->
6. <------------------------------------------------------- HTTP 2xx response

At an illustrative 160 ms RTT, the arithmetic looks like this:

Step	TLS 1.3	TLS 1.2
TCP + TLS handshake complete	~320 ms	~480 ms
First response byte (best case, zero processing time)	~480 ms	~640 ms

These numbers are far below a 5-second or 10-second timeout on a healthy path. The real danger is variance: loss, congestion, and slow handlers stack on top of that baseline. A sender that sees a failure retries, and a large burst of retries can add load to an endpoint that is already struggling. Providers use exponential backoff to soften this, but your endpoint still absorbs the duplicates.

2. Loss Hurts More on Long Paths

Traffic over the public internet crosses multiple autonomous systems (AS) with varying congestion. Two effects make loss more costly on long paths:

Recovery time scales with RTT. Loss in the middle of a transfer is typically recovered after roughly one RTT, so a 160 ms path recovers far more slowly than a 10 ms one. Loss-based congestion control such as CUBIC also shrinks the congestion window after loss, and on long paths it takes longer to grow back.
Handshake loss is the most expensive kind. If a SYN is lost, RFC 6298 sets the initial retransmission timeout at one second. A single dropped SYN can add a full second before the connection even starts.

3. Longer Paths Mean More Exposure

The farther a connection travels over networks you don’t control, the more opportunities there are for congestion, rerouting, and jitter. Internet paths are also often asymmetric, meaning the forward and return routes differ. That is normal, and it applies to anycast too. Anycast doesn’t remove path problems. It shortens the portion of the path where they can hurt a new connection.

How BGP Anycast Works

With anycast, the same IP prefix is announced from multiple physical locations (Points of Presence, or PoPs). Each network on the internet picks a route to that prefix using BGP, so traffic from a given sender typically flows to whichever announcing location its network prefers.

                      Sender (US-West)          Sender (Europe)
                            |                         |
                     (BGP-selected route)      (BGP-selected route)
                            v                         v
                  +------------------+      +------------------+
                  | PoP: San Jose    |      | PoP: Frankfurt   |
                  | 198.51.100.0/24  |      | 198.51.100.0/24  |
                  +--------+---------+      +--------+---------+
                           \                       /
                            \    private / warm   /
                             v    connections    v
                          +--------------------------+
                          |  Central origin / queue  |
                          +--------------------------+

(198.51.100.0/24 is a documentation-only prefix and is not routable on the real internet.)

“Nearest” Means Nearest in BGP, Not in Kilometers

BGP selects routes using policy first, then attributes such as AS-path length. It knows nothing about geography. Most of the time the chosen PoP is close to the sender, but not always. A sender on a network with unusual peering can land at a distant PoP. Anycast catchments are therefore something to measure and tune, not something to assume.

Operators influence catchments by choosing where to announce, which transit providers and exchange points (IXPs) to peer with, and by using techniques such as AS-path prepending or BGP communities.

Terminate Connections at the Edge

Instead of completing TCP and TLS across an ocean, the sender completes them with the nearest PoP. If the sender is 10 ms from that PoP, the handshake costs roughly 20 ms instead of several hundred.

Local handshake: TCP and TLS 1.3 complete in about two short RTTs.
Local receipt: The edge reads the POST body over a short path.
Backbone forwarding: The edge hands the event to your systems over pre-established, long-lived connections on a path you control. This might be a private backbone or a provider network such as AWS’s.

Loss Recovery Happens Over a Shorter Segment

The sender-to-edge leg is still the public internet, but it is shorter. Because retransmissions are paced by RTT, recovery on a 10 ms path is much faster than on a 160 ms path. The long-haul leg moves onto infrastructure with more predictable behavior. It can still suffer loss, but it is no longer the part the sender’s timeout clock is watching.

Managed Anycast vs. Running Your Own

Announcing your own anycast prefix requires an ASN, provider-independent address space (a /24 is the smallest IPv4 prefix generally accepted in the global routing table), and BGP relationships at every site. Many teams use a managed service instead:

Cloudflare runs its network on anycast. It was connected to more than 300 cities as of its 2023 announcement.
AWS Global Accelerator provides two static anycast IPv4 addresses announced from multiple AWS edge locations. Traffic enters the AWS network near the sender and travels to your regional endpoint over AWS’s backbone.

Check where your actual senders dispatch from before investing. If most of your webhook volume originates in one or two regions, a regional endpoint near those regions may deliver most of the benefit at lower cost.

Architecture of an Anycast-Fronted Webhook Ingress

An anycast ingress stack has three layers: edge routing, edge processing, and internal transport and buffering.

+-----------------------------------------------------------------------------------+
| 1. ANYCAST EDGE ROUTING                                                           |
|    - Prefix announced via BGP (e.g., BIRD or ExaBGP)                              |
|    - L4 load balancing inside the PoP (e.g., Maglev-style consistent hashing,     |
|      IPVS, or XDP/DPDK-based balancers)                                           |
+-----------------------------------------------------------------------------------+
                                       |
                                       v
+-----------------------------------------------------------------------------------+
| 2. EDGE PROCESSING                                                                |
|    - TLS termination, HTTP/1.1 + HTTP/2 (HTTP/3 optional)                         |
|    - Signature verification (optional)                                            |
|    - Durable enqueue, then 2xx acknowledgment                                     |
+-----------------------------------------------------------------------------------+
                                       |
                                       v
+-----------------------------------------------------------------------------------+
| 3. TRANSPORT AND BUFFER                                                           |
|    - Persistent connections over private links or a provider backbone             |
|    - Durable queue (Kafka, Redpanda, SQS, etc.), consumed asynchronously          |
+-----------------------------------------------------------------------------------+

Layer 1: BGP at the Edge

Each PoP runs a BGP daemon such as BIRD or ExaBGP that announces the anycast prefix to upstream providers and peers. A minimal BIRD 2 sketch looks like this:

# Illustrative BIRD 2 sketch. All ASNs and prefixes are placeholders.
protocol static anycast_vip {
    ipv4;
    # Anycast service address, bound to a loopback interface on the PoP's servers
    route 198.51.100.0/24 via "lo";
}

protocol bgp upstream_1 {
    local as 65001;
    neighbor 192.0.2.1 as 64512;

    ipv4 {
        import none;
        export filter {
            if net = 198.51.100.0/24 then accept;
            reject;
        };
    };
}

In production you would use your own public ASN and prefix, add session authentication, prefix limits, and RPKI validation, and tune announcements per upstream.

Layer 2: Fast-Ack Ingestion

To keep the sender’s clock short, the edge should do as little as possible before acknowledging:

Terminate TLS.
Optionally verify the signature using a cached secret, so obviously forged traffic is rejected cheaply. Each provider signs differently:
- GitHub sends X-Hub-Signature-256, an HMAC-SHA256 of the raw body prefixed with sha256=.
- Shopify sends X-Shopify-Hmac-Sha256, a base64-encoded HMAC-SHA256 of the raw body.
- Stripe sends a Stripe-Signature header containing a timestamp and signature(s), computed over the timestamp and the payload. Its libraries also enforce a timestamp tolerance.In every case, verify against the raw, unparsed body.
Durably enqueue the event.
Return a 2xx. The acknowledgment should never come before the event is durably stored.

Example: Edge Ingress Handler for GitHub Webhooks

import { createHmac, timingSafeEqual } from "node:crypto";

interface WebhookRequest {
  headers: Record<string, string | undefined>;
  body: string; // raw, unparsed body
}

// Application-defined: durably writes to a queue near this PoP.
declare function publishToQueue(msg: {
  rawBody: string;
  headers: Record<string, string | undefined>;
  receivedAt: number;
  pop: string | undefined;
}): Promise<boolean>;

export async function handleGitHubWebhook(req: WebhookRequest): Promise<Response> {
  const signature = req.headers["x-hub-signature-256"];
  const secret = process.env.GITHUB_WEBHOOK_SECRET;

  if (!signature || !secret) {
    return new Response(JSON.stringify({ error: "Unauthorized" }), { status: 401 });
  }

  // 1. Verify the HMAC over the raw body using a constant-time comparison
  const expected = "sha256=" + createHmac("sha256", secret).update(req.body).digest("hex");
  const a = Buffer.from(signature);
  const b = Buffer.from(expected);

  if (a.length !== b.length || !timingSafeEqual(a, b)) {
    return new Response(JSON.stringify({ error: "Invalid signature" }), { status: 403 });
  }

  // 2. Durably enqueue before acknowledging
  const ok = await publishToQueue({
    rawBody: req.body,
    headers: req.headers,
    receivedAt: Date.now(),
    pop: process.env.POP_REGION_CODE,
  });

  if (!ok) {
    // A non-2xx makes the sender retry, which is what you want if nothing was stored
    return new Response(JSON.stringify({ error: "Queue unavailable" }), { status: 503 });
  }

  // 3. Acknowledge; heavy processing happens asynchronously downstream
  return new Response(JSON.stringify({ status: "accepted" }), { status: 202 });
}

Two caveats apply to this design:

Where the queue lives matters. If every PoP must synchronously write to one distant queue, you have moved the long round trip from before the acknowledgment to inside it. Write to a regional or PoP-local durable buffer and replicate or forward asynchronously, or accept the cross-region write latency as a known cost.
Duplicates are normal. Senders retry on timeouts and ambiguous failures, and Shopify’s documentation notes that duplicate deliveries can occur. Deduplicate downstream, using identifiers such as GitHub’s X-GitHub-Delivery header or the event ID in a Stripe payload.

Anycast vs. Unicast: What Changes

Dimension	Single-region unicast	Anycast edge ingress
Connection setup cost	Full RTT to the origin region for TCP plus TLS (e.g., ~320 ms at 160 ms RTT with TLS 1.3)	RTT to the nearest PoP (often tens of milliseconds or less, depending on catchment)
Ingress points	One region	Many PoPs; the count depends on your provider or deployment
Sensitivity to long-haul loss	High: recovery scales with the full RTT	Lower for the sender-facing segment; the backbone leg is under your control
Failover	DNS-based failover, bounded by TTLs and client caching	BGP withdrawal, typically taking tens of seconds to converge and sometimes minutes
DDoS absorption	Concentrated at one location	Traffic is spread across PoPs by BGP catchment
Operational complexity	Low	Higher: BGP operations, per-PoP health, catchment monitoring

Operational Challenges and Mitigations

1. Route Changes Can Break In-Flight Connections

Anycast sends each packet toward whichever PoP the network currently prefers. If routing shifts mid-connection, subsequent packets can arrive at a different PoP that has no state for that TCP connection, and it answers with a reset.

How common is this? A measurement study by Wei and Heidemann covering more than 9,000 vantage points and 11 anycast services found that about 1% of vantage points saw frequent flips between anycast sites. TCP connections were disrupted far less often, in about 0.15% of vantage point and service combinations. Those results come from DNS root services, so your own measurements may differ.

Practical mitigations:

Keep connections short. Webhook deliveries are brief request/response exchanges, so the exposure window is small. A rare reset simply causes a sender retry, which your deduplication already handles.
Use consistent hashing inside each PoP. Maglev-style balancing keeps a connection on the same backend server within a PoP. Within-PoP ECMP does not help when traffic moves between PoPs.
Consider cross-PoP forwarding only if you need it. Encapsulating and forwarding misdirected packets to the PoP that owns the connection (for example, over GRE or GUE tunnels) is possible, but it adds significant complexity.
Don’t count on QUIC to solve this. QUIC connection IDs support migration, but it only helps if the sender’s HTTP client speaks HTTP/3. A server that receives packets for a connection it doesn’t know still cannot continue it without extra routing infrastructure.
Lean on idempotency and retries. Providers already retry failed deliveries, so a robust consumer treats a rare reset as just another duplicate.

2. Health Checks and Route Withdrawal

If a PoP’s application fails while BGP keeps announcing the prefix, traffic still arrives and is dropped. Withdraw the route when the local service is unhealthy.

Run health checks that cover the ingress service, its link to the queue, and upstream connectivity.
Have the checker withdraw the announcement (ExaBGP’s process API is commonly used for this) when checks fail, and re-announce when they recover.
Add hysteresis so a flapping service doesn’t cause a flapping route. Frequent announcements and withdrawals can trigger route-flap damping at upstream providers.

Be realistic about timing. Withdrawing a route isn’t instant, because the news must propagate across the internet. Measurements discussed by APNIC’s Geoff Huston put typical BGP convergence at tens of seconds, and older academic studies recorded minutes in bad cases. Vendors that advertise anycast failover in under 30 seconds are describing good-case behavior. Sub-second failover is not a safe assumption.

During that window, senders will see timeouts or resets and rely on their retry policies. Stripe’s three days and Shopify’s eight attempts over four hours are far longer than any realistic convergence period.

 PoP A: healthy ---- announces prefix ----> stays in service
 PoP B: unhealthy -- withdraws prefix ----> traffic shifts to other PoPs
                                             as the withdrawal propagates
                                             (tens of seconds typical; can be longer)

3. Edge-to-Origin Backhaul

Terminating at the edge solves the sender-facing leg. The data still has to reach your origin.

Keep persistent, pre-established connections from each PoP to your origin or queue so forwarding never pays a fresh handshake.
Where possible, use a private or provider backbone, such as AWS’s network behind Global Accelerator or a dedicated interconnect, rather than the public internet.
Make backhaul failures survivable. The edge should buffer or fail over when the link to the origin degrades, and it should return a 5xx if it truly cannot store the event, so the sender retries.

When Anycast Is (and Isn’t) Worth It

Anycast ingress is a strong fit when:

Senders are geographically diverse, or you receive webhooks from many sources worldwide.
Providers enforce tight timeouts, such as Shopify’s five seconds.
You already need a global edge for DDoS absorption or for static IPs.

It is less compelling when:

Nearly all senders dispatch from a single region you can colocate with.
Your real bottleneck is slow handler code, not the network. Moving to a queue-then-process pattern fixes that regardless of anycast.
You don’t have the operational capacity to run BGP at multiple sites and would need to depend on a managed provider anyway.

Conclusion

Webhook senders enforce short acknowledgment windows: five seconds for Shopify and ten for GitHub, with Stripe asking for a quick 2xx and not publishing a number. A single-region endpoint makes distant senders pay several long round trips just to connect, and it exposes them to loss recovery that scales with distance.

BGP anycast moves the sender’s handshake to a nearby PoP, shortens the segment where loss recovery happens, and lets you move events to your origin over connections you control. It does not give you instant failover or immunity from route changes. The realistic goal is to accept events locally and durably, acknowledge quickly, deduplicate downstream, and treat the occasional reset or slow convergence as a normal retry.

Originally published at https://instawebhook.com/blog/anycast-routing-for-global-webhook-ingress-minimizing-latency-and-packet-loss-fo

Zero-Copy Webhook Parsing: Optimizing Rust and Go Ingress Pipelines for 100k RPS

InstaWebhook — Thu, 22 Oct 2026 02:11:40 +0000

Imagine a webhook ingestion service, such as a durable intake layer like InstaWebhook that accepts events, stores them and queues delivery work outside the request path. Suppose it has to absorb bursts of 100,000 requests per second (RPS). At that rate, small per-request costs add up quickly. A parser that makes 15 allocations per request produces 1.5 million allocations per second, which the allocator, and in Go the garbage collector, has to absorb.

This article is about the parsing step of the pipeline. It shows how to verify signatures before parsing, borrow from the request buffer in Rust with serde, cut allocations in Go with easyjson, and what has changed in Go’s standard library as of Go 1.27. The 100k RPS figure is a design target for illustration, not a benchmark result. Measure your own payloads before drawing conclusions.

1. Where the Allocations Come From

A conventional parse lifecycle has several hidden costs:

Buffering the body. The request body has to be read into a contiguous buffer. In Go, io.ReadAll grows a fresh slice on every call. In Rust frameworks, the body is collected into a buffer as well.
Per-field allocations. A naive decoder allocates an owned String (Rust) or string (Go) for every key and value it keeps.
Reflection and interface boxing (Go). Go’s encoding/json historically inspects struct fields and types at runtime, which costs CPU and allocations.
Parsing before authenticating. If you build objects from an unauthenticated body, you spend CPU and memory on traffic you were going to reject anyway.

You can see the allocation effect in the benchmarks published by the easyjson project, which compare it against encoding/json. These numbers come from the project’s README, were measured with Go 1.6 (2016), and use a Twitter API sample. The “small” case is an 80-byte slice of that sample and the “regular” case is the full 13 kB document.

Unmarshal benchmark	Library	Speed	Allocs/op	Bytes/op
Small (80 B)	`encoding/json`	9.7 MB/s	14	720
Small (80 B)	`easyjson`	67 MB/s	3	128
Regular (13 kB)	`encoding/json`	22 MB/s	218	10,229
Regular (13 kB)	`easyjson`	125 MB/s	128	9,794

At 100k RPS, 14 allocations per request means 1.4 million allocations per second, and 218 allocations per request means about 21.8 million. The table also shows that the regular-sized document still costs easyjson 128 allocations. Code generation reduces allocations but does not eliminate them, which is why the string-handling options below matter.

2. Verify First, Parse Second

Every major webhook provider signs deliveries with an HMAC, so you can reject bad traffic before any JSON is parsed. The schemes differ in detail:

GitHub sends X-Hub-Signature-256, formatted sha256=<hex>, computed as HMAC-SHA256 over the raw body. GitHub’s documentation recommends a constant-time comparison. The header is absent if no secret was configured on the webhook.
Stripe sends a Stripe-Signature header containing a timestamp (t=) and one or more v1= signatures. The signed string is the timestamp, a literal ., and the raw body. A five-minute timestamp tolerance is the common default for rejecting replays.
Shopify sends X-Shopify-Hmac-Sha256, an HMAC-SHA256 of the raw body encoded as base64 rather than hex.

Two rules follow from this. Always compute the HMAC over the exact bytes received, never over a re-serialized object, because re-encoding changes the bytes and breaks the signature. And always compare signatures in constant time. In Rust, the hmac crate’s verify_slice does this. In Go, use hmac.Equal.

3. Rust: Borrowing from the Request Buffer

Serde supports zero-copy deserialization. If a struct field is &'a str, serde_json can return a slice that points directly into the input buffer instead of allocating a String.

There is one important catch. A string can only be borrowed if it contains no escape sequences. If a JSON string contains something like \n or \", the decoder must rewrite it, so it cannot hand back a pointer into the original bytes. With a bare &str field, deserialization then fails with an error. Serde cannot detect this at compile time, so the failure appears at runtime on real traffic.

For untrusted input such as webhooks, use Cow<'a, str> with #[serde(borrow)] instead. It borrows when it can and falls back to an owned String only when unescaping is required. Both the Cow type and the attribute are needed, because serde defaults to the owned variant otherwise. (#[serde(borrow)] is implicit for &str and &[u8] fields, but not for Cow.)

This also means axum’s standard Json extractor is not the right tool for borrowed types, because it requires owned data. Take the body as Bytes and call serde_json::from_slice yourself, or use the JsonDeserializer extractor from axum-extra, which exists specifically for zero-copy deserialization.

The payload below is illustrative. Real GitHub, Stripe and Shopify events are larger and nested.

use std::{borrow::Cow, sync::Arc};

use axum::{
    body::Bytes,
    extract::{DefaultBodyLimit, State},
    http::{HeaderMap, StatusCode},
    routing::post,
    Router,
};
use hmac::{Hmac, Mac};
use serde::Deserialize;
use sha2::Sha256;
use tokio::sync::mpsc;

type HmacSha256 = Hmac<Sha256>;

#[derive(Debug, Deserialize)]
struct WebhookPayload<'a> {
    #[serde(borrow)]
    event_id: Cow<'a, str>,
    #[serde(borrow)]
    action: Cow<'a, str>,
    timestamp: u64,
    #[serde(borrow)]
    actor: Cow<'a, str>,
}

#[derive(Clone)]
struct AppState {
    secret: Arc<[u8]>,
    queue: mpsc::Sender<Bytes>,
}

fn verify_github(secret: &[u8], header: &str, body: &[u8]) -> bool {
    let Some(hex_sig) = header.strip_prefix("sha256=") else { return false };
    let Ok(sig) = hex::decode(hex_sig) else { return false };
    let Ok(mut mac) = HmacSha256::new_from_slice(secret) else { return false };
    mac.update(body);
    mac.verify_slice(&sig).is_ok() // constant-time comparison
}

async fn ingest_webhook(
    State(state): State<AppState>,
    headers: HeaderMap,
    body: Bytes,
) -> StatusCode {
    // 1. Authenticate the raw bytes before parsing anything.
    let Some(sig) = headers
        .get("x-hub-signature-256")
        .and_then(|v| v.to_str().ok())
    else {
        return StatusCode::UNAUTHORIZED;
    };
    if !verify_github(&state.secret, sig, &body) {
        return StatusCode::UNAUTHORIZED;
    }

    // 2. Hand off a cheap, reference-counted handle to the same buffer.
    //    Bytes::clone() bumps a refcount; it does not copy the payload.
    match state.queue.try_send(body) {
        Ok(()) => StatusCode::ACCEPTED,
        Err(_) => StatusCode::SERVICE_UNAVAILABLE, // queue full: apply backpressure
    }
}

async fn worker(mut rx: mpsc::Receiver<Bytes>) {
    while let Some(body) = rx.recv().await {
        // The parsed struct borrows from `body` and lives only inside this scope.
        if let Ok(event) = serde_json::from_slice::<WebhookPayload>(&body) {
            tracing::info!(id = %event.event_id, action = %event.action, "event received");
        }
    }
}

#[tokio::main]
async fn main() {
    let (tx, rx) = mpsc::channel::<Bytes>(10_000);
    tokio::spawn(worker(rx));

    let state = AppState {
        secret: Arc::from(b"replace-with-your-webhook-secret".as_slice()),
        queue: tx,
    };

    let app = Router::new()
        .route("/webhook", post(ingest_webhook))
        .layer(DefaultBodyLimit::max(1024 * 1024)) // cap request bodies at 1 MiB
        .with_state(state);

    let listener = tokio::net::TcpListener::bind("0.0.0.0:8080").await.unwrap();
    axum::serve(listener, app).await.unwrap();
}

A few details in this code matter:

Lifetimes. A struct with a borrowed field cannot outlive the buffer it borrows from. Because it cannot be sent across tasks on its own, send the Bytes handle instead. Bytes is a cheaply cloneable, reference-counted buffer, so queueing it does not copy the payload.
Body limits. Axum’s Bytes extractor refuses bodies larger than 2 MB by default. Set an explicit limit that matches your largest legitimate payload, as shown above.
Hot-path logging. Avoid println! on the hot path. Use a structured logger such as tracing with sampling.
Scope of the claim. “Zero-copy” here applies to string fields during parsing. The framework still buffers the body, the HMAC helper here decodes the hex signature into a small Vec, and escaped strings fall back to owned Strings. Verify allocation counts with a profiler such as dhat rather than assuming zero.

4. Go: Generated Parsers with `easyjson`

encoding/json has traditionally relied on runtime reflection. easyjson avoids that by generating UnmarshalEasyJSON and MarshalEasyJSON methods at build time. Its README claims it outperforms encoding/json by a factor of 4–5x and other JSON packages by 2–3x. It also generates standard MarshalJSON/UnmarshalJSON methods, so types remain compatible with encoding/json, but you only get the speedup by calling easyjson.Unmarshal directly.

Some practical details:

The generator does not work on package main files, so put your types in a separate package.
Add //go:generate easyjson -all webhook.go and run go generate. This produces webhook_easyjson.go.
By default, string fields are copied out of the input buffer, which costs allocations. A field tagged json:"actor,nocopy" makes the string refer to the original buffer memory instead. That is only safe if the decoded object does not outlive the buffer. The README notes this works well for short-lived objects, and strings that need unescaping are processed normally.
A string field with the intern option is deduplicated, which helps for fields that take only a few distinct values, such as an action name.
easyjson uses unsafe for no-copy conversions during unmarshaling. You can build without it using the easyjson_nounsafe build tag.
Its own limitations are documented: object keys are case-sensitive, there is no true streaming, and the parser does only the minimum work needed to skip unmatched values rather than fully validating them.

// webhook.go
package webhook

//go:generate easyjson -all webhook.go

type Payload struct {
    EventID   string `json:"event_id"`
    Action    string `json:"action,intern"`
    Timestamp uint64 `json:"timestamp"`
    Actor     string `json:"actor"`
}

// handler.go
package webhook

import (
    "bytes"
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "net/http"
    "strings"
    "sync"

    "github.com/mailru/easyjson"
)

const maxBody = 1 << 20 // 1 MiB

var bufPool = sync.Pool{New: func() any { return new(bytes.Buffer) }}

func validSignature(secret []byte, header string, body []byte) bool {
    const prefix = "sha256="
    if !strings.HasPrefix(header, prefix) {
        return false
    }
    got, err := hex.DecodeString(header[len(prefix):])
    if err != nil {
        return false
    }
    mac := hmac.New(sha256.New, secret)
    mac.Write(body)
    return hmac.Equal(got, mac.Sum(nil)) // constant-time comparison
}

func Handler(secret []byte) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        buf := bufPool.Get().(*bytes.Buffer)
        buf.Reset()
        defer bufPool.Put(buf)

        // Bound the body size and reuse a pooled buffer instead of io.ReadAll.
        if _, err := buf.ReadFrom(http.MaxBytesReader(w, r.Body, maxBody)); err != nil {
            http.Error(w, "payload too large or unreadable", http.StatusRequestEntityTooLarge)
            return
        }
        body := buf.Bytes()

        // Authenticate the raw bytes first.
        if !validSignature(secret, r.Header.Get("X-Hub-Signature-256"), body) {
            http.Error(w, "invalid signature", http.StatusUnauthorized)
            return
        }

        var p Payload
        if err := easyjson.Unmarshal(body, &p); err != nil {
            http.Error(w, "invalid JSON", http.StatusBadRequest)
            return
        }

        // Strings in p are copies (no "nocopy" tags), so p is safe to use
        // after the buffer returns to the pool. If you add "nocopy", it is not.
        w.WriteHeader(http.StatusAccepted)
    }
}

The pooled buffer cuts the per-request allocation of the body slice. That is useful, but be careful: the moment you tag a field nocopy, you must finish using the decoded object before the deferred Put runs.

5. What Changed in Go Recently

Two Go releases change the picture for this kind of service:

Green Tea garbage collector (Go 1.26). It was an experiment in Go 1.25 and is enabled by default in Go 1.26. The release notes say it improves marking and scanning of small objects and expect a 10–40% reduction in GC overhead for programs that lean heavily on the garbage collector. That helps allocation-heavy parsers, but it does not remove the benefit of allocating less.
encoding/json/v2 (Go 1.27). Go 1.27 was released on August 19, 2026. It makes encoding/json/v2 and encoding/json/jsontext generally available, and encoding/json itself is now backed by the v2 implementation, with the stated goal of faster unmarshaling. The v2 API has stricter defaults than v1, such as rejecting duplicate object keys and invalid UTF-8, so don’t assume switching is behavior-neutral. One independent benchmark found encoding/json/v2 unmarshaling 1.5x to 2.3x faster than the original implementation, with mixed marshaling results depending on the workload.

The practical consequence is that the headline gap between easyjson and the standard library may be smaller today than the 2016-era README numbers suggest. Benchmark easyjson against Go 1.27’s encoding/json and encoding/json/v2 on your actual payloads before adding a code generator to your build.

If you are considering json-iterator/go as a reflection-free alternative, note that it is not actually reflection-free. It builds and caches per-type decoders using reflection (through reflect2), and its repository was archived on December 15, 2025. For a new service, easyjson or the standard library are safer choices.

6. Measure It Yourself

Treat any numbers in this article, including the ones above, as starting points. To get figures that apply to your service:

Use a realistic payload. Benchmark with real, anonymized webhook bodies from the providers you integrate with, not a synthetic struct.
Count allocations. In Go, run go test -bench . -benchmem and watch allocs/op. In Rust, use a profiler like dhat or a counting allocator in tests.
Load test end to end. Parsing is one stage. Network I/O, TLS, signature verification, queue writes and storage often dominate once parsing is fast.
Watch tail latency. Compare p99 latency and GC pause or allocator behavior under sustained load, not only averages.

Key Takeaways

Authenticate raw bytes before parsing. It is cheaper, safer, and the only way to get the signature right, since all three major providers sign the exact body they send.
In Rust, use Cow<'a, str> with #[serde(borrow)] for untrusted JSON. Bare &str fields fail at runtime on any escaped string.
Enqueue Bytes, not parsed structs. A reference-counted buffer handle crosses task boundaries cheaply.
In Go, easyjson reduces allocations but does not remove them. Its nocopy and intern tags cut them further, at the price of buffer-lifetime discipline.
Re-benchmark on current Go. Go 1.26’s default GC and Go 1.27’s v2-backed encoding/json both change the baseline.
Set explicit body-size limits in both languages, and measure with your own traffic.

References

Originally published at https://instawebhook.com/blog/zero-copy-webhook-parsing-optimizing-rust-and-go-ingress-pipelines-for-100k-rps

DEV Community: InstaWebhook

Memory-Mapped Webhook Buffers: Using `mmap` and Ring Buffers to Cut Kernel-User Crossings at 500k RPS

1. Where the time goes at 500k RPS

The traditional receive path

A system call is not a context switch

2. What mmap and ring buffers actually buy you

3. A correct SPSC shared-memory ring

4. Pairing the ring with io_uring for network ingress

5. Production hardening

Backpressure and durability

Be careful with file-backed mmap

NUMA and CPU affinity

Check the rest of the path

6. What measurements actually show

Summary

References

Building a Resilient Webhook Ingress Proxy in Go: Handling Slow Consumers Without Thread Starvation

1. The Anatomy of the Webhook Ingress Problem

The Naive Anti-Pattern: Unbounded Goroutines

2. Architecture Blueprint: Decoupling Ingress and Egress

3. Sizing the Pool: Little’s Law Again

4. Deep Dive 1: A Bounded Worker Pool

5. Deep Dive 2: Non-Blocking Enqueue and Backpressure

6. Deep Dive 3: Connection Pooling in Go’s HTTP Client

7. A Complete, Runnable Example

Design notes on the code

Try it against a slow consumer

8. Observability: Know When You Are Absorbing Pain

9. Idempotency: Assume Duplicates

10. Security Notes

11. Advanced Enhancements

12. Conclusion

Sources and Further Reading

High-Performance Ingress & Network Architecture: Building Resilience at Scale

1. Know the Contract Before Tuning the Network

2. HTTP/3 and QUIC: What They Fix and What They Don’t

Where TCP-based HTTP hurts

What QUIC changes

Where HTTP/3 Datagrams fit

Server sketch in Go (quic-go)

3. Preventing Slowloris and Slow-Read Attacks

The attack patterns

Key principle: most timeouts measure gaps, not totals

NGINX example

Go net/http example

Rust (Actix Web) example

Related protocol-level attacks

4. Surviving Traffic Bursts: Durable Buffering with mmap

The problem

What an mmap ring buffer gives you

What it does not give you

A correct single-producer, single-consumer ring in Go

Choosing a buffering strategy

5. Putting It Together

Further Reading

Deterministic Reordering of Concurrent Webhooks: Handling Out-of-Order State Transitions with CRDTs

Introduction: The Multi-Region Webhook Delivery Dilemma

1. The Anatomy of Out-of-Order Webhook Failures

Common Failure Modes in Multi-Region Ingress

2. Foundations: CRDTs and Strong Eventual Consistency

State-based CRDTs (CvRDTs)

Operation-based CRDTs (CmRDTs)

Logical Clocks

Deterministic Tie-Breaking

3. Architecture of a Deterministic Reordering Layer

4. Reference Implementation (TypeScript)

5. Production Resilience Patterns

A. Unbounded buffers and missing events

B. Clock drift and sequence sources

C. Tombstone garbage collection

D. When to skip the CRDT

Summary: Approaches Compared

Conclusion

References

High-Velocity Webhook Engineering: Low-Level Ingress, Dynamic Flow Control, and Multi-Tenant Isolation

1. Ingress with io_uring: Multishot Receive and Provided Buffers

Where the standard epoll model spends its time

What io_uring changes

A correction on “zero-copy”

How the data flows

2. What `mmap` and ring buffers actually buy you

4. Pairing the ring with `io_uring` for network ingress

Be careful with file-backed `mmap`

Server sketch in Go (`quic-go`)

Go `net/http` example

4. Surviving Traffic Bursts: Durable Buffering with `mmap`

What an `mmap` ring buffer gives you

1. Ingress with `io_uring`: Multishot Receive and Provided Buffers

What `io_uring` changes

Part 1: RAM-Disk Ingress Buffers — Using Linux `tmpfs` and Shared Memory to Absorb Webhook Bursts

How `tmpfs` behaves

Ring buffers and `io_uring`

HTTP semantics: `429` and `Retry-After`