DEV Community: kholnnaw

Choosing the right throttling algorithm for a multi‑tenant TypeScript API

kholnnaw — Wed, 21 Oct 2026 08:37:50 +0000

in a SaaS platform that exposes a public GraphQL endpoint, each tenant is allowed a fixed number of requests per minute. the naive implementation—just checking a counter at the start of each resolver—collapsed under burst traffic, causing 429 spikes and downstream database overload.

Token bucket vs. leaky bucket

the core problem is two‑fold: we need a deterministic way to limit traffic per tenant, and we need that limiter to be cheap enough to run inside the request path without adding noticeable latency. two classic algorithms fit the bill:

token bucket – fills a bucket with tokens at a steady rate; a request consumes a token, allowing short bursts as long as tokens remain.
leaky bucket – treats incoming requests as water flowing into a bucket that leaks at a constant rate; excess traffic is discarded, producing a smoother request stream.

Both can be implemented in pure TypeScript with an in‑memory map or a distributed cache for horizontal scaling. the choice hinges on the desired traffic shape and the cost of state synchronization.

interface Bucket {
  tokens: number;
  lastRefill: number;
}

const LIMIT = 120; // requests per minute per tenant
const REFILL_RATE = LIMIT / 60_000; // tokens per ms
const buckets = new Map<string, Bucket>();

function allow(tenantId: string): boolean {
  const now = Date.now();
  const bucket = buckets.get(tenantId) ?? { tokens: LIMIT, lastRefill: now };

  // refill based on elapsed time
  const elapsed = now - bucket.lastRefill;
  bucket.tokens = Math.min(LIMIT, bucket.tokens + elapsed * REFILL_RATE);
  bucket.lastRefill = now;

  if (bucket.tokens >= 1) {
    bucket.tokens -= 1;
    buckets.set(tenantId, bucket);
    return true;
  }
  // out of tokens – reject request
  return false;
}

The snippet shows a simple in‑memory token bucket. for a distributed system you would replace the Map with a Redis hash and use Lua scripts to keep the refill‑and‑check operation atomic.

Trade‑offs and edge cases

Burst tolerance – token bucket permits short spikes, which is useful for UI‑driven interactions but can still overwhelm downstream services if the burst size exceeds their capacity. leaky bucket smooths traffic but may reject legitimate user bursts, hurting perceived responsiveness.
Clock drift – both algorithms rely on timestamps. if nodes have unsynchronized clocks, token counts diverge. using a single source of truth (e.g., Redis server time) eliminates the drift at the cost of an extra network hop.
State overflow – a naive map grows with every tenant that ever hits the endpoint. you need eviction (e.g., setTimeout to purge idle buckets) or a TTL in the cache to keep memory bounded.
Distributed consistency – in a multi‑instance deployment, race conditions can allow a tenant to exceed its limit if two instances read the same token count concurrently. atomic scripts or compare‑and‑set primitives guard against this.

When to pick which?

if your service can absorb short bursts and you value lower latency, token bucket is the default. if downstream APIs are fragile or you need to guarantee a smooth request rate, leaky bucket (or a hybrid that caps burst size) is safer.

how do you handle throttling in your services, and which algorithm has proven most reliable under real load?

avoiding race conditions in async cascade deletes for event‑sourced services

kholnnaw — Sun, 18 Oct 2026 01:49:22 +0000

you've got an event‑sourced aggregate that emits a UserDeleted event, and several downstream services react by cleaning up related rows in their own stores. the cleanup runs in background workers, so a second request that references the same user can slip through the cracks and hit a foreign‑key violation.

the race condition

in a typical setup each service subscribes to the UserDeleted topic and issues a cascade delete against its own tables. because the subscription processing is asynchronous, a request that arrives after the UserDeleted event is published but before the worker finishes will see stale data. the DB then throws an error, and the client gets a 500 instead of a graceful "not found".

a deterministic approach

the trick is to make the delete operation idempotent and observable before the request reaches the service. one pattern is to store a tombstone flag in a shared cache (e.g. Redis) as part of the same transaction that publishes the event. workers check the flag before proceeding, and request handlers read it to short‑circuit.

// publish + tombstone atomically
await redis.multi()
  .set(`tombstone:user:${userId}`, '1', 'EX', 300) // 5 min TTL
  .publish('user.deleted', JSON.stringify({ userId }))
  .exec();

workers then do:

const flag = await redis.get(`tombstone:user:${userId}`);
if (!flag) return; // nothing to clean up
await db.transaction(async trx => {
  await trx('orders').where({ userId }).delete();
  await trx('profiles').where({ userId }).delete();
});
await redis.del(`tombstone:user:${userId}`);

trade‑offs

latency vs consistency – adding the cache lookup adds a few milliseconds to every request, but it eliminates sporadic DB errors.
cache eviction – the TTL must be long enough to cover the slowest worker, otherwise a stale request could still hit the DB.
failure handling – if the worker crashes after deleting rows but before clearing the tombstone, subsequent requests will be rejected even though the data is gone. a periodic reconciler that scans for orphaned tombstones can mitigate this.
testing – simulate the race by issuing a delete and a read in parallel within integration tests; assert that the read returns a 404 rather than a DB exception.

when to apply

if your service experiences occasional 500 spikes after bulk deletions, or if you already use a message broker for domain events, the tombstone pattern is a low‑overhead way to enforce eventual consistency without locking the DB.

anyone using a similar approach with different stores (e.g. DynamoDB streams) or who prefers optimistic concurrency controls instead? how do you keep the delete path safe without sacrificing throughput?

Refactoring a Distributed Cache for Strong Reads When Eventual Consistency Fails

kholnnaw — Thu, 15 Oct 2026 03:02:36 +0000

we hit a subtle bug in our order‑service after a recent traffic surge: occasional duplicate invoices appeared in the admin UI. the root cause? a cache layer that was designed for eventual consistency but was being used for read‑after‑write scenarios.

The Problem

our stack uses a Node.js API gateway, a PostgreSQL write store, and a Redis cluster as a read‑through cache. writes go through a transaction, then an async job publishes an order.updated event to a Kafka topic. the consumer updates the Redis entry. under low load the latency between the DB commit and the cache refresh is negligible, but under load the lag grows to several seconds. the UI, which reads directly from Redis for latency reasons, sometimes shows stale data, leading to duplicate processing downstream.

Why Not Just Switch to Strong Consistency?

we could have forced every read to hit the DB, but that defeats the purpose of the cache. another option is to use Redis' wait command to block until replicas acknowledge the write, but that only helps with replication lag, not with our asynchronous event pipeline. the real challenge is to preserve the low‑latency read path while guaranteeing that any read immediately after a write sees the latest state.

A Pragmatic Refactor

instead of treating the cache as a fire‑and‑forget layer, we turned it into a write‑through store for critical entities. the API now performs the DB write and updates Redis in the same request, using a transaction‑like pattern. we still keep the async Kafka flow for eventual propagation to other services, but the cache is now authoritative for the write‑through path.

// write‑through helper – keeps DB and Redis in sync
async function updateOrder(id: string, payload: Partial<Order>) {
  // 1️⃣ start DB transaction
  await db.transaction(async trx => {
    await trx('orders').where({ id }).update(payload);
    // 2️⃣ update Redis inside the same logical unit
    const cacheKey = `order:${id}`;
    const fresh = { ...(await trx('orders').where({ id }).first()), ...payload };
    await redis.set(cacheKey, JSON.stringify(fresh));
  });
  // 3️⃣ fire async event for downstream consumers
  await kafka.produce('order.updated', { id, ...payload });
}

Trade‑offs & Edge Cases

Latency vs. Consistency – the write‑through path adds a few milliseconds to the request, but the UI now reads correct data.
Cache Invalidation – if another service updates the order directly in the DB (bypassing the API), the cache can become stale. we mitigated this by adding a TTL of 30 seconds and a background poller that reconciles mismatches.
Failure Scenarios – if Redis is unavailable, the transaction still commits to the DB, but we must decide whether to fail the request or proceed without caching. in our case we fail fast, returning a 503, because a stale cache is worse than a brief outage.
Memory Pressure – keeping a large write‑through cache can increase memory usage. we tuned Redis maxmemory-policy to volatile-lru and scoped the cache to only high‑traffic entities.

The Takeaway

by moving from an eventual‑only cache to a write‑through model we regained strong read guarantees without sacrificing overall performance. the pattern does introduce extra complexity around failure handling and cache hygiene, but those concerns are manageable with clear contracts.

how do you handle read‑after‑write consistency in your high‑traffic services, and what patterns have you found most sustainable?