DEV Community: Krrish Kumar

Saved to Postgres, never queued: the transactional outbox for Node

Krrish Kumar — Sun, 01 Nov 2026 08:37:36 +0000

Here's a bug that never shows up in tests. A request saves an order to Postgres, then calls queue.add('send-receipt', ...). The deploy restarts the pod between those two lines, or Redis blips for two seconds, and now you have a paid order that never gets a receipt, an invoice or a warehouse pick. Nothing errors. Nothing retries. The two writes were never one unit, and nobody notices until a customer emails.

I'm Krrish Kumar, a backend engineer. I wrote this with AI assistance for drafting and editing; the code and every number below come from a test script I ran against PostgreSQL 18.6 and Redis, with BullMQ 5.81.5 and 6.5.3. The fix is an old pattern called the transactional outbox, and this post builds a small version of it in Node, plus the three mistakes I made on the way.

Why "just retry" doesn't work

There are two orders you can write in, and both are broken:

You do	Crash between the two writes	Result
`COMMIT`, then `queue.add()`	after the commit	Order saved, job never created
`queue.add()`, then `COMMIT`	after the add	Job runs for an order that was rolled back

You can't wrap Redis and Postgres in one transaction. Even BullMQ 6, which can now keep its queues in PostgreSQL, doesn't help here: its backend interface "intentionally exposes no connection or transaction type", so queue.add() runs on its own connection, outside your transaction.

The outbox gets around this by not writing to the queue in the request at all. The request writes the order and a row describing the job in the same database transaction. A separate relay process copies outbox rows into the queue later. If the transaction commits, both rows exist. If it rolls back, neither does.

The table

CREATE TABLE outbox (
  id           bigserial PRIMARY KEY,
  topic        text NOT NULL,
  payload      jsonb NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now(),
  published_at timestamptz
);

-- Only unpublished rows are indexed, so the relay's query stays cheap
-- even when the table holds millions of published rows.
CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;

The write side

export async function createOrder(pool, { customer, totalPaise }) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    const { rows: [order] } = await client.query(
      'INSERT INTO orders (customer, total_paise) VALUES ($1, $2) RETURNING id',
      [customer, totalPaise],
    );
    await client.query(
      `INSERT INTO outbox (topic, payload) VALUES ('order.created', $1)`,
      [{ orderId: order.id, customer }],
    );
    await client.query('NOTIFY outbox'); // delivered only if the transaction commits
    await client.query('COMMIT');
    return order.id;
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }
}

The NOTIFY is optional, and it's free: Postgres holds notifications sent inside a transaction until it commits, and drops them on rollback. The relay uses it to wake up immediately instead of waiting for its next poll.

The relay

export async function relayBatch(pool, queue, { limit = 100 } = {}) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    const { rows } = await client.query(
      `SELECT id, topic, payload FROM outbox
        WHERE published_at IS NULL
        ORDER BY id
        LIMIT $1
        FOR UPDATE SKIP LOCKED`,
      [limit],
    );
    for (const row of rows) {
      await queue.add(row.topic, row.payload, { jobId: `outbox-${row.id}` });
    }
    await client.query(
      'UPDATE outbox SET published_at = now() WHERE id = ANY($1)',
      [rows.map((r) => r.id)],
    );
    await client.query('COMMIT');
    return rows.length;
  } catch (err) {
    await client.query('ROLLBACK').catch(() => {});
    throw err;
  } finally {
    client.release();
  }
}

Three things carry the weight here:

FOR UPDATE SKIP LOCKED lets you run more than one relay. Each one locks the rows it picked, and the others skip them instead of waiting. In my test, 4 relays drained 500 rows with exactly 500 queue.add() calls between them, split 100/150/150/100.
The row is marked published only after the add succeeds. If the process dies after queue.add() but before COMMIT, the transaction rolls back, the lock goes away, and the next run picks the same row again. That means the relay is at least once: the same row can reach the queue twice.
jobId: outbox-<id> is what makes "twice" harmless. BullMQ ignores an add() whose jobId already exists. I simulated the crash (throw after the adds, before the commit), ran the relay again, and the queue held one job, not two.

One trade-off: the transaction stays open while you talk to Redis. With a batch of 100 that's a few milliseconds, but keep batches small so one slow Redis call doesn't hold locks for long.

The loop around it drains, then sleeps until a NOTIFY arrives or a poll timer fires, whichever comes first:

export async function runRelay(pool, queue, { pollMs = 5000, signal } = {}) {
  const listener = await pool.connect();
  let wake;
  const nudge = () => wake?.();
  listener.on('notification', nudge);
  await listener.query('LISTEN outbox');
  try {
    while (!signal?.aborted) {
      while ((await relayBatch(pool, queue)) > 0) {}
      await new Promise((resolve) => {
        wake = resolve;
        setTimeout(resolve, pollMs);
        signal?.addEventListener('abort', resolve, { once: true });
      });
    }
  } finally {
    listener.removeListener('notification', nudge);
    await listener.query('UNLISTEN outbox').catch(() => {});
    listener.release();
  }
}

With the poll timer set to 60 seconds, a new order was in the queue 9 ms after its commit, so NOTIFY did the work. Keep the poll anyway. A notification sent while the relay is reconnecting is simply lost, and the poll is what catches it.

Mistake 1: a "last seen id" cursor

My first relay remembered the highest id it had published and asked for WHERE id > $lastSeen. It looks fine, and it loses rows.

bigserial hands out ids when a row is inserted, not when its transaction commits. Transaction A inserts id 1 and is still working. Transaction B inserts id 2 and commits. The relay reads, sees only id 2, and sets lastSeen = 2. Then A commits. Id 1 is now in the table, and id > 2 will never return it. My test reproduced it: the first read saw [2], the second saw [], and row 1 was stuck forever.

Asking WHERE published_at IS NULL instead has no cursor to get ahead of. The same test with that query published both rows.

Mistake 2: `removeOnComplete: true`

The jobId dedupe only works while BullMQ still has the job. With removeOnComplete: true (or { count: 0 }), a completed job is deleted at once, and a later add() with the same jobId is treated as new. In my test the processor ran twice for outbox-1.

That matters for the outbox, because the duplicate add comes from a relay retry, and the first job may already be done by then. Keep completed jobs for longer than any retry could take:

const worker = new Worker('orders', processor, {
  connection,
  removeOnComplete: { age: 24 * 3600 }, // seconds
});

With that setting, a second add() of the same jobId inside the window was ignored, and the processor ran once. This narrows the window but doesn't close it. If your consumer sends money or email, have it check orderId before acting too.

Mistake 3: a colon in the jobId

I reached for a colon out of habit. BullMQ rejects it: Custom Id cannot contain :, because : is its Redis key separator. A bare number fails too, with Custom Id cannot be integers, so jobId: String(row.id) doesn't work either. A prefix with a dash (outbox-42) passes both checks, on BullMQ 5 and 6.

I hit mistakes 2 and 3 often enough that I wrote lint rules for them in eslint-plugin-bullmq, a small plugin I maintain. On a file that sets jobId: `outbox:${row.id}` together with removeOnComplete: true, it reports both:

5:52  error    BullMQ uses ':' as its Redis key separator, so a custom jobId containing ':' is rejected ...  bullmq/valid-job-id
5:72  warning  removeOnComplete: true deletes the job as soon as it finishes, so a later add() with the same jobId runs again ...  bullmq/no-jobid-dedupe-with-remove-on-complete

You don't need it for any of this. A code review checklist with the same two lines works too.

Cleaning up

Published rows pile up. Delete them in batches from a daily job, so one big DELETE doesn't hold locks or bloat the table:

DELETE FROM outbox
 WHERE id IN (
   SELECT id FROM outbox
    WHERE published_at < now() - interval '7 days'
    LIMIT 5000
 );

Run it in a loop until it deletes 0 rows. Keep the retention longer than your BullMQ removeOnComplete age, so you can still trace a job back to its row.

When you don't need this

If losing an occasional job is fine (analytics pings, cache warming), skip the outbox. If the work can live entirely in Postgres, a Postgres-backed job queue that shares your connection, such as graphile-worker or pg-boss, can enqueue inside your own transaction, and you get the same guarantee without a relay. The outbox is for when the job has to land somewhere else: Redis, Kafka, a webhook, another service.

Checklist

Write the business row and the outbox row in one transaction. Never call the queue from the request.
Relay with WHERE published_at IS NULL ... FOR UPDATE SKIP LOCKED, not an id cursor.
Mark rows published only after the queue accepted them, and expect duplicates.
Use a deterministic jobId with no : and not a bare number.
Keep completed jobs longer than a relay retry could take, and make consumers idempotent anyway.
NOTIFY for speed, poll for safety.
Delete published rows in batches.

Disclosure: I'm Krrish Kumar, and I maintain eslint-plugin-bullmq. The test script behind the numbers here ran on PostgreSQL 18.6, BullMQ 5.81.5 and 6.5.3, ioredis and node-postgres.

Testing webhook handlers against duplicates, races and out-of-order delivery

Krrish Kumar — Thu, 29 Oct 2026 05:38:58 +0000

Most webhook handlers get tested once: send one event, check for a 200, move on. Production doesn't send one event. Stripe's docs say so. Under Handle duplicate events: "Webhook endpoints might occasionally receive the same event more than once." Under event ordering: "Stripe doesn't guarantee the delivery of events in the order that they're generated." And in live mode Stripe retries failed deliveries "for up to three days with an exponential back off."

So the handler you need is one that gives the right business result when the same event arrives twice, when two copies arrive at the same moment, and when a refund shows up before the purchase it refunds. A 200 status tells you none of that.

This post builds a small handler, shows how each of those cases breaks it, and then shows how to test for them in a normal test run.

The delivery cases worth testing

Case	What happens in production	What breaks
Sequential duplicate	A retry after a timeout, or the same event sent twice	Credits granted twice
Concurrent duplicate	Two copies hit two instances at once	"Check then insert" dedupe races and both copies pass
Out of order	A refund arrives before the purchase	The refund is lost, the purchase is granted
Forged signature	Anyone can POST to your URL	Unverified events change state
Missing signature	A health checker or scanner hits the URL	Handler crashes with a 500
Stale timestamp	A captured request is replayed later	Old event accepted
Malformed JSON	Truncated body	Unhandled `JSON.parse` returns a 500, so the provider retries

The last four are about input checking. The first three are where most real bugs are, so most of this post is about them.

The race in "check then insert"

Here's a handler that looks correct. It verifies the signature, parses the body safely, and skips events it has seen:

if (db.processed.has(event.id)) return new Response('duplicate');
await grantCredits(user, 100);      // a database call
await db.markProcessed(event.id);   // another database call

Send the event twice, one after the other, and it works. The second request sees the id and stops.

Send two copies at the same time and both requests run the has() check before either one reaches markProcessed(). Both see "not processed", and both grant credits. Any await between the check and the write opens this window, and with a real database there's always one.

The fix is to make the claim and the check one atomic step, before the work:

INSERT INTO processed_events (event_id) VALUES ($1)
ON CONFLICT (event_id) DO NOTHING
RETURNING event_id;
-- no row returned means another request already claimed it

The Redis version is SET webhook:evt_123 1 NX EX 259200. If the work goes into a queue, a BullMQ job with jobId set to the event id does the same thing, as long as the completed job isn't removed straight away.

Out of order: record facts, don't do arithmetic

Suppose the handler adds 100 credits on checkout.session.completed and subtracts 100 on charge.refunded, never going below zero. If the refund arrives first, it subtracts from zero and does nothing. Then the purchase arrives and adds 100. The customer ends up refunded and still holding credits.

The fix is to store what you know about the object (paid, refunded) and work out the result from that, so the order of arrival doesn't matter. Another option is to fetch the current object from the API when an event arrives and act on that instead of the payload. Stripe's docs mention using the API to retrieve objects for the same reason.

Testing all of it in one run

You could write each of these tests by hand. Each one needs a correctly signed request (Stripe uses an HMAC-SHA256 over timestamp.body in a t=...,v1=... header), a way to reset your state, and a check on business state afterwards rather than on status codes.

I packaged that as webhook-crash-lab. You give it a handler (any (req: Request) => Response function, or a URL), a secret, an event, and two functions: one to reset your state and one to read it. It sends the seven cases above (eight scenarios, because a valid single delivery is the baseline), and compares your state to what you said it should be.

Here's a complete, runnable file. The handler does the two fixes from above: it claims the event id before any await, and it records facts per checkout session instead of adding and subtracting.

// crash-test.mjs
import { runCrashLab, toText, verifySignature } from 'webhook-crash-lab';

const SECRET = 'whsec_test_secret';
const db = { processed: new Set(), sessions: new Map(), ledger: [] };

const sessionOf = (event) => event.data.object.session ?? event.data.object.id;
const credits = () =>
  [...db.sessions.values()].filter((s) => s.paid && !s.refunded).length * 100;

async function handler(req) {
  const raw = await req.text();
  if (!verifySignature(raw, req.headers.get('stripe-signature'), SECRET)) {
    return new Response('bad signature', { status: 400 });
  }
  let event;
  try {
    event = JSON.parse(raw);
  } catch {
    return new Response('invalid json', { status: 400 });
  }

  // Claim the event id before any await.
  if (db.processed.has(event.id)) return new Response('duplicate');
  db.processed.add(event.id);

  // Record facts about the session instead of adding and subtracting.
  const s = db.sessions.get(sessionOf(event)) ?? { paid: false, refunded: false };
  if (event.type === 'checkout.session.completed') s.paid = true;
  if (event.type === 'charge.refunded') s.refunded = true;
  db.sessions.set(sessionOf(event), s);
  db.ledger.push(event.id);

  return new Response('ok');
}

const checkout = { id: 'evt_1', type: 'checkout.session.completed', data: { object: { id: 'cs_1' } } };
const refund = { id: 'evt_2', type: 'charge.refunded', data: { object: { id: 'ch_1', session: 'cs_1' } } };

const result = await runCrashLab({
  target: handler,
  secret: SECRET,
  event: checkout,
  adapter: {
    reset: () => {
      db.processed.clear();
      db.sessions.clear();
      db.ledger = [];
    },
    observe: () => ({ credits: credits(), ledgerEntries: db.ledger.length }),
  },
  expect: { once: { credits: 100, ledgerEntries: 1 } },
  sequence: { events: [checkout, refund], expected: { credits: 0 } },
});

console.log(toText(result));
process.exitCode = result.summary.fail > 0 ? 1 : 0;

npm i -D webhook-crash-lab
node crash-test.mjs

webhook-crash-lab: checkout.session.completed

  PASS  single-valid           One valid delivery
  PASS  duplicate-sequential   Same event delivered again (retry)
  PASS  duplicate-concurrent   Same event delivered in parallel
  PASS  out-of-order           Related events arrive out of order
  PASS  invalid-signature      Forged signature
  PASS  missing-signature      No signature header
  PASS  stale-timestamp        Old, replayed delivery
  PASS  malformed-json         Broken JSON body

  8 passed, 0 failed, 0 not tested

Now break it. Replace the claim-and-record block with the "check then insert" version and plain arithmetic (add credits: 0 to db, reset it in reset(), and have observe() return db.credits):

if (db.processed.has(event.id)) return new Response('duplicate');
await new Promise((r) => setTimeout(r, 5)); // stands in for a DB round trip
if (event.type === 'checkout.session.completed') db.credits += 100;
if (event.type === 'charge.refunded') db.credits = Math.max(0, db.credits - 100);
db.ledger.push(event.id);
db.processed.add(event.id);

The run now reports:

  PASS  single-valid           One valid delivery
  PASS  duplicate-sequential   Same event delivered again (retry)
  FAIL  duplicate-concurrent   Same event delivered in parallel
        business state: The effect was applied more than once or not at all: credits: expected 100, got 500; ledgerEntries: expected 1, got 5.
        Why: "Check then insert" dedupe races: two requests both see "not processed" and the effect is granted twice.
        Fix: Claim the event id atomically (INSERT ... ON CONFLICT DO NOTHING, Redis SET NX, BullMQ jobId = event.id) before doing the work.
  FAIL  out-of-order           Related events arrive out of order
        business state: Final state is wrong after out-of-order delivery: credits: expected 0, got 100.
        Why: Stripe does not guarantee event order. A refund that arrives first must not be lost.
        Fix: Record late-arriving facts (tombstones / status with timestamps) or re-fetch the object from the API before applying.
  ...
  6 passed, 2 failed, 0 not tested

Note that the sequential duplicate still passes. That's why a single "send it twice" test isn't enough.

The concurrent scenario sends 5 copies in parallel. The out-of-order scenario sends your sequence in reverse and checks the final state against sequence.expected.

A few details that matter for using it in CI:

If you leave out the adapter, expect.once or sequence, those checks are reported as "not tested", not passed.
Each delivery has a timeout (10 seconds by default), so a handler that hangs fails the scenario instead of hanging the run.
npx webhook-crash-lab init writes a config file, and npx webhook-crash-lab run exits with code 1 when anything fails, so it can be a CI step.
For GitHub webhooks, set provider: 'github' to sign with X-Hub-Signature-256. GitHub signatures have no timestamp, so the stale-timestamp case is reported as not tested.
Point reset() at a test database. It's going to wipe whatever it touches.

To see a broken handler and a fixed one side by side without writing anything:

npx webhook-crash-lab demo

Letting a coding agent run the loop

A lot of webhook handlers are now written by coding agents, and they tend to pass the single-event test and stop. There's a companion MCP server, webhook-crash-lab-mcp, that gives an agent four tools: run_crash_lab, run_demo, explain_report and list_scenarios. Failures come back as structured data with the failed checks, the reason and a suggested fix, so the agent can fix the handler and run it again.

claude mcp add webhook-crash-lab -- npx -y webhook-crash-lab-mcp

Any MCP client that can start a stdio server works the same way, with npx -y webhook-crash-lab-mcp as the command.

What this kind of testing doesn't cover

It tests the invariants you write down. If your observe() doesn't read the table where the bug is, the run won't see it. It also doesn't test Stripe's own behaviour or your production infrastructure. Two instances behind a load balancer can still race on things that a single test process can't reproduce, such as an in-memory dedupe set, which only works in one process. Use a database constraint or Redis for the real claim, as in the SQL above. The in-memory Set in the example is fine for a test because Node runs the check and the add() without anything in between.

Checklist

Verify the signature on the raw body, before parsing. Use your provider's library where one exists. Stripe's libraries reject timestamps older than 5 minutes by default.
Return 400, not 500, for missing signatures and bad JSON.
Claim the event id atomically, before doing the work.
Store facts about objects instead of applying deltas, or re-fetch the object.
Test for business state, not status codes, and include a concurrent duplicate and a reversed sequence.

Disclosure: I'm Krrish Kumar, and I maintain webhook-crash-lab and webhook-crash-lab-mcp.

BullMQ in production: the retry, stalled-job and Redis connection mistakes that bite

Krrish Kumar — Sun, 25 Oct 2026 09:08:45 +0000

BullMQ is easy to start with. You create a Queue, a Worker and a Redis connection, and jobs flow. Most of the problems show up later, under load, during a deploy or when Redis restarts.

This post covers common BullMQ mistakes, most of them called out in the official docs, why each one happens, and how to fix it. Everything here is checked against the BullMQ source (v6.4.2) and the official going-to-production guide. At the end there's a lint config that catches four of them before you ship.

1. Sharing one ioredis connection with `maxRetriesPerRequest` left at the default

This is the most common one. (In BullMQ 6, ioredis is an optional peer dependency, so install both: npm i bullmq ioredis.)

import { Queue, Worker } from 'bullmq';
import IORedis from 'ioredis';

const connection = new IORedis(process.env.REDIS_URL);
const queue = new Queue('emails', { connection });
const worker = new Worker('emails', processEmail, { connection });

A Worker sits in blocking Redis commands while it waits for jobs. ioredis has a per-command retry limit, maxRetriesPerRequest, which defaults to a finite number. If Redis drops for a few seconds, a blocking command can hit that limit and fail instead of waiting for the reconnect.

BullMQ handles this two different ways depending on what you pass:

If you pass connection options, BullMQ sets maxRetriesPerRequest: null for workers and logs a warning if you set something else.
If you pass an ioredis instance, it can't change the instance's options, so the Worker constructor throws BullMQ: Your redis options maxRetriesPerRequest must be null.

The fix is to give workers their own connection with the right setting:

const workerConnection = new IORedis(process.env.REDIS_URL, { maxRetriesPerRequest: null });
const worker = new Worker('emails', processEmail, { connection: workerConnection });

Producers are the opposite case. An API route that calls queue.add() should fail fast if Redis is down, so the client gets an error instead of a request that hangs. Keep a finite maxRetriesPerRequest on the connection your API uses.

2. No `'error'` listener

const worker = new Worker('emails', processEmail, { connection });
// no worker.on('error', ...)

Worker and Queue are Node.js event emitters. When an emitter emits 'error' and nobody is listening, Node throws, which is how a short Redis blip becomes a crashed process. The going-to-production guide says to attach an error handler to both:

worker.on('error', (err) => logger.error({ err }, 'worker error'));
queue.on('error', (err) => logger.error({ err }, 'queue error'));

This is separate from the 'failed' event, which fires when your processor throws for a specific job.

3. Assuming jobs retry by default

They don't. A job with no attempts option runs once, and if the processor throws, it goes straight to failed. From BullMQ's retry docs: "you should use the attempts setting with a value larger than 1."

Set it once as a queue default, with backoff, so a flaky downstream API doesn't get hammered:

const queue = new Queue('emails', {
  connection,
  defaultJobOptions: {
    attempts: 5,
    backoff: { type: 'exponential', delay: 1000 },
    removeOnComplete: { age: 24 * 3600, count: 1000 },
    removeOnFail: { age: 7 * 24 * 3600, count: 5000 },
  },
});

attempts counts the total number of runs, not the number of retries. For errors that will never succeed on retry (bad input, a deleted user), throw BullMQ's UnrecoverableError so the job fails immediately instead of using up its attempts.

The removeOnComplete and removeOnFail settings are there for a different reason. Without them, every finished job stays in Redis forever.

4. Breaking jobId deduplication

A custom jobId is BullMQ's simplest way to avoid duplicate jobs. If a webhook gets delivered twice, you add the job with the event id, and the second add() is ignored. Two things commonly break it.

First, the id format. BullMQ uses : as its Redis key separator, and in the v6.4 source, Job throws Custom Id cannot contain : for most ids with a colon, and Custom Id cannot be integers for ids that are integer strings like '42':

await queue.add('welcome', data, { jobId: `user:${id}` }); // throws
await queue.add('welcome', data, { jobId: String(id) });   // throws if id is numeric
await queue.add('welcome', data, { jobId: `user-${id}` }); // fine

These throw at runtime, often in a code path your tests don't cover.

Second, deduplication only works while the original job still exists. This combination quietly turns it off:

await queue.add('charge', data, { jobId: `evt-${event.id}`, removeOnComplete: true });

removeOnComplete: true deletes the job as soon as it finishes. When the same event is delivered again five minutes later, there's no job with that id any more, so BullMQ adds a new one and the work runs twice. Keep completed jobs for at least your dedupe window, with removeOnComplete: { age: <seconds> }, or use BullMQ's deduplication: { id, ttl } option, which is built for this.

5. Stalled jobs from a blocked event loop

When a worker picks up a job, it takes a lock in Redis and renews it while the job runs. The defaults in the v6.4 Worker constructor are a 30-second lockDuration and a 30-second stalledInterval. If the lock isn't renewed in time, BullMQ treats the job as stalled and moves it back to waiting so another worker can run it. With the default maxStalledCount of 1, a job that stalls a second time fails with "job stalled more than allowable limit."

There are two common causes:

CPU-bound work in the processor. The stalled jobs guide explains that a processor which keeps the CPU busy blocks the event loop, and the lock can't be renewed. The fix is to break the work up, or use a sandboxed processor (a file path instead of a function) so it runs in a separate process.
Killing workers during deploys. If the process is killed mid-job, the lock expires and the job stalls. Listen for SIGTERM and SIGINT and call worker.close(), which stops fetching new jobs and waits for active ones to finish.

for (const sig of ['SIGINT', 'SIGTERM']) {
  process.once(sig, async () => {
    await worker.close();
    await workerConnection.quit();
    process.exit(0);
  });
}

A stalled job runs again, so processors need to be safe to run twice. That's a good rule for queue workers anyway.

One Redis setting: `maxmemory-policy`

This isn't in your code, but the guide calls it out: set Redis maxmemory-policy to noeviction. With any eviction policy, Redis can delete queue keys when it runs low on memory, and jobs disappear. If you use a managed Redis, check what policy your plan sets.

Catching these with a linter

Four of these mistakes can be found by reading the code, so I wrote them up as ESLint rules in eslint-plugin-bullmq:

Rule	Catches
`worker-connection-max-retries`	an ioredis connection used by a `Worker` or `QueueEvents` without `maxRetriesPerRequest: null` (mistake 1)
`require-error-listener`	a `Worker` or `Queue` with no `'error'` listener (mistake 2)
`valid-job-id`	a literal or template `jobId` containing `:`, or an integer (mistake 4)
`no-jobid-dedupe-with-remove-on-complete`	`jobId` combined with `removeOnComplete: true`, including through `defaultJobOptions` (mistake 4)

Setup with ESLint's flat config:

npm i -D eslint eslint-plugin-bullmq

// eslint.config.js
import bullmq from 'eslint-plugin-bullmq';

export default [bullmq.configs.recommended];

Running it against this file:

// worker.js
import { Queue, Worker } from 'bullmq';
import IORedis from 'ioredis';

const connection = new IORedis(process.env.REDIS_URL);
const queue = new Queue('emails', { connection });
queue.on('error', console.error);
const worker = new Worker('emails', async () => {}, { connection });

export async function signup(id) {
  await queue.add('welcome', { id }, { jobId: `user:${id}`, removeOnComplete: true });
}

gives this (messages shortened):

  5:20  error    This ioredis connection is used by a BullMQ Worker but does not set maxRetriesPerRequest: null ...  bullmq/worker-connection-max-retries
  8:16  error    This Worker has no 'error' listener ...                                                           bullmq/require-error-listener
 11:47  error    BullMQ uses ':' as its Redis key separator, so a custom jobId containing ':' is rejected ...       bullmq/valid-job-id
 11:61  warning  removeOnComplete: true deletes the job as soon as it finishes ...                                 bullmq/no-jobid-dedupe-with-remove-on-complete

Two rules come with suggestions you can apply from the editor, adding maxRetriesPerRequest: null and replacing : with -. For TypeScript files, add it after typescript-eslint's config so ESLint picks up .ts files.

The linter can't catch mistakes 3 and 5, since a missing attempts might be deliberate and a blocked event loop only shows at runtime.

Starting from a template instead

If you're starting a new service, create-bullmq-app scaffolds an API and a separate worker with these fixes already in place. It has separate Redis options for producers (maxRetriesPerRequest: 3) and workers (null), 'error' listeners on every worker, attempts with exponential backoff and bounded removeOnComplete/removeOnFail as queue defaults, and graceful shutdown on SIGINT/SIGTERM. The generated project has eslint-plugin-bullmq turned on, so it starts out passing these rules.

npm create bullmq-app@latest my-service

Checklist

Workers get a connection with maxRetriesPerRequest: null. The API connection stays finite.
Every Worker and Queue has an 'error' listener.
attempts and backoff are set, plus removeOnComplete and removeOnFail with limits.
Custom job ids have no : and aren't plain integers, and completed jobs live at least as long as your dedupe window.
Processors don't block the event loop, and workers close on SIGTERM.
Redis runs with maxmemory-policy noeviction.

Disclosure: I'm Krrish Kumar, and I maintain eslint-plugin-bullmq and create-bullmq-app.

How npm worms steal your credentials, and the order to rotate them in

Krrish Kumar — Thu, 22 Oct 2026 19:31:57 +0000

Almost every npm supply-chain advisory ends with the same instruction: rotate every credential the machine could access. It sounds simple until you try it at 2 AM. Which credentials did the machine have? Where are they stored? And does the order matter?

The order does matter. In at least one recent case, doing the obvious thing first deletes your home directory.

This post walks through what a malicious package can read on a typical developer machine, why worms in particular go after some credentials before others, and a rotation order that holds up. The worked example at the end uses a small open-source CLI, but the order applies whether you use a tool or a checklist on paper.

How a malicious package gets to run

npm packages can declare lifecycle scripts in package.json: preinstall, install and postinstall. When you run npm install, npm runs those scripts with your user's permissions. No prompt, no sandbox.

{
  "name": "innocent-looking-lib",
  "version": "1.2.4",
  "scripts": {
    "preinstall": "node ./setup.js"
  }
}

Whatever is in setup.js can read any file your user can read and make network requests. That is the whole attack. Two real examples from this month show what the payloads do in practice.

On October 8, 2026, tensorlake@0.5.144 was published from a compromised maintainer account. According to the project's own advisory (GHSA-8g63-53c9-65j2), the payload ran from a preinstall hook and harvested "GitHub and npm tokens, cloud provider credentials (AWS, GCP), Kubernetes and Vault tokens, SSH and GPG keys, .env files, browser-saved passwords, cryptocurrency wallets, and AI tool and MCP configuration." It then used the stolen tokens to republish itself into other packages the victim could write to. The version was live for about an hour and forty minutes, and it shipped with valid npm provenance because the attacker published through the project's own release workflow.

A few days earlier, @subql/common@5.8.3 shipped a postinstall hook that decoded and ran an obfuscated payload. The OSV entry MAL-2026-17571 has the analysis.

Neither of these is new as a technique. CISA's September 2025 alert on the Shai-Hulud worm describes the same pattern: steal tokens, then use them to publish infected versions of the victim's own packages.

What a stealer reads first

Most of these payloads don't need anything clever. Developer credentials sit in well-known plaintext files:

File	What's in it
`~/.npmrc`	`//registry.npmjs.org/:_authToken=...`, your publish token
`~/.config/gh/hosts.yml`, `~/.git-credentials`	GitHub tokens, when no OS keyring is in use
`~/.aws/credentials`	AWS access keys, often long-lived
`~/.kube/config`	cluster tokens and client keys
`~/.ssh/id_*`	private keys, sometimes without a passphrase
`.env` files in your projects	database URLs, payment keys, webhook secrets
MCP configs (`~/.cursor/mcp.json`, Claude Desktop config, etc.)	API keys pasted directly into `env` blocks
`process.env` in CI	whatever secrets the job passes to the install step

You can check your own exposure for npm in one line:

grep -E '_authToken|_auth|_password' ~/.npmrc

If that prints a literal token (not ${NPM_TOKEN}), any install script on this machine can read it.

Why the order matters

When a machine is compromised, you're racing two things: the malware that may still be running, and the stolen credentials that are already being used elsewhere. Rotating in the wrong order can let the attacker re-steal what you just rotated, or set off something worse.

Here is the reasoning, step by step.

1. Stop the malware first. If the payload installed something persistent, every credential you rotate from that machine gets stolen again. The tensorlake payload made this concrete. Its advisory says it installs a service named gh-token-monitor "that deletes the user's home directory if the stolen GitHub token is revoked," and tells you to remove it before rotating any GitHub token. So the first step is to disconnect the machine, remove persistence, and do every rotation after this from a different, clean machine.

On Linux the advisory's removal steps are:

systemctl --user disable --now gh-token-monitor.service
rm -rf ~/.config/gh-token-monitor ~/.local/bin/gh-token-monitor.sh

On macOS:

launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/com.user.gh-token-monitor.plist
rm ~/Library/LaunchAgents/com.user.gh-token-monitor.plist

Also check crontab -l, systemd user units, LaunchAgents and .vscode/tasks.json for anything you didn't add.

2. Package registry tokens next. This is what makes a worm a worm. A stolen npm token lets the attacker publish an infected version of every package you maintain, which then runs on every machine that installs it. Every minute that token is live, the infection can spread past you. Revoke it:

npm token list
npm token delete <token-id>

npm's own docs note that revocation can take up to an hour to take effect. Then run npm view <your-package> time for each package you maintain and look for versions you didn't publish. The same logic covers PyPI, RubyGems and crates.io tokens.

3. GitHub. GitHub tokens let the attacker push commits, add workflows and read private repos. They come after npm because they're slightly less direct as a spreading channel, and they come after step 1 because of the hostage-service trick above. Revoke tokens at github.com/settings/tokens and OAuth apps at /settings/applications, then gh auth logout and gh auth login.

4. Cloud and infrastructure. AWS, GCP, Azure, Terraform, Kubernetes. These are often the most damaging credentials, but they don't spread the worm to other people, so they come after the spreading channels. For AWS, create a new key before you delete the old one so you don't break running systems, and check CloudTrail for reads of SSM Parameter Store and Secrets Manager during the exposure window.

5. Deploy platforms and CI. Anything stored in a platform's env store, plus the secrets your CI passes to steps. Remember to redeploy, because most services read environment variables at startup and keep the old values until they restart.

6. AI provider keys, SSH keys, and application secrets. Still important, still need doing, but they don't let the attack spread.

So the order is: stop the code that's running, close the channels the attack uses to spread, then clean up everything else, most damaging first.

Worked example: encoding the order in a tool

I wrote this order into a small CLI called secret-blast-radius, so a scan can turn "rotate everything" into a concrete list. The interesting part is how little code the ordering needs. Each finding the scanners produce has a category, and the categories map to a table in src/index.js:

export const ROTATION = {
  persistence: { order: 0, label: 'Stop the malware first (worm services, malicious packages/extensions)', /* steps, verify, docs */ },
  npm:         { order: 1, label: 'Package registry tokens: npm, PyPI, RubyGems, crates.io, Composer (stops a worm republishing your packages)' },
  github:      { order: 2, label: 'GitHub tokens and sessions' },
  cloud:       { order: 3, label: 'Cloud and IaC credentials (AWS, GCP, Azure, Terraform, Pulumi)' },
  kubernetes:  { order: 4, label: 'Kubernetes and container registries' },
  paas:        { order: 5, label: 'Deploy platforms (Railway, Vercel, Fly, Render, Heroku, Netlify)', redeploy: true },
  env:         { order: 6, label: 'Secrets present in the environment / CI', redeploy: true },
  ai:          { order: 7, label: 'AI provider keys and MCP servers' },
  ssh:         { order: 8, label: 'SSH keys' },
  dotenv:      { order: 9, label: 'Application secrets in .env files', redeploy: true },
  // ...
};

(Trimmed. Each entry also carries steps, a verify check that the old credential is dead, and links to the provider's docs.)

The runbook is then a filter and a sort. It keeps only the categories where the scan actually found something, so you don't get a 12-step plan for a machine that only has an npm token and an SSH key:

export function runbook(result) {
  const cats = [...new Set(result.findings.map((f) => f.category))]
    .filter((c) => ROTATION[c])
    .sort((a, b) => ROTATION[a].order - ROTATION[b].order);
  return cats.map((c) => ({ category: c, ...ROTATION[c], items: result.findings.filter((f) => f.category === c) }));
}

Findings in the normal report are sorted by severity first and rotation order second, so two critical findings show up with the persistence one above the npm token.

If you write your own tooling, one choice I'd copy is that findings meaning "you're already infected" go into the persistence category, not their own. A known-bad version in a lockfile, a worm marker dependency, or a VS Code task that runs a disguised file all land in step 0. If you've installed a malicious version, the first job is cleaning the machine, not rotating keys from it.

The scanner checks the plaintext locations from the table above and prints a short fingerprint (the first 8 hex characters of a SHA-256) instead of the value, so you can tell two tokens apart without the tool ever printing a secret.

To try it:

npx secret-blast-radius            # scan and print findings plus the rotation order
npx secret-blast-radius --runbook  # Markdown checklist with steps and verify checks
npx secret-blast-radius --harden   # read-only audit of settings that would have limited the damage

Making the next one hurt less

Rotation is cleanup. A few settings reduce what the next payload can reach:

Set ignore-scripts=true in ~/.npmrc and allow-list the few packages that need build scripts. pnpm already blocks dependency build scripts by default.
Add a release-age cooldown so freshly published versions aren't installed straight away. pnpm has minimumReleaseAge. The tensorlake version was live for under two hours.
Keep tokens out of plaintext files. gh auth login uses the OS keyring when one is available. For publishing from CI, use npm trusted publishing instead of a long-lived NPM_TOKEN.
Don't pass secrets to the npm install step in CI. Give them only to the steps that need them.

One caveat on provenance: it tells you where a package was built, not whether the code is safe. The tensorlake advisory points out that the malicious version carried valid provenance, because the attacker used the project's own workflow.

Stop the malware, revoke the tokens that let it spread, then rotate the rest from a clean machine.

Disclosure: I'm Krrish Kumar, and I maintain secret-blast-radius.

DEV Community: Krrish Kumar

Saved to Postgres, never queued: the transactional outbox for Node

Why "just retry" doesn't work

The table

The write side

The relay

Mistake 1: a "last seen id" cursor

Mistake 2: removeOnComplete: true

Mistake 3: a colon in the jobId

Cleaning up

When you don't need this

Checklist

Testing webhook handlers against duplicates, races and out-of-order delivery

The delivery cases worth testing

The race in "check then insert"

Out of order: record facts, don't do arithmetic

Testing all of it in one run

Letting a coding agent run the loop

What this kind of testing doesn't cover

Checklist

BullMQ in production: the retry, stalled-job and Redis connection mistakes that bite

1. Sharing one ioredis connection with maxRetriesPerRequest left at the default

2. No 'error' listener

3. Assuming jobs retry by default

4. Breaking jobId deduplication

5. Stalled jobs from a blocked event loop

One Redis setting: maxmemory-policy

Catching these with a linter

Starting from a template instead

Checklist

How npm worms steal your credentials, and the order to rotate them in

How a malicious package gets to run

What a stealer reads first

Why the order matters

Worked example: encoding the order in a tool

Making the next one hurt less

Mistake 2: `removeOnComplete: true`

1. Sharing one ioredis connection with `maxRetriesPerRequest` left at the default

2. No `'error'` listener

One Redis setting: `maxmemory-policy`