DEV Community: DevOps Daily

Base64 Explained: 3 Bytes In, 4 Characters Out, Zero Secrecy

DevOps Daily — Fri, 30 Oct 2026 09:00:00 +0000

Base64 is in more of your config than you think. Kubernetes Secrets store their values in it, a signed JWT in an Authorization header is three base64url strings joined by dots, kubeconfig files carry certificates in it, and CI systems ask you to paste base64 when a file has to fit in a variable. Most of the time it works. When it does not, the bug is usually small, like an extra newline or a value encoded twice, and it takes an afternoon to find because the broken value looks almost the same as the right one.

This post shows how base64 works, why its output is a third bigger, and what it does not protect. Then it walks through the bugs that keep coming back. Every command and output below comes from a real run.

TLDR

Base64 maps every 3 bytes to 4 characters from a 64-character alphabet. Anyone can reverse it with one command. It is an encoding, not encryption.
A Kubernetes Secret's data field is plain base64. kubectl get secret -o yaml plus base64 -d shows the value to anyone with read access.
The output is 33.3% bigger than the input. GNU base64 also adds a newline every 76 characters, which makes it 35.1%. Gzip wins most of it back on the wire.
The common bugs: a trailing newline from echo, wrapped output pasted into YAML or an env var, double encoding with --from-literal or stringData, and the URL-safe alphabet without padding in JWTs.

Prerequisites

A Linux shell with GNU coreutils for base64 and basenc. We used coreutils 9.1 on Debian 12.
openssl and python3 for the JWT and comparison examples.
kubectl for the Secret examples. Every kubectl command here uses --dry-run=client, so you do not need a cluster. We used kubectl v1.37.1.

How base64 turns 3 bytes into 4 characters

Base64 uses 64 symbols, and 64 is 2 to the power of 6, so each output character carries exactly 6 bits. Three bytes are 24 bits, and 24 bits split cleanly into four 6-bit groups. That is the whole algorithm: take 3 bytes, cut them into four 6-bit numbers, and look each number up in the alphabet.

This script prints each step for the string k8s:

import base64

data = b"k8s"
bits = "".join(f"{b:08b}" for b in data)
groups = [bits[i:i + 6] for i in range(0, len(bits), 6)]

print("bytes ", " ".join(f"{b:08b}" for b in data))
print("6 bits ", " ".join(groups))
print("values ", " ".join(str(int(g, 2)) for g in groups))
print("base64 ", base64.b64encode(data).decode())


bytes 01101011 00111000 01110011
6 bits 011010 110011 100001 110011
values 26 51 33 51
base64 azhz

The standard alphabet from RFC 4648 is:

 0-25 A B C ... Z
26-51 a b c ... z
52-61 0 1 2 ... 9
62 +
63 /

So 26 is a, 51 is z and 33 is h, and k8s becomes azhz. Decoding runs the same steps backwards. There is no key and nothing to guess, which is why base64 is not a way to hide anything.

Padding: what the = signs mean

When the input length is not a multiple of 3, the last group is short. Base64 fills the missing bits with zeros and adds = so the output length stays a multiple of 4:

printf 'k' | base64
printf 'k8' | base64
printf 'k8s' | base64


aw==
azg=
azhz

One leftover byte gives 2 characters plus ==. Two leftover bytes give 3 characters plus =. Padding carries no data, which is why some formats, JWT among them, drop it. That saves a few bytes and causes a bug we will get to below.

Because the mapping is fixed, common inputs have recognizable prefixes. JSON that starts with {" followed by a letter encodes to a string starting with eyJ, so eyJ... is almost always JSON, often a JWT. Base64-encoded PEM text (-----BEGIN ...) starts with LS0tLS1CRUdJTi.

The 33% overhead, measured

Every 3 input bytes become 4 output characters, so the encoded size is 4 × ceil(n / 3). For large inputs that is 4/3 of the original, an overhead of 33.3%. GNU base64 also wraps its output at 76 characters by default, which adds one newline for every 76 characters.

We encoded 1 MiB of random data to check the math:

head -c 1048576 /dev/urandom > blob.bin
wc -c < blob.bin
base64 blob.bin | wc -c # default: wrapped at 76 characters
base64 -w0 blob.bin | wc -c # -w0: no wrapping
base64 blob.bin | wc -l


1048576
1416501
1398104
18397

The formula gives the same numbers: 4 × ceil(1048576 / 3) is 1,398,104 characters, and ceil(1398104 / 76) is 18,397 lines, each ending in a newline, for 1,416,501 bytes in total. That is 1.3333 times the input without wrapping and 1.3509 times with it.

1 MiB of random data, before and after base64

	Value	Series
raw	1048576 bytes	raw
base64 -w0	1398104 bytes	base64
base64 (wrapped)	1416501 bytes	base64
base64 -w0	gzip -9	1058004 bytes

Measured with wc -c on Debian 12, GNU coreutils 9.1 and gzip -9. The base64 overhead is 33.3% (35.1% with line wrapping); after gzip it is 0.9% on the wire.

On the wire, compression gets most of it back. Base64 only uses 6 of every 8 bits, and gzip notices:

gzip -9 -c blob.bin | wc -c
base64 -w0 blob.bin | gzip -9 | wc -c


1048763
1058004

For this sample, gzip cut the overhead from 33.3% to 0.9%. The overhead still counts wherever the encoded size is what matters: memory, environment variables, cookies and request headers. A JWT is sent in a header on every call, and header size limits apply to its encoded form: each claim costs 4/3 of its size, and nginx's default large_client_header_buffers allows 8 KB per header line.

Kubernetes Secrets are base64, not encryption

A Secret's data field holds base64 strings. Create one locally to see what the API server would store:

kubectl create secret generic db-creds \
  --from-literal=username=app \
  --from-literal=password='hunter2-prod' \
  --dry-run=client -o yaml


apiVersion: v1
data:
  password: aHVudGVyMi1wcm9k
  username: YXBw
kind: Secret
metadata:
  name: db-creds

Getting the value back takes one pipe. On a cluster you would use kubectl get secret db-creds -o jsonpath='{.data.password}'; here we read the same field from the dry run:

kubectl create secret generic db-creds --from-literal=password='hunter2-prod' \
  --dry-run=client -o jsonpath='{.data.password}' | base64 -d; echo


hunter2-prod

Base64 is there so binary data such as keystores and TLS keys fits inside JSON and YAML. It does nothing for secrecy.

Warning

Anyone who can get or list Secrets in a namespace can read every value in it, and so can anyone who can create a pod there that mounts them. A Secret manifest committed to Git is the password in plain sight. What protects Secrets is RBAC, encryption at rest for etcd, and keeping the values out of Git with a tool such as Sealed Secrets, SOPS or an external secret store.

For a longer walkthrough, see How to Decode a Kubernetes Secret and the ConfigMaps and Secrets part of our Kubernetes guide.

Bug 1: the newline from echo

echo adds a newline, and base64 encodes it like any other byte:

echo 'hunter2-prod' | base64
printf '%s' 'hunter2-prod' | base64


aHVudGVyMi1wcm9kCg==
aHVudGVyMi1wcm9k

The first value ends in Cg==, which is the encoded \n. Decode it and look at the bytes:

echo 'aHVudGVyMi1wcm9kCg==' | base64 -d | od -c


0000000 h u n t e r 2 - p r o d \n
0000015

Put that value in a Secret and the password is now hunter2-prod plus a newline. The app sends 13 bytes, the database expects 12, and you get an authentication error for a password that looks right in every log line.

The same thing happens with files. A file written with echo ends in a newline, and --from-file keeps it:

echo 'hunter2-prod' > password.txt
kubectl create secret generic db-creds --from-file=password=password.txt \
  --dry-run=client -o jsonpath='{.data.password}'; echo

printf '%s' 'hunter2-prod' > password.txt
kubectl create secret generic db-creds --from-file=password=password.txt \
  --dry-run=client -o jsonpath='{.data.password}'; echo


aHVudGVyMi1wcm9kCg==
aHVudGVyMi1wcm9k

Use printf '%s' whenever the exact bytes matter, and pipe a suspect value through base64 -d | od -c to see the last byte.

Bug 2: wrapped output in YAML and env vars

GNU base64 breaks its output into 76-character lines. A database URL is long enough to hit that:

DB_URL='postgresql://app:hunter2-prod@db.internal:5432/orders?sslmode=require'
printf '%s' "$DB_URL" | wc -c
printf '%s' "$DB_URL" | base64


69
cG9zdGdyZXNxbDovL2FwcDpodW50ZXIyLXByb2RAZGIuaW50ZXJuYWw6NTQzMi9vcmRlcnM/c3Ns
bW9kZT1yZXF1aXJl

Now put it into a manifest the way many scripts do, with a heredoc:

cat <<EOF > secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: db-url
data:
  url: $(printf '%s' "$DB_URL" | base64)
EOF
kubectl create --dry-run=client -f secret.yaml -o yaml


error: error parsing secret.yaml: error converting YAML to JSON: yaml: line 8: could not find expected ':'

The second line of the value landed at column 0, and YAML read it as a new key. In an env var or a CI variable the newline does not fail this loudly: it ends up inside the value. Turn wrapping off with -w0:

cat <<EOF > secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: db-url
data:
  url: $(printf '%s' "$DB_URL" | base64 -w0)
EOF
sed -n 's/^ url: //p' secret.yaml | base64 -d


postgresql://app:hunter2-prod@db.internal:5432/orders?sslmode=require

Even simpler: let kubectl create secret generic db-url --from-literal=url="$DB_URL" --dry-run=client -o yaml write the manifest, and the encoding is never your problem.

macOS and openssl

macOS ships a different base64. According to its man page, it does not wrap by default (-b sets a line length, and -w is accepted for GNU compatibility). Older macOS releases decode only with -D, and on them -d is a debug flag. A script that has to run on both should encode with a pipe that works everywhere:

printf '%s' "$DB_URL" | base64 | tr -d '\n'


cG9zdGdyZXNxbDovL2FwcDpodW50ZXIyLXByb2RAZGIuaW50ZXJuYWw6NTQzMi9vcmRlcnM/c3NsbW9kZT1yZXF1aXJl

openssl base64 is another option, but it wraps at 64 characters and has a sharper edge when decoding:

printf '%s' "$DB_URL" | openssl base64
printf '%s' "$DB_URL" | base64 -w0 | openssl base64 -d | wc -c; echo "rc=${PIPESTATUS[2]}"
printf '%s' "$DB_URL" | base64 -w0 | openssl base64 -d -A; echo


cG9zdGdyZXNxbDovL2FwcDpodW50ZXIyLXByb2RAZGIuaW50ZXJuYWw6NTQzMi9v
cmRlcnM/c3NsbW9kZT1yZXF1aXJl
0
rc=0
postgresql://app:hunter2-prod@db.internal:5432/orders?sslmode=require

Without -A, OpenSSL 3.0 decoded our 92-character single line to zero bytes and exit code 0. No error, just an empty value. If you use openssl base64 for single-line data, pass -A both ways.

Bug 3: double encoding

--from-literal, --from-file and stringData all take plain text and encode it for you. Give them a value that is already encoded and it gets encoded again:

kubectl create secret generic db-creds \
  --from-literal=password="$(printf '%s' 'hunter2-prod' | base64)" \
  --dry-run=client -o jsonpath='{.data.password}'; echo

kubectl create secret generic db-creds \
  --from-literal=password="$(printf '%s' 'hunter2-prod' | base64)" \
  --dry-run=client -o jsonpath='{.data.password}' | base64 -d; echo


YUhWdWRHVnlNaTF3Y205aw==
aHVudGVyMi1wcm9k

The pod now gets aHVudGVyMi1wcm9k as its password. The giveaway is a first decode that looks like base64 instead of the value you stored. Decode it again to confirm:

echo 'YUhWdWRHVnlNaTF3Y205aw==' | base64 -d | base64 -d; echo


hunter2-prod

The rule: data takes base64, and stringData, --from-literal and --from-file take the raw value. In a Helm chart, b64enc belongs in data:, never in stringData:.

Not every double encoding is a bug. PEM is itself base64 with header lines, so a TLS Secret's tls.crt field is encoded twice on purpose:

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -keyout tls.key -out tls.crt -days 30 -subj '/CN=orders.internal' 2>/dev/null
kubectl create secret tls orders-tls --cert=tls.crt --key=tls.key \
  --dry-run=client -o jsonpath='{.data.tls\.crt}' | cut -c1-40
kubectl create secret tls orders-tls --cert=tls.crt --key=tls.key \
  --dry-run=client -o jsonpath='{.data.tls\.crt}' | base64 -d | head -2


LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1J
-----BEGIN CERTIFICATE-----
MIIBijCCAS+gAwIBAgIUazfeROKqRj7HLheO41+kH88CZagwCgYIKoZIzj0EAwIw

That LS0tLS1CRUdJTi prefix is the -----BEGIN from earlier. The certificate-authority-data field in a kubeconfig works the same way.

base64url and JWTs

Two characters of the standard alphabet cause trouble outside of files. + means a space in a form-encoded query string, and / is a path separator. RFC 4648 defines a second alphabet, base64url , that uses - and _ instead:

printf '\xfb\xff\xbf' | base64
printf '\xfb\xff\xbf' | basenc --base64url


+/+/
-_-_

This is not only about odd bytes. The database URL from earlier encodes to ...cnM/c3Ns... in standard base64 and ...cnM_c3Ns... in base64url. In a URL path, that / splits the value in two.

JWTs use base64url and also drop the = padding (RFC 7515). This script, saved as make-jwt.sh, builds a real HS256 token with shell tools:

#!/usr/bin/env bash
set -euo pipefail

b64url() { basenc --base64url -w0 | tr -d '='; }

header=$(printf '%s' '{"alg":"HS256","typ":"JWT"}' | b64url)
payload=$(printf '%s' '{"sub":"deploy-bot","role":"admin","exp":1793318400}' | b64url)
signature=$(printf '%s' "$header.$payload" \
  | openssl dgst -sha256 -hmac 'not-a-real-key' -binary | b64url)

echo "$header.$payload.$signature"


eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJkZXBsb3ktYm90Iiwicm9sZSI6ImFkbWluIiwiZXhwIjoxNzkzMzE4NDAwfQ.mirpAmeTzGcS59Fo6JImIz4IBPSFSqpDJy1qltzzzuA

Both JSON parts start with eyJ. Now try to read the payload the obvious way:

chmod +x make-jwt.sh
TOKEN=$(./make-jwt.sh)
echo "$TOKEN" | cut -d. -f2 | base64 -d; echo " rc=$?"


{"sub":"deploy-bot","role":"admin","exp":1793318400}base64: invalid input
 rc=1

The payload part is 70 characters, not a multiple of 4, so GNU base64 prints what it could decode and then exits with 1. The JSON looks complete, which makes this easy to miss: under set -e the script stops here, and without it the script carries on with output that may be cut short. basenc --base64url -d fails the same way, because it also expects padding. Translate the alphabet and put the padding back first:

b64url_decode() {
  local s
  s=$(tr '_-' '/+') # back to the standard alphabet
  case $(( ${#s} % 4 )) in # put the stripped padding back
    2) s="$s==" ;;
    3) s="$s=" ;;
  esac
  printf '%s' "$s" | base64 -d
}

echo "$TOKEN" | cut -d. -f2 | b64url_decode; echo
echo "$TOKEN" | cut -d. -f1 | b64url_decode; echo


{"sub":"deploy-bot","role":"admin","exp":1793318400}
{"alg":"HS256","typ":"JWT"}

In Python, base64.urlsafe_b64decode(part + "=" * (-len(part) % 4)) does the same in one line.

Anyone holding a signed JWT like this one can read its claims, because they are only base64url. The signature stops people from changing them, not from reading them, so never put a secret in a JWT payload. (Encrypted JWTs, JWE, are a different format with five parts.) For a quick look at a token, our JWT decoder runs in the browser, and the base64 tool does the same for plain values.

Data URIs

A data URI puts a whole file inside a URL: data:<media type>;base64,<data>. Browsers then skip the extra request. We built one from this site's 180x180 Apple touch icon, saved as icon.png:

wc -c < icon.png
printf 'data:image/png;base64,%s' "$(base64 -w0 icon.png)" | wc -c
printf 'data:image/png;base64,%s' "$(base64 -w0 icon.png)" | cut -c1-60


1167
1578
data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAALQAAAC0CAYAAA

1,167 bytes became 1,578 characters: the 33% plus the 22-character prefix. Use -w0 again so the URI stays on one line in HTML or CSS. The image also loses its own cache entry and is downloaded again with every page or stylesheet that inlines it. That is fine for a few small icons and bad for anything large or shared across pages.

For SVG, which is already text, base64 is not always the best choice. Our 110-byte SVG made a 174-character base64 data URI and a 143-character URL-encoded one (data:image/svg+xml,%3Csvg ..., with single quotes inside the SVG).

Summary

Base64 is a fixed mapping from bytes to 64 safe characters. It makes binary data fit in text formats, costs a third more space, and hides nothing. The bugs come from a short list of mistakes:

Base64 does not protect Secrets. RBAC, encryption at rest and keeping values out of Git do.
echo adds a newline. Use printf '%s' and check suspect values with base64 -d | od -c.
GNU base64 wraps at 76 characters. Use -w0, or | tr -d '\n' where macOS is involved. With openssl base64, pass -A both ways.
data takes base64; stringData, --from-literal and --from-file take raw values. If the first decode still looks like base64, check for double encoding.
JWTs use base64url without padding. Swap -_ back to +/ and restore the = before decoding.
Check the exit code of base64 -d. GNU prints partial output before it fails, and the Rust coreutils in Ubuntu 26.10 behave differently on bad input.

Originally published at devops-daily.com.

15 Minutes to 5: What Happens When You Move iOS CI From M1 to M4

DevOps Daily — Thu, 29 Oct 2026 09:00:00 +0000

The Wikipedia iOS app's build and its 503 unit tests took a median 922 seconds (15.4 minutes) on GitHub's standard macos-26 runner, an Apple M1 with 3 vCPU. The same job, with the same Xcode, simulator and xcodebuild commands, took a median 337 seconds (5.6 minutes) on a Bitrise Build Hub M4 Large with 5 vCPU. That is 2.7 times faster, over 17 runs on each side, every run cold.

In our first article this was one row in a pipeline benchmark. Here we look only at the iOS job: which phase gets faster, what a run costs at list price, whether a bigger Mac helps, and what you change in a workflow to move it.

The-DevOps-Daily/gha-runner-benchmark on GitHub

TLDR

The build gained the most minutes: 538 s on the M1 against 145 s on the M4 Large (3.7x). The unit tests went from 370 s to 86 s (4.3x).
Cost per run at list price: a median $0.99 on GitHub (16 billed minutes at $0.062) and $0.21 on the M4 Large (7 billed minutes at $0.03).
A bigger M4 adds little: the 10 vCPU M4 XLarge was 9% faster than the M4 Large and costs twice as much per minute. The 14 vCPU M4 Pro XLarge took 23% less time, and it is on the Enterprise plan only.
Machines start on demand: iOS jobs waited a median 38 seconds for a Build Hub machine, against 8 seconds on GitHub.
The workflow change is the runs-on line: it points at a Build Hub pool. The setup around it is a GitHub App, a pool and an image with your Xcode. GitHub Actions still schedules the job and keeps the logs and secrets.

Prerequisites

An iOS project that already builds in GitHub Actions with xcodebuild
A way to read job times: the Actions tab, or the gh CLI for many runs
For Build Hub: a Bitrise workspace and the Build Hub GitHub App installed on the repository

The setup

Side	Runner	Machine	List price
GitHub-hosted	`macos-26`	Apple M1 (virtual), 3 vCPU, 7 GB	$0.062 a minute
Build Hub, self-serve Pro Tier 1	M4 Large	Apple M4 (virtual), 5 vCPU, 14 GB	$0.03 a minute

These are not same-size machines: the M4 Large has more cores and twice the memory. We compared the standard macOS runner on GitHub with the smallest Mac on Build Hub: the default choice on each side.

The workload. The Wikipedia iOS app at commit ca8aaeb9, its own "Run Unit Tests" job: resolve Swift packages, xcodebuild build-for-testing, then test-without-building on an iPhone 17 simulator with iOS 26.5 and Xcode 26.5. In the separate runs below, all 503 tests passed every time, and in the pipeline runs every iOS job succeeded. Every run started cold, with no DerivedData or package cache on either side. Cold runs compare the whole environment, not only the chip: the images, macOS patch versions, storage and network also differ.

How we ran it. Bitrise provided the Build Hub workspace for these tests. The 17 runs per side come from the pipeline benchmark on 16 and 17 October 2026, dispatched in pairs. The phase split and the bigger Macs come from separate runs of the same job on 15 and 16 October: 7 on GitHub and 3 on each Build Hub machine. Those are fewer samples, so compare their rows with each other, not with the 17-run numbers. All runs and the raw results are in the repository above.

Where the minutes go

Wikipedia iOS job by phase, median seconds

	Value	Series
Resolve packages	64s	GitHub M1 3 vCPU
Resolve packages	27s	Build Hub M4 Large
Build for testing	538s	GitHub M1 3 vCPU
Build for testing	145s	Build Hub M4 Large
Unit tests	370s	GitHub M1 3 vCPU
Unit tests	86s	Build Hub M4 Large

Separate runs of the iOS job on 15 and 16 October 2026: GitHub macos-26 7 runs, Build Hub M4 Large 3 runs. Cold, no cache. Source: results/ios-wikipedia-unit-tests.json.

Phase	GitHub M1, 3 vCPU	Build Hub M4 Large, 5 vCPU	Faster by
Resolve packages	64 s	27 s	2.4x
Build for testing	538 s	145 s	3.7x
Unit tests	370 s	86 s	4.3x

The build is where most of the minutes are, and it is where most of them were saved: 393 seconds per run. Compiling Swift is CPU work, and the M4 Large has a newer chip and two more cores. The unit tests gained the most in relative terms. They run in the iOS simulator, which needs CPU and memory too, and the M1 runner has 7 GB for Xcode, the simulator and the test host together. We did not measure memory pressure, so that is a possible factor, not a finding.

Package resolution gained least. It downloads the same packages on both sides, and we did not time the network part separately. The phases do not add up to the whole job: checkout, Xcode selection and the other steps take the rest, a median 87 seconds per run on GitHub and 69 seconds on the M4 Large.

What a run costs

At list price, each job's billed time is rounded up to a whole minute and multiplied by the per-minute price. Build Hub bills per started minute and includes the machine's start-up, which the job's timestamps do not show, so we estimated it from the wait before each job, as in the first article. That is why the M4 Large bills 7 minutes for a 337 second job. Over the 17 pipeline runs per side:

Side	Median job time	Billed minutes	Cost per run
GitHub `macos-26`	922 s	16	$0.99
Build Hub M4 Large	337 s	7	$0.21

Two things to keep in mind:

On GitHub, public repositories pay nothing for standard runners. In a private repository, standard macOS minutes come out of your plan's included minutes first; the $0.99 is the price after those are used.
Build Hub is sold as a monthly tier. If you do not use the budget you pay for, the real price per run is higher than $0.21.

A team that runs the iOS job 20 times a day would, at these list prices, spend about $20 a day on GitHub and about $4 on Build Hub, and use about 3 hours less job time. That is total job time, not the time any one person waits: jobs run in parallel, and other jobs may be slower. Check your own run count before you trust that number: rounding makes short jobs cost more than they look.

Is a bigger Mac worth it?

We ran the same job on the larger Build Hub Macs too.

Machine	Plan	Runs	Median job time	Cost per run
GitHub `macos-26`: M1, 3 vCPU, 7 GB	GitHub	7	1,026 s (805 to 1,305)	$1.12 (18 billed minutes)
Build Hub M4 Large: M4, 5 vCPU, 14 GB	Self-serve	3	323 s (314 to 335)	$0.18 (6 billed minutes)
Build Hub M4 XLarge: M4, 10 vCPU, 28 GB	Self-serve	3	295 s (293 to 301)	$0.36 (6 billed minutes)
Build Hub M4 Pro XLarge: M4 Pro, 14 vCPU	Enterprise	3	250 s (246 to 251)	No per-minute list price

Twice the cores and twice the memory, from the M4 Large to the M4 XLarge, saved 28 seconds (9%) and doubled the cost per run. For this app, the Large gave most of the gain at half the XLarge's cost per run. The M4 Pro XLarge, with the M4 Pro chip and 14 vCPU, took 73 seconds less than the Large (23%). It is on the Enterprise plan, which Bitrise sells by concurrency and not by minutes, so we cannot give a list price per run for it. Billed minutes for the Build Hub rows include the estimated machine start-up, as above.

Your app may be different. A project with many independent modules can use more cores than the Wikipedia app did. Run your own job three times on each size before you pick one.

Moving one job

GitHub Actions still runs the workflow: it schedules the job, streams the logs and passes the secrets. The work is in the setup, and the workflow itself changes little.

Install the Build Hub GitHub App on the repository, and create a pool in your Bitrise workspace with the machine type and an image that has your Xcode version.
Point runs-on at the pool's label. Ours was bitrise-runner-ios-m4-large.
Select the Xcode version in the job, so that both sides use the same one.

Images name the Xcode app differently (Xcode_26.5.app, Xcode-26.5.app or Xcode.app), so our job picks the app by the version it reports, and fails if it is not there:

jobs:
  ios-tests:
    runs-on: [bitrise-runner-ios-m4-large] # was: macos-26
    env:
      XCODE_VERSION: '26.5'
    steps:
      - name: Select Xcode
        run: |
          for app in /Applications/Xcode*.app; do
            if ["$(DEVELOPER_DIR="$app/Contents/Developer" xcodebuild -version 2>/dev/null | head -1)" = "Xcode $XCODE_VERSION"]; then
              sudo xcode-select -s "$app/Contents/Developer"
              break
            fi
          done
          test "$(xcodebuild -version | head -1)" = "Xcode $XCODE_VERSION"

The xcodebuild steps did not change at all. Two checks are worth adding before the first real run:

The simulator exists. xcrun simctl list devices available should list the device and the iOS version in your -destination. If the image lacks the runtime, the job fails, so check it before the first real run.
Expect a short wait for a machine. A self-serve pool starts a machine when a job arrives. Our iOS jobs waited a median 38 seconds before they started, which is not in the job times above.

What we could not conclude

Chip or cores. The M1 runner has 3 vCPU and the M4 Large has 5. We did not run a 3 vCPU M4, so we cannot say how much of the 2.7x comes from the chip and how much from the cores and the memory. Going from 5 to 10 M4 cores, with twice the memory, saved only 9%, so beyond 5 cores the size did little for this app.
Warm builds. Every run was cold. With a DerivedData or package cache, both sides get faster, and the gap may shrink. We did not measure that.
Other apps. This is one app at one commit. A smaller app saves fewer minutes, and a larger one may save more.
GitHub's larger Mac runners. GitHub also sells larger macOS runners on paid plans. We compared the standard one, which every team gets.

Summary

Moving the Wikipedia iOS job from GitHub's M1 runner to a Build Hub M4 Large took it from 15.4 to 5.6 minutes and from $0.99 to $0.21 per run at list price, with the same workflow steps. Most of the time came out of the build and the simulator tests. A bigger M4 added little; the M4 Pro XLarge took 23% less time than the M4 Large. If your iOS job is the slowest one in your pipeline, measure it on one larger machine before you change anything else, and keep the Xcode version pinned so the comparison stays fair.

The workflows, the dispatch scripts and every result are in the benchmark repository. The setup for Build Hub is in Bitrise's documentation.

Originally published at devops-daily.com.

GitLab.com's 60-Requests-an-Hour Limit Is Live, and the Headers Won't Warn You

DevOps Daily — Thu, 29 Oct 2026 09:00:00 +0000

GitLab.com now limits requests without credentials to 60 per hour per IP address, at least for part of its traffic. On the evening of October 28 we sent 140 anonymous API requests to GitLab's own gitlab-runner project in under ten minutes, and every one of them came back 200. Then we pointed the same loop at a public project in a personal namespace. After 58 requests, the 59th came back 429 Too Many Requests, and our IP stayed blocked for more than 40 minutes.

Three things from that evening matter if you run CI or automation against GitLab.com:

What counts depends on what you ask for. While we were blocked, requests to projects in the gitlab-org and inkscape groups still worked, and requests to personal projects and to /api/v4/version did not. If you test against a big open source project, you may conclude that nothing has changed.
Successful responses do not show the new limit. The last 200 before the block carried RateLimit-Remaining: 476. That number belongs to the old 500-per-minute throttle. The new limit only appears in the 429 itself.
Retry-After follows the clock, the block does not. Every Retry-After pointed to the end of the current clock hour. The block ended about an hour after the first request that counted. Our first explanation for that was wrong, and a test with predictions written down in advance showed it.

Everything below comes from one IP in one overnight session, October 28 to 29 UTC, with the scripts and raw results in a public repository, so you can run the same checks from your own network.

TL;DR

GitLab announced the Free and unauthenticated limits for October 19, 2026, and the limits for Premium and Ultimate for January 2027. Anonymous traffic gets 60 requests per hour per IP.
From our IP, the budget was exactly 60 requests. The count started with the first request and reset about an hour later. Requests to the gitlab-org group did not count, 401 responses did, and blocked requests did not extend the block. The 429 carried RateLimit-Name: unauthenticated_traffic_per_ip.
Nothing in a successful response warns you. Track your own request counts, and alert on 429 responses with that RateLimit-Name.
Do not plan around Retry-After from this throttle. It points to the end of the clock hour, and the block ends about an hour after the first request of the current count.
A separate limit, most likely at Cloudflare, stopped bursts of 401 responses after 25 requests, with a 429 that looks nothing like GitLab's.
Behind one NAT address, every runner shares those 60 requests. Find anonymous curl, wget and Dockerfile ADD calls to gitlab.com and give them a token. CI_JOB_TOKEN covers raw files, releases, packages, artifacts and archives.

Prerequisites

A GitLab.com account or group, and CI jobs or scripts that talk to gitlab.com
bash, curl and Python 3.9 or later to run the scripts in the companion repository
Access to your runners' egress IP addresses, if you want to reason about shared limits

What GitLab announced

GitLab's announcement ties GitLab.com rate limits to the subscription plan. The numbers are on the rate limits page, which still called them proposed when we read it on October 28:

Caller	Sustained limit	Burst limit (API)	Starts
No credentials, per IP	60 per hour	none listed	October 19, 2026
Free, per signed-in user	5,000 per hour	100 per minute	October 19, 2026
Premium, per signed-in user	15,000 per hour	1,250 per minute	January 2027
Ultimate, per signed-in user	25,000 per hour	2,000 per minute	January 2027

Free also has a separate 200-per-minute limit for web traffic. Personal access tokens, OAuth tokens and CI/CD job tokens all move a request from the anonymous limit to your plan's limit. Git over HTTPS is not counted against the anonymous limit. GitLab Self-Managed and GitLab Dedicated are not affected. The old per-minute limits stay in place, and GitLab applies the lower of the two.

The hourly limit matters more than the burst limit. At full burst speed, you reach the hourly cap long before the hour is over:

Minutes at full burst speed until the hourly limit

	Value
Free (5,000/h, 100/min)	50 min
Premium (15,000/h, 1,250/min)	12 min
Ultimate (25,000/h, 2,000/min)	12.5 min

Sustained limit divided by burst limit, from GitLab's rate limits page, retrieved 2026-10-28. Per signed-in user, API traffic.

How we measured

The companion repository holds the scripts and every result file:

The-DevOps-Daily/gitlab-rate-limit-check on GitHub

probe.sh sends requests with a one-second pause between them and writes the status code, the cache status and the rate-limit headers of each response to a TSV file stamped with the script's hash. All requests came from one residential IP address over IPv4, with no token, on October 28 between 22:38 and 23:38 UTC and on October 29 between 01:00 and 04:03 UTC.

probe.sh against a project in a personal namespace

$ PROJECT=dwt1%2Fdotfiles ./probe.sh api 70
results/api-anon-20261028T225259Z.tsv
     58 200 BYPASS  throttle_unauthenticated_api    500
     12 429 BYPASS  unauthenticated_traffic_per_ip
$ sed -n '58,61p' results/api-anon-20261028T225259Z.tsv
56  22:54:21    200 BYPASS  throttle_unauthenticated_api    500 478 
57  22:54:23    200 BYPASS  throttle_unauthenticated_api    500 477 
58  22:54:24    200 BYPASS  throttle_unauthenticated_api    500 476 
59  22:54:26    429 BYPASS  unauthenticated_traffic_per_ip 334

The columns are request number, UTC time, status, Cloudflare cache status, RateLimit-Name, RateLimit-Limit, RateLimit-Remaining and Retry-After.

Finding 1: the limit depends on the project you request

Before that run we had already sent 140 anonymous API requests to gitlab-org/gitlab-runner and 70 cache-busted raw-file requests to the same project, in just over ten minutes. All 210 returned 200, and the only throttle in the headers was the old throttle_unauthenticated_api or throttle_unauthenticated_web, at 500 requests per minute.

The run against dwt1/dotfiles, a popular public project in a personal namespace, was blocked after 58 requests. That hour we had also made two requests to endpoints outside any project, /api/v4/version and a /api/v4/projects search. If those count, the block came after exactly 60 requests, which also suggests nothing else on that address was sending anonymous requests to GitLab.com. While the IP was blocked, we sent one request to each of six targets:

Target	Status while blocked
`gitlab-org/gitlab-runner`	200
`gitlab-org/gitlab`	200
`inkscape/inkscape`	200
`dwt1/dotfiles` (personal namespace)	429
`CalcProgrammer1/OpenRGB` (personal namespace)	429
`/api/v4/version` (no project)	429

Our reading is that the new anonymous limit depends on the namespace you request, most likely on its plan: gitlab-org is GitLab's own group, Inkscape is a large open source group, and requests to both went through. We cannot see a namespace's plan without credentials, so this is an inference from the pattern, not a fact we could check.

To test that reading, we waited until the IP had been quiet for more than an hour and ran count-test.sh at 01:00:45 UTC on October 29. It sends 10 requests to gitlab-org/gitlab-runner, then 10 to /api/v4/version, then requests to dwt1/dotfiles until the first 429. If the gitlab-org requests do not count and the others do, the last step gets 50 successes before the block. It got exactly 50:

count-test.sh from a clean start

$ ./count-test.sh
results/count-test-20261029T010045Z.tsv
10 paid 200
50 personal 200
1 personal 429
10 unscoped 401
personal-namespace successes before the first 429: 50 (prediction: 50)

paid is the script's label for the gitlab-org step, after our guess about why it is not counted. The same total would come out if the gitlab-org requests counted and the /api/v4/version requests did not, but the first evening rules that out: more than 200 gitlab-org requests in the same hour as the 58 personal ones did not trigger a block. The /api/v4/version requests all returned 401 Unauthorized, so a request that fails still uses up the budget.

So if you want to know whether your automation will be throttled, test it against the projects it really uses. A quick check against a large open source project tells you nothing.

Finding 2: successful responses do not show the new limit

GitLab's advice is to watch RateLimit-Remaining and slow down as it drops. For this limit, that does not work. Every successful response in our runs described only the old per-minute throttle. Request 58 got RateLimit-Remaining: 476, and request 59 got a 429.

The 429 was the first response that mentioned the new limit. It carried RateLimit-Name: unauthenticated_traffic_per_ip and a Retry-After, but no RateLimit-Limit or RateLimit-Remaining.

So you have two options. Count your own anonymous requests, or alert on the first 429 with that RateLimit-Name. The repository's glcurl wrapper logs one JSON line per request without the token, and budget.py reads those logs and reports each caller's busiest minute and busiest hour against the plan limits.

Finding 3: the count resets an hour after it starts, not at the top of the hour

The first 429, at 22:54:26 UTC, said Retry-After: 334. That points to 23:00:00, the top of the hour. We waited until 23:00:45 and tried again. Requests to gitlab-org still worked, but the first request that counts was blocked, and this time Retry-After was 3,540 seconds, pointing to midnight. Then we sent one request every three minutes to see when the block really ended:

UTC	Status	`Retry-After`
22:54:26	429 (first)	334
23:01:01	429	3,540
23:02:07	429	3,473
23:20:10	429	2,390
23:35:12	429	1,488
23:38:12	not limited	none

Every Retry-After was the time until the end of the current clock hour, but the block ended between 23:35 and 23:38. The first request that counted that evening was a one-off /api/v4/version call. It has no result file, but our terminal log puts it between 22:38:02 and 22:38:03, and an hour after that falls inside the three minutes in which the block ended.

Our first explanation was a sliding window: count the requests of the last 3,600 seconds, and let the caller back in as the oldest one drops out. We tested it a few hours later. The count test had blocked the IP at 01:02:29, and its first counted request was at 01:01:01. predict-window.py worked out what three counting models expect, and we committed its predictions before window-test.sh ran:

Request	Clock hour	Sliding 60 minutes	Weighted estimate	GitLab
One at 01:50:00	429	429	429	429, `Retry-After: 600`
One at 02:00:30	allowed	429	allowed	429, `Retry-After: 3570`
Back to back from 02:01:45	59 allowed	35 to 46 allowed	1 allowed	25 allowed, then a different 429

The weighted estimate is a common shortcut in rate limiters: the previous hour's count, weighted by the part of it still inside the last 3,600 seconds, plus the current hour's count. Half a minute into the new clock hour we were still blocked, and Retry-After now pointed to 03:00:00. A minute later, 25 requests went through, and then the burst was stopped by the limit in the next section. The sliding window matched, and we wrote it up that way.

The second review of this post found the hole. A counter that starts with the first request and resets completely 3,600 seconds later, the way a Redis INCR with an EXPIRE on the first increment works, fits the same three rows. The two models only disagree while old requests are leaving a sliding window, and our first test of that was cut off at request 26 by the same limit. So we ran another, with requests that return 200 and do not trip that limit. At 03:04:00 the sliding window predicted 25 more requests, and the resetting counter 35. GitLab allowed 35:

series-test.sh against a personal project, predictions committed at 03:03:46

$ URL='https://gitlab.com/api/v4/projects/dwt1%2Fdotfiles' START=$(date -u -d 03:04:00 +%s) ./series-test.sh
results/series-test-20261029T030400Z.tsv
     35 200 throttle_unauthenticated_api
      1 429 unauthenticated_traffic_per_ip

So the sliding window was wrong. The first test could have told the two apart, but the other limit stopped it after 25 requests. What matched every response from the hourly limit: a count that starts with the first request that counts, allows 60, and resets about an hour after that first request.

We tested that once more, with the predictions committed first. The count that series ran into started with a request sent at 03:02:00.5, so it should reset at about 04:02:00, not at 04:00:00, where Retry-After pointed:

reset-test.sh, predictions committed at 03:06:45

$ TIMES="04:00:30 04:01:50 04:02:10" BURST_AT=04:02:15 ./reset-test.sh
results/reset-test-20261029T030649Z.tsv
2026-10-29T04:00:30.666Z 429 unauthenticated_traffic_per_ip 3569
2026-10-29T04:01:50.122Z 429 unauthenticated_traffic_per_ip 3490
2026-10-29T04:02:10.586Z 200 throttle_unauthenticated_api
1 x 429 unauthenticated_traffic_per_ip
59 x 200 throttle_unauthenticated_api

Both blocked responses now pointed Retry-After to 05:00:00, about 59 and 58 minutes away. Less than 21 seconds later the block was gone, and all 60 came back at once: one at 04:02:10 and 59 in the burst that followed. predict-window.py --compare checks each response against what each model expects, given the requests GitLab had accepted before it. Only the resetting counter matched all 63 responses in this run. The clock hour matched 61, because it expected the two requests after 04:00:00 to go through, and its forecast before the run had expected 57 in the burst, not 59.

Two more things follow from the data. Blocked requests did not extend the block: we got 38 blocked responses the first evening, and it still ended about an hour after 22:38. And the budget is exactly 60. After the 25-request burst at 02:01:45, burst-test.sh sent 25 more back to back, waited 75 seconds, and then sent requests with a 1.5-second pause between them. With 50 used, it should get 10 more, and it got exactly 10:

burst-test.sh: 25 + 25 + 10 = 60

$ ./burst-test.sh
results/burst-test-20261029T020328Z.tsv
     25 C   401 throttle_unauthenticated_api
      1 C   429 
     10 D   401 throttle_unauthenticated_api
      1 D   429 unauthenticated_traffic_per_ip

So Retry-After from this throttle is wrong in both directions. At 22:54 it pointed about 35 to 38 minutes too early. At 04:01:50 it said 3,490 seconds, and the block ended less than 21 seconds later. If you keep a log of your anonymous requests, the block ends about an hour after the first request of the current count, and then all 60 come back at once. Since blocked requests did not extend the block, checking again every few minutes is safe. The better fix is to stop sending anonymous requests from that address and switch to a token.

For the other limits, backoff.py in the repository waits Retry-After, uses capped exponential backoff with full jitter when a 429 has no headers, and slows down when RateLimit-Remaining gets low.

Finding 4: a second limit we could not pin down

Every run of requests to /api/v4/version that got that far stopped at request 26 with a different 429: no RateLimit-Name, and Retry-After: 60. Two were back-to-back bursts that took about ten seconds, and one was a slower series that took 48. We saved one of these responses in full. Like every gitlab.com response it carried server: cloudflare, but none of the headers that GitLab's own 429 had, such as gitlab-lb, gitlab-sv, x-runtime and the RateLimit-* headers. So it most likely came from Cloudflare, not from GitLab. The body said:

The service is temporarily unavailable due to elevated error rates. Please retry after a short wait.

Every request in those runs had returned 401, because /api/v4/version needs a token. The message suggests a limit on errors, but 60 back-to-back 404 responses for a missing file in gitlab-org/gitlab-runner, at the pace of the fast bursts, did not trip it. Neither did 60 200 responses sent four at a time in about eight seconds. So it does not apply one request rate or one error count to every route. It may count 401 responses, or apply only to that endpoint, or leave gitlab-org alone. Our tests cannot tell these apart.

It matters for anything that retries quickly on authentication errors. Our runs got this 429 after 25 failed requests, and it carries none of the headers your GitLab rate-limit handling looks for. When a 429 has no RateLimit-Name, log its body before you decide how long to wait.

A method bug: cached raw files never reach the limiter

Our first raw-file run looked strange. All 70 requests returned 200, and RateLimit-Remaining stayed at 494 for the whole run. The probe did not record cache headers yet, so two minutes later we checked by hand:

cf-cache-status: HIT
age: 48
cache-control: max-age=60, public, must-revalidate, stale-while-revalidate=60, stale-if-error=300, s-maxage=60
ratelimit-remaining: 494
ratelimit-reset: 1792878180

That is an excerpt of one response, saved in the repository. Cloudflare answered from its cache, so the rate-limit headers were a frozen copy: that ratelimit-reset is 21:43 UTC on October 24, four days before our run. Those requests most likely never reached GitLab's limiter. With a unique query string on each request, all 70 responses were cf-cache-status: MISS and the counter moved again. probe.sh now adds that query string by default and records the cache status of every response.

The API responses all said cf-cache-status: BYPASS, so API calls reach GitLab every time. Do not count on the cache for raw files either: you cannot tell in advance which requests it will absorb.

Where it hurts: many runners behind one address

60 requests per hour is plenty for a laptop. It is not much for a fleet of self-hosted runners that leave the network through one NAT gateway, because every anonymous request from every runner counts against the same IP.

Here is a worked example, not a measurement. Twelve parallel jobs per pipeline, a pipeline every ten minutes, and each job fetches an install script and makes two API calls without a token. examples/make_nat_log.py writes that hour as a log, and budget.py compares it with the limits. It groups anonymous requests by the egress address they leave from, and token requests by the GitLab user behind the token:

$ python3 budget.py examples/runner-nat.jsonl
caller requests max/min limit max/hour limit use
anon@runners-nat 216 36 500 216 60 360% <-- over
token:release-bot 90 90 100 90 5000 90% <-- near

The busiest minute, 36 requests, is far below the old 500-per-minute limit. Each pipeline makes 36 anonymous requests, so if all of them count and the hour starts with an unused budget, the second pipeline takes the total past 60. The release bot with a token is fine on the hourly limit, but 90 requests in 45 seconds is close to the Free burst limit of 100 per minute.

The usual sources of anonymous traffic in CI are:

install scripts fetched with curl ... | sh from a /-/raw/ URL
release binaries downloaded from /-/releases/ links
source archives from /-/archive/
ADD https://gitlab.com/... in a Dockerfile, which runs on every uncached build
API calls through $CI_API_V4_URL without a JOB-TOKEN header

Find the anonymous calls before GitLab does

find_anon_calls.py scans CI files, shell and PowerShell scripts, Dockerfiles and Makefiles for these calls and tells you what to use instead. Git over HTTPS is skipped, because it is not counted against the anonymous limit. Run against three of the repository's test fixtures:

$ python3 find_anon_calls.py tests/fixtures/.gitlab-ci.yml tests/fixtures/Dockerfile tests/fixtures/install.sh
tests/fixtures/.gitlab-ci.yml:8: [raw] anonymous call to https://gitlab.com/acme/tools/-/raw/main/install.sh
    fix: use GET /api/v4/projects/:id/repository/files/:path/raw?ref=:ref with JOB-TOKEN or PRIVATE-TOKEN
tests/fixtures/.gitlab-ci.yml:12: [api] anonymous call to $CI_API_V4_URL/projects/42/releases
    fix: add a token header: JOB-TOKEN: $CI_JOB_TOKEN in CI (packages, releases, raw files, artifacts, archives) or a project/group access token for other endpoints
tests/fixtures/.gitlab-ci.yml:20: [archive] anonymous call to https://gitlab.com/acme/lib/-/archive/v2.0/lib-v2.0.tar.gz
    fix: use GET /api/v4/projects/:id/repository/archive with JOB-TOKEN or PRIVATE-TOKEN
tests/fixtures/Dockerfile:3: [raw] anonymous call to https://gitlab.com/acme/tools/-/raw/main/healthcheck.sh
    fix: use GET /api/v4/projects/:id/repository/files/:path/raw?ref=:ref with JOB-TOKEN or PRIVATE-TOKEN
tests/fixtures/Dockerfile:5: [raw] anonymous call to https://gitlab.com/acme/tools/-/raw/main/motd
    fix: use GET /api/v4/projects/:id/repository/files/:path/raw?ref=:ref with JOB-TOKEN or PRIVATE-TOKEN
tests/fixtures/install.sh:3: [download] anonymous call to https://gitlab.com/acme/cli/-/releases/v1.4.0/downloads/cli-linux-amd64
    fix: download through the API (releases, job artifacts, packages) with a token
6 anonymous GitLab call(s) found

It exits with status 1 when it finds anything, so you can add it as a CI job. It decides per command, so a token on one command does not cover the next one on the same line. It is still a pattern matcher, not a parser: it misses URLs built at runtime and calls in languages it does not read.

The fix: send a token

In a GitLab CI job you already have one. According to GitLab's job token documentation, CI_JOB_TOKEN can read raw files, releases and release links, packages, job artifacts and repository archives. For a project outside your own, add your project to the target project's job token allowlist. Public and internal projects allow some of these reads without it.

The most common change is the install script:

# Before: anonymous, counts against the runner's IP
curl -fsSL https://gitlab.com/acme/tools/-/raw/main/install.sh | sh

# After: authenticated with the job token, counts against your plan's limits.
# 1234 is the numeric ID of the acme/tools project; the file path is URL-encoded.
curl -fsSL --header "JOB-TOKEN: $CI_JOB_TOKEN" \
  "$CI_API_V4_URL/projects/1234/repository/files/install.sh/raw?ref=main" | sh

For endpoints the job token cannot reach, such as reading project or group details, use a project or group access token with the read_api scope, stored as a masked CI/CD variable. Outside CI, use a personal access token with the narrowest scope that works. And if a third-party tool cannot authenticate at all, GitLab asks you to contact limits@gitlab.com.

What we could not conclude

One IP, one night. GitLab may enforce the limit differently in other regions, at other times, or for other address ranges.
Namespace plans. We inferred that paid groups are not counted yet. We cannot see a namespace's plan without credentials.
The exact rule. A counter that resets about an hour after its first request matched every response from the hourly limit, including two runs predicted in advance. Close variants fit the same data, such as a reset an hour after the start of that request's minute, or a few seconds short of an hour. A clock-hour window, a weighted estimate and a sliding window each failed at least one prediction. Other implementations could fit the same data, and GitLab can change it at any time.
The second limit. We could trigger it only with 401 responses from /api/v4/version. We do not know what it counts, and that it runs at Cloudflare is an inference from the headers.
Authenticated limits. We had no GitLab token on the test machine, so we did not test the per-user Free limits of 5,000 per hour and 100 per minute.
January. We could not test the Premium and Ultimate limits, which start in January 2027, or whether requests to paid groups start counting against the anonymous limit then.

Summary

The new anonymous limit is live on GitLab.com for at least some of the traffic, and it is easy to miss. It did not apply to the projects we tested first. Successful responses carried no sign of it, and the Retry-After on the 429 did not say when the block would end.

Before it shows up as a failed pipeline:

Run find_anon_calls.py over your CI files, scripts and Dockerfiles.
Give every gitlab.com call a token, starting with CI_JOB_TOKEN where it is allowed.
Count requests per runner egress IP, not per job.
Alert on 429 responses with RateLimit-Name: unauthenticated_traffic_per_ip. Do not wait for their Retry-After: in our tests the block ended about an hour after the first request of the current count from that address.
Before January, check your busiest hour against the Premium or Ultimate limits with budget.py.

Originally published at devops-daily.com.

Email Bounce Types Explained: Practice Sorting Hard, Soft and Blocked Bounces in Your Browser

DevOps Daily — Tue, 27 Oct 2026 12:03:39 +0000

550 5.7.1 Message rejected due to content restrictions

Hard bounce? The 5 says permanent, so a bounce handler built on "5xx means dead" files it with the dead addresses and suppresses the recipient forever. But the code is not about the address. It says a content filter at their end objected to your message, maybe over a link shortener, and the same template will likely hit that filter for other readers behind it. The usual rule, "5xx is hard, 4xx is soft", is a hint, not a verdict. This post covers the rest: where a bounce comes from in the SMTP conversation, how to read one, and which ones to suppress. Disclosure: I help build the two simulators below; they are free, run in the browser, and need no signup.

Where a bounce comes from

Most bounces start as a negative SMTP reply. If the receiving server says no during the session, your relay turns that into a bounce for you. If it accepts the message and delivery fails later, RFC 5321 says it must send a notification of its own, from the null sender <>, to your envelope MAIL FROM address. The rest come from your own relay with no reply at all: a domain that does not exist, a connection that never opens, a queue that expires. The SMTP Flow Simulator plays an illustrative timeline of the conversation in three scenarios, mixing SMTP commands with the DNS and delivery steps around them. The actors (App, SMTP relay, DNS auth, Recipient MX, Mailbox) light up in turn, and a trace pane prints each step as a C: and S: pair:

Successful transactional email: nine steps, from a password reset submitted on port 587 through EHLO, STARTTLS, a second EHLO, AUTH, MAIL FROM and RCPT TO, DATA, the SPF, DKIM and DMARC checks, and the relay to the recipient's MX, to inbox placement. (The simulator shows the auth checks as one step before the hand-off. In real delivery, your side signs DKIM and the receiving server runs the checks as the message arrives.)
Temporary greylisting retry: the recipient's MX answers RCPT TO with 451 4.7.1 Try again later, the relay backs off ("retry in 10 minutes, then 30, then 2 hours"), and the second attempt is accepted.
Authentication failure: 535 5.7.8 Authentication credentials invalid at AUTH, then 530 5.7.0 Authentication required when the app tries MAIL FROM anyway. Both codes come from RFC 4954. That is a rejection, not a bounce: the message never left.

Step through them with Play or Step and read the protocol notes, especially "A 250 after DATA means the next server accepted responsibility" and "Accepted does not always mean inboxed".

You can watch the start of a real conversation yourself. Receiving servers listen on port 25, and dig +short MX gmail.com gives gmail-smtp-in.l.google.com as the most preferred host (preference 5). Many home ISPs and cloud providers block outbound port 25, so this can time out. From my connection it worked, and this is the real output, with my IP swapped for a documentation address:

$ python3 -c '
import smtplib
s = smtplib.SMTP("gmail-smtp-in.l.google.com", 25, timeout=10)
print(s.ehlo("test.invalid")[1].decode())
s.quit()'
mx.google.com at your service, [203.0.113.10]
SIZE 157286400
8BITMIME
STARTTLS
ENHANCEDSTATUSCODES
PIPELINING
CHUNKING
SMTPUTF8

The line to notice is ENHANCEDSTATUSCODES. The server announces that its replies carry a second, more precise code, and that code is where bounce triage starts.

How to read a bounce

Take 550 5.1.1 User unknown. It has three parts:

The reply code, 550, defined in RFC 5321. The first digit says 4 (try again later) or 5 (do not repeat this as-is).
The enhanced status code, 5.1.1, defined in RFC 3463 as class.subject.detail. The middle number says what the problem is about: 1 addressing, 2 mailbox, 3 mail system, 4 network and routing, 5 protocol, 6 content, 7 security or policy.
The text, free-form and often the most useful part.

The class says how long. The subject says what kind of problem it is. A few lines of Python give you a first look:

import re

SUBJECT = {"0": "other", "1": "addressing", "2": "mailbox", "3": "mail system",
           "4": "routing", "5": "protocol", "6": "content", "7": "policy"}

def first_look(reply):
    m = re.search(r"\b([45])\.(\d{1,3})\.(\d{1,3})\b", reply)
    if not m:
        return "no enhanced code, read the text"
    cls, subject, _ = m.groups()
    when = "permanent" if cls == "5" else "temporary"
    return f"{when}, about {SUBJECT.get(subject, 'other')}"

Run against three replies:

550 5.1.1 User unknown         -> permanent, about addressing
452 4.2.2 The email account th -> temporary, about mailbox
550 5.7.1 Message rejected due -> permanent, about policy

For 5.1.1, "permanent, about addressing" means a dead address: suppress it. Other addressing codes need the detail digit, since 5.1.7 and 5.1.8 are about your sender address, not theirs. "Permanent, about policy" usually points at your side: content, authentication or sending reputation. Suppressing the recipient fixes none of those. The IANA registry lists every assigned code.

Four kinds, five actions

The Bounce Triage Simulator turns this into practice. It shows twelve bounces one at a time, each with the recipient, the provider (Gmail, Outlook, Postfix, Proofpoint and others) and the raw response. For each one you answer two questions. "What kind of failure is this?": Hard bounce, Soft bounce, Blocked, or Complaint. "What do you do with it?": Suppress permanently, Retry with backoff, Fix the message then retry, Slow down then retry, or Stop sending and fix the cause. The last one is the answer for a relay denial and a blocklist listing. Press "Triage it" to see what the code means, plus a box titled "What getting it wrong costs you".

Get the kind right but the action wrong and it tells you "Right diagnosis, wrong treatment", the exact mistake it is built to catch. The rounds that teach the most:

550-5.7.26 Unauthenticated email from yourdomain.com is not accepted due to domain's DMARC policy. A 5xx that is an authentication bug on your side. RFC 7372 named 5.7.26 "Multiple authentication checks failed", and Gmail uses it for DMARC failures. Receivers that enforce your DMARC policy keep rejecting mail that fails this way until you fix SPF or DKIM alignment.
421 4.7.0 Too many messages from your IP. Please try again later. A 421 also closes the connection. The right action is to slow down. Retrying at the same rate makes it worse.
450 4.7.1 ... Greylisted, try again in 300 seconds. Treat it as a failure and you never reach a whole class of small self-hosted domains, because the retry is the test.
A Yahoo feedback loop report with Feedback-Type: abuse. Not a bounce at all: the mail was delivered and the reader pressed "report spam".

Complaints get the hardest line. Suppress them at once and for good. Google's sender guidelines require every sender to keep the spam rate in Postmaster Tools below 0.3%, and recommend staying under 0.1%.

What to suppress

Seven of the twelve rounds are 5xx, and four of those seven should not be suppressed. The policy:

Suppress permanently: hard bounces (5.1.1 user unknown, 5.1.2 domain not found, 5.2.1 disabled mailbox) and every complaint.
Retry with backoff: soft bounces (4.2.2 over quota, greylisting, 4.3.0 server errors), within your relay's queue lifetime. RFC 5321 says the give-up time generally needs to be at least 4 to 5 days. It also says the retry interval SHOULD generally be at least 30 minutes, with smarter schedules when the sender knows why delivery failed, so the SMTP Flow Simulator's first greylisting retry after 10 minutes is within the rules. One expired message does not prove the address is dead.
Keep the recipient, investigate: 5.7.x content and authentication blocks, blocklist listings, rate limits and relay errors. Most are fixed on your side (template, DNS, send rate, delisting), some need the recipient's mail admin, and none is fixed by resending the same message unchanged.

The first digit tells you how long. The middle digit tells you what kind of problem it is. Play the three SMTP scenarios to see where replies come from, then triage the twelve bounces and count how many 5xx codes you would have suppressed by mistake. To type the conversation by hand, one command at a time, read Send an Email by Hand: The Raw SMTP Conversation.

Which bounce code has fooled your bounce handling? Tell us in the comments.

Moving Authorization Out of Your App Code: If-Statements vs Cerbos vs SpiceDB, Measured

DevOps Daily — Tue, 27 Oct 2026 09:00:00 +0000

Many products rebuild their permissions three times: first roles, then teams, then per-document sharing. Somewhere after that, a rule arrives that nobody planned for. The usual advice is to move authorization out of the application code before that happens, into a policy engine or a Zanzibar-style relationship graph. We wanted to know what that move costs and what it buys, so we built the same permission model three ways and grew it in four steps:

if-statements in the application, with a SQL query for lists.
Cerbos policy files, evaluated by a local Cerbos policy decision point (PDP).
SpiceDB , an open source implementation of Google's Zanzibar design: a schema plus relationships.

A hand-written table of 84 expected decisions checked all three at every step. Then we timed them on a Raspberry Pi 4 with 10,000 documents. The numbers describe this setup: a Pi, one request at a time, the app database in the same process. Both the milliseconds and the ratios between the designs belong to this setup; a network, a different database or other hardware can change either.

What we found:

A single check is cheap only in-process. The if-statements answered in 0.14 ms. Cerbos took 3.8 ms and SpiceDB 10.8 ms (median p50). On this Pi a call to either server that does no authorization work already costs about 2.4 ms, and Cerbos's own log puts its handling time at 0.63 ms. By the bar we set before measuring (p99 under 5 ms), a check through either engine is not cheap on this machine.
Listing is where they really differ."Which of these 10,000 documents can this member view?" took 4.4 ms as SQL, 11.9 to 31.6 ms through a Cerbos query plan turned into SQL (depending on two lines of our own adapter), and 404 ms through SpiceDB when its cache was cold (77 ms warm). For an admin who can view all 10,000, SpiceDB took 3.3 seconds cold.
The late rule cost little code in every design: 22 changed lines in one file for the if-statements, 15 in two files for Cerbos, 13 in two files for SpiceDB. It never stayed out of the application: Cerbos needed a new attribute in every request, and SpiceDB needed a new relationship synced from the database. In SpiceDB it also made a cold list about 3.5 times slower.
Leave out part of the late rule and every design fails open. The table caught each of the three faults we injected, and it also caught SpiceDB returning duplicate documents in a list.

TLDR

Same model, three designs, four steps, one table of 84 hand-written decisions. All three matched it at every step.
Single check, median p50 and p99: if-statements 0.14 / 0.24 ms, Cerbos 3.8 / 7.9 ms, SpiceDB 10.8 / 19.4 ms on new checks and 2.8 ms p50 on repeated ones, because SpiceDB caches results.
List for a typical member (1,020 of 10,000 visible): SQL 4.4 ms, Cerbos 11.9 to 31.6 ms, SpiceDB 77 ms warm to 404 ms cold. SpiceDB's own docs suggest a different pattern above about 10,000 results, and for an admin's first page it took 18 ms.
Lines changed over the four steps: 83 for if-statements, 211 for Cerbos, 187 for SpiceDB. The engines cost more up front; by step 4 a new rule cost them fewer lines than the if-statements (15 and 13 against 22).
The engines did not remove application changes. Every new rule needed data, and the data lives in your app.
Keep the if-statements while one app owns the rules and you can test them as a table. Move out when several services need the same answer or the relationships nest deeply.

Prerequisites

Node.js 22.13 or later (the demo uses the built-in node:sqlite)
Linux on arm64 or amd64. The demo downloads pinned Cerbos and SpiceDB release binaries and checks their SHA-256. No Docker.
Basic familiarity with role-based access control and SQL joins
The demo repository, which contains every commit, script and recorded run in this post:

The-DevOps-Daily/authz-three-ways on GitHub

The model and the four steps

The app has users, teams, documents and shares in SQLite. The actions are view, edit and delete. We wrote the rules for all four steps into CLAIM.md before writing any code, together with what result would prove our claim wrong:

Step	Rule
1. Roles	Each user has one org role. `viewer` can view every document, `editor` can view and edit, `admin` can view, edit and delete.
2. Teams	Each document belongs to a team. `viewer` and `editor` now apply only to documents of the user's own teams. `admin` is unchanged.
3. Sharing	A document can be shared with one user as `view` or `edit`, whatever that user's teams are. A share never grants delete.
4. Late rule	Contractors get no access at all to documents outside their own teams. This overrides every earlier grant, including `admin` and shares.

We picked the late rule before implementing anything. It is a realistic one: security asks for it after an audit, and it cuts across every grant that already exists, so it tests whether a design can say "no" over the top of many "yes" paths.

The expected-decisions table, fixture/expected.csv, has seven users, two teams, four documents and four shares, and one row per (user, action, document) with the expected answer at each step:

user,action,document,step1,step2,step3,step4
cora,view,d4,allow,deny,allow,deny
kai,delete,d1,allow,allow,allow,deny
nell,view,d2,allow,deny,allow,allow

Cora is an editor in eng and a contractor, with an edit share on d4, a sales document. Kai is an admin and a contractor in sales. Nell is a viewer with no team and a view share on d2. harness/correctness.js runs every row against each design at each step, and also asks each design for the full list of documents each user can view, which must match the table's view rows.

Three designs

if-statements everything in the app
- check() JavaScript, up to 4 queries
- list() one SQL query
- SQLite the app's own tables
Cerbos stateless policy engine
- App reads attributes from SQLite
- Cerbos PDP policy YAML, gRPC
- Plan to SQL PlanResources, your adapter
SpiceDB Zanzibar-style graph
- App syncs rows to relationships
- SpiceDB schema, gRPC
- Relationships SpiceDB's own copy

The three designs put the rule and the data in different places, and that difference explains almost every number below.

if-statements (approaches/ifs/index.js): one check() function that reads the user, the document, the team membership and any share from SQLite, and one list() function that is a SQL query. The rule is written twice: once in JavaScript and once in SQL.
Cerbos (approaches/cerbos/): the rules live in YAML policies, with derived roles such as team_editor and shared_viewer. Cerbos stores no application data, so for every check the app reads the same rows from SQLite and sends them as principal and resource attributes. For lists, Cerbos's PlanResources API returns a condition tree, and our plan-to-sql.js turns it into a SQL WHERE.
SpiceDB (approaches/spicedb/): the rules live in a schema, and the data lives in SpiceDB as relationships such as document:d1#team@team:eng. sync.js copies the app's rows into SpiceDB. A check is one CheckPermission call with no database read in the app. A list is LookupResources.

Both engines ran on 127.0.0.1 and were called over gRPC through their official Node clients, one client each for the whole run, without TLS: Cerbos 0.57.0 with policies on disk, and SpiceDB 1.57.0 with its in-memory datastore, which its docs describe as intended for testing integrations. Each design has one commit per step in the repo, tagged step<N>-ifs, step<N>-cerbos and step<N>-spicedb, so every line count below comes from git diff --numstat.

The same rule, three ways

Here is the late rule, "contractors get nothing outside their own teams", as each design wrote it. These are excerpts from the step 4 commits; // ... marks lines left out.

Step 4: contractors get nothing outside their own teams

if-statements: check()

const inTeam = Boolean(isMember.get(doc.team_id, userId));
// Contractors get nothing outside their own teams, whatever else grants it.
if (user.contractor && !inTeam) return false;
// A share grants access whatever the user's role and teams are.
const share = getShare.get(docId, userId);
if (share && SHARE_ACTIONS[share.level].includes(action)) return true;
// Roles other than admin apply only to documents of the user's teams.
if (user.role !== 'admin' && !inTeam) return false;
return ROLE_ACTIONS[user.role].includes(action);

if-statements: list()

// Contractors see only documents of their own teams.
const contractorFilter = `(:contractor = 0 OR d.team_id IN
  (SELECT team_id FROM team_members WHERE user_id = :user))`;
const allDocs = db.prepare(`SELECT d.id FROM documents d WHERE ${contractorFilter}`);
const visibleDocs = db.prepare(`
  SELECT d.id FROM documents d
  JOIN team_members m ON m.team_id = d.team_id
  WHERE m.user_id = :user
  UNION
  SELECT s.document_id FROM shares s
  JOIN documents d ON d.id = s.document_id
  WHERE s.user_id = :user AND ${contractorFilter}`);

Cerbos: policy

# Contractors get nothing outside their own teams, whatever else grants it.
- actions: ["*"]
  effect: EFFECT_DENY
  roles: ["*"]
  condition:
    match:
      all:
        of:
          - expr: request.principal.attr.contractor == true
          - expr: "!(request.resource.attr.team in request.principal.attr.teams)"

Cerbos: app

const getUser = db.prepare('SELECT id, role, contractor FROM users WHERE id = ?');
// ...
return { id: user.id, roles: [user.role], attr: { teams, contractor: user.contractor === 1 } };

SpiceDB: schema

definition org {
    // ...
    relation contractor: user
}

definition document {
    // ...
    // Contractors get nothing outside their own teams, whatever else grants it.
    permission outside_contractor = org->contractor - team->member

    permission view = (org->admin + team->view_docs + viewer + editor) - outside_contractor
    permission edit = (org->admin + team->edit_docs + editor) - outside_contractor
    permission delete = org->delete_docs - outside_contractor
}

SpiceDB: sync

for (const u of db.prepare('SELECT id, role, contractor FROM users').all()) {
  out.push(touch('org', ORG, u.role, 'user', u.id));
  if (u.contractor) out.push(touch('org', ORG, 'contractor', 'user', u.id));
}

Three things stand out:

The if-statements wrote the rule more than once: in check(), and again in each list query that can return a document from outside the user's teams (the admin query and the shares branch). Miss one and the list leaks while every check passes.
Cerbos needed no change to the plan adapter. The planner folds the principal's side of the condition into constants, so for Cora the deny rule becomes an eq on request.resource.attr.team with the value eng, ANDed with her grants, which the adapter already handled (results/final/plans-step4.json). It did need one more attribute from the app.
SpiceDB's rule is one line of schema, an exclusion: org->contractor - team->member. But SpiceDB knows only what the app writes into it, so sync.js needed a new relationship type too.

What each step cost in code

Lines changed in each design, per step

	1: roles	2: teams	3: sharing	4: contractors
if-statements	29 lines	12 lines	20 lines	22 lines
Cerbos	73 lines	82 lines	41 lines	15 lines
SpiceDB	133 lines	31 lines	10 lines	13 lines

Lines added plus deleted in each design's own commit, from git diff --numstat. The shared data-model commits (29, 26, 9, 5 lines) are the same for all three and not counted. Step 2 for SpiceDB includes the de-duplication fix the table forced (8 of its 31 lines).

Step	if-statements	Cerbos	SpiceDB
1. Roles	1 file, 29 lines	3 files, 73 lines	3 files, 133 lines
2. Teams	1 file, 12 lines	4 files, 82 lines	3 files, 31 lines
3. Sharing	1 file, 20 lines	4 files, 41 lines	2 files, 10 lines
4. Late rule	1 file, 22 lines	2 files, 15 lines	2 files, 13 lines
Total	83 lines	211 lines	187 lines

The shapes differ more than the totals. The if-statements were cheapest to start, and from step 2 on each rule cost more than the last (12, 20, 22 lines), because every rule touched both check() and the list query. Cerbos paid most at step 2, when the first condition appeared and we had to write the 43-line plan-to-SQL adapter. SpiceDB paid most at step 1: client setup, paging for LookupResources and the sync code. After that, sharing cost SpiceDB 10 lines.

Two caveats keep these numbers honest. Line counts depend on style, so read the diffs in the repo rather than trusting a total. And our if-statements are the best case: one module with one check(). If your checks are spread over many handlers, the late rule is many edits.

A single check

One permission check, 10,000 documents (median p50, tick at p99)

	Value	p99	Series
if-statements, in-process	0.14ms	0.24ms	if-statements
Cerbos over gRPC	3.83ms	7.91ms	Cerbos
SpiceDB over gRPC, new checks	10.78ms	19.37ms	SpiceDB
SpiceDB over gRPC, repeated checks	2.83ms	15.62ms	SpiceDB
No-op call to Cerbos	2.4ms	6ms	No-op call
No-op call to SpiceDB	2.44ms	6.31ms	No-op call

Raspberry Pi 4, step 4 model, concurrency 1. Checks: median across 3 runs of 10,000 each. No-op calls: one run of 5,000 each. "New checks" are random (user, action, document) triples; "repeated checks" draw from 100 triples, so SpiceDB's cache answers most of them. A no-op call does no authorization work. Absolute numbers are from a Pi: compare the bars with each other.

We timed 10,000 random (user, action, document) checks per run, three runs, one at a time, with the three designs interleaved on every sample so that anything else on the Pi slowed all of them equally. All three gave the same answer on every sample that completed.

At the median, the in-process check (0.144 ms) was about 27 times faster than Cerbos (3.83 ms) and about 75 times faster than SpiceDB (10.8 ms). Where that time goes:

A request to either server that does no authorization work (Cerbos ServerInfo, SpiceDB ReadSchema) took 2.40 and 2.44 ms at the median on this Pi. A Node-to-Node TCP echo between two processes took 0.33 ms, so most of that is request handling in the gRPC client and server, not the loopback network.
Cerbos logs its own handling time for each request. Its median was 0.63 ms, against 3.8 ms seen by the client.
So for Cerbos, a request that does nothing already costs most of what a check costs. For SpiceDB on new checks, it is about a quarter of the 10.8 ms. We did not measure where the rest goes inside SpiceDB, and a ReadSchema call is only a reference point, not the same work as a check minus the graph.
SpiceDB's cache removes most of that difference. When the same 100 checks repeated, as they do when people reopen the same documents, its p50 dropped to 2.8 ms, close to the no-op call, while Cerbos stayed at 3.8 ms.

One thing the chart hides: the if-statements' check runs up to four queries against SQLite in the same process. With Postgres across a network, each of those queries is a network round trip. The Cerbos design pays the same queries and the PDP call on top. The SpiceDB design reads nothing from the app database for a check. In that layout the gap between the in-process design and the engines could be smaller than it is here. We did not measure it.

One of 60,000 SpiceDB checks failed with UNAVAILABLE: Stream closed with error code NGHTTP2_REFUSED_STREAM. SpiceDB closes client connections after 30 seconds by default (--grpc-max-conn-age), which is our best explanation, and the Node client did not retry it. We counted the failure and left it out of the timings. In production, give the client a retry policy for UNAVAILABLE.

Listing what a user can see

This is the question every list page asks, and the designs answer it in completely different ways:

if-statements: a SQL query with a UNION of team documents and shared documents, filtered for contractors.
Cerbos: PlanResources returns a condition tree for this user, our adapter turns it into a WHERE, and SQLite runs it.
SpiceDB: LookupResources walks the graph backwards from the user and streams document IDs. The app then fetches those rows from its own database, a step we did not time.

List the documents a typical member can view (1,020 of 10,000)

	Value	Series
if-statements: SQL	4.4ms	if-statements
Cerbos: plan to SQL, IN subquery	11.9ms	Cerbos
Cerbos: plan to SQL, as committed	31.6ms	Cerbos
SpiceDB: LookupResources, warm	77.2ms	SpiceDB
SpiceDB: LookupResources, cold	403.9ms	SpiceDB

Median ms on a Raspberry Pi 4, step 4 model. "As committed" turns each share condition into a correlated EXISTS; the IN-subquery variant changes only those two lines. SpiceDB bars come from a separate cache experiment with one unlimited LookupResources call: "cold" is the first call in a new 5 s revision window, "warm" repeats it at the same revision.

10,000 documents	Visible	if-statements	Cerbos (as committed)	SpiceDB, warm	SpiceDB, cold
Admin	10,000	25.3 ms	29.9 ms	691 ms	3,279 ms
Member	1,020	4.4 ms	31.6 ms	77 ms	404 ms
Contractor	510	2.4 ms	9.6 ms	41 ms	247 ms

The SQL and Cerbos columns are medians of the run medians from results/final/list-*.json. The SpiceDB columns come from a separate cache experiment, harness/spicedb-list-cache.js, which separates the two cases on purpose, because SpiceDB's list time depends on its cache more than on anything else. It differs from our adapter's list call in two ways: it makes one LookupResources call with no limit instead of pages of 1,000, and it reads with minimize_latency and then at_exact_snapshot instead of at_least_as_fresh. Timed through the adapter itself, the member's list had a median of 75 ms and the admin's 803 ms, mostly warm calls; see below.

SpiceDB LookupResources, step 4, 10,000 documents

$ nice -n 19 node harness/spicedb-list-cache.js --tree=.steps/step4 --out=results/final/spicedb-list-cache.json
admin visible 10000 first in a new 5 s window: median 3278.544 ms same revision again: median 691.44 ms
member visible 1020 first in a new 5 s window: median 403.896 ms same revision again: median 77.2 ms
contractor visible 510 first in a new 5 s window: median 246.668 ms same revision again: median 41.139 ms
problems (wrong answers or a reused revision): 0

SpiceDB caches results per revision. By default it reads at a revision rounded to a 5-second window (--datastore-revision-quantization-interval), and the in-memory datastore's revisions are timestamps, so every 5 seconds the read revision moves and the cache starts cold, even with no writes. Which case a request gets depends on timing. In our runs of 20 back-to-back admin lists (results/final/list-*.json), most found a warm cache, with a median of about 800 ms, and the slowest took about 3.7 s.

Four more things we measured about lists:

Our Cerbos adapter was the slow part, not Cerbos. The committed adapter turns "this user is in the document's viewers" into a correlated EXISTS, which SQLite runs once per document. Changing those two adapter lines to an uncorrelated d.id IN (SELECT document_id FROM shares WHERE ...) cut the member list from 31.6 ms to 11.9 ms, with the same answers (scripts/variant.js). The plan is only as fast as the SQL you turn it into, and that SQL is yours to write. Cerbos publishes reference adapters for Prisma, Drizzle, SQLAlchemy and others; we used none, because we used plain SQLite.
Page size matters for LookupResources. Our list code asks LookupResources for pages of 1,000, the default of SpiceDB's --max-lookup-resources-limit flag, and follows the cursor. For the admin that took 0.8 seconds over 11 calls. Pages of 100 took 7.2 seconds over 106 calls, and a single call with no limit took 0.7 seconds (results/final/lookup-pages.json).
SpiceDB's docs already say this. Its guide to protecting a list endpoint suggests LookupResources when a user can see fewer than about 10,000 resources, and otherwise fetching a page of candidates from your own database and filtering them with CheckBulkPermissions. We timed that for "the first 50 documents in ID order", against our adapter listing every visible ID with LookupResources, sorting them and keeping 50 (harness/spicedb-page.js). For the admin the bulk checks took 18 ms with one call, against 786 ms for the full LookupResources list. For the contractor, who can see 5% of the documents, it took 167 ms over 11 bulk calls, against 37 ms. So the right pattern depends on how much of the data each user can see.
The late rule made SpiceDB's cold lists about 3.5 times slower. We ran the cold and warm list test against the code of every step (results/steps/step<N>-spicedb-list-cache.json). From step 3 to step 4, the admin's cold list went from 928 to 3,204 ms, the member's from 108 to 390 ms and the contractor's from 64 to 227 ms. The warm lists did not move (686 to 696, 73 to 70, 42 to 38 ms). Thirteen lines of schema and sync were cheap to write; the exclusion they added is not cheap to evaluate.

What the table caught

The 84-row table was the most useful file in the repo. It caught two things we would otherwise have shipped.

SpiceDB returns duplicates. At step 2, Kai, an admin who is also a member of sales, got d3 and d4 twice from our adapter's paged LookupResources calls: once through the admin path and once through the team path. This is expected behaviour: an issue in the SpiceDB repository explains that two paths to the same resource can produce two results, calls duplicates a known limitation of cursored calls, and recommends de-duplicating on the client. We added a Set and recorded the failing run in results/caught/.

Every design fails open when part of the late rule is missing. scripts/break.js leaves out one part of the step 4 change in each design and runs the table again:

leave out part of the late rule

$ node scripts/break.js > results/breaks.txt
$ grep -v '^ ' results/breaks.txt
ifs-forgot-the-list-query mismatches 2 (list) caught
cerbos-forgot-the-attribute mismatches 10 (check, list) caught
spicedb-forgot-the-sync mismatches 10 (check, list) caught

if-statements: check() has the rule, but list() kept its step 3 query. Every check passes, and Cora's list still shows the sales document shared with her.
Cerbos: the policy has the deny rule, but the app does not send contractor. The condition cannot match a missing attribute, so the deny never fires. Cerbos can validate attributes against a JSON schema; with reject enforcement, its docs say every action of an invalid request is denied. With contractor marked as required, that would turn this failure from open to closed. We did not test it.
SpiceDB: the schema has the exclusion, but sync.js does not write org#contractor. With no relationship there is nothing to exclude.

None of the three designs made these omissions safe. The table of expected decisions caught all three, which is why we would run it against every change.

The costs this benchmark leaves out

Keeping SpiceDB in sync. sync.js copies the whole database at start. A real app must write every change, such as a new share or a new contractor, to both its database and SpiceDB, and a crash between the two writes leaves them disagreeing. AuthZed calls this the dual-write problem and lists ways to handle it, from a transactional outbox to making SpiceDB the source of truth. Cerbos and the if-statements read the app's data at request time, so they do not have this problem.
Running a server. Both engines are another service to deploy, monitor, upgrade and keep available, or a hosted product to pay for.
A real datastore. SpiceDB on Postgres or CockroachDB will have different numbers from the in-memory datastore.
Fetching the rows. SpiceDB's list returns IDs. A list page then needs a second query for titles and dates.

When the if-statements are the right answer

In this repo the if-statements were the fastest design on every timing we took for all three, and the smallest in total code. They are the right answer more often than vendor pages suggest. Keep them when:

One application owns the rules. Everything lives in one codebase and one database, and one team changes both. A policy engine earns its keep when several services must answer the same question the same way.
Your rules are mostly filters on your own tables. Teams, owners and shares are joins. SQL already plans them, and your list endpoint keeps pagination, sorting and search.
You can keep the checks in one module. Our ifs design is one file with one check() and one list(). That is the best case. If your checks are spread over fifty handlers, the late rule is fifty edits, and our line counts do not apply to you.
You test the rules as data. The expected-decisions table is what made each step safe, and it works for if-statements as well as for policies.

The if-statements stop being enough when:

More than one service or language needs the same answer. Copying the rules into each one is how they drift apart.
People outside the team need to read or change the rules, for example a security review of what contractors can see. A separate policy file or schema may be easier for them to review than a SQL UNION inside a JavaScript function, depending on what they already read.
You need a record of decisions. Cerbos can write decision logs (off by default, and off in our runs). Homemade code has none unless you build it.
The relationships nest, such as folders inside folders or groups inside groups. Walking a graph is what SpiceDB is built for, and it is awkward to write and to index in SQL. Our model has no nesting, so this test does not cover it.

The hosted options

Several companies offer managed authorization services or control planes, some built on these engines and some on others. Some run the decision service for you; others manage policies while evaluation stays in your environment. We did not test any of them, so this is only what their own documentation says they offer:

Cerbos Hub is a control plane for the open source Cerbos PDP. Per its docs, it tests, signs and distributes policy bundles to your PDPs, which keep running in your own network, and it offers an embedded PDP compiled to WebAssembly for evaluation in browsers, edge functions and other JavaScript environments.
AuthZed sells hosted SpiceDB. Per its docs, AuthZed Cloud is a self-service platform with usage-based pricing, and AuthZed Dedicated is a private deployment in the cloud provider and regions you choose. For lists too large for LookupResources, its list guide describes AuthZed Materialize: its Event Streams feature, in early access for Dedicated customers at the time of writing, lets your service keep a local table of permissions so that filtering becomes a JOIN.
Oso Cloud is Oso's authorization service, with policies written in the Polar language. Its local authorization mode returns SQL, including conditions that filter a list query, which your application runs against its own PostgreSQL or MySQL database; Oso does not read that data.
Permit.io offers a managed cloud PDP, or a PDP container that you run next to your services and that evaluates policies with OPA or Cedar, per its docs. It lists RBAC, ABAC and ReBAC models, and bulk checks for lists and tables.
WorkOS FGA extends WorkOS RBAC with roles scoped to resources in a parent-child hierarchy, per its docs. Your app registers resources with WorkOS, and the API checks access and lists the resources a user can access.

The trade we measured carries over to each of them: wherever the rule lives, it has to meet the data it needs. Either the data travels with every request, or someone keeps a copy in sync, or, as Oso's local authorization does, the rule becomes a query that runs where the data already is.

What we could not conclude

Production latency. Everything ran on one Raspberry Pi 4, at concurrency 1, with the app database in the same process. Neither the milliseconds nor the ratios predict a production system; they show how the designs differ under the same conditions.
Tuned servers. We did not tune either engine. Both ran with their defaults, except that we turned off telemetry and SpiceDB's metrics endpoint and set SpiceDB's log level to warn. Cerbos, at its default info level, writes two log lines per request; we did not measure what that costs it. Everything, servers included, ran at the lowest CPU priority (nice -n 19) on a Pi shared with other work, and each timed run first waited for a quiet machine: a 1-minute load average under 1.25 for the step 4 runs and under 1.5 for the per-step runs, giving up after 10 and 15 minutes. Each result file records the load average it started at.
Throughput. We measured one request at a time. We did not measure how many checks per second each design sustains.
Deep hierarchies. Our model has no nested folders or groups, which is where a relationship graph should do best and recursive SQL is hardest. This test does not cover it.
Why the exclusion costs SpiceDB so much on a cold list. We measured that it does, not where the time goes inside SpiceDB.
The cause of the one failed SpiceDB call. The 30-second connection age is our best explanation; we did not confirm it.

Summary

We set out to show that a single check is cheap in all three designs and that listing and the late rule separate them. By the rules we wrote in CLAIM.md before measuring, that claim is wrong, because its first half failed. A single check is cheap only in-process: through either engine it cost a network call, and on this Pi that call alone was about 2.4 ms. The second half held. For the member, a Cerbos plan turned into SQL took 2.7 to 7 times as long as the hand-written SQL, and SpiceDB's LookupResources 18 to 92 times as long, warm to cold. The cache, the page size and our own SQL adapter moved those numbers as much as the engines did. The late rule cost every design 13 to 22 lines and still needed a change in the application, because every rule needs data and the data lives in your app.

If one app owns the rules, keep them in code, keep them in one module, and test them with a table like fixture/expected.csv. When you do move them out, plan the data path first: what each request must carry for Cerbos, what must be kept in sync for SpiceDB, and how your list pages will get their answers.

Originally published at devops-daily.com.

LiteLLM 1.107.0 Was Built From a Lookalike Fork: What PyPI Provenance Can and Cannot Tell You

DevOps Daily — Tue, 27 Oct 2026 09:00:00 +0000

On October 21, 2026, litellm==1.107.0 was on PyPI for about 1 hour 43 minutes, with a credential stealer and a remote-code loader in all seven wheels. Reading github.com/BerriAI/litellm would not have shown you anything: LiteLLM's own release pipeline built the release from a lookalike fork, Berri-AI/litellm, one hyphen away from the real name.

So we asked a narrower question: which record could have told you where the artifact came from? We checked PyPI, the container registries and the 1,000 most downloaded PyPI projects. The short answer:

The Docker images' build provenance named the fork. It was there to read.
PyPI had nothing to check. litellm published PEP 740 attestations on 15 releases in March and April 2026 and on none of the 214 releases since. Those 15 name the private release repository, BerriAI/project-releaser, not the source repository. That is the repository whose release workflows the attacker ran, so an attestation on 1.107.0 would most likely have named the same publisher as every legitimate release.
Across the 1,000 most downloaded PyPI projects, 39.0% of latest releases carry an attestation , and 33.4% carry one from the repository their own metadata names.

This post covers what happened, how to check your environments, lock files, caches and images, what to rotate, the provenance gap with the numbers, and what you can enforce today. Every number we measured comes from a file in the companion repo. Incident facts, such as the timeline, come from the sources we link, and the repo's SOURCES.md lists each one.

The-DevOps-Daily/pypi-provenance-check on GitHub

TLDR

What shipped: 7 wheels (cp310-abi3, Linux, macOS, Windows) with cook.pth, cook.py and sitecustomize.py, plus container images under v1.107.0, latest, main-latest and main-stable on GHCR and Docker Hub. Clean versions: 1.106.2, 1.107.0rc1 and 1.107.0rc2.
Check without starting the suspect Python. The payload runs whenever an interpreter starts with that site-packages on its path, and that includes pip. Our checker runs under python3 -I -S and reads files only. It finds the payload in venvs, lock files, pip and uv caches and image tarballs.
Rotate everything the host could reach: cloud role credentials, every provider key stored in the LiteLLM proxy database, the master and virtual keys, DATABASE_URL, CI and registry tokens, SSH and TLS keys.
The provenance gap: a PyPI publish attestation names the workflow that uploaded the file, not the source it built. pypi-attestations verify against BerriAI/litellm fails even on a legitimate litellm release. In the files we sampled from the 396 attested projects in our survey, we found no SLSA provenance attestation, the PyPI type that can record a source repository.
23 projects stopped attesting. They attested an earlier release and not their latest one. Today, 12 of them upload with uv publish, which uploads attestations but does not create them.
What works today: a release cooldown (exclude-newer = "7 days" in uv, --uploaded-prior-to P7D in pip 26.1+), hash-locked installs, image digests plus a provenance source check, and publisher pins for packages that attest. As of October 2026, neither pip nor uv verifies attestations at install time.

Prerequisites

Python 3.11 or later on the hosts you want to check (the scripts use only the standard library)
Read access to your virtual environments, lock files, pip and uv caches, CI logs and container images
Optional: uv, to run the pypi-attestations verifier, and docker or podman, to export images for scanning

What happened

LiteLLM is a Python SDK and proxy that puts many LLM providers behind one OpenAI-style API. The proxy usually stores every upstream provider key in its database, which made it a valuable target. The timeline below comes from the maintainers' incident post and their attack-chain post, which they call the canonical account.

Time (UTC, Oct 21, 2026)	Event
10:08	Branch protection on the private release repo `BerriAI/project-releaser` rejects commit `9cf97e9` ("ci: release from a fork source repository"). A stolen token of a GitHub organization owner disables the protection and force-pushes it.
10:46:39	A staging run starts with source `Berri-AI/litellm` at commit `20535d8f`, which "exists only on the attacker's fork".
11:11:18	A second stolen maintainer token approves the production gate.
11:39:51 to 11:40:13	Seven `litellm==1.107.0` wheels are uploaded to PyPI.
11:46:56	Images are pushed under `v1.107.0`, `latest`, `main-latest` and `main-stable`.
11:53:49 to 11:54:01	A third stolen token deletes the workflow run records.
12:06:41	Upwind opens issue #47826.
13:23	PyPI removes 1.107.0.
13:51	The GHCR tags are gone.
14:20	The Docker Hub tags are removed or repointed to clean images.

The way in was ordinary. On May 27, a config file with the master key of the LiteLLM Agent Platform was committed to a public repository. The commit was force-pushed over, but the key was not revoked. On October 10, the attacker used it to read a GitHub CLI OAuth token that belonged to an organization owner. More maintainer tokens were taken from repository secrets the same day. A maintainer summed it up in the issue as "use of insufficiently scoped keys combined with failure to revoke after accidental exposure."

The payload sits in litellm-1.107.0.data/purelib/, so the installer puts it in the root of site-packages:

cook.pth contains import cook, so cook.py runs every time that interpreter starts. We covered this startup trick in detail in our post on the Shai-Hulud Hades wave on PyPI.
cook.py is a stealer that calls itself privcollect. It encrypts what it collects and POSTs it to http://www.litellm.dev/healthcheck, a lookalike domain registered on October 13.
sitecustomize.py forks a daemon that holds /tmp/remote_code_loader.lock. Every hour, it reads a domain from raw.githubusercontent.com/letters9813/pythonclient-test/main/domain.txt and runs the code at http://<domain>/codepy. The issue reports that the file has been empty since October 20, so this stage was dormant.

Early reports said the v1.107.0 git tag was clean. The maintainers later wrote that "there is no official v.1.107.0 source tag". The tag list we pulled from BerriAI/litellm on October 26 has v1.107.0-dev.1 to -dev.3 and v1.107.0-rc.1 and -rc.2, and no v1.107.0. Either way, the malicious lines were never in BerriAI/litellm. The lookalike repository now returns 404.

This is the second LiteLLM poisoning this year. In March, versions 1.82.7 and 1.82.8 were published with a stolen PyPI token, which Chainguard attributes to TeamPCP, the group in our PyTorch Lightning post. There is no evidence that links the two incidents.

Check if you were hit

You are exposed if any of these happened:

A pip install or uv resolve picked litellm==1.107.0 between 11:39:51 and 13:23 UTC. This includes unpinned litellm and litellm[proxy] requirements, and lock files that were refreshed in that window.
You ran an image pulled by one of the four tags between 11:46:56 and 13:51 UTC (GHCR) or 14:20 UTC (Docker Hub), or you built an image FROM one of those tags in that window.

Pulling or unpacking an image does not start Python, so it does not run the payload. Any start of the image's Python with normal site initialization does: a container whose entrypoint runs it, a RUN step in a derived build that runs it (pip install counts), or docker exec ... python.

The most important rule comes from the issue: do not start with pip install. Any Python process that starts in the affected environment runs cook.pth and sitecustomize.py again, and pip is a Python process.

With nothing but a shell

# The payload files in any site-packages or dist-packages on this host (all mounted filesystems)
find / \( -path /proc -o -path /sys \) -prune -o \
  \( -path '*/site-packages/*' -o -path '*/dist-packages/*' \) \
  \( -name 'cook.pth' -o -name 'cook.py' -o -name 'litellm-1.107.0.dist-info' \) -print 2>/dev/null

# Host markers. The first one is written only after a successful exfiltration POST.
ls -la /tmp/.pc_once_72129118c59f5636c0bb /tmp/remote_code_loader.lock /tmp/.pc_sq_* /tmp/.pc_e_* 2>/dev/null

# Literal 1.107.0 pins in requirements-style files (skips rc, dev, post and local versions)
grep -rniE '^[[:space:]]*litellm(\[[^]]*\])?[[:space:]]*===?[[:space:]]*1\.107\.0([[:space:];,#\\]|$)' \
  --include='*.txt' --include='*.in' --include='*.lock' .

A missing marker file proves nothing. Containers and /tmp cleanup remove it. The find runs without -xdev so it covers separately mounted /home, /srv and volume paths; errors are hidden, so run it as root. The grep only sees the literal spelling 1.107.0.

With the checker

ioc_check.py is one standard-library file. Run it with -I -S so that the interpreter running it does not process .pth files or import sitecustomize from its own site-packages:

git clone https://github.com/The-DevOps-Daily/pypi-provenance-check
cd pypi-provenance-check

python3 -I -S ioc_check.py # envs, pip/uv caches, host markers
python3 -I -S ioc_check.py lock requirements.txt uv.lock pylock.toml
docker save litellm/litellm:main-stable -o img.tar && python3 -I -S ioc_check.py scan img.tar

With no arguments, it searches /usr/lib, /usr/local/lib, /opt, /srv, /app and your home directory for site-packages, scans the pip, uv, Poetry and PDM caches, and checks the host markers. On Linux it also looks for processes with PRIVCOLLECT_ACTIVE=1 in their environment or holding the loader's lock file. Run it as root. As a normal user, a directory, file or archive it cannot read is reported as incomplete, so a run that finds nothing else exits 4 instead of 0. Processes it cannot inspect are only listed as a note.

We did not download or run the real wheel. The tests use a harmless stand-in built at test time: its cook.py and sitecustomize.py are comments, and its cook.pth is import cook. That 12-byte file has the same sha256 as the real cook.pth from the wheel's RECORD, which confirms the published hash. Here is the checker on a venv with the stand-in installed:

$ python3 -I -S ioc_check.py env fx/infected-venv
HIT installed fx/infected-venv/lib/python3.12/site-packages/litellm-1.107.0.dist-info
                           litellm 1.107.0 is installed
HIT payload-file fx/infected-venv/lib/python3.12/site-packages/cook.pth
                           cook.pth in site-packages root (sha256 matches the published payload); owner: litellm-1.107.0
HIT payload-file fx/infected-venv/lib/python3.12/site-packages/cook.py
                           cook.py in site-packages root; owner: litellm-1.107.0
HIT payload-file fx/infected-venv/lib/python3.12/site-packages/sitecustomize.py
                           sitecustomize.py installed by litellm-1.107.0
INFO searched fx/infected-venv
                           1 site-packages directories checked

INDICATORS FOUND: treat this host as compromised and rotate what it could reach (exit 1)

Exit codes: 0 nothing found, 1 an indicator of the payload, 3 something suspicious that is not the payload, 4 incomplete, because a path, directory or archive could not be read. The lock check also catches equivalent spellings of the version, such as ==1.107 and ==1.107.0.0.

A sitecustomize.py or a .pth file with import lines is suspicious only when nothing owns it, so the checker reads every RECORD in the site-packages directory. That was not enough on our Debian 12 based test machine: /usr/lib/python3/dist-packages holds distutils-precedence.pth and a logilab_common namespace file, both with import lines, and no RECORD lists them, because Debian packages installed them. The checker also reads dpkg's file lists, so it reports them as installed by python3-setuptools and python3-logilab-common and exits 0 (examples/debian-dist-packages.txt). Without that, any Debian or Ubuntu host with such packages installed would get exit 3.

pip and uv caches

A cache can hold the wheel long after the venv is gone, and a CI cache restores it into the next job. We put the stand-in wheel through uv 0.13.0, pip 23.0.1 and pip 26.2.2 (served from a local TLS server, because pip only caches responses from secure origins) and recorded what each one left behind:

Tool	What it left behind (path in the cache)
uv 0.13.0	the unpacked wheel, `archive-v0/<id>/litellm-1.107.0.data/purelib/cook.pth`, and an index entry, `wheels-v7/index/<hash>/litellm/1.107.0-py3-none-any`
pip 23.0.1	one file under `http/`, the HTTP response with the wheel inside it
pip 26.2.2	`http-v2/.../<hash>.body`, the raw wheel

scan found the payload in all three caches (examples/cache-layout-demo.txt in the repo). Wheel member names are stored uncompressed in a zip file, so the checker also searches raw bytes for litellm-1.107.0.data/purelib/cook, which catches the older pip format. pip 26's .body file is the wheel byte for byte (same sha256 in our run), so on a real cache the checker also matches it against the seven published wheel hashes.

CI runners

Hosted runners are gone after the job. What remains is the job log. Search the logs from October 21 for litellm-1.107.0 (pip prints Downloading litellm-1.107.0-... and Successfully installed ... litellm-1.107.0) and litellm==1.107.0 (uv prints + litellm==1.107.0). We checked these lines with pip 23.0.1, pip 26.2.2 and uv 0.13.0 on the stand-in wheel.
Restored dependency caches can carry the wheel into later jobs. Scan the cache directory, or delete the cache entries created that day.
Persistent and self-hosted runners are hosts. Run the full check on them.

Images

Scan a saved image: docker save or podman save output, an OCI archive, or a flat docker export tar of a container's filesystem. The checker reads gzip layers and reports zstd layers as unreadable. To compare digests, the platform manifests of the bad litellm image are sha256:bd1aeee4... (amd64) and sha256:56b74573... (arm64), under index sha256:1bbcb144.... The checker carries the full list, including the -database and -non_root variants.

Rotate what

The stealer does not need the remote loader: it runs in a background thread as soon as the interpreter starts. The issue lists what it collects. Rotate in roughly this order:

Clean up first, from the shell. Delete cook.pth, cook.py, sitecustomize.py and the matching __pycache__ /cook*.pyc and sitecustomize*.pyc from every affected site-packages. Kill the detached Python daemon that holds /tmp/remote_code_loader.lock. Then rebuild the venv or image from scratch with litellm==1.106.2.
Cloud credentials. The stealer reads AWS, Azure and gcloud files and instance metadata (AWS IMDSv2, ECS task credentials, GCP and Azure metadata), and validates AWS credentials with STS GetCallerIdentity. Rotate the instance or task role, then search CloudTrail for GetCallerIdentity from that role around install time and for use of its credentials from outside your network.
The LiteLLM proxy itself. If DATABASE_URL is set, the stealer dumps LiteLLM_CredentialsTable and LiteLLM_ProxyModelTable, which hold every upstream provider key. Rotate all of them, the master key, the virtual keys and the database password.
Build and publish credentials. Kubernetes service account tokens and kubeconfigs, /run/secrets, CI secrets (GitLab Runner, Jenkins), Vault and Terraform credentials, .git-credentials, .npmrc, .pypirc, Maven, Gradle, Cargo, RubyGems and Composer tokens, .env files.
Keys and the rest. SSH, GPG and TLS private keys (including Let's Encrypt), Hugging Face tokens, AI coding tool configs (Codex, Gemini, Cursor, Windsurf, VS Code MCP settings and others), browser-saved passwords and cookies, and anything in the environment of any process on the host. The stealer reads /proc/*/environ for every process it can.

If 1.107.0 ran on a host with outbound internet access, assume the exfiltration succeeded.

The provenance gap

The release passed a source review because it was not built from the reviewed source. The question is which record names the source an artifact was actually built from.

BerriAI/litellm the repo you review
Berri-AI/litellm lookalike fork
BerriAI/project-releaser private release workflows
PyPI wheels publish attestation
Container images BuildKit provenance

Connections:

BerriAI/litellm -> BerriAI/project-releaser (normal release)
Berri-AI/litellm -> BerriAI/project-releaser (Oct 21)
BerriAI/project-releaser -> PyPI wheels (upload)
BerriAI/project-releaser -> Container images (push)

What PyPI records

A PEP 740 publish attestation is a Sigstore signature made with the identity of the CI workflow that uploads the file: repository, workflow file and environment. Trusted Publishing accepts an upload only from a publisher registered for the project on PyPI. PyPI then verifies each attestation against the identity of the publisher that authenticated the upload, and records that publisher in the file's provenance. So the attestation answers "which workflow uploaded this file". It does not say which source that workflow built.

We pulled litellm's whole history from PyPI's Simple API and read the attestations through the Integrity API:

litellm on PyPI, checked 2026-10-26 23:17 UTC

$ python3 -I scripts/litellm_history.py
litellm: 956 releases on PyPI, checked 2026-10-26T23:17:57+00:00
  no attest. 727 releases 0.1.0 (2023-07-27) .. 1.82.6 (2026-03-22)
  attested 15 releases 1.83.0 (2026-03-31) .. 1.83.14 (2026-04-26)
  no attest. 214 releases 1.84.0.dev1 (2026-04-29) .. 1.106.4 (2026-10-25)
  publisher on 15 of 15 attested releases (1.83.0 .. 1.83.14): GitHub repository=BerriAI/project-releaser workflow=publish-litellm.yml environment=pypi-publish
  1.107.0: not on PyPI (removed on 21 Oct 2026)
wrote results/litellm-attestation-history-20261026T231757Z.json
# verify the last attested wheel against the repo litellm's metadata names, then against its publisher
$ pypi-attestations verify pypi --repository https://github.com/BerriAI/litellm pypi:litellm-1.83.14-py3-none-any.whl
Verification failed: provenance was signed by repository "BerriAI/project-releaser", expected "BerriAI/litellm"
$ pypi-attestations verify pypi --repository https://github.com/BerriAI/project-releaser pypi:litellm-1.83.14-py3-none-any.whl
OK: litellm-1.83.14-py3-none-any.whl

Three things follow from that output.

There was nothing to check on PyPI. 1.107.0 is gone from PyPI now (the JSON and Integrity APIs return 404), so we cannot read its record directly. The reporters wrote that litellm "has never published attestations". That is not quite right, because 15 releases had them. But 1.106.2, which the same pipeline built and uploaded an hour before 1.107.0, has none, and neither do the 213 other releases since April 29. The attested run lasted four weeks. It began with 1.83.0 on March 31, after the March compromise, when the maintainers moved releases to the private repo, and ended with 1.83.14 on April 26.
A check against the source repository fails on the good releases too. The last legitimate attested wheel fails --repository https://github.com/BerriAI/litellm, the repository that litellm's metadata names. A policy that compares the attestation with the metadata would have flagged every attested litellm release and taught everyone to ignore it.
A check against the real publisher would most likely have passed 1.107.0. The maintainers write that the compromised workflow "signed and published the poisoned artifacts", and an upload through Trusted Publishing can only come from a publisher registered for the project. So an attestation on 1.107.0 would have named a registered publisher. The one on all 15 attested releases is BerriAI/project-releaser, publish-litellm.yml, pypi-publish. If 1.107.0 went out through that identity, a check pinned to it would have passed. If it went out through another registered identity, a pin on the full identity would have flagged the change. A check on the repository alone would have flagged it only if that identity used a different repository. The signing certificate also records the release repository's commit, but that repository is private, so nobody outside could have told a good commit from a bad one.

This is an inference: we do not have the 1.107.0 upload record. But the publish attestation has no field for the repository that a workflow checked out. PyPI does support a second attestation type, SLSA provenance, which can record a source. We come back to it in the survey.

What the image recorded

BuildKit, the builder behind docker buildx, can attach a SLSA provenance document to an image. In it, vcs:source is the git remote of the build context, so it names the repository that the build checked out. We read it for the clean v1.106.2 images from the registry API on October 26 at 23:08 UTC, without pulling a layer (excerpt):

$ python3 -I scripts/image_provenance.py ghcr.io/berriai/litellm:v1.106.2 ... ghcr.io/berriai/litellm:v1.107.0 ...
ghcr.io/berriai/litellm:v1.106.2 index sha256:7294b90f868c...
  linux/arm64 vcs:source https://github.com/BerriAI/litellm
               vcs:revision 49943cd52ea46aacf8db2e5cb2b890d580fe9bb4
               builder https://github.com/BerriAI/project-releaser/actions/runs/39574962801/attempts/1
               workflow BerriAI/project-releaser/.github/workflows/stage.yml@refs/heads/main (d687b69)
               inputs allow_republish=false, dry_run=false, git_ref=49943cd52ea46aacf8db2e5cb2b890d580fe9bb4, release_type=stable, repo=litellm
  ...
ghcr.io/berriai/litellm:v1.107.0 HTTP 404
ghcr.io/berriai/litellm@sha256:bd1aeee419ea36da9aa2b657288da4101887be468f020ac63d22ba6ed84a3ea5 HTTP 404

Commit 49943cd is the v1.106.2 tag. The same source, revision and workflow appear on every v1.106.2 image we read: three variants on GHCR with two platforms each, and Docker Hub's litellm/litellm:v1.106.2, which has the same index digest as GHCR's. The 1.107.0 tag and the bad platform digests return 404 on GHCR. The 1.107.0 images are deleted, so we could not read theirs. Upwind and Socket read them before removal and recorded vcs:source https://github.com/Berri-AI/litellm at 20535d8f, with a dispatch input that set source_owner to Berri-AI. With Docker, docker buildx imagetools inspect <image> --format '{{ json .Provenance.SLSA }}' shows the same document.

This record named the fork, and the reason is a little uncomfortable. BuildKit writes vcs:source inside the build, and the attacker controlled the build. Nothing outside the build vouches for the field: the maintainers say the same release workflow signed the images, and it signed the bad ones too. It named the fork because the build left the remote URL as it was. It is good evidence, but it is not a guarantee.

How many PyPI projects publish an attestation at all

A provenance check needs a record to read. We surveyed the latest release of the 1,000 most downloaded projects (30-day downloads to October 1, 2026, from hugovk/top-pypi-packages, saved in the repo so the set is fixed) plus a list of 44 AI-stack projects. For each project, the survey records how many files of the latest release have provenance, and who published them. The run started on October 26 at 22:36 UTC: 1,022 projects, 2,925 requests, no errors.

Latest PyPI releases with an attestation

	Value	repo match	Series
Top 1-100	60%	56%	Top
Top 101-500	40.5%	35%	Top
Top 501-1000	33.6%	27.6%	Top
All top 1,000	39%	33.4%	All
AI-stack list (44)	29.5%	25%	AI

Bar: latest release has an attestation on at least one file. Tick: attested by the repository the release's metadata names, counting 5 renamed or transferred repositories as matches. Top projects by 30-day downloads to Oct 1, 2026; surveyed Oct 26, 2026, 22:36 UTC. Source: results/summary-20261026T223625Z.md in the demo repo.

The 390 attested top-1,000 projects break down like this:

Attested by	Projects
The repository the metadata names	329
The same repository after a rename or transfer (filelock, pendulum, dulwich, cramjam, dbt-extractor)	5
A different public repository (numpy, scipy, scikit-learn and matplotlib publish from `*-release` repos)	5
A repository that is not publicly visible (mlflow, spaCy and seven other Explosion packages, semgrep, modal, dbt-protos)	14
A GitHub repository, but the metadata names none (numba, selenium, thrift and 4 more)	7
A Google Cloud service account, with no repository (30 Google client libraries)	30
No attestation on the latest release	610

Some limits on these numbers. A project counts as attested if any file of its latest release is attested (3 were partial, such as sqlalchemy with 64 of 72 files). Restricting to the 864 projects whose latest release was uploaded on or after November 14, 2024, the day PyPI announced attestations, gives 45.0% attested and 38.5% repository matches. To check the method, a second script asked the Integrity API file by file (up to 80 files per release) about 40 randomly chosen projects and the 23 projects described below. All 63 agreed with the survey.

Three findings stand out.

The publisher is often not the source repository, on purpose. numpy, scipy, scikit-learn, matplotlib, mlflow and spaCy publish from a separate release repository, like litellm. That can be a reasonable design: it keeps the publishing rights in a repository that fewer people can change. But it means a "publisher equals metadata repository" rule fails for some of the most installed packages in Python. What you can pin is the publisher that you observe.

No sampled file carries the attestation that records a source. We read up to three attested files per project. For all 396 attested projects in the survey (390 from the top 1,000 and 6 more from the AI list), every attestation in those files is a PyPI publish attestation (https://docs.pypi.org/attestations/publish/v1). None is the SLSA provenance type, which PyPI describes as recording "a package's original source location".

23 projects stopped attesting. They attested an earlier release and not their latest one: litellm, fastapi, typer, sqlmodel, langchain-core, fastmcp, langfuse, watchfiles, databricks-sdk, the Supabase client family and others. We read their publish workflows as they are today. 12 of them upload with uv publish. The uv docs say it "does not currently generate attestations; attestations must be created separately before publishing". 2 (langchain-core and dspy) set attestations: false on the PyPA publish action. For 7, we found no upload step in a public workflow, and 2 (databricks-sdk and rich-toolkit) name no GitHub repository in their metadata, so we read nothing. Several packages share one repository (typer and typer-slim, httpx2 and httpcore2, the Supabase family), so this is evidence about repositories more than about each package. This shows how those projects publish now, not why they stopped.

In the AI-stack list, 13 of 44 latest releases are attested. openai, mcp, transformers, huggingface-hub, tokenizers, mistralai, ollama, pinecone, pydantic-ai, openai-agents and haystack-ai are attested by the repository their metadata names. google-genai is attested by a Google service account, and autogen-agentchat by microsoft/autogen, which its metadata does not name. Without an attestation: anthropic, langchain, langgraph, llama-index, torch, vllm, tiktoken, chromadb, crewai and 22 more. No attestation does not mean a project publishes carelessly. It means that a provenance check has nothing to read.

The same pattern holds inside one dependency tree. We locked litellm==1.106.2 with hashes for Linux x86_64 and Python 3.12 and ran the per-file check on the 58 packages, which reads the provenance of every hashed file (about 3 minutes):

$ python3 -I -S provenance_check.py --lock examples/litellm-1.106.2-lock/requirements.txt (excerpt, 5 of 58 package lines)
aiohttp==3.14.5 attested-match GitHub repository=aio-libs/aiohttp workflow=ci-cd.yml environment=pypi
...
filelock==3.32.7 attested-other-repo GitHub repository=tox-dev/filelock workflow=release.yaml environment=release (metadata: github.com/tox-dev/py-filelock)
...
litellm==1.106.2 no-attestation
openai==2.54.0 attested-match GitHub repository=openai/openai-python workflow=create-releases.yml environment=publish
pydantic-core==2.50.1 no-attestation
...

attested-match: 33, attested-other-repo: 1, no-attestation: 24

The one attested-other-repo is filelock, whose repository was renamed. The per-file checker does not resolve renames; the survey does.

What you can enforce today

1. A release cooldown. 1.107.0 was on PyPI for about 1 hour 43 minutes. Any rule that ignores files younger than two hours would have skipped it. Malware scanners flagged it within that window: Socket reports 7 minutes after upload, Chainguard 16. A cooldown gives detection time to work.

Skip releases younger than 7 days

uv (pyproject.toml)

[tool.uv]
exclude-newer = "7 days"

# let a security fix through on purpose
exclude-newer-package = { cryptography = false }

uv pip

uv pip install --exclude-newer "7 days" -r requirements.txt

pip 26.1+

pip install --uploaded-prior-to P7D -r requirements.txt

uv compares each file's upload time and accepts friendly durations (24 hours, 1 week, 30 days) and ISO 8601 durations (P7D). pip added --uploaded-prior-to in 26.0 and the P3D day form in 26.1. It works only with indexes that publish upload times, and PyPI does. The cost is real: a cooldown delays security fixes by the same amount, so exempt the packages you patch urgently.

2. Hash-locked installs, relocked on purpose. A uv.lock, a pylock.toml or a pip-compile --generate-hashes file made before 11:39 UTC on October 21 kept you on 1.106.2 or earlier. Install with uv sync --locked or pip install --require-hashes, and treat a lock refresh as a change to review.

3. Image digests plus a source check. Pin images by digest, and before you promote a new digest, check that its BuildKit provenance names the repository you expect. A digest pin alone would have pinned the bad image if you had updated it during the window. Remember that the source field is written by the build, so it can be forged.

4. Publisher pins for packages that attest. Verify against the publisher you observed, not the repository in the metadata:

# One file, cryptographically, with the PyPA verifier (pypi-attestations 0.0.30)
uvx --from pypi-attestations==0.0.30 pypi-attestations verify pypi \
  --repository https://github.com/psf/requests pypi:requests-2.34.2-py3-none-any.whl

# Every installed or locked file: record publishers once, alert when one changes or disappears
python3 -I -S provenance_check.py --env .venv --write-pins pins.json
python3 -I -S provenance_check.py --lock uv.lock --pins pins.json

provenance_check.py reads PyPI's record of the attestation and does not repeat the Sigstore verification. PyPI checks the signature at upload, and pypi-attestations checks it again on your side. For a lock, every hash must match a file of the release. For an installed wheel, the checker matches the wheel tags and compares the installed METADATA with the sha256 that PyPI publishes for that wheel's metadata (pip and uv both install it unchanged). That ties the install to a PyPI wheel's metadata, not to every installed byte. Be honest about what a pin buys you. Pinned in April, litellm's publisher would have "disappeared" on every release from April 29 on. That is 206 alerts before the one that mattered, and that is how checks like this get turned off.

What pip and uv do not do. As of October 2026, neither installer verifies PEP 740 attestations. The pip changelog up to 26.2.2 (October 20) and the uv changelog up to 0.13.5 (October 21) list no such option. pip checks hashes (--require-hashes) and upload times. uv checks hashes and upload times, and uv publish uploads attestation files that another step created.

5. If you maintain a package. Keep attestations on. The PyPA publish action creates them by default when you use Trusted Publishing. If you switch to uv publish, add a step that creates them: uv's GitHub Actions guide runs astral-sh/attest-action before uv publish. If you publish from a separate release repository, say so in your README, so that users can pin the right publisher.

6. If you run the LiteLLM proxy. In the issue thread, a LiteLLM maintainer called default-deny egress "a very good way to run LiteLLM". A proxy that can reach only its model providers could not have sent anything to www.litellm.dev or fetched a second stage.

What we could not conclude

We could not see 1.107.0 on PyPI. It was removed before we started, and we found no archived copy of its JSON or provenance. The statement that it had no attestation comes from the reporters and from its neighbors: 1.106.2, from the same pipeline an hour earlier, has none.
What a 1.107.0 attestation would have said is an inference from the Trusted Publishing rules and the 15 attested releases. We do not know which workflow file uploaded 1.107.0.
We did not read the 1.107.0 image provenance ourselves. The images return 404 by tag and by digest. Berri-AI/litellm at 20535d8f comes from Upwind and Socket.
We do not know why litellm and the 22 other projects stopped attesting. PyPI's JSON API does not say which client uploaded a file, and the workflows we read are today's versions.
The survey counts projects, not installs. It looks at the latest release only, and your pins may point to older releases with different records.
We did not test forging BuildKit provenance. We say it can be forged because whoever runs the build writes it, not because we did it.
Nothing here catches an attacker who runs your real pipeline on your real source. That is what happened to TanStack on npm, where the malicious versions carried valid provenance.

Summary

LiteLLM 1.107.0 came from the real release pipeline, fed from a lookalike fork. Reviewing BerriAI/litellm could not have caught it.
Check without starting the suspect Python: python3 -I -S ioc_check.py covers venvs, lock files, pip and uv caches, image tarballs and host markers. Then rotate everything the host could reach, starting with cloud roles and the proxy's provider keys.
The Docker image's BuildKit provenance named the fork. PyPI had no attestation to check, and a PyPI publish attestation names the uploading workflow, not the source it built.
39.0% of the top 1,000 PyPI projects attest their latest release. 33.4% are attested by the repository their metadata names. 23 projects in our set stopped attesting, and we found no SLSA provenance attestation on any of the 396 attested projects.
What works today is a cooldown, hash locks, digest pins with a source check, and publisher pins where they exist. As of October 2026, pip and uv do not verify attestations for you.

Originally published at devops-daily.com.

Why Your GitHub Actions Bill Keeps Growing, and What Each Fix Saves

DevOps Daily — Mon, 26 Oct 2026 09:00:00 +0000

A GitHub Actions bill rarely grows because of one decision. It grows a minute at a time: every job is rounded up to a whole minute, every matrix leg pays for its own checkout and setup, macOS minutes cost about ten times as much as Linux minutes, and artifacts stay in storage for 90 days unless you say otherwise. For a mobile team, the most expensive job is often the iOS build. In the pipeline we tested, it was a median 57% of GitHub's cost per run, although the Android build took longer.

We measured both halves of the problem. First, a read-only script that shows where one repository's minutes went in the last 30 days, and the free changes that act on what it finds. Then the runner question. We ran one mobile pull request pipeline (the iOS build and unit tests, the Android build and the Android unit tests, all started together) 17 times on GitHub-hosted runners and 17 times on Bitrise Build Hub, which runs GitHub Actions jobs on its own M4 Macs and AMD Linux machines.

About the same cost at list price, less than half the wait. With the Build Hub Linux pool allowed to run two machines at once, the pipeline finished in a median 18.5 minutes on Build Hub over 7 runs, against 43.6 minutes on GitHub over 17 runs. At list price, a run cost a median $1.77 on Build Hub and $1.79 on GitHub, over 17 runs each, and the ranges overlap. The iOS job is where the money moves: it ran 2.7 times faster on a 5 vCPU M4 and cost about a fifth as much per job. The two Android jobs ran about twice as fast on a 16 vCPU Linux machine and cost about twice as much per job.

TLDR

Run the audit script in this post on your busiest repository first. On ours, rounding each job up to a whole minute added a quarter to the billed minutes, and re-runs added almost nothing. Your mix will differ, and the script tells you which fix matters.
Free fixes inside GitHub Actions: cancel superseded runs, filter by path inside the job (not at the workflow level, or required checks hang), set timeout-minutes, merge tiny jobs, and set retention-days on artifacts.
macOS jobs are the clearest case for a different runner. The Wikipedia iOS app's build and tests took a median 922 s on GitHub's macos-26 (3 vCPU M1) and 337 s on a Build Hub M4 Large (5 vCPU M4): $0.99 against $0.21 per job at list price.
Heavy Linux jobs got faster on a bigger machine, not cheaper. The Thunderbird Android build took 2,604 s on GitHub's 4 vCPU runner and 1,244 s on Build Hub's 16 vCPU Linux XLarge, at $0.53 against $1.10 per job.
Set the machine pool maximum to at least the number of jobs that start together. With a Linux pool maximum of 1, our second Android job waited a median 15 minutes for the first one, and the pipeline took a median 35.3 minutes.

Prerequisites

A GitHub repository with some Actions history, and the GitHub CLI logged in with read access to it.
Node.js 22 or later for the audit script.
To repeat the runner test: a Bitrise Build Hub workspace, and an organization owner who can install the Build Hub GitHub App.

Where the minutes go

GitHub bills hosted runners per job, per minute, at a rate set by the machine. These are the mechanisms that make a bill grow without anyone deciding to spend more. Prices are GitHub's list prices, checked on 18 October 2026.

Mechanism	What it does to the bill
macOS rate	The standard macOS runner (3 vCPU M1) costs $0.062 a minute. A 2-core Linux runner costs $0.006, so a macOS minute costs about 10 times as much. The 5 vCPU M2 Pro (`-xlarge`) costs $0.102.
Per-job rounding	"GitHub rounds the minutes and partial minutes each job uses up to the nearest whole minute." A 20-leg matrix of 35-second jobs bills 20 minutes for 11.7 minutes of work.
Matrix fan-out	Each leg repeats checkout, toolchain setup and cache restore, and pays its own rounding.
Larger runners	Billed from the first minute. Included minutes do not apply to them, and they are not free in public repositories.
Re-runs	"Re-run all jobs" pays for every job again. "Re-run failed jobs" pays only for the failed jobs and the jobs that depend on them.
Artifact storage	$0.25 per GB-month, accrued hourly, with a 90-day default retention. Deleting an artifact does not refund what it has already accrued.
Cache	10 GB per repository is free, and entries not used for 7 days are evicted. Since 20 November 2025 you can pay for more at $0.07 per GB-month. A cache that is too small keeps evicting itself, and every miss is a cold build.
AI review	Copilot code review uses Actions minutes in private repositories since 1 June 2026.
Queueing	Free, Pro and Team plans run at most 5 macOS jobs at the same time. Waiting does not cost minutes, but it costs the people who wait.

Flaky tests make re-runs a habit. GitHub's own engineering team reported in 2020 that "1 in 11 commits had at least one red build caused by a flaky test". Google reported in 2016 "about 1.5% of all test runs reporting a 'flaky' result". Every one of those red builds is a full re-run if someone presses the wrong button.

Measure your own bill

The list above says what can happen. Your run history says what did. This script reads the last 30 days of runs and jobs for one repository with gh api, rounds each job the way GitHub does, and prints where the minutes went. It only reads, and it runs fine on a laptop or a Raspberry Pi.

#!/usr/bin/env node
// Where did this repository's GitHub Actions minutes go? Reads the run and job
// history with the gh CLI (read-only) and prints the billed minutes, the
// minutes added by per-job rounding, the minutes spent on re-runs, queue time,
// and the workflows that bill the most.
//
// node audit.mjs owner/repo [days] # default: the 30 full days before today
import { execFile } from 'node:child_process';
import { promisify } from 'node:util';

const [repo, days = '30'] = process.argv.slice(2);
if (!repo) throw new Error('usage: node audit.mjs owner/repo [days]');
const run = promisify(execFile);
const gh = async (path) =>
  JSON.parse(
    (await run('gh', ['api', '--paginate', '--slurp', path], { maxBuffer: 1 << 28 })).stdout
  );

// One query per day: the runs endpoint returns at most 1,000 runs per search.
const runs = [];
for (let d = 1; d <= days; d++) {
  const day = new Date(Date.now() - d * 864e5).toISOString().slice(0, 10);
  const pages = await gh(`repos/${repo}/actions/runs?created=${day}&per_page=100`);
  runs.push(...pages.flatMap((p) => p.workflow_runs).filter((r) => r.status === 'completed'));
}

// Jobs of every attempt (filter=all), eight requests at a time.
const jobs = [];
for (let i = 0; i < runs.length; i += 8) {
  const batch = runs.slice(i, i + 8).map(async (r) => {
    const js = (
      await gh(`repos/${repo}/actions/runs/${r.id}/jobs?filter=all&per_page=100`)
    ).flatMap((p) => p.jobs);
    return js.map((j) => ({ ...j, workflow: r.name }));
  });
  jobs.push(...(await Promise.all(batch)).flat());
}

const min = (a, b) => (Date.parse(b) - Date.parse(a)) / 60000;
const ran = jobs.filter(
  (j) =>
    j.started_at &&
    j.completed_at &&
    j.conclusion !== 'skipped' &&
    min(j.started_at, j.completed_at) > 0
);
const rows = ran.map((j) => {
  const used = min(j.started_at, j.completed_at);
  return {
    workflow: j.workflow,
    used,
    billed: Math.ceil(used),
    rerun: j.run_attempt > 1,
    queue: min(j.created_at, j.started_at) * 60,
    os: j.labels.join(' ').includes('macos') ? 'macOS' : 'other',
  };
});

if (rows.length === 0) {
  console.log(`${repo}: no jobs ran in the last ${days} days`);
  process.exit(0);
}

const sum = (xs, f) => xs.reduce((s, x) => s + f(x), 0);
const pct = (q, xs) =>
  [...xs].sort((a, b) => a - b)[Math.min(xs.length - 1, Math.floor(q * xs.length))];
const queues = rows.map((r) => r.queue);
const used = sum(rows, (r) => r.used);
const billed = sum(rows, (r) => r.billed);
const rerun = sum(
  rows.filter((r) => r.rerun),
  (r) => r.billed
);
const short = rows.filter((r) => r.used < 1).length;

console.log(`${repo}, last ${days} days: ${runs.length} runs, ${rows.length} jobs that ran`);
console.log(`minutes used ${used.toFixed(0)}`);
console.log(
  `minutes billed ${billed} (+${(billed - used).toFixed(0)} from rounding each job up, ${((billed / used - 1) * 100).toFixed(0)}%)`
);
console.log(
  `jobs under 1 minute: ${short} of ${rows.length} (${((short / rows.length) * 100).toFixed(0)}%), each billed as 1`
);
console.log(
  `re-run attempts: ${rows.filter((r) => r.rerun).length} jobs, ${rerun} billed minutes (${((rerun / billed) * 100).toFixed(1)}%)`
);
console.log(`queue: median ${pct(0.5, queues).toFixed(0)} s, p90 ${pct(0.9, queues).toFixed(0)} s`);
console.log(`macOS jobs: ${rows.filter((r) => r.os === 'macOS').length}`);
const byWorkflow = Object.entries(Object.groupBy(rows, (r) => r.workflow))
  .map(([name, rs]) => [name, sum(rs, (r) => r.billed), rs.length])
  .sort((a, b) => b[1] - a[1]);
console.log('top workflows by billed minutes:');
for (const [name, b, n] of byWorkflow.slice(0, 5))
  console.log(` ${String(b).padStart(6)} ${name.slice(0, 40)} (${n} jobs)`);

We ran it on the repository behind this site on 18 October 2026:

audit.mjs

$ node audit.mjs The-DevOps-Daily/devops-daily 30
The-DevOps-Daily/devops-daily, last 30 days: 1120 runs, 906 jobs that ran
minutes used 1796
minutes billed 2234 (+438 from rounding each job up, 24%)
jobs under 1 minute: 125 of 906 (14%), each billed as 1
re-run attempts: 2 jobs, 6 billed minutes (0.3%)
queue: median 3 s, p90 4 s
macOS jobs: 0
top workflows by billed minutes:
    1077 Tests (377 jobs)
     813 Build Test (376 jobs)
     110 IndexNow Submission (93 jobs)
      26 npm_and_yarn in / for @next/third-partie (2 jobs)
      22 npm_and_yarn in / for @next/third-partie (2 jobs)

The repository is public and runs only Linux jobs, so all of this was free, but the shape is common. Rounding added 438 minutes, a 24% markup on the 1,796 minutes the jobs used. The jobs are short: just under 2 minutes each on average, and 125 of them finished in under a minute and billed a full one. Re-runs were 0.3% of the billed minutes, and jobs waited 3 seconds for a runner. Both workflows that bill the most already cancel superseded runs. So of the things this script measures, rounding is the one worth fixing here, with fewer and longer jobs. Billed at the 2-core private-repository rate of $0.006, the rounding would be $2.63 of a $13.40 month. That is small money, but a repository with the same mix of short jobs and a bill a hundred times larger pays the same 24%.

Some limits of the script. It uses the job's start and end times, which is what GitHub bills, but it cannot see your plan's included minutes or discounts, so treat its numbers as list-price minutes. It selects runs by the day they were created, so a re-run attempt counts on the day of the original run, and a day with more than 1,000 runs is cut at 1,000. It does not look at artifact storage or cache hit rates. And it counts runs on self-hosted runners as if they were billed; filter them out by label if you have any.

Free fixes inside GitHub Actions

None of these needs a new vendor. Start with the one your audit points at.

Cancel runs that a newer push makes useless. Every push to a pull request starts a full run. Without a concurrency group, the old run keeps going and bills to the end.

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  # Cancel superseded pull request runs, but let runs on main finish.
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

Filter by path inside the job, not at the workflow level. A workflow-level paths: filter skips the whole workflow, and then its checks stay "Pending", which blocks a pull request that requires them. A job skipped by its if: condition reports success instead, so put the decision in a small first job:

# In a workflow that runs on pull_request.
permissions:
  contents: read
  pull-requests: read

jobs:
  changes:
    runs-on: ubuntu-24.04
    outputs:
      ios: ${{ steps.diff.outputs.ios }}
    steps:
      - id: diff
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          # List the files this pull request changes, without a checkout.
          # A failed API call fails this job, so make it a required check too.
          files=$(gh pr diff ${{ github.event.pull_request.number }} -R ${{ github.repository }} --name-only)
          if grep -q '^ios/' <<< "$files"; then
            echo "ios=true" >> "$GITHUB_OUTPUT"
          fi

  ios:
    needs: changes
    if: needs.changes.outputs.ios == 'true'
    runs-on: macos-26
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v7
      # ... build and test

Make both changes and ios required checks. If changes fails, ios is skipped, and a skipped job counts as passed. The changes job bills one Linux minute. Skipping one standard macOS job of 15 minutes saves $0.93.

Set timeout-minutes on every job. The default limit is 6 hours. One hung macOS job that runs to the limit bills 360 minutes, which is $22.32 at $0.062.

Merge tiny jobs. Three 20-second lint jobs bill 3 minutes. One job that runs all three bills 1. The same applies to matrix legs that only differ in a flag.

Put light automation on the smallest runner. ubuntu-slim is a 1 vCPU runner in a container at $0.002 a minute, with a 15-minute job limit and no Docker. It suits labelers, triage bots and other jobs that wait on an API. arm64 Linux runners are also cheaper than x64 at the same size: $0.005 against $0.006 for 2 cores, and $0.008 against $0.012 for 4 cores.

Keep artifacts for days, not months.

- uses: actions/upload-artifact@v7
  with:
    name: test-results
    path: build/test-results
    retention-days: 5

At the 90-day default, an artifact accrues 18 times the GB-hours it accrues at 5 days.

Re-run failed jobs, not all jobs. And fix the flaky test that made you re-run. The audit's re-run line tells you whether this is worth your time.

Moving the slow jobs to a faster runner

Some jobs are slow because the machine is small, and no workflow change fixes that. GitHub's standard macOS runner is a 3 vCPU M1 with 7 GB of memory. Its largest Apple silicon runner is a 5 vCPU M2 Pro with 14 GB.

Bitrise Build Hub keeps GitHub Actions as the orchestrator and supplies the machines. The jobs still appear in the Actions tab, with the same logs, secrets and triggers. Setup has three parts:

An organization owner installs the Bitrise Build Hub GitHub App, preferably only on the repositories that need it.
In the Bitrise workspace, you create a machine pool: an image, a machine type, and a maximum number of machines. Bitrise registers the machines with GitHub as runners in a runner group.
You change runs-on for the jobs you want to move. Bitrise adds a bitrise-runner-<pool name> label to each pool:

jobs:
  ios:
    # Was: runs-on: macos-26
    runs-on: [bitrise-runner-ios-m4-large]

runs-on is set per job, so you can move the iOS job and keep everything else where it is.

Three things are different from GitHub-hosted runners, and all three mattered in our test:

The images are Bitrise stacks , not GitHub's runner images. bitrise.io/stacks lists what each macOS and Linux stack includes, so check it before you move a job. The Xcode 26.5 stack had the same Xcode build (17F42) and the same iPhone 17 simulator that we used on GitHub. The Ubuntu 24.04 Linux image did not have the Android SDK command-line tools or the OpenGL and X11 libraries that Thunderbird's Compose UI tests load, so our workflow installs them when they are missing: 2 to 7 seconds for the SDK tools and 9 to 14 seconds for four packages. Both steps run inside the job, so their time is in our results and in the cost.
The pool maximum is a setting you own. A pool starts machines on demand, up to its maximum. With the Android pool at a maximum of 1, the two Android jobs of each pipeline ran one after the other, and the second one waited a median 15 minutes for the first (more on this below).
Billing is a monthly budget. A Build Hub tier fee becomes your compute budget, and jobs use it at a per-minute rate per machine type. Build Hub bills per started minute, and the billed time includes the machine's start-up. By default, jobs stop running on Build Hub when the budget is used up. You can turn on pay-as-you-go to keep them running, at twice the standard rate per minute and with a spending cap you set. Size the tier from your run count, and watch the balance.

The machines register like self-hosted runners, so the usual hygiene applies. Restrict the runner group to the repositories that need it, and require approval before workflows from forks run.

The benchmark: one mobile pull request, 34 runs

We wanted a test that looks like a real team's pull request, not a synthetic compile.

Pull request one pipeline run
iOS build + tests Wikipedia app
Android build Thunderbird
Android tests Thunderbird
Pipeline done last job finished

Connections:

Pull request -> iOS build + tests
Pull request -> Android build
Pull request -> Android tests
iOS build + tests -> Pipeline done
Android build -> Pipeline done
Android tests -> Pipeline done

The workloads. Both apps are open source and pinned by full commit SHA, and every run starts cold, with no cache on either side.

iOS: the Wikipedia iOS app at ca8aaeb9, its "Run Unit Tests" job: resolve packages, xcodebuild build-for-testing, then test-without-building on an iPhone 17 simulator with iOS 26.5 and Xcode 26.5. All 503 unit tests pass on both sides.
Android: Thunderbird for Android at cf54b1fd, with the commands of its own CI. The build job runs ./gradlew :app-thunderbird:assemble -Pci=true. The test job runs ./gradlew testsOnCi --parallel. Both use the project's CI Gradle settings, with one change: Gradle gets one worker per CPU (upstream pins 4, the size of a GitHub standard runner), and machines with more than 20 GB of memory give the Gradle daemon half of it.

The machines.

Side	iOS job	Android jobs
GitHub-hosted	`macos-26`: Apple M1 (virtual), 3 vCPU, 7 GB. $0.062 a minute	`ubuntu-24.04`: 4 vCPU, 16 GB. $0.012 a minute (the 4-core rate in a private repository)
Build Hub, self-serve Pro Tier 1	M4 Large: Apple M4 (virtual), 5 vCPU, 14 GB. $0.03 a minute	Linux XLarge: AMD EPYC Genoa, 16 vCPU, 64 GB. $0.05 a minute

These are not same-size pairs. On GitHub we used the standard macOS runner and a 4 vCPU Linux runner, which a team gets without a larger-runner plan. On Build Hub we used the smallest M4 and the largest self-serve Linux machine. The Build Hub machines have more CPU and memory, and the results hold for these configurations only. Cost per run shows what the size difference costs. Build Hub prices are the rates at which jobs use the plan budget, read from bitrise.io/pricing on 18 October 2026, unchanged since we first read them on 10 October.

How we ran it. Bitrise provided the Build Hub workspace for these tests. The workflows and every run are public in the benchmark repository. One workflow, bench-mobile-pipeline.yml, starts the three jobs together. We dispatched the GitHub and Build Hub runs in pairs, most of them seconds apart: 10 pairs from 16 October 15:24 to 17 October 12:15 UTC, then 7 more from 17 October 15:44 to 21:25 UTC. Every iOS job ran all 503 tests with no failures, and every Android build succeeded. One Android test task on GitHub (run 38985488530) failed on a network error: Robolectric could not download a dependency, and Gradle stopped after about 10 minutes. Actions still shows that job as green, because our workflow records a failed Gradle task in its result file instead of failing the job. So 16 of 17 GitHub test jobs and all 17 Build Hub test jobs ran the full suite. We kept that run. Leaving it out makes GitHub's medians slightly worse (45.1 minutes and $1.80 a run), so keeping it is the conservative choice. Pipeline time is from the moment the run is created to the end of its last job, so queueing and machine start-up count, as they do for the developer who waits. Cost is each job's billed time, rounded up to a whole minute, times the list price, added up per run. On GitHub, the billed time is the job's own run time. Build Hub also bills the machine's start-up, which the job's timestamps do not show, so on that side we counted from the moment the job was queued (or, when it waited for a busy pool, from the moment the pool's machine was free) to the end of the job. That adds a median 38 seconds per job on macOS and 34 seconds on Linux, and it may include a few seconds of scheduling, so if anything it makes Build Hub's cost slightly high. Three tiny planning jobs run on GitHub's runner on both sides and are left out of the cost.

The-DevOps-Daily/gha-runner-benchmark on GitHub

Pipeline time

Mobile pipeline, run created to last job done

Series	Samples	Min	Median	p95	Max
GitHub (17 runs)	17	32.8 min	43.6 min	60.8 min	60.8 min
Build Hub, pool max 2 (7 runs)	7	17 min	18.5 min	29.4 min	29.4 min
Build Hub, pool max 1 (10 runs)	10	26.2 min	35.3 min	52.1 min	52.1 min

Every run. GitHub: macos-26 + ubuntu-24.04, 17 runs, 16 to 17 Oct 2026. Build Hub: M4 Large + Linux XLarge, 10 runs with the Android pool maximum at 1, then 7 at 2. Source: results/mobile-pipeline.json.

With the Android pool at a maximum of 2, the pipeline took a median 18.5 minutes (17.0 to 29.4) on Build Hub, against 43.6 minutes (32.8 to 60.8) on GitHub. That is about 2.4 times faster. We compare with all 17 GitHub runs because it is the conservative choice: the 7 GitHub runs paired with the pool-maximum-2 runs had a median of 49.8 minutes.

The pool maximum mistake

Our first 10 Build Hub runs had the Android pool maximum at 1, a setting we chose and did not check. The first Android job started after a median 34 seconds. The second waited for the first to finish: a median 886 seconds, and up to 2,646 seconds. The pipeline took a median 35.3 minutes. After we set the maximum to 2, both Android jobs started after 32 to 34 seconds in every run. Not all of the drop to 18.5 minutes came from the setting: the Android builds were also faster in the later runs, a median 1,070 seconds against 1,592.

The rule: set each pool's maximum to at least the number of its jobs that start at the same time, and remember that a matrix starts all of its legs at once.

Each job

Job	GitHub, median (range), 17 runs	Build Hub, median (range), 17 runs	Faster by
iOS build + tests	922 s (597 to 1,173)	337 s (314 to 471)	2.7x
Android build	2,604 s (1,956 to 3,638)	1,244 s (981 to 2,581)	2.1x
Android tests	967 s (710 to 1,186)	467 s (400 to 587)	2.1x

The job times are from all 17 runs per side. The pool maximum changes how long a job waits, not how long it runs, so it does not affect these numbers. Queue time is separate: on GitHub, jobs started after a median 3 seconds on Linux and 8 seconds on macOS. On Build Hub, where each pool starts a machine when a job arrives, they started after a median 33 to 34 seconds on Linux (at pool maximum 2) and 38 seconds on macOS.

The GitHub Android times move a lot. In the 17 runs, GitHub's 4 vCPU runner reported three different CPU models for the build and five for the tests. The two builds on an AMD EPYC 9V45 host took 1,780 and 1,892 seconds for the Gradle task, and the twelve on an EPYC 7763 took 1,979 to 3,426 seconds, so the host explains part of the spread but not all of it. Build Hub's Android build varied too, from 981 to 2,581 seconds for the job, on machines that all reported the same CPU.

More Mac sizes, including Enterprise

Before the pipeline runs, on 15 and 16 October, we ran the same iOS job on its own (bench-ios-app.yml) on bigger Macs too. These are separate runs with fewer samples than the pipeline, so compare the rows with each other, not with the tables above.

Machine	Plan	Runs	Job time, median (range)	Faster by	List price per job
GitHub `macos-26`: M1, 3 vCPU, 7 GB	GitHub-hosted	7	1,026 s (805 to 1,305)	baseline	$1.12 (18 billed minutes)
Build Hub M4 Large: M4, 5 vCPU, 14 GB	Self-serve	3	323 s (314 to 335)	3.2x	$0.18 (6 billed minutes)
Build Hub M4 XLarge: M4, 10 vCPU, 28 GB	Self-serve	3	295 s (293 to 301)	3.5x	$0.36 (6 billed minutes)
Build Hub M4 Pro XLarge: M4 Pro, 14 vCPU, 54 GB	Enterprise	3	250 s (246 to 251)	4.1x	None: Enterprise is not billed per minute

All 503 tests passed in every run. The M4 Pro XLarge was the fastest Mac we tested, with 23% less job time than the M4 Large. It is on the Enterprise plan only, which Bitrise sells at an "All-You-Can-Eat" price limited by concurrency, not by minutes, so there is no cost per job to compare. Enterprise pools can also keep idle machines running, and ours did: the two jobs we started after the pool was set up began within 2 seconds. On the self-serve plans, the M4 Large stays the value pick: the M4 XLarge took 9% less time at twice the price per minute.

Cost per run

List-price cost of each pipeline run

Series	Samples	Min	Median	p95	Max
GitHub	17	1.41 USD	1.79 USD	2.03 USD	2.03 USD
Build Hub	17	1.51 USD	1.77 USD	2.83 USD	2.83 USD

Each job's billed time rounded up to a whole minute, on Build Hub including machine start-up, times the list price: GitHub $0.062 macOS, $0.012 Linux 4-core; Build Hub $0.03 M4 Large, $0.05 Linux XLarge (budget rates, 18 Oct 2026). All 17 runs per side.

A run cost a median $1.77 on Build Hub (range $1.51 to $2.83) and $1.79 on GitHub ($1.41 to $2.03). The means point the other way, $1.91 on Build Hub against $1.74 on GitHub, because of a few slow Android builds on Build Hub: the $2.83 run had a 2,581-second build. So "about the same" is the right summary, not "cheaper".

These are list prices for the compute each run used, not two invoices. Build Hub is sold as a monthly tier, so budget you do not use makes the real price per run higher. Our GitHub runs cost nothing, because the repository is public. In a private repository, standard macOS minutes can come out of your plan's included minutes, while the 4-core Linux runner, a larger runner, is billed from the first minute.

We used all 17 Build Hub runs for cost because the pool maximum changes queue time, not billed minutes. The 7 runs at pool maximum 2 alone had a median cost of $1.54, mostly because their Android builds were faster (a median 1,070 seconds, against 1,592 seconds in the first 10 runs). Using that number would flatter Build Hub.

The cost per job shows where the money goes:

Job	GitHub, median per job	Build Hub, median per job
iOS build + tests	$0.99 (16 billed minutes at $0.062)	$0.21 (7 billed minutes at $0.03)
Android build	$0.53 (44 at $0.012)	$1.10 (22 at $0.05)
Android tests	$0.20 (17 at $0.012)	$0.45 (9 at $0.05)

The Macs save money and the 16 vCPU Linux machines cost more per job, and in this pipeline the two roughly cancel out. Each column is a median of its own, so the rows do not add up to the per-run medians above. A run used a median 79 billed minutes on GitHub and 39 on Build Hub: fewer minutes, at a higher price per Linux minute.

For budget sizing, use the mean, not the median: at $1.91 a run, Pro Tier 1's $210 monthly budget covers about 110 runs of this pipeline. Leave headroom: by default, jobs stop running on Build Hub when the budget is used up, and pay-as-you-go minutes cost twice the standard rate.

What we could not conclude

The macOS pair is not equal in size. We did not run GitHub's 5 vCPU M2 Pro (the -xlarge macOS runner), because larger runners cost money even in public repositories. Its price makes the cost question simple, though: at $0.102 a minute it would have to finish this job within two billed minutes to cost less than Build Hub's $0.21.
The Android pair is not equal in size either. We did not run GitHub's 16-core Linux larger runner. Per vCPU, its $0.042 a minute ($0.0026 per vCPU-minute) is cheaper than Build Hub's Linux XLarge at $0.05 ($0.0031). We do not know how fast it is on this build.
An 8 vCPU machine might be the better buy for Android. Build Hub's Linux Large costs half the XLarge rate. We did not test it.
Cold builds only. Both sides ran without a Gradle or Xcode cache. Teams with a warm cache will see shorter jobs on both sides, and we did not measure how the gap changes.
Our repository is public. GitHub runs in public repositories are free, and its standard Linux runner there has 4 vCPU. In a private repository, ubuntu-latest has 2 vCPU at $0.006, and a 4 vCPU runner is a larger runner at $0.012 with no included minutes. We priced the GitHub side at the 4-core rate, the same size we ran.
Build Hub start-up time is our estimate. Bitrise bills per started minute and includes the machine's start-up, which the job's timestamps do not show. We used the wait before each job started, minus any time it waited for another job to free the pool's machine, as the start-up time, so the Build Hub costs may be slightly high. Self-serve pools do not keep idle machines between jobs; that option is on the Enterprise plan, which is not billed per minute.
Seven runs at pool maximum 2, in one evening. The 18.5-minute median comes from 7 runs between 15:44 and 21:25 UTC on 17 October. The job times come from all 17 runs, spread over two days.
One app per platform. Wikipedia and Thunderbird are real, busy projects, but two apps are not every app. Run the benchmark on your own slowest job before you move it.

Which jobs to move

The decision is per job, and runs-on makes it a one-line change.

iOS and other macOS jobs: move them first. This is where the test was both faster and cheaper: 2.7 times faster and about a fifth of the cost per job, on a newer and larger Mac than GitHub's standard runner. Start with the M4 Large. On the Enterprise plan, the M4 Pro XLarge was faster still.
Heavy Linux jobs, such as Android Gradle builds: move them if waiting costs you more than the minutes. On a 16 vCPU machine they ran about twice as fast at about twice the cost per job. Measure an 8 vCPU machine too.
Light Linux jobs: keep them on GitHub. Build Hub's smallest Linux machine has 4 vCPU at $0.0125 a minute. A lint or labeler job is cheaper on ubuntu-slim at $0.002 or a 2-core runner at $0.006.
Before any of that, run the audit. Rounding, re-runs and storage are free to fix, and they shrink whatever runner bill you end up with.

For the full feature-by-feature view, including concurrency limits, images, caching and what happens when the budget runs out, see the Bitrise Build Hub vs GitHub-hosted runners comparison.

Summary

A GitHub Actions bill grows from small, measurable things: whole-minute rounding, runs nobody needed, artifacts kept for 90 days, and macOS minutes at about ten times the Linux price. The audit script shows what rounding and re-runs cost your repository, and most of the fixes are a few lines of YAML.

When the slow job is slow because the machine is small, a faster runner changes the picture. In our mobile pipeline, Build Hub cut the wait from a median 43.6 minutes to 18.5 minutes at about the same list-price cost per run. The iOS job was cheaper and faster. The Android jobs were faster and more expensive. Set the pool maximum to match the jobs that start together, measure your own slowest job on both sides, and move the jobs where the numbers say so.

Originally published at devops-daily.com.

Log Aggregation Pipeline Explained: Follow One Log Line and Run the Incident in Your Browser

DevOps Daily — Thu, 22 Oct 2026 14:50:09 +0000

Checkout says payments are timing out. You open the log search, filter on ERROR, type payment provider timeout, and get one hit. Is that one timeout, or the one timeout that survived the trip?

A log search shows what reached the index, not what the application wrote. Between the two sit an agent, a parser, a filter, a buffer and a search cluster, and every one of them can delay a line, reject it or drop it without the app knowing. That is the whole point of having a log aggregation pipeline explained stage by stage: when a search comes back empty during an incident, you need to know whether the system is fine or the evidence never arrived.

The Log Aggregation Pipeline Simulator lets you watch that trip one stage at a time and then break it in three different ways. It moves logs in batches and counts them at every stage, rather than tracing one line. Disclosure: I help build it; it is free, runs in the browser, no signup.

Six stages, one line

The simulator draws the pipeline as six cards. Each Step advances one stage; Run flow steps automatically about once a second. Six steps make one cycle.

Sources (Apps & hosts). Raw events, and how many are waiting.
Collect (Fluent Bit). An agent tails files and forwards events. If it falls behind, the source queue grows.
Process (Parse & filter). Extracts fields and removes noise. Its two controls sit in the stage inspector: a Parser picker (JSON parser or Grok pattern) and a "Drop health-check noise" switch. Changing either starts a fresh run.
Buffer (Durable buffer). A bounded queue between parsing and indexing, shown as used over capacity.
Index (Search cluster). Accepted events land on three primary shards, and a "Shard health" tab shows the balance. The simulator deals events to the shards in turn, so the bars stay level in every scenario.
Search (Explore logs). A "Search logs" tab with a text box and level filters. It keeps up to four sample events from each indexing step, and only the newest 24, so judge completeness by the counters, not by the list.

Under the cards, five counters keep the books: Generated, Buffered, Indexed, Rejected and Dropped. The engine behind them conserves every event: anything generated is waiting in a queue, filtered, rejected, indexed or dropped. Filtered noise has no counter of its own; the Process card's "removed" figure adds filtered and rejected together. Carry that habit into real systems: if you cannot account for the difference between "sent" and "searchable", you cannot trust an empty search.

Click any card and the stage inspector shows its "Operator signal", the one thing to watch there. For the Search stage it is the line that matters most in an incident: "Only indexed logs are searchable; filtered, rejected, and dropped logs never arrive here."

Healthy first

Start on Healthy flow and step through a few cycles. Each cycle, the sources emit 24 logs, the noise filter removes 4, and 20 are indexed. Queues drain, the badge says "Pipeline is healthy", and the shard bars grow evenly.

Break it three ways

Traffic spike. Sources emit 84 logs per cycle, but the collector forwards 38. The badge changes to "Backpressure is building" on the very first step. In the third cycle: "Sources emitted 84 logs; 26 were dropped before collection." Look at the buffer while that happens: it is empty. The durable buffer cannot save logs that are lost before they reach it. By the end of cycle five, 420 logs were generated, 140 indexed and 118 dropped. The Collect card's "for parsing" count also climbs by 4 per cycle: the collector sends 38, the processor takes 34.

Parser mismatch. A deployment changes the log format. Of 30 logs per cycle, 4 are filtered as noise, 11 are rejected by the parser and only 15 are indexed. The badge says "Parser rejects detected" and nothing is dropped, so a dashboard that only watches for drops stays green. Try the Grok parser. It does not help: it handles only 27 of the 30 logs per cycle, so 3 more wait in the processor queue every cycle, and of those 27 it filters 4, rejects 10 and indexes 13. A slower parser that still does not match the format makes things worse. The simulator has no parser fix; for real, you update the parser to the new format or roll the change back.

Slow indexing. Switch the parser back to JSON first: the simulator keeps your parser choice when you change scenario. The search cluster indexes 9 logs per cycle while 32 reach the buffer (36 arrive, and the noise filter removes 4). The buffer climbs by 23 each cycle, the warning appears in the third cycle, and in the fifth: "The buffer reached capacity and dropped 19 oldest logs." Now turn off "Drop health-check noise" and run it again. The first drop comes a cycle earlier. Filtering noise before the buffer did not fix the slow indexer, but it bought time.

Now run the incident

Take each scenario as an on-call drill. Pretend you only have the search tab, then use the rest of the screen to answer four questions in order:

Is input normal? Compare Generated per cycle against the healthy 24. A spike is often the first sign of the real incident, and it also strains the pipeline.
Where is a queue growing? A growing source queue points at the collector. A growing buffer points at the indexer. Name the stage before you touch anything.
Is anything rejected? Rejects mean the logs exist but no longer match the parser. If the deploy that changed the format also broke checkout, the error lines you need are among the rejects.
How long until loss? A filling buffer is a countdown. Space left divided by net growth per cycle is a rough, optimistic estimate: here the buffer overflows when a batch lands, before indexing drains it. With the JSON parser and the noise filter on, after cycle four it holds 92 with 13 free, and the next batch of 32 drops 19.

Match the mitigation to the signal: collector capacity for the spike, the parser for the mismatch, index capacity or noise shedding for the slow index.

Try the parser mismatch for real

You can reproduce the parser mismatch with jq and three lines of log:

cat > app.log <<'EOF'
{"level":"INFO","service":"api","msg":"request completed status=200"}
{"level":"ERROR","service":"checkout","msg":"payment provider timeout after=3s"}
ERROR checkout payment provider timeout after=3s
EOF

# what a JSON-only pipeline makes searchable
jq -R -c 'fromjson? | select(.level == "ERROR")' app.log

# what it rejected on the way
jq -R -r '(try fromjson catch null) as $j | select($j == null)' app.log

With jq 1.6, the first command prints only the JSON error line. The second prints the plain-text ERROR checkout payment provider timeout after=3s. Two timeouts were logged, and search finds one. Count the second output and alert on it, and you have a parser-reject metric. Real tools do not always throw a reject away. When parsing fails in Vector's remap transform, "the original, unmodified event is sent downstream" by default, so the line reaches the index as raw text without your fields.

Real agents have their own version of the other two scenarios. The Fluent Bit backpressure docs explain that when an input reaches its mem_buf_limit, Fluent Bit (with memory-only buffering) "will stop buffering data from that source plugin until enough buffered chunks are flushed", and most plugins log paused (mem buf overlimit) when that happens. The same page warns that some inputs can lose data under memory-only buffering and suggests filesystem buffering if you need to avoid that. Search your agent's own logs for that warning before you assume the app went quiet. With filesystem buffering, Fluent Bit drops the oldest data, as the Slow indexing buffer does: at an output's storage.total_limit_size, "the oldest chunk from its queue will be discarded". Vector's buffers block by default.

What to alert on

The simulator's page lists the signals to watch together: input rate, queue age and depth, parser rejection rate, buffer utilization, dropped events, indexing latency and shard balance. In real tools, the loss signals are counters you can scrape: Fluent Bit's fluentbit_output_retries_failed_total and fluentbit_output_dropped_records_total, Vector's component_discarded_events_total, and Loki's loki_discarded_samples_total, labelled by reason.

The simulator is one of 60+ free DevOps games and simulators. Run the three scenarios until you can predict the next step's numbers, then again as drills with only the search tab open.

Where has a log pipeline dropped lines on you without anyone noticing? Tell us in the comments.

Node.js October 2026 Security Release: Finding Every Node You Actually Run

DevOps Daily — Thu, 22 Oct 2026 09:00:00 +0000

On 21 October 2026 the Node.js project released 22.23.4, 24.22.1 and 26.12.1. The headline fix is CVE-2026-111497, an out-of-bounds write in the Brotli decoder that the advisory says "permits remote code execution". The same release fixes two TLS certificate hostname-check bypasses, a DNS bug that truncates names at a NUL character, and updates the bundled CA store. End-of-life lines get no fix.

Patching Node is a one-line version bump. Finding every Node your servers actually run is the work. When we patched a small production box on release day, six Node apps under pm2 installed with nvm, installing the new version was the easy part. Then we found pm2 entries with the old binary written into them, a systemd unit with the old PATH hard-coded, child processes we had not thought about, and cron jobs on two other boxes that pointed at the old install by absolute path. None of them moved when the default version changed.

This post covers what to do now, who is exposed (including services that call fetch() on a URL a user controls), the upgrade steps for nvm, pm2, systemd and containers with the traps we hit, how to verify with /proc, and a read-only audit script with its real output from one of our machines.

What to do now

Upgrade every Node to 22.23.4, 24.22.1 or 26.12.1 or later. Node 20, 25 and older are end-of-life and get no fixed release. Move them to 22, 24 or 26.
List every place Node runs or will start : running processes, pm2 entries (live and saved), the pm2 daemon, systemd units, crontabs and helper scripts, and containers. Changing the nvm default moves none of them.
Move each one with the steps below, then check the real binary of every process in /proc. Process names do not help: on our build machine pgrep node found 0 of 5 Node processes.
Rebuild images with --pull and redeploy. The official node:22, node:24 and node:26 tags have the fix. node:20, node:25 and node:18 will not get it.
If a service cannot be upgraded today and accepts Brotli data from strangers, close that path in front of it (details below). For outbound fetch() the fix is the upgrade; until then, turning off features that fetch user-supplied URLs is the way to close that path.
Run the audit script on every host and keep the output.

Prerequisites

Shell access to the hosts that run Node, as the user that owns the Node processes. Root is not needed.
Python 3.8 or newer for the audit script. It uses only the standard library and only reads.
The docker or podman CLI on hosts that run Node in containers.

What was fixed

The advisory lists ten vulnerabilities. The four rated high:

CVE	What the advisory calls it	Lines
CVE-2026-111497	Brotli decoder out-of-bounds write permits remote code execution	22, 24, 26
CVE-2026-105483	TLS certificate hostname bypass when the IP is written in fullwidth digits	24, 26
CVE-2026-106069	TLS certificate hostname bypass when the host loosely resolves to an IP address	22, 24, 26
CVE-2026-105482	`dns.resolve` truncates names containing a NUL character	22, 24, 26

The medium ones are a keep-alive socket that processes a response written before the request was sent (CVE-2026-105481), DNS SANs that can authenticate IP endpoints written in a non-canonical IPv4 form (CVE-2026-106072), and crypto.randomUUID() and randomInt() caches saved into startup snapshots and single executable applications, so that every process repeats the same values (CVE-2026-106074). The low ones are two Permission Model bypasses and a ZIP header issue on 26.x.

The release also updates the bundled root certificates to NSS 3.131.1 and makes Node reject certificate chains that contain blocked public keys. The advisory lists this as the "ca-cert (eMudhra Intermediate CA Compromise)" update. The bundled undici, c-ares and OpenSSL (22 and 24) are updated too.

Two things the advisory does not give you yet. There is no technical detail on any of the bugs, and on the morning of 22 October all ten CVE IDs were still in the RESERVED state at CVE.org, with no NVD record. So there is no CVSS score; "high" is the Node.js project's rating.

The timeline in UTC, from the release metadata:

21 Oct, 10:32 to 10:36 : the fixed binaries on nodejs.org/dist are dated in this window. The GitHub releases were published at 10:38.
10:48 : the update to the official Docker images was committed.
about 11:35 : the Debian-based official images were built. Docker Hub shows the amd64 and arm64 images as pushed at about 14:40.
19:27 : the Brotli project released 1.2.1, 1.1.1 and 1.0.10, each with "CVE-2026-111497: out-of-bounds write permits remote code execution" in the notes. So the bug is in the Brotli decoder itself, and Node shipped the upstream fix as a cherry-picked commit.
22:58 to 00:08 : the Alpine-based official images were built, about 12 to 13.5 hours after the release.

Who is exposed

Brotli: any Node process that decodes Brotli someone else chose

The decoder is compiled into the node binary and sits behind node:zlib. Every way into it counts, and three are common:

Your code decompresses Brotli directly : zlib.brotliDecompress(), brotliDecompressSync() or createBrotliDecompress() on uploads, archives, cache entries or messages.
Your service fetches URLs. fetch() in Node is the bundled undici, and it decodes responses that say Content-Encoding: br with zlib.createBrotliDecompress. It goes by the response's header, whatever the request asked for, so setting Accept-Encoding: identity does not keep Brotli away from a hostile server. axios in Node does the same unless you pass decompress: false. Link previews, webhook testers, feed readers, "import from URL" features and AI agents that browse are all services that fetch a URL a user chose.
Your service accepts compressed request bodies. body-parser 2 decompresses br request bodies that it parses, and its inflate option defaults to true. Express 5 ships body-parser 2, so express.json() on Express 5 decodes Brotli bodies sent with a JSON content type, from any client. Express 4 uses body-parser 1.x, which handles only gzip and deflate.

Reaching the decoder is not the same as being exploitable, and the advisory does not say what input triggers the bug. These are the paths to upgrade or close first. Because the decoder is in the binary, no npm update fixes it. Upgrading the undici package in your package.json, for example, still decodes through the old Node's zlib.

If you cannot upgrade a service today, the interim options close only the paths you control:

// Express 5: refuse compressed request bodies (415) until Node is upgraded.
// This also refuses gzip and deflate request bodies.
app.use(express.json({ inflate: false }));


# In front of any Node service: refuse Brotli request bodies.
if ($http_content_encoding ~* "br") {
    return 415;
}

Distribution builds are different. Debian's nodejs package (we checked the trixie build of 20.19.2) is built with --shared-brotli against the system libbrotli, so on Debian the Brotli fix comes from the distribution's brotli package, not from the Node version number. At the time we checked, neither Debian's security tracker nor Ubuntu's notices listed CVE-2026-111497 yet. The audit script reports such builds as unknown rather than guessing.

TLS hostname checks and DNS

When Node connects over TLS it checks the server's certificate against the host you asked for. Both high-severity TLS bugs are in that check when the host is, or loosely resolves to, an IP address written in a non-standard form: fullwidth digits (24 and 26 only) or a form that "loosely resolves to an IP address" (all lines). The fix makes Node reject non-canonical IP addresses. The advisory gives no more detail than that. Our reading is that the services to look at first are the ones where someone else chooses the host string: webhook senders, URL fetchers and integrations that connect to customer endpoints.

The DNS bug truncated names at a NUL byte in dns.resolve*(), and the fix rejects such input. Code that resolves names it got from users is where this matters.

End-of-life lines

"End-of-Life versions are always affected when a security release occurs," says the advisory. Node 20 reached end of life on 30 April 2026 and Node 25 on 1 June 2026. Neither gets a fixed release, and the official node:20 and node:25 image tags stay on 20.20.2 and 25.9.0.

Why changing the default moves nothing

nvm installs the new version next to the old one, which is what you want: the old one stays for rollback. nvm alias default 24.22.1 then changes what new shells get. Nothing that is already running reads that alias, and nothing that stored a path to the old install reads it either.

nvm install 24.22.1 next to the old version
nvm alias default new shells get it
What runs the new Node? check /proc, not config

Outcomes:

New shells, and pm2 apps restarted with --update-env from one
pm2 daemon, cluster workers, absolute interpreters, systemd unit, cron, images: still old

On the production box, each of these needed its own step:

pm2 entries with an absolute interpreter. Several entries had the old Node binary written into them as the interpreter, and the ones started as npm or npx had their script path under the old version's bin directory. Changing the default did not move them. They had to be deleted and recreated with the new interpreter.
pm2 entries with a bare node interpreter took the new binary only after pm2 restart <name> --update-env from a shell where the new Node came first in PATH.
The pm2 systemd unit that pm2 startup wrote had the old version's PATH and the old pm2 path hard-coded. After editing it, pm2 update moved the pm2 daemon itself, which restarts every app at once. The sites on that box were down for under a minute while the Next.js apps started.
Child processes. Checking /proc/<pid>/exe for every process turned up chains such as npx to tsx to node, and npm start to next-server. Each process in a chain has its own binary.
Cron jobs and helper scripts on two other boxes had absolute paths to the old Node.

What pm2 really does on an upgrade

To check each of those behaviors on its own, lab/pm2-pin-lab.sh in the repo starts four throwaway apps in a separate PM2_HOME from a shell where Node v24.14.0 comes first in PATH: bare (pm2 start server.js), via-npm (pm2 start npm -- start, where the start script is node server.js), abs (--interpreter set to the old binary) and clustered (-i 1, cluster mode). Then it puts v24.22.1 first in PATH, as a changed nvm default does, and records the real binary of every process after each step. A plain pm2 restart left all four on v24.14.0. This is the state after pm2 restart <name> --update-env for each app:

pm2-pin-lab.sh, step B2

# excerpt of results/pm2-lab-20261022T063002Z.txt (2026-10-22T06:30:02Z, aarch64, pm2 6.0.14)
# old: v24.14.0, new: v24.22.1 first in PATH; a plain restart left all four apps on v24.14.0
$ lab/pm2-pin-lab.sh <v24.14.0 bin> <v24.22.1 bin>
$ pm2 restart bare --update-env
$ pm2 restart via-npm --update-env
$ pm2 restart abs --update-env
$ pm2 restart clustered --update-env

== B2. pm2 restart --update-env from the new shell
daemon pid 1937570 node v24.14.0 <v24.14.0>/bin/node
bare interpreter=node script=<app>/server.js
           saved PATH finds node in <v24.22.1>/bin
  pid 1938284 node v24.22.1 <v24.22.1>/bin/node server.js
via-npm interpreter=<v24.14.0>/bin/node script=<v24.14.0>/bin/npm
           saved PATH finds node in <v24.22.1>/bin
  pid 1938375 node v24.14.0 <v24.14.0>/bin/node npm start
    pid 1938462 dash (not node) /usr/bin/dash sh -c node server.js
      pid 1938468 node v24.22.1 <v24.22.1>/bin/node node server.js
abs interpreter=<v24.14.0>/bin/node script=<app>/server.js
           saved PATH finds node in <v24.22.1>/bin
  pid 1938455 node v24.14.0 <v24.14.0>/bin/node server.js
clustered interpreter=node script=<app>/server.js (cluster mode)
           saved PATH finds node in <v24.22.1>/bin
  pid 1938518 node v24.14.0 <v24.14.0>/bin/node server.js

bare moved. via-npm is split: npm itself still runs on the old Node because pm2 stored the old binary as its interpreter, while its child node server.js got the new one from PATH. abs did not move. clustered did not move either, even though its saved PATH now finds v24.22.1: in cluster mode the daemon forks the workers with its own Node binary, so they move only when the daemon moves. All four steps, from results/pm2-lab-20261022T063002Z.txt:

App	After `restart`	After `restart --update-env`	After delete and recreate	After `pm2 update`
`bare` (bare `node`)	old	new	not recreated: new	new
`via-npm` (npm path stored)	old	npm old, app new	new	new
`abs` (absolute interpreter)	old	old	new	new
`clustered` (cluster mode)	old	old	not recreated: old	new
pm2 daemon	old	old	old	new

pm2 update restarted every app in one command: all four PIDs changed. On the production box that was the step where the sites went down. In the lab the command took 2.3 seconds; that measures the command, not how long the apps were unavailable, and real apps take as long as they take to start.

Upgrade steps: nvm, pm2 and systemd

Run these as the user that owns the pm2 daemon.

# 1. Install next to the old version and bring global packages (pm2) along.
nvm install 24.22.1 --reinstall-packages-from=24.14.0
nvm alias default 24.22.1

# 2. Work from a shell where the new Node comes first in PATH.
nvm use 24.22.1
command -v node && node --version

# 3. See what each pm2 entry is pinned to.
pm2 jlist | python3 -c 'import json, sys
for a in json.load(sys.stdin):
    e = a["pm2_env"]
    print(a["name"], e.get("exec_mode"), e.get("exec_interpreter"), e.get("pm_exec_path"), sep=" ")'

Then move each entry according to what step 3 shows:

# Fork mode, interpreter "node": pick up the new PATH.
pm2 restart api --update-env

# Interpreter is an absolute path to the old node, or the script is
# .../v24.14.0/bin/npm or npx: delete and start it again from this shell
# (or from your ecosystem file with the same name and arguments).
pm2 delete web
pm2 start npm --name web -- start

# Save the list that a reboot brings back.
pm2 save

# Regenerate the startup unit: the old one hard-codes the old PATH and pm2 path.
pm2 unstartup # prints a sudo command; run it
pm2 startup # prints a sudo command; run it

# Move the daemon, and every cluster-mode app with it.
# This restarts ALL apps at once: do it in a quiet window.
pm2 update

The pm2 docs ask for the unstartup and startup pair after every Node upgrade. The new unit is pinned again, this time to the new version. That is fine, as long as something checks it at the next upgrade.

Two details are easy to miss. pm2 save writes the saved list, including each entry's environment, so if you save before --update-env the reboot copy keeps the old PATH. And the daemon only reads the systemd unit when systemd starts it. If someone started pm2 by hand, the fixed unit applies at the next boot, not now.

Cron jobs and helper scripts

Cron runs with a minimal PATH, so jobs that need Node usually name it by absolute path, and that path names one install. The audit script below reads only your own crontab -l, so search the system crontabs and your scripts on every host too:

# Paths inside a versioned nvm, fnm, n, volta or asdf install
sudo grep -rnE '/(\.nvm/versions/node|node-versions|n/versions/node|\.volta/tools/image/node|installs/nodejs)/v?[0-9]+\.[0-9]+\.[0-9]+' \
  /etc/crontab /etc/cron.d /var/spool/cron /etc/systemd/system /usr/local/bin ~/bin 2>/dev/null

Update the path in each match. A job that points at a version you later uninstall stops running, which at least fails loudly; one that points at the old version you kept for rollback keeps running vulnerable and says nothing.

Containers

A container keeps the Node it was built with. Rebuild with --pull so the builder fetches the new base image instead of a cached one, deploy by the new tag or digest, and check what is running:

docker build --pull -t registry.example.com/app:2026-10-22 .

docker ps --format '{{.ID}} {{.Names}}' | while read -r id name; do
  printf '%-30s %s\n' "$name" "$(docker exec -e NODE_OPTIONS= "$id" node --version 2>&1)"
done

node --version inside a container reports the node on the container's PATH. If an image carries more than one Node, docker exec <id> readlink /proc/1/exe shows which binary the container's main process runs (when the image has readlink).

The repo's registry_scan.py read the Node version of 24 official node tags for amd64 and arm64 from registry metadata only (manifest and image config, no layers) at 06:23 UTC on 22 October. Every 22, 24 and 26 tag we sampled, and the latest, lts and alpine aliases, shipped the fixed version. How long they took:

Official node images: hours from the release to build and to push

	Value	Series
Debian-based amd64: built	1 h	built
Debian-based amd64: pushed	4.1 h	pushed
Debian-based arm64: built	1 h	built
Debian-based arm64: pushed	4.1 h	pushed
Alpine amd64: built	12.4 h	built
Alpine amd64: pushed	13.1 h	pushed
Alpine arm64: built	13.5 h	built
Alpine arm64: pushed	16.1 h	pushed

Reference: fixed binaries on nodejs.org/dist by 10:36 UTC on 21 Oct 2026. Built: the image config's created time. Pushed: Docker Hub's last_pushed time for that platform image, which is metadata, not proof of when a tag first served it. Latest value in each group of tags. Source: results/registry-20261022T062338Z-summary.txt.

If you rebuilt an Alpine-based image on the afternoon of 21 October, check it again: by this metadata the fixed Alpine images were built 12 to 13.5 hours after the release. The end-of-life tags will stay where they are: node:18 (18.20.8, built March 2025), node:20 (20.20.2, built April 2026) and node:25 (25.9.0, built April and May 2026).

Verify with /proc, not with names

pm2 renames its daemon to PM2 v6.0.14: God Daemon and Next.js renames its server to next-server, and on our build machine the plain node processes showed up as MainThread. Those names are what pgrep matches and what top shows. On that machine pgrep -c node returned 0 while five processes ran a node binary (results/pgrep-vs-proc-20261022T063334Z.txt). The kernel's link to the executable does not lie:

# Every process whose binary is node, with the binary it really runs (root sees all users)
for d in /proc/[0-9]*; do
  exe=$(readlink "$d/exe" 2>/dev/null) || continue
  case "${exe% (deleted)}" in */node|*/nodejs) ;; *) continue ;; esac
  printf '%-8s %-70s %s\n' "${d#/proc/}" "$exe" "$(tr '\0' ' ' < "$d/cmdline" | cut -c1-50)"
done

# The version of one of them, read from the binary without running it.
# Official builds compile in their headers URL. Do not run another user's
# binary with --version as root: that runs their code as root.
sudo grep -a -o -m1 -E 'node-v[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?-headers\.tar\.gz' /proc/12345/exe
# node-v24.14.0-headers.tar.gz

# Its children, which have their own binaries
ps -o pid,args --forest -g "$(ps -o sid= -p 12345 | tr -d ' ')"

An exe that ends in (deleted) means the file was replaced on disk after the process started, so the process still runs the old code until it restarts. For more on that pattern, and on proving a runtime is patched when the version string can mislead, see PHP2Shell: How to Check That Every PHP You Run Is Actually Patched.

Run the audit script

The-DevOps-Daily/node-runtime-audit on GitHub

node_runtime_audit.py is one file of standard-library Python. Its sources are /proc, the output of pm2 jlist and crontab -l, systemd unit files, and node --version inside running containers; besides those it only runs systemctl show for pm2 units and docker or podman ps and inspect, and looks at file metadata. For every running process it reads the real binary through /proc/<pid>/exe and the version compiled into that file, which works for deleted binaries and for processes inside containers. By default it never runs a binary it finds on the host, and it does not open the Node binaries that pm2 entries, crontabs and units name: their version comes from a running process that uses the same file, or from the path as a hint, where a path that names an unfixed version counts as vulnerable and one that names a fixed version stays unknown. Files are pinned with O_PATH and checked before they are read, so a planted FIFO or device cannot stall it. Command lines and container names are printed with control characters escaped. Run it as the user that owns the Node processes; root is not needed.

This is the real output on the Raspberry Pi we build on, run as the user that owns the Node processes, with --redact, which hides user names and app names. By the time of this run, the fnm default, the crontab and the pm2 systemd unit had already been moved to v24.22.1:

node_runtime_audit.py on the Pi

# 2026-10-22T07:10:33Z Raspberry Pi (Debian 12, aarch64), Node via fnm, pm2 6.0.14, run as the user that owns the Node processes
$ python3 node_runtime_audit.py --redact
node-runtime-audit: Node.js security release of 2026-10-21, fixed releases v22.23.4, v24.22.1, v26.12.1
lines without a fix (end-of-life) count as vulnerable
note: covers the processes, crontab and pm2 of USER; run it as each user that owns Node processes

Running Node processes (real binary from /proc/<pid>/exe)
  [vulnerable] pid 1883303 v24.14.0 USER /home/USER/.local/share/fnm/node-versions/v24.14.0/installation/bin/node
      cmd: PM2 v6.0.14: God Daemon (/home/USER/.pm2)
      parents: systemd(1)
      why: v24.14.0 < v24.22.1
  [vulnerable] pid 1883371 v24.14.0 USER /home/USER/.local/share/fnm/node-versions/v24.14.0/installation/bin/node
      cmd: node /home/USER/projects/app-1/node_modules/.bin/next dev
      parents: bun(1883370) <- node(1883303) <- systemd(1)
      why: v24.14.0 < v24.22.1
  [vulnerable] pid 1883390 v24.14.0 USER /home/USER/.local/share/fnm/node-versions/v24.14.0/installation/bin/node
      cmd: next-server (v16.2.6)
      parents: node(1883371) <- bun(1883370) <- node(1883303) <- systemd(1)
      why: v24.14.0 < v24.22.1
  [vulnerable] pid 1884634 v24.14.0 USER /home/USER/.local/share/fnm/node-versions/v24.14.0/installation/bin/node
      cmd: node /home/USER/projects/app-1/.next/dev/build/postcss.js 38963
      parents: node(1883390) <- node(1883371) <- bun(1883370) <- node(1883303)
      why: v24.14.0 < v24.22.1
  183 processes of other users not inspected: run the script as their owner

pm2 live list (pm2 jlist, PM2_HOME=/home/USER/.pm2)
  [unknown] pm2 CLI /home/USER/.bun/bin/pm2 is writable by group or others (mode 777): not run, live list skipped
  [vulnerable] daemon pid 1883303 v24.14.0 /home/USER/.local/share/fnm/node-versions/v24.14.0/installation/bin/node

Installed Node versions (version managers)
  fnm in /home/USER/.local/share/fnm
      installed: v24.14.0 (running), v24.14.1 (from path), v24.22.1 (from path)
      default alias -> v24.22.1 (from path)
  `node` on this shell's PATH -> v24.22.1 (from path) /home/USER/.local/share/fnm/node-versions/v24.22.1/installation/bin/node
  for information only: these do not change the exit code

Crontab: absolute Node paths
  [unknown] crontab -l (USER):33
      /home/USER/.local/share/fnm/node-versions/v24.22.1/installation/bin/node -> v24.22.1 (from path) (pinned to one install)
      why: from path: v24.22.1 (a hint, not proof: binaries outside /proc are not read)
  [unknown] crontab -l (USER):53
      /home/USER/.local/share/fnm/node-versions/v24.22.1/installation/bin/node -> v24.22.1 (from path) (pinned to one install)
      why: from path: v24.22.1 (a hint, not proof: binaries outside /proc are not read)

systemd units: absolute Node paths
  [unknown] /etc/systemd/system/pm2-USER.service:12
      /home/USER/.local/share/fnm/node-versions/v24.22.1/installation/bin -> v24.22.1 (from path) (pinned to one install)
      why: from path: v24.22.1 (a hint, not proof: binaries outside /proc are not read)
      unit: pm2-USER.service ActiveState=inactive MainPID=0
      note: the unit is not what runs pm2 now; its PATH applies at the next boot

Containers
  no docker or podman CLI found

result: VULNERABLE: something runs or will start an unfixed Node (exit 1)
$ echo $?
1

The fnm default, the shell's node, both cron jobs and the systemd unit point at the v24.22.1 install. The script reports the cron and unit paths as unknown rather than patched, because no running process uses that binary and it does not open it. The four processes that run are v24.14.0. The pm2 daemon is not running under the systemd unit (ActiveState=inactive), so the unit we had fixed is not what runs it now. The one pm2 app is started by bun, and the Node processes under it run the app's node_modules/.bin/next script. A separate check (scripts/pm2-path-evidence.py, results/pm2-path-20261022T070504Z.txt) shows that the process the pm2 daemon started and the saved pm2 list both have the v24.14.0 install first on PATH, which is where those Node processes come from. If the app is resurrected from that saved list after a reboot, it gets the same PATH; we did not reboot to confirm. The fix here is pm2 restart <app> --update-env from a fixed shell, then pm2 save, then moving the daemon.

The run found one more thing: the pm2 CLI on that machine is world-writable (mode 777). The script will not run a CLI that someone else could have changed, so it skipped the live pm2 list and reported it as unknown. Check the permissions of your Node installs and global CLIs while you are in there.

The exit code is 1 when anything runs or will start a vulnerable Node, otherwise 2 when anything could not be checked, and 0 when it found nothing vulnerable or unknown within what it covers, including no Node at all. Unknown never counts as patched: a pre-release build, a distribution build linked to the system Brotli, a binary without an embedded version, a unit it could not read, a failed pm2 jlist, crontab -l or docker ps, a config path that only names a fixed version, or a cron path to a Node that no longer exists all give 2. It only sees what the user running it can see, so run it as each user that owns Node processes. To run it across hosts without copying it first:

for h in deploy@web1 deploy@web2 deploy@web3; do
  ssh "$h" 'python3 - --json' < node_runtime_audit.py > "node-${h#*@}.json"
  echo "$h exit $?"
done

The tests build fake hosts with the cases a version check gets wrong: a fake node that would leave a marker file if the script ever ran it, a binary replaced while it runs, a container binary, a Debian-style build, a pm2 entry whose config is fixed but whose process was never restarted, npm pinned to the old install while its child gets the new one, a cluster-mode app, an old binary in a directory named after a fixed version, end-of-life and pre-release versions, a cron job on a removed version, a shell wrapper that mentions a fixed version, units owned by other users, and FIFOs, devices, hard links and symlinks planted where a file is expected. One test starts a real process, deletes its binary and checks that the script still reads the running file.

Rollback

Keep the old version installed until every check passes. Rolling back is the same work in reverse: nvm alias default 24.14.0, then the same --update-env, delete-and-recreate, pm2 save, startup unit and pm2 update steps from a shell where the old version comes first. For containers, redeploy the previous image digest. A rollback puts the vulnerable version back, so treat it as a few hours to fix a regression, not as a resting state.

Once the audit shows nothing on the old version, nvm uninstall 24.14.0. Anything still pinned to it will then fail to start instead of quietly running the old code, and the audit script reports those paths as missing.

What we could not check

The bugs themselves. The advisory gives no technical detail, and we did not test whether any code path reaches the Brotli flaw. The exposure section is based on which code calls the decoder.
Scores. The CVE records were still reserved when we checked, so there is no CVSS score.
Distribution packages. Debian and Ubuntu had not published anything for CVE-2026-111497 when we checked. Check your distribution's nodejs and brotli advisories.
Containers on a real host. The Pi has no Docker, so the container checks are covered by the test fixtures only. The registry scan read real image metadata.
The downtime number."Under a minute" comes from the production upgrade. The lab's pm2 update took 2.3 seconds because its apps do nothing.
Bare node in cron jobs and units. The script checks absolute paths and PATH= lines; it does not resolve a bare node through the job's PATH. It reads only your own crontab -l, not the system crontabs or the saved pm2 list, and it does not follow EnvironmentFile=.
Other platforms. The registry scan covers linux/amd64 and linux/arm64 only. The script needs Linux and /proc, and does not cover Kubernetes, stopped containers, or Node binaries renamed to something other than node (single executable applications, Electron).

Checklist

Every host: Node 22.23.4, 24.22.1 or 26.12.1 or later installed; Node 20, 25 and older replaced
Every pm2 entry: absolute interpreters and npm or npx script paths recreated, bare node entries restarted with --update-env, then pm2 save
pm2 startup unit regenerated (pm2 unstartup, pm2 startup) and the daemon moved with pm2 update in a quiet window
Cluster-mode apps checked after the daemon moved
Crontabs, systemd units and helper scripts searched for absolute paths to the old install
Images rebuilt with --pull (Alpine-based ones checked after the late rebuild) and redeployed by new tag or digest
Services that fetch user-supplied URLs or accept compressed bodies upgraded first; interim measures where they cannot be
Every Node process checked by /proc/<pid>/exe, not by name, including child processes
Node installs and global CLIs (pm2, npm) writable only by their owner
Audit script output saved per host, and the old version uninstalled once it is clean

Every fact in this post is listed with its source in the repo's SOURCES.md.

Originally published at devops-daily.com.

A Service Catalog Nobody Updates Can Be Worse Than None: We Checked 217 Entries Against GitHub

DevOps Daily — Thu, 22 Oct 2026 09:00:00 +0000

We took 217 service catalog entries that 30 organizations publish on GitHub, in the Backstage catalog-info.yaml format, and checked each one against what GitHub says today. For one in ten (21 entries), GitHub contradicted a record: in 9 the catalog entry itself was wrong (a repository renamed while the catalog kept the old name, a repository reference without its owner, a dead link, no owner at all), and in 12 the CODEOWNERS file next to it named people or teams GitHub no longer accepts. That is a floor, not a total. For three entries in four, the owner could not be checked at all from outside the organization. And where the catalog and CODEOWNERS both named an owner for the same file, the two agreed, with every owner resolving, only half the time.

That matters most at the worst moment. During an incident, someone opens the catalog to find out who owns the failing service. With no entry, they go and ask. With a wrong entry, they can page a team that was reorganized away, or a person who left, and do it with confidence because the catalog said so. We did not measure that happening; it is the reason the measurement is worth doing. This post shows the checker we built, how we picked the sample, what it found, the bugs two reviews found in our own method, and the habits that keep a catalog true.

TL;DR

217 Component entries from 214 public repositories in 30 GitHub organizations, checked on 2026-10-22: 21 (9.7%) had at least one record GitHub contradicts, 9 of them in the catalog entry itself and 12 only in the CODEOWNERS file next to it. 62 (28.6%) had only warnings, and 134 (61.8%) had nothing we could find.
Ownership is where the records disagree. Of 73 entries whose catalog file is also covered by a CODEOWNERS rule, 37 agree and every owner resolves. 14 name owners that GitHub cannot resolve, and 22 name different owners (in 7 of those, one team name contains the other, which may be the same team).
159 entries (73.3%) have an owner nobody outside the org can check: GitHub teams are only visible to members, and some owners come from other directories. With a token that can read your org, the script checks teams and members directly.
Links barely rotted: 2 of 152 were dead. But only 70 entries had any link, and none was titled as a runbook or a dashboard. We could not test the links that probably matter most during an incident.
We expected older catalog files to drift more. Drift was 8.0% for files changed in the last year and 11.4% for older ones, a gap this sample cannot tell apart from chance.
The fixes are boring and they work: derive owners from the source of truth, check catalog-info.yaml in CI, use GitHub's own CODEOWNERS validation, and retire entries nobody claims.

Prerequisites

Python 3.10 or later (the checker uses only the standard library)
A GitHub token in GITHUB_TOKEN, or a logged-in gh CLI. To check owners inside your own org, the token needs to read the org's teams and members
A catalog in the Backstage catalog-info.yaml format, or the public sample in the repository

The-DevOps-Daily/service-catalog-drift on GitHub

Why a catalog drifts

A catalog entry is usually written once, by a scaffolder template, on the day the repository is created. After that, teams get reorganized and renamed, people move on, repositories are renamed or transferred, and documentation moves from one wiki to the next. None of those changes touch catalog-info.yaml.

None of them break anything either. Backstage's own FAQ, answering whether a plugin should reject entities that fail validation, says: "Never throw errors due to "soft" errors, in particular relations not matching an existing target". So an owner that points at a team that no longer exists loads and shows up like any other. GitHub redirects a renamed repository, so a catalog that still uses the old name keeps working, and nobody has a reason to look. The catalog stays green while it drifts away from the truth.

That is why a stale catalog can be worse than none. An empty owner field sends you looking. A wrong one can send you to the wrong place, quickly.

What the checker compares

drift.py reads Backstage Component entities and checks five things against GitHub, all read-only:

Check	Drift (GitHub contradicts it)	Warning (look at it)	Cannot tell from outside
owner	no `spec.owner`; with org access, a team that does not exist or has no members	a team GitHub cannot resolve in the CODEOWNERS files that name it; a `user:` with no GitHub account of that name	a team no CODEOWNERS file names; a user who is not a public member; a group in another directory
repo	the slug redirects (renamed or transferred); a `github.com/project-slug` that is not `owner/repo`; archived while the catalog does not say `deprecated`	no human commit on the default branch in a year; a root catalog file that describes a different repository	a repository that is not visible (deleted or private)
codeowners	the CODEOWNERS rule for the catalog file names owners GitHub's validation cannot resolve	CODEOWNERS and the catalog name different owners	GitHub's validation not readable
links	404 or 410; a GitHub file missing at a ref that exists		401, 403, 429, 5xx, timeouts, internal hostnames, pages below a GitHub repository
lifecycle		`production` or `experimental` with no human commit in a year; no lifecycle

catalog-info.yaml Component entities
Owner teams API or CODEOWNERS errors
Repository redirects, archive, commits
Links public addresses only
Report reasons and exit code

A drift on the CODEOWNERS check is about the file that routes code reviews, not about the catalog entry itself. We count both because both are ownership records people rely on, but we report them separately below. A warning is a fact worth a look: a library that works may not need a commit for a year. "Cannot tell" is its own status and never counted as either.

Checking an owner from outside the org

Inside your org, checking an owner is a lookup: does the team exist, does it have members, is this user a member. From outside, GitHub hides teams. But GitHub validates CODEOWNERS files, and it publishes the result. For a public repository, anyone can call GET /repos/{owner}/{repo}/codeowners/errors, and it lists every owner GitHub cannot resolve, with messages such as:

Unknown owner on line 3: make sure the team @org/team exists, is publicly visible, and has write access to the repository

So if any CODEOWNERS file in the org names the catalog's owner team, GitHub can tell us whether that team resolves. The checker looks in the entry's own repository first, then runs one code search for the team in the org's other CODEOWNERS files. A team that resolves exists. A team that does not resolve might be deleted, renamed, secret, or only lack write access there, so on the owner check it is a warning, not drift. A team that no CODEOWNERS file names stays "cannot tell". All of this assumes Backstage group names are GitHub team slugs, which is what Backstage's GitHub org provider produces; groups from other namespaces are not mapped.

Links come from strangers

The links in a catalog file are written by whoever wrote the file, so link checks are opt-in (--check-links) and careful. The token only ever goes to api.github.com, and github.com links are checked through the API with every path segment validated and quoted. Everything else gets a HEAD, then a GET, with no credentials and no cookies, at most one request per second per host. The host name is resolved once, every address must be public, and the request goes to that pinned address, so a second DNS answer cannot point it at an internal service. Every redirect hop is checked the same way.

The sample

We wanted catalogs that real organizations maintain, not tutorials. The selection, all recorded in the repository:

GitHub code search for filename:catalog-info.yaml path:/ (files up to 20,000 bytes) on 2026-10-22, between 07:08 and 07:23 UTC. Code search returns at most 1,000 results per query, so sample.py splits the query by file size into 16 ranges. GitHub reported 8,864 matches; we fetched 9,761 unique repositories (code search counts are estimates).
3,579 of those repositories belong to personal accounts and were dropped. The other 6,182 belong to 984 organizations. Code search returned no forks.
53 organizations had a root catalog-info.yaml in 20 or more repositories. Many turned out to be test, demo or QA organizations for developer portals, so we kept the 30 that have a website on their GitHub profile and at least one repository with 10 or more stars. That rule keeps out obvious test orgs; it does not prove that every sampled repository is a real service, and a few are demos or samples. The excluded orgs are stored as hashes, because many are named after a person.
From each of the 30, 8 repositories at random (seed 20261022): 240 files. One was a scaffolder template, one was not valid YAML, and 24 held no Component entity, only other kinds. That leaves 217 Component entries from 214 repositories.

The 30 organizations are a mix of companies, public bodies, research institutes and open-source projects: bcgov, broadinstitute, cds-snc, Clever, contentful, cultureamp, DEFRA, dell, DiamondLightSource, dwp, eduNEXT, elastic, equinor, grafana, hmcts, Invoca, Kajabi, kartverket, open-dxp, openedx, pantheon-systems, prime-radiant-inc, radiorabe, reearth, sendbird, snyk, sourcefuse, spotify, Unity-Technologies and vtex-apps.

Every organization contributed 8 repositories, but a repository can hold several entries or none, so organizations contribute between 1 and 10 entries. The shares in this post are pooled over entries. Averaged per organization instead, the drift share is 10.7% rather than 9.7%.

So the population this describes is narrow: root-level catalog-info.yaml files in public repositories of organizations that use them widely. Catalog files elsewhere in a repository, catalogs kept in a separate repository, and private catalogs are not in it.

The run

One pass makes about 1,700 GitHub API requests, cached on disk per token for 24 hours. --redact keeps people out of the output: user logins become user:<redacted>, and owner and team names become a hash with a key that is random per run, which is why you see group:<56b023> below and different hashes in the committed results. The results name organizations, repositories and services, never a person.

drift.py on the public sample

$ python3 drift.py --sample sample/sample-2026-10-22.json --check-links --redact --quiet > report.txt 2>/dev/null
$ echo $?
1
$ tail -7 report.txt
217 entries from 238 files (1 templates, 1 unparsed, 0 unreadable skipped)
  drift 21 (9.7%), warn 62, ok 134
  owner drift 1, ok 53, unverifiable 159, warn 4
  repo drift 6, ok 168, warn 43
  codeowners drift 14, n/a 144, ok 37, warn 22
  links drift 2, n/a 147, ok 63, unverifiable 5
  lifecycle n/a 38, ok 148, warn 31
$ awk '/^(DRIFT|WARN) /{p=($2=="com.unity.services.samples.relationships"||$2=="changelog-github"||$2=="kaizen-design-system"||$2=="loopback4-microservice-catalog")} p' report.txt
DRIFT com.unity.services.samples.relationships (Unity-Technologies/com.unity.services.samples.friends:catalog-info.yaml) owner group:<56b023>
         repo drift Unity-Technologies/com.unity.services.samples.relationships moved to Unity-Technologies/com.unity.services.samples.friends
         codeowners drift CODEOWNERS line 2 names owners GitHub cannot resolve: @Unity-Technologies/<072a85>
DRIFT changelog-github (cultureamp/changelog-github:catalog-info.yaml) owner group:<ffb463>
         repo drift github.com/project-slug is not a valid owner/repo
DRIFT kaizen-design-system (cultureamp/kaizen-design-system:catalog-info.yaml) owner group:<159434>
         codeowners drift CODEOWNERS line 4 names owners GitHub cannot resolve: @cultureamp/<15544f>
         links drift 1 of 5 links dead: other https://cultureamp.design/<path 642625a6c6> (HTTP 404)
DRIFT loopback4-microservice-catalog (sourcefuse/loopback4-microservice-catalog:catalog-info.yaml) owner group:<046fce>
         owner warn GitHub cannot resolve @sourcefuse/<046fce> in the CODEOWNERS files that name it (deleted, renamed, secret, or without write access)
         codeowners drift .github/CODEOWNERS line 5 names owners GitHub cannot resolve: @sourcefuse/<046fce>

The exit code is 1 because drift was found, so the same command can fail a CI job. The four entries above show the main kinds. The Unity sample was renamed from relationships to friends, its catalog entry still uses the old name, and line 2 of its CODEOWNERS names a team GitHub cannot resolve. cultureamp's changelog tool sets github.com/project-slug to the repository name without its owner, which Backstage's GitHub integrations cannot use. In cultureamp's design system, a CODEOWNERS team does not resolve and the Storybook link answers 404. In sourcefuse's microservice catalog, the catalog's own owner team is the one GitHub cannot resolve in the same repository's CODEOWNERS: drift for the CODEOWNERS file, a warning for the owner.

What it found

What 217 public catalog entries got wrong

	Value	Series
CODEOWNERS names an owner GitHub cannot resolve (14)	6.5%	Drift
Repository renamed, catalog has the old name (4)	1.8%	Drift
project-slug is not owner/repo (2)	0.9%	Drift
Dead link (2)	0.9%	Drift
No owner at all (1)	0.5%	Drift
No human commit in a year (39)	18%	Warning
Says production or experimental, idle a year (28)	12.9%	Warning
Catalog and CODEOWNERS name different owners (22)	10.1%	Warning
Owner team does not resolve where it is named (3)	1.4%	Warning

Share of all 217 entries with each finding; an entry can have more than one. 30 GitHub organizations, 8 random repositories each, checked 2026-10-22. Source: results/summary-20261022T091627Z.json in the demo repository.

Ownership is where the records disagree

Most findings are about who owns what. The clearest view is the 73 entries whose catalog file is also covered by a CODEOWNERS rule, so there are two records of the owner to compare:

summarize.py

$ python3 summarize.py results/run-20261022T091627Z.json | sed -n '/^catalog vs CODEOWNERS/,/Fisher/p'
catalog vs CODEOWNERS: 73 entries with both; agree and resolve 37 (50.7%); unknown owners 14 (teams 8, users 6)
  disagree: 4 catalog names a team, CODEOWNERS people
  disagree: 11 different team names
  disagree: 7 one team name contains the other
drift by file age: under 1 year {'entries': 112, 'drift': 9, 'pct': 8.0}, 1 year or more {'entries': 105, 'drift': 12, 'pct': 11.4}, Fisher p = 0.493

73 entries with both a catalog owner and a CODEOWNERS rule

	Value	Series
Agree, every owner resolves	37	Agree
One team name contains the other	7	Disagree
Different team names	11	Disagree
Catalog names a team, CODEOWNERS people	4	Disagree
CODEOWNERS owner GitHub cannot resolve	14	Cannot resolve

"One name contains the other" is, for example, a short catalog name inside a longer GitHub team slug; it may be the same team, but the data cannot show it. Unresolvable owners are what GitHub's CODEOWNERS validation reports for the rule that covers the catalog file.

37 agree. The other 36, almost half, either name different owners or name owners GitHub cannot resolve. In 7 of them one team name contains the other: in four snyk repositories, for example, the catalog uses a short team name that is part of the longer GitHub team slug in CODEOWNERS. Those may well be the same team. Leave them out and 29 of 73 (40%) still disagree or cannot be resolved. Of the 14 with unresolvable owners, 8 involve a team and 6 involve individual users.

Two caveats keep this from being a verdict on the catalog. CODEOWNERS says who reviews code, which is not always who owns the service, so a disagreement means one record is stale or the two mean different things. And GitHub reports a team as unknown when it was deleted or renamed, is secret, or has no write access to the repository; any of those means the team does not review that code, but only the first two mean it is gone.

Unresolvable owners repeat. In vtex-apps, the same unknown team sits on line 3 of the CODEOWNERS in three of the eight sampled repositories, which looks like a file copied from a template and never updated. Four organizations hold 14 of the 21 drifted entries, and 20 of the 30 had none.

The owner field itself could be checked for only 58 entries (27%): 37 resolved in the entry's own CODEOWNERS, 15 in another repository's, 3 named a team GitHub could not resolve in any CODEOWNERS file that names it, 2 were checked through GitHub's user API, and 1 had no owner at all. Backstage requires spec.owner on a Component, so that entry probably never loaded into anyone's catalog. The 159 we could not check are 142 groups and 17 users: hidden teams, groups that do not look like GitHub teams, users who are not public members of the org, and user IDs that are not GitHub logins.

Repositories that moved

4 entries (1.8%) point at a repository by a name it no longer has. GitHub redirects the old name, so links and API calls keep working. But GitHub's documentation warns that if anyone creates a new repository with the old name, "redirects to the renamed repository will no longer work", and the catalog entry then points at someone else's code. 2 more entries set github.com/project-slug to a bare repository name without the owner, and 2 live in one repository but describe another, which is what a catalog file copied from a sibling repository and never edited looks like.

We found no archived repositories. That is a gap in the sample, not good news: none of the 211 repositories the checker read was archived (each entry records the flag), a code search we tried with archived:true returned nothing (so archived repositories are probably missing from the index), and Backstage's GitHub discovery skips them by default (allowArchived defaults to false).

Idle, but still "production"

39 entries (18.0%) live in repositories with no human commit on the default branch for over a year, and 2 more where the last 30 commits are all from bots. We do not count commits authored by Dependabot, Renovate or other bots as activity. 28 of the 39 still say production (19) or experimental (9). A stable library can go a year without a change, so this is a warning, not drift. An experimental service with no human commit for a year is a decision nobody made.

Links

Only 70 of 217 entries (32%) had any link at all. Of 152 links, 132 answered, 2 were dead, and 18 could not be checked: 12 pointed at hostnames that do not resolve in public DNS (internal tools, most likely), 5 answered 403, and 1 was a page below a GitHub repository that we do not check. The two dead ones were a Storybook page and a vulnerability report page on a vendor's site.

None of the 152 links was titled as a runbook, an on-call schedule or a dashboard. Public catalogs may simply not publish them, or may title them differently; either way we could not test those links, and they are the ones you would want to work during an incident. If you run this on your own catalog, the link check is where we would expect your numbers to differ most from ours.

Does a catalog decay with age?

Drift by when the catalog file last changed

	Value
Under 6 months (81)	9.9%
6 to 12 months (31)	3.2%
1 to 2 years (48)	12.5%
Over 2 years (57)	10.5%

Share of entries with at least one drift, by the date of the last commit that touched catalog-info.yaml. Under one year: 9 of 112. One year or more: 12 of 105. Fisher's exact test p = 0.49, which also treats entries as independent although entries from one organization share habits.

We expected a clear slope and did not find one: 8.0% for files changed in the last year against 11.4% for older files. Fisher's exact test gives p = 0.49, and it assumes the entries are independent, which they are not quite: entries from one organization share templates and habits. So this sample cannot tell the difference from chance. Drift and warnings together are far more common in older files (24.7% and 19.4% under a year, 54.2% and 54.4% over), but mostly by construction: a repository with no commits for a year also has a catalog file nobody touched for a year. The median catalog file in the sample was last changed 352 days before we checked it.

A single snapshot also cannot see the worst decay. Entries that rotted badly enough to be noticed may have been deleted, and deleted entries are not in a sample of files that still exist.

The bugs two reviews found in our checker

The first version of the checker and of this post had problems, found by our own runs and by a second review of the code and the draft. Several changed numbers you just read, and some of the corrections made the result look worse, others better.

An unresolvable team was called a missing owner. The first version reported a catalog owner as drift when GitHub could not resolve the team in a CODEOWNERS file, including one in another repository. But a team can exist and simply lack write access there. That is now a warning on the owner, and only the broken CODEOWNERS file counts as drift. Three entries moved from owner drift to a warning; two of them had no other drift.
An invalid repository reference was quietly replaced. Two entries set github.com/project-slug to a repository name without its owner. The first version fell back to the repository the file lives in and reported nothing. It now reports the annotation as drift, which added two drifted entries just as the first fix removed two. The headline stayed at 21 by coincidence.
A user ID that looks like a GitHub login. One organization names users by an ID from its own directory, three letters and five digits, which is also a valid GitHub login. GitHub had no account by that name, and the first version reported a departed owner. A missing account is now a warning, unless you pass --users-are-github-logins for a catalog where you know users are GitHub logins.
GitHub's two error formats. For a user, the CODEOWNERS error says make sure @user exists; for a team, make sure the team @org/team exists, is publicly visible, .... The first version only parsed the user form, and for teams it fell back to marking every owner on the line as unknown. It changed no result in this sample, but on another catalog it would have.
A YAML reader that was too forgiving. The checker has its own small YAML reader to stay dependency-free. One sampled file had description: at the wrong indentation, which swallowed the annotations: block below it. PyYAML rejects that file, and our reader accepted it. It now raises the same error, and a cross-check against PyYAML on all 240 files agrees on every field the checker reads, with the one invalid file rejected by both. The review also found that it decoded some escapes in quoted strings wrongly; it now handles them or refuses the file.
Too generous with links and CODEOWNERS. Any page below an existing GitHub repository counted as a working link, a redirect loop counted as an answer, docs/* in CODEOWNERS also matched files in subdirectories (GitHub's own docs say it does not), and an unreadable CODEOWNERS validation could still report agreement. Each now reports "cannot tell" or matches GitHub's rules.
People in the output. A bare owner such as owner: someone is treated as a team by Backstage, but it is sometimes a person's login, and one owner was an email address. An early run printed them. That file was deleted, and owner and team names are now hashed with a key that is never written down.

Smaller fixes: a link with a space in it stopped an early run; cached answers now expire after a day and are kept per token; CODEOWNERS lines with invalid syntax are skipped, as GitHub skips them; a file link is not called dead when its ref might contain a slash; and an error in one entry, or a 401 or 5xx from GitHub, makes the run exit with 2 instead of stopping or passing. None of these changed a count in this sample. Every superseded run that left a file is kept in results/ with a note on what changed.

False positives, and what we could not check

A team GitHub cannot resolve is not always deleted. GitHub's message covers deleted, renamed, secret, and teams without write access to that repository.
Different names can be the same people. A parent team and a child team, or a Backstage group name that differs from the GitHub team slug, show up as a disagreement. That is why disagreement is a warning.
A link that answers is not always alive. A 200 can be a login page, or a "page not found" that answers 200. We count both as working. A 403 from a site that blocks scripts is "cannot tell".
Bot detection is by commit author. A merge commit by a person counts as human activity even when it merges a bot's change, so "no human commit in a year" can undercount idle repositories.
Owner coverage is uneven. We could check 27% of owners, mostly those named in some CODEOWNERS file. Those may be better or worse kept than the rest; we cannot tell which.
Private catalogs are not in the sample. Public repositories may be kept better or worse than internal ones, and internal catalogs carry the runbook, on-call and dashboard links we could not test.

How to keep a catalog honest

Derive owners, do not type them

Many of the disagreements above look like two hand-typed names for the same thing. Load groups from the source of truth instead: Backstage's GitHub org provider mirrors GitHub teams as Groups on a schedule, and the catalog products sync from identity providers. Then the owner in catalog-info.yaml and the handle in CODEOWNERS are the same string, and a machine can compare them:

# catalog-info.yaml
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payments-api
  annotations:
    github.com/project-slug: acme/payments-api # owner/repo, not just the repository name
spec:
  type: service
  lifecycle: production
  owner: group:default/payments # the GitHub team slug, not a display name


# .github/CODEOWNERS: the same team, so the two can be checked against each other
* @acme/payments

Check the catalog file in CI

Drift arrives two ways: someone edits the file badly, or the world changes while nobody touches it. A pull request check catches the first, a weekly scheduled run catches the second. The repository ships this workflow in examples/:

name: catalog drift

on:
  pull_request:
    paths: ['catalog-info.yaml', '.github/CODEOWNERS', 'CODEOWNERS', 'docs/CODEOWNERS']
  schedule:
    - cron: '17 6 * * 1'
  workflow_dispatch:

permissions:
  contents: read

jobs:
  drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/checkout@v5
        with:
          repository: The-DevOps-Daily/service-catalog-drift
          ref: main # pin this to a commit SHA you have read
          path: .catalog-drift
      - uses: actions/setup-python@v6
        with:
          python-version: '3.13'
      - name: check catalog-info.yaml
        env:
          # The default token cannot read org teams; use a token with Members: read for full owner checks
          GITHUB_TOKEN: ${{ secrets.CATALOG_DRIFT_TOKEN || github.token }}
          REPO: ${{ github.repository }}
          # on a pull request, read CODEOWNERS as the pull request changes it
          REF: ${{ github.event.pull_request.head.sha || github.sha }}
        run: python .catalog-drift/drift.py catalog-info.yaml --repo "$REPO" --ref "$REF" --check-links --no-org-search --cache-hours 0

It exits with 1 on drift or on a catalog file that does not parse, and with 2 when it could not check what it was given, so a broken run never passes quietly.

Use the check GitHub already runs

You do not need our script to find the worst cases. GitHub already validates every CODEOWNERS file, and one call per repository lists what it cannot resolve:

# Owners GitHub cannot resolve in this repository's CODEOWNERS
gh api repos/OWNER/REPO/codeowners/errors \
  --jq '.errors[] | "line \(.line): \(.message | split("\n")[0])"'

Loop it over every repository in your org once a month. Each line it prints is an owner or a rule GitHub cannot use, which can leave reviews without the reviewer you meant, and is often a sign that the catalog owner is stale too.

Give the scorecard consequences

Every catalog product has scorecards, and so does Backstage through the Tech Insights plugin. The useful checks are the ones in this post: the owner resolves, the runbook link answers, the lifecycle matches the activity. Put the result where people already work, such as a comment on the pull request or a weekly list sent to each owning team, and not only on a dashboard.

Retire what nobody claims

An entry whose owner does not resolve and whose repository has been idle for a year may have nobody to fix it. One workflow we would suggest: open an issue for the last known owner, and if nobody confirms ownership within 30 days, mark the entry deprecated; remove it 30 days later unless someone claims it. A missing entry sends people asking. A wrong one can send them to the wrong place.

Build or buy

Every catalog product handles freshness with some mix of the same three tools: sync from a source of truth, automatic discovery, and scorecards. Here is what each one's public documentation says, in alphabetical order, with one documented limit each where we found one:

Backstage (open source) mirrors GitHub teams and users with its org provider, finds catalog-info.yaml files with GitHub discovery, and removes orphaned entities. Checks on catalog data, including the owner field, come from the Tech Insights community plugin. Limit: as quoted above, relations that point at nothing are accepted.
Cortex has scorecard rules such as "has an owner" and "has a runbook", syncs ownership daily from identity providers, Git hosts and HR systems, and auto-archives entities whose repository was deleted or archived. Limit: in GitOps mode, a YAML file that is moved rather than deleted is not archived.
Humanitec says in its own docs that it is not an internal developer portal: the Platform Orchestrator is the backend of a platform and complements portals such as Backstage, Port and Cortex.
OpsLevel has a component ownership check in its rubric, syncs GitHub teams every 12 hours, detects services from repositories and suggests owners from commit authors, and removes repositories deleted at the Git provider. Limit: the team sync page says deleted GitHub teams and users are not synced.
Port has scorecards with a "has a team" rule, syncs users and teams from SSO and Git providers, deletes entities that are gone from the source on a full resync, and has a guide for mapping CODEOWNERS to teams. Limit: entities of a resource type removed from the mapping are not deleted automatically.
Roadie is managed Backstage: catalog-info.yaml is its native format, it discovers files from Git providers, runs Tech Insights scorecards, and its Catalog Completeness beta suggests owners for repositories. Limit: deleting a location does not remove the entities found through it.

If you already run Backstage, the checks in this post fit in a standard-library Python script that runs in CI. What the products add is the plumbing around them: syncing from HR and identity systems, a place where scores are shown, and someone else keeping it running. Whichever way you go, test it the way we tested ours. The repository has a fixture catalog with one of each kind of drift; point your tool at something like it and confirm it fails on each one. A tool you have only seen report "all good" has not shown you that it can find anything.

What we could not conclude

Whether catalogs decay with age. The difference we saw (8.0% against 11.4%) is within chance for this sample, and one snapshot cannot measure a rate.
Whether three owners in four are right. 159 owners could not be checked from outside. Among the 58 we could check, 1 entry had no owner and 4 drew a warning, and that rate does not carry over to the rest.
Whether a disagreement means a wrong owner. CODEOWNERS and the catalog can mean different things by "owner".
Whether runbooks and on-call links rot. None were identified in public catalogs.
How private catalogs compare. Every entry here is in a public repository.
Whether a wrong entry ever caused a bad page."Can be worse than none" is an argument about how people use catalogs during incidents, not something this data measures.

Summary

Check a catalog against the systems it describes, not against itself. Ours found 21 entries out of 217 where GitHub contradicts the entry or its CODEOWNERS, all of which looked fine in their own files.
Ownership is where the records disagree. Where two records of the owner existed, half disagreed or named an owner GitHub could not resolve.
From outside an org, most owners cannot be checked at all. Run the check where the token can see your teams, and map directory groups to GitHub teams explicitly.
Make owners the same string everywhere by deriving them from GitHub teams or your identity provider, then compare catalog and CODEOWNERS in CI.
Run the check on a schedule too: a team deleted in a reorg changes no file.
Retire entries nobody claims. A gap sends people asking; a wrong answer can send them to the wrong place.

The checker, the sample, every result and the tests are in The-DevOps-Daily/service-catalog-drift. Run it against your own catalog, with a token that can see your teams, and see how your numbers compare.

Originally published at devops-daily.com.

PHP2Shell: How to Check That Every PHP You Run Is Actually Patched

DevOps Daily — Thu, 22 Oct 2026 09:00:00 +0000

On 21 October 2026 the PHP project released PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35 to fix CVE-2026-109618 and CVE-2026-109619, the pair now called PHP2Shell. Both are rated critical, CVSS v4.0 9.2. Unauthenticated remote code execution has been demonstrated against Apache with mod_php before any application code runs, and PHP says every installation that serves web requests is at risk.

So update. The trouble is that "we updated" can be true while the server is still vulnerable. Ubuntu and RHEL fix PHP without changing the version number, so php -v says the same thing before and after. PHP-FPM and Apache keep running the old code until they restart. A fixed base image fixes nothing until your image is rebuilt and the old containers are gone. This post shows how to find every PHP you run, how to prove each one is fixed, and what the interim settings do and do not block. It comes with a read-only check script, and a registry scan of 39 popular PHP images: by their own metadata, 8 still shipped a vulnerable PHP 4.4 hours after the scheduled release, and 3 still did at 5.5 hours.

What to do now

Update PHP everywhere to 8.5.12, 8.4.27, 8.3.36 or 8.2.35, or to your distribution's fixed package. On Ubuntu and RHEL the fixed package keeps the old upstream version, so check the package version, not php -v.
Restart every PHP process : PHP-FPM, Apache with mod_php, php-cgi, LiteSpeed and FrankenPHP. Then confirm nothing still runs the replaced binary.
Rebuild and redeploy containers , then check what the running containers contain, not what the registry has.
If you cannot update today , set session.upload_progress.enabled=0 (and the other settings below) in php.ini or the FPM pool, restart, and treat it as partial. PHP says only the update is a complete fix.
PHP 8.1 and older get no upstream release. Use a vendor backport or move to 8.2 or newer.
Run the check on every host and keep the output as your evidence.

Prerequisites

Shell access to the hosts that run PHP, and root if you want to see processes owned by other users.
Python 3.6 or newer for the check script. It uses only the standard library.
The docker or podman CLI on hosts that run PHP in containers, and kubectl for clusters.

What was fixed

PHP published two advisories:

CVE-2026-109618 , GHSA-w8hw-w9fx-j79w, "RCE via OOM corruption". Out-of-memory handling can interrupt PHP while request data is only partly initialized, and request cleanup can then treat that data as valid.
CVE-2026-109619 , GHSA-6wmf-4rv5-p3rp, "RCE via stale global state". State owned by one request can survive after its memory is freed, and a later request handled by the same worker can use it.

Branch	Fixed in	Note
8.5	8.5.12
8.4	8.4.27
8.3	8.3.36
8.2	8.2.35	End of life on 31 December 2026
8.1 and older	no upstream release	Source backports for 7.4, 8.0 and 8.1 only

The timeline, all on 21 October unless noted (UTC):

20 October : PHP announced the release for 21 October at 14:00 UTC and reserved the right to release earlier if details became public.
02:29 : VirusTotal saw the first submission of a leaked archive with technical details.
13:34 : the GitHub advisories were published. Ubuntu's USN-8948-1 followed at 13:56.
14:00 : the scheduled release time. The php.net announcements carry no time of day; NVD published both CVEs at 14:17.
16:03 : the last of the 16 official php tags we scanned for 8.2 to 8.5 (linux/amd64) was built with a fixed PHP. Docker Hub showed the new version tags as pushed between 16:48 and 17:31.

The first report took about ten weeks to reach a release. Calif, which credits the discovery to Calif, OpenAI and Wiz, dates discovery to 1 August 2026, the first report to PHP to 11 August and the second to 21 August. Calif traces the earliest affected code path to PHP 5.4.0, and says a related out-of-memory issue found in a 2024 audit was left unfixed because exploitation was not proven. In OpenAI's account, Greg Brockman wrote that OpenAI had spent the past two months "leading a coalition to mitigate" the bug, found as part of its Patch the Planet initiative. OpenAI's own write-up is here.

Who is really exposed

PHP's advisory lists "all PHP installations serving web requests, including PHP-FPM, Apache with mod_php, php-cgi, FrankenPHP and LiteSpeed" and puts mod_php at the top. The GitHub advisories go into more detail per server API (SAPI):

SAPI	What the PHP advisories say
Apache mod_php	Highest risk. Unauthenticated RCE validated over HTTP with default limits, before any application code runs.
PHP-FPM	At risk. The CVE-2026-109619 session path was validated against FPM, but it needs a suitable application operation. For CVE-2026-109618, source analysis says FPM is susceptible, but no complete FPM exploit was shown. The advisory also names per-request settings sent by the web server (`PHP_VALUE` or `PHP_ADMIN_VALUE`) that produce a warning as a risk for FPM.
php-cgi	Listed as at risk by php.net. The advisories do not discuss it separately.
FrankenPHP	Listed. Source analysis says that, like mod_php, it is likely susceptible to CVE-2026-109618.
LiteSpeed (LSAPI)	Listed. Application-specific exploits are described as likely but not validated.
CLI	Not named. The bugs are in web request handling. But `php -S` serves HTTP from the CLI binary, and the CLI usually comes from the same source package as FPM and mod_php, so an old CLI is a sign of what else on the host is old.

Red Hat rates both bugs Important, not Critical, because "Standard Red Hat Enterprise Linux deployments use PHP-FPM rather than mod_php." That is a statement about the default, not about your server. PHP names mod_php as the highest-risk setup, so if you run mod_php on RHEL, put it first even though Red Hat's rating is Important.

Note

The sources disagree on the affected range. PHP's advisories and the CVE.org records list only 8.2 to 8.5. NVD marks every version below 8.2.35 as vulnerable, with no lower bound. Google Cloud says "All versions of PHP from 5.4 through 8.5.11", and PHP itself shipped source backports for 7.4, 8.0 and 8.1. Treat every PHP older than the fixed releases as affected, including end-of-life branches.

Why `php -v` does not tell you

Debian ships the new upstream release, so on Debian php -v changes when you are fixed: bookworm gets 8.2.35 and trixie gets 8.4.27. Ubuntu and RHEL backport the patch into the version they already ship. Ubuntu 24.04's fixed php8.3 still reports 8.3.6. So does the vulnerable one.

To show this with real packages, the repo's scripts/ubuntu-backport-demo.sh downloads php8.3-fpm and php8.3-common from the Ubuntu archive, from before the fix (.12) and with the fix (.13). It extracts them without installing or running anything, reads the PHP version compiled into php-fpm8.3, and searches the package changelog for the CVE IDs:

ubuntu-backport-demo.sh

$ scripts/ubuntu-backport-demo.sh
# 2026-10-21T18:26:56Z Ubuntu 24.04 php8.3 packages from http://archive.ubuntu.com/ubuntu/pool
sha256 60187aae56db2056e916d99ba4854415993429f12732ace1b5eb5225bb33f584 php8.3-fpm_8.3.6-0ubuntu0.24.04.12_amd64.deb
sha256 85e4fc03ff84fef7dfed923c99d1a2e0e6f49147a50bcd5eaca4e7cd05a835fa php8.3-common_8.3.6-0ubuntu0.24.04.12_amd64.deb
sha256 ae395e2c15c3b96da56da807a204f771a589eb5bf54a3e5e239b1f50ade2815b php8.3-fpm_8.3.6-0ubuntu0.24.04.13_amd64.deb
sha256 801cae4e842e1b9e9d60a542bf18ba831bf504b2f3070404892bf3f8fb943ea4 php8.3-common_8.3.6-0ubuntu0.24.04.13_amd64.deb

package 8.3.6-0ubuntu0.24.04.12
  php-fpm8.3 embeds: PHP/8.3.6
  changelog top: php8.3 (8.3.6-0ubuntu0.24.04.12) noble-security; urgency=medium
  PHP2Shell IDs: none
package 8.3.6-0ubuntu0.24.04.13
  php-fpm8.3 embeds: PHP/8.3.6
  changelog top: php8.3 (8.3.6-0ubuntu0.24.04.13) noble-security; urgency=medium
  PHP2Shell IDs: CVE-2026-109618 CVE-2026-109619

Both binaries carry the same version. Only the package version and its changelog say which one has the fix. This is what the fixed packages looked like on the evening of 21 October:

System	Fixed package	Does `php -v` change?
Debian 12 bookworm	php8.2 8.2.35-1~deb12u1 (DLA-4848-1)	Yes, to 8.2.35
Debian 13 trixie	php8.4 8.4.27-1~deb13u1 (DSA-6574-1)	Yes, to 8.4.27
Ubuntu 26.04	php8.5 8.5.4-0ubuntu1.5 (USN-8948-1)	No, stays 8.5.4
Ubuntu 24.04	php8.3 8.3.6-0ubuntu0.24.04.13	No, stays 8.3.6
Ubuntu 22.04	php8.1 8.1.2-1ubuntu2.28	No, stays 8.1.2
Ubuntu 14.04 to 20.04 (ESM)	php5, php7.0, php7.2, php7.4 (USN-8948-2)	No
RHEL 8.10 EUS, php:7.4 module	php-7.4.33-8.module+el8.10.0+25117+30337cca (RHSA-2026:87080)	No, stays 7.4.33
RHEL 8 php:8.2, RHEL 9, RHEL 10	No fix listed at 16:49 UTC
packages.sury.org (Debian 12, arm64)	Still at 8.2.34, 8.3.35, 8.4.26 and 8.5.11 at about 18:10 UTC	Yes, when it lands
rpms.remirepo.net	php.net: "The repo will not have updates today"	Yes, when it lands

The same trap works in the other direction. A scanner that compares only the upstream version reports fixed Ubuntu and RHEL packages as vulnerable.

The right check compares the package version with your distribution's fixed version for that package, or finds both CVE IDs in the changelog of the package that owns the binary. The examples use Ubuntu 24.04's php8.3-fpm; take the fixed version for your release and package from its USN, DSA or RHSA:

# Ubuntu 24.04, php8.3-fpm from the Ubuntu archive (fixed in USN-8948-1)
dpkg --compare-versions "$(dpkg-query -W -f='${Version}' php8.3-fpm)" ge 8.3.6-0ubuntu0.24.04.13 \
  && echo fixed || echo NOT fixed

# Which package owns the binary that actually runs, and what its changelog says
dpkg-query -S "$(readlink -f /usr/sbin/php-fpm8.3)"
zcat /usr/share/doc/php8.3-fpm/changelog.Debian.gz | head -1
for id in CVE-2026-109618 CVE-2026-109619; do
  zgrep -q "$id" /usr/share/doc/php8.3-fpm/changelog.Debian.gz && echo "$id listed" || echo "$id NOT listed"
done

# RHEL, AlmaLinux, Rocky Linux: ask about the package that owns the binary
rpm -qf "$(readlink -f /usr/sbin/php-fpm)"
for id in CVE-2026-109618 CVE-2026-109619; do
  rpm -qf --changelog "$(readlink -f /usr/sbin/php-fpm)" | grep -q "$id" && echo "$id listed" || echo "$id NOT listed"
done

# Alpine packages carry the upstream version
apk info -v | grep -E '^php[0-9]+(-fpm|-cgi|-apache2|-embed)?-[0-9]'

Two details make the changelog check less simple than it looks:

On Debian and Ubuntu, /usr/share/doc/php8.3-fpm is usually a symlink to php8.3-common's docs. If an upgrade was partial and php8.3-fpm was held back, you read the newer package's changelog and conclude the old FPM binary is fixed. Check that the first line of the changelog names the version that dpkg-query -W php8.3-fpm reports. The check script refuses a changelog whose top entry does not match the installed package.
Container images often strip /usr/share/doc. Then there is no changelog to read, and the package version against the distribution's tracker is the only evidence.
A changelog line is a mention, not proof. A fix that was later reverted would still be listed. Treat it as supporting evidence next to the package version.

Restart, or the old code keeps running

Upgrading the package replaces the files on disk. The PHP-FPM master and its workers, and every Apache child with mod_php loaded, still have the old binary mapped in memory. Linux marks a mapped file that has been replaced as (deleted), which gives you a direct check:

# Run as root, or processes owned by other users are not visible.
# PHP-FPM, php-cgi, LiteSpeed and FrankenPHP processes on a replaced binary
sudo ls -l /proc/[0-9]*/exe 2>/dev/null | grep -E '(php|frankenphp).*\(deleted\)'

# Apache processes that still map a replaced libphp
sudo grep -l 'libphp.*(deleted)' /proc/[0-9]*/maps 2>/dev/null

Restart the service (systemctl restart php8.3-fpm, systemctl restart apache2), run both commands again until they print nothing, and then check the version or package of the binary the service runs. An empty result only says no replaced file is still mapped; it does not say the file on disk is the fixed one. needrestart on Debian and Ubuntu and dnf needs-restarting on RHEL also list processes that need a restart after an upgrade. PHP's own advice for the interim settings is to "restart every worker" as well, because a setting changed in php.ini does not reach a worker that is already running.

Containers: rebuild, redeploy, then check what is running

A container keeps the PHP it was built with. Google's guidance for Cloud Run says it plainly: services without automatic base image updates "must be rebuilt and redeployed", and "Container images you build yourself are your responsibility."

Rebuild with --pull, so the builder fetches the new base instead of a cached one:

docker build --pull -t registry.example.com/app:2026-10-21 .

If your Dockerfile pins a digest (FROM php:8.4-fpm@sha256:...), update the pin. If it pins a version tag, move to 8.4.27-fpm or later. Then deploy the new tag or digest. On Kubernetes, kubectl rollout restart with a reused tag and imagePullPolicy: IfNotPresent starts the new pods from the image already cached on the node, which is the old one.

Then check the containers that are running, not the image you meant to deploy:

# Docker hosts (use podman the same way): the PHP CLI version in each running container
docker ps --format '{{.ID}} {{.Image}} {{.Names}}' | while read -r id image name; do
  printf '%-30s %-40s ' "$name" "$image"
  docker exec "$id" php -r 'echo PHP_VERSION, PHP_EOL;' 2>&1 || echo "PHP CLI check failed"
done

# Kubernetes: the image digests your pods are actually running
kubectl get pods -A -o jsonpath='{..imageID}' | tr ' ' '\n' | sort -u

The CLI version is a first look, not proof for the web server: a container can have a fixed CLI and an older FPM or mod_php, so check the binary that serves requests (php-fpm -v, or its package) as well. The repo's registry_scan.py takes repository-qualified digests (registry/repository@sha256:...) as arguments and reads the PHP version from each image's metadata without pulling it. Some runtimes report imageID in other forms, such as a bare digest, which you need to map back to a repository first. That works for images built FROM php, which carry ENV PHP_VERSION. For images that install PHP from distribution packages, look inside a running container, because the version is not in the metadata.

How fast the popular images were rebuilt

We wanted to know how long "pull the new image" takes to become true, so registry_scan.py read the PHP version of 39 popular images from registry metadata only: the manifest and the image config, a few KB each, checked against their digests, never a layer. The version is the one the image declares in ENV PHP_VERSION (or Bitnami's APP_VERSION), which images built FROM php inherit. An image could replace PHP after that, so this is what the images report, not a look at the binary. The list is the official php image for 8.1 to 8.5, the official application images built on it, and widely used community images. images.txt in the repo says why each one is there. All results are for linux/amd64.

We ran the scan twice, at 18:21 UTC and at 19:32 UTC, 4.4 and 5.5 hours after the scheduled release:

39 popular PHP images by the PHP version their metadata reports

	Value	Series
18:21 UTC: vulnerable	8	vulnerable
18:21 UTC: unknown	2	unknown
18:21 UTC: patched	29	patched
19:32 UTC: vulnerable	3	vulnerable
19:32 UTC: unknown	2	unknown
19:32 UTC: patched	34	patched

linux/amd64. PHP version from ENV PHP_VERSION (or Bitnami APP_VERSION) in each image config, read without pulling layers. Fixed: 8.5.12, 8.4.27, 8.3.36, 8.2.35; 8.1 and older count as vulnerable. Unknown: no full version in the metadata. Files: results/scan-20261021T182129Z.json, results/scan-20261021T193200Z.json.

All 16 official php tags we scanned for 8.2 to 8.5 reported a fixed PHP in both runs, built 1.8 to 2.1 hours after the scheduled release.
At 18:21, 8 images still shipped a vulnerable PHP : php:8.1-fpm and php:8.1-apache (8.1.34, last built in December 2025; there is no fixed 8.1 release to rebuild with), joomla:latest (8.4.26), nextcloud:latest and nextcloud:fpm (8.5.11), phpmyadmin:latest (8.3.35), yourls:latest (8.5.11) and laravelsail/php84-composer:latest (8.4.1, built in November 2024). The Laravel documentation uses that image to run Composer, not to serve traffic.
At 19:32, 3 were still vulnerable : the two php:8.1 tags and laravelsail/php84-composer:latest. joomla, both nextcloud tags, phpmyadmin and yourls had fixed images by then.
2 images could not be judged from metadata in either run: thecodingmachine/php:8.4-v5-fpm sets PHP_VERSION=8.4 without the patch level, and linuxserver/nextcloud installs PHP from Alpine packages and records no PHP version at all.

Hours from the scheduled 14:00 UTC release to the image build

Series	Samples	Min	Median	p95	Max
Official php 8.2 to 8.5 (16)	16	1.83 h	1.935 h	2.06 h	2.06 h
Official app images (13)	13	2.47 h	2.53 h	4.08 h	4.08 h
Community images (5)	5	2.23 h	3.68 h	4.05 h	4.05 h

Patched images only, by the 'created' time in each linux/amd64 image config, from results/scan-20261021T193200Z.json. This is when the image was built, not when the registry started serving it. Images still on a vulnerable PHP are not on this chart.

One result changes how you should read a registry. Docker Hub listed joomla:latest as pushed at 15:46 UTC, after the release. Its amd64 image was built at 01:52 UTC with PHP 8.4.26. So a fresh "last pushed" time is not evidence that the image you pull has a fixed PHP. The build time is not either: the fixed joomla, phpmyadmin and yourls images that the registry served at 19:32 were built between 16:28 and 16:36 UTC, but at 18:30 the registry still served the old ones for amd64. nextcloud:latest and nextcloud:fpm looked the same at 18:21; by 18:30 their amd64 images had moved to 8.5.12. To check the method, at 18:30 UTC we compared the amd64 digest from the Docker Hub API with the registry's own index for those three images and two others: they agreed for all five (results/method-check-20261021T183037Z.txt).

The interim settings and their limits

If you cannot update today, PHP lists settings that reduce exposure to specific paths. Its own words: they "reduce exposure to specific attack paths but do not make an installation safe."

For CVE-2026-109619, turn off session upload progress, and on PHP 8.5 fatal error backtraces. Set them where they apply before PHP parses the request: php.ini, the FPM pool, or the Apache config. PHP says not to rely on ini_set() or .user.ini, which apply too late.

; php.ini (and the php.ini of every SAPI: cli, fpm, apache2, cgi)
session.upload_progress.enabled = 0
; PHP 8.5 only
fatal_error_backtraces = 0


; PHP-FPM pool, for example /etc/php/8.3/fpm/pool.d/www.conf
php_admin_flag[session.upload_progress.enabled] = off


# Apache with mod_php
php_admin_flag session.upload_progress.enabled off

Then restart every worker. What this does and does not do, per the advisory:

It "prevents the demonstrated early session producing and consuming operations. It does not mitigate the other lifecycle paths." fatal_error_backtraces=0 "blocks the confirmed fatal backtrace path only."
PHP lists it under CVE-2026-109619 only. For CVE-2026-109618 PHP suggests enforcing memory_limit=-1 so applications cannot override it, and warns that an underlying allocation failure can still reach the bug. With no PHP memory limit, use a container or cgroup limit as the availability boundary, and do not use RLIMIT_AS or RLIMIT_DATA, which can cause the failure you are trying to avoid.
Hiding warnings does not help. Changing error_reporting, display_errors or log_errors, or using @, does not protect against CVE-2026-109619. Instead, remove per-request settings that produce warnings or deprecation notices, including PHP_VALUE or PHP_ADMIN_VALUE lines your web server passes to FPM.
If your application does not use multipart form data, PHP suggests blocking it at the frontend. It also suggests a low post_max_size (8M for applications that do not need file uploads, 32M for those that do) and proxy limits that also cover requests without a usable Content-Length.
Edge rules help but do not replace the update. Help Net Security reports that Cloudflare deployed WAF rules for the known exploit vectors to all customers, including Free, and VirusTotal reports its CTO saying that only updating PHP fully protects a site.

PHP also suggests keeping session.upload_progress.enabled=0 after you patch if you do not use upload progress bars, "as more attackers try to find flaws in this hardly-used and hardly-tested feature."

PHP 8.1 and older

The PHP project does not release fixes for 8.1 or older. It published minimal source backports for 7.4, 8.0 and 8.1 for people who build from source, and points everyone else to their vendor. Some vendors have shipped them: Ubuntu for 22.04's PHP 8.1 and, under ESM, for 14.04 to 20.04, and Red Hat for the RHEL 8.10 EUS php:7.4 module.

The official php:8.1 tags we scanned still reported 8.1.34, built in December 2025. Google's guidance says App Engine applications on PHP 5.5, 7.2, 7.3, 7.4 and 8.1 remain vulnerable and must move to 8.2 or newer. VirusTotal counted about 758,000 internet-facing hosts that announce an end-of-life PHP version from 5.4 to 8.1, out of about 1.1 million that announce a version at all. That counts exposed PHP, not confirmed vulnerable hosts, but it shows how much of the long tail has no simple update.

If you are on 8.2, the same applies on 31 December 2026, when it reaches end of life.

Run the check

The-DevOps-Daily/php2shell-check on GitHub

php2shell_check.py is one file of standard-library Python, so you can copy it to a server and run it. For each PHP it finds, it reports the SAPI, the version compiled into the binary or Apache module (so mod_php is checked without running Apache; -v is the fallback for binaries without a readable string), the package that owns the file, and whether that package's changelog names both vulnerabilities by CVE or GHSA ID. A changelog only counts if it belongs to the installed package and dpkg --verify or rpm -V confirms the binary still matches that package. It also reports the interim settings from a fresh php -i, the FPM pool files, or the parsed php.ini files for mod_php, plus host web server lines that set them or pass PHP_VALUE. A running service started with other options can differ. Then it lists running PHP processes and marks any that still run a replaced binary. Through the first container CLI it finds (docker, then podman), it checks the PHP binaries inside each running container that has a PHP process, and adds an unknown result when no checked binary covers the kind of PHP process running there (for example an FPM process but only a CLI binary found). The interim settings are reported only: they never turn a vulnerable result into a pass.

This is the real output on the Raspberry Pi we build on, which has the PHP 8.4.23 CLI from packages.sury.org and no web server:

php2shell-check on the Pi

# 2026-10-21T18:45:30Z Raspberry Pi, Debian 12, PHP CLI from packages.sury.org, no web server, run as a normal user
$ python3 php2shell_check.py
php2shell-check: fixed releases 8.2.35, 8.3.36, 8.4.27, 8.5.12

PHP on disk
  [vulnerable] /usr/bin/php8.4 cli
      version: 8.4.23 (embedded string)
      exposure: CLI: not a web SAPI unless it runs `php -S`; same package line often ships FPM/mod_php
      also as: /bin/php, /bin/php8.4, /usr/bin/php
      package: php8.4-cli 8.4.23-1+0~20260703.52+debian12~1.gbp817a0c (dpkg); CVE IDs in changelog: none
      why: upstream 8.4.23 < 8.4.27
      session.upload_progress.enabled: On [php8.4 -i]
      memory_limit=-1 post_max_size=8M

Running PHP processes
  none

Containers
  none checked
note: no docker or podman CLI found; container processes are judged by the version in /proc

result: VULNERABLE or restart needed (exit 1)
$ echo $?
1

The exit code is 1 when anything is vulnerable or needs a restart. Otherwise it is 2 when anything could not be checked, and 0 when every PHP it found is patched or it found no PHP at all (it says which). Anything left unknown, such as a changelog that lists only one of the two CVEs on an older version, a pre-release build, or an unreadable process, gives 2 rather than 0, unless something else is vulnerable, which gives 1. It only knows the paths and names it searches, so a 0 means "nothing vulnerable found", not proof that no other PHP exists. To run it across hosts without copying it first:

for h in web1 web2 web3; do
  ssh "$h" 'sudo python3 - --json' < php2shell_check.py > "php2shell-$h.json"
  echo "$h exit $?"
done

The test suite builds fake hosts with the cases a version check gets wrong: an Ubuntu backport, the same version without the update, a partial upgrade whose symlinked changelog belongs to another package, an old binary copied over a patched package, a changelog with only one of the two CVEs, a source build where FPM was left on the old version, a process running a deleted binary, and containers on old and new images.

How to prove you are fixed

For each host, keep evidence of all of these, not just the first:

Package : the upstream version is at or above the fixed release, or the package version is at or above your distribution's fixed version.
Processes : no PHP-FPM, php-cgi, LiteSpeed or FrankenPHP process runs a (deleted) binary, and no Apache process maps a (deleted) libphp.
Containers : the binary that serves requests inside each running container (php-fpm, mod_php, FrankenPHP) has a fixed version or a fixed package, not just the CLI. The image digest you deployed is the one running.
Images : every image you build FROM a PHP base has been rebuilt since the fix. The registry scan of its digest is a quick inventory check; the container check above is the proof.
Exposure while vulnerable : if a host was internet-facing and unpatched, check it for compromise. The NCSC asks for this, and Wiz suggests looking for unusual multipart requests followed by memory-limit errors or repeated HTTP 500 responses, unexpectedly large or malformed PHPSESSID response cookies, and unexplained worker churn.

What we could not check

Containers on a real host. The Pi we build on has no Docker, so the container checks are covered by the test fixtures only. The registry scan read real images.
Web servers. The Pi runs PHP from the command line only. mod_php and FPM detection are covered by the tests.
Whether the interim settings block anything. We did not test any part of the bugs, and the tool does not. PHP's own text is the source for what the settings cover.
Later distribution updates. The distribution table is as of the evening of 21 October. At 16:49 UTC Red Hat listed a fix only for the RHEL 8.10 EUS php:7.4 module, the Debian 12 arm64 index on packages.sury.org still had the old versions at about 18:10 UTC, and PHP said Remi would not have updates that day.
Other architectures. The registry scan reads linux/amd64 only. We did not measure arm64 images.
Unusual install paths. The script searches distribution paths, Remi, cPanel, Plesk, CloudLinux alt-php, LiteSpeed, aaPanel and $PATH. PHP elsewhere, .htaccess overrides and stopped containers are not covered.

Checklist

Every host: PHP package updated, checked against the distribution's fixed version, not php -v
Every PHP-FPM pool, Apache, php-cgi, LiteSpeed and FrankenPHP service restarted, with no (deleted) PHP binaries left in /proc
Every image built FROM a PHP base rebuilt with --pull and redeployed by new tag or digest
Running containers checked inside, including the binary that serves requests, and their image digests matched to what you deployed
PHP 8.1 and older: vendor backport installed or upgrade to 8.2+ scheduled, with interim settings in place until then
Interim settings applied where updates wait, in php.ini or the pool, with workers restarted
session.upload_progress.enabled=0 kept where upload progress bars are not used
Internet-facing hosts that were unpatched checked for signs of compromise
Check script output saved per host as evidence

Every fact in this post is listed with its source in the repo's SOURCES.md, together with the points where the sources disagree.

Originally published at devops-daily.com.

What Keeps a Neon Compute Awake, and What It Costs

DevOps Daily — Wed, 21 Oct 2026 09:00:00 +0000

We pointed a health check at a Neon compute every 5 minutes, the same as Neon's default scale-to-zero timeout. The compute never suspended. In 40 minutes it took 8 checks, woke once for the first one, and stayed awake the whole time. Over 30 days that is 180 CU-hours at the smallest compute size: about $19 on the Launch plan, $40 on Scale, and on the Free plan it uses up the 100 CU-hour allowance by day 17.

The suspect most people reach for first, an idle connection left open by the app, does not keep a compute awake. We held idle connections, direct and pooled, and the compute suspended every time, a few seconds later than with nothing open. What the suspend did do was cut those connections, including the pooled ones.

This post measures which ordinary clients keep a Neon compute running and which do not, how long a compute takes to suspend, what your app sees when it does, and what each pattern costs per month. Everything runs against real computes in one Neon project, watched through the Neon API, and the scripts are public so you can run them on your own account.

TL;DR

Idle connections do not stop scale to zero. With a 60-second timeout the compute suspended 66 to 75 seconds after the last query with nothing open, and 81 to 83 seconds with an idle direct or pooled connection held. The suspend dropped every held connection, pooled ones included.
An open transaction keeps it awake for as long as it stays open. A session left idle in transaction, direct or through the pooler, kept the compute up for the whole 6-minute window, 4 runs out of 4. Setting idle_in_transaction_session_timeout = '30s' let it suspend again.
A monitoring agent that queries every 20 seconds keeps it awake , even when it only reads pg_stat_database on the postgres database.
A LISTEN connection does not keep it awake. It is dropped on suspend, and a notification sent after the suspend never reached it (2 of 2 runs).
Health checks are the expensive one. With the default 5-minute timeout, a check every 5 minutes kept the compute awake for the whole 40-minute run. In our runs the suspend came 312 to 327 seconds after a wake, so each check arrived before it. A check every 6 minutes still kept it awake 90.6% of the time, and every 10 minutes 52.1%.
A wake adds roughly half a second to a second. End-to-end checks (connect, authenticate, query, close) that woke the compute had medians of 847 to 1,134 ms per run; checks on an awake compute took 308 to 339 ms, from a client 48 ms from the region.

Prerequisites

A Neon account and an API key (Account settings > API keys). The 60-second timeout used in part of the test needs a plan that lets you change it (Scale); everything with the default 5-minute timeout works on any plan.
Node.js 22 or newer.
Basic Postgres knowledge: sessions, transactions, pg_stat_activity.

How Neon decides a compute is idle

Neon's documentation says a compute scales to zero after 5 minutes of inactivity, that you can change the timeout on paid plans, and that a connected logical replication subscriber prevents the suspend. It does not say what counts as activity.

The answer is in the compute's own code. Every compute runs compute_ctl, and its activity monitor checks Postgres every 500 ms. It moves the "last active" time forward when:

The totals in pg_stat_database change: the summed active_time (time spent running queries) or the number of sessions, across all databases except postgres and the templates. This catches a query that started and finished between two checks, and every new connection. The code calls this path experimental but says it is on for everyone.
Any client backend is in a state other than idle. That includes active (a query is running) and idle in transaction (a transaction is open but doing nothing). Backends of Neon's own cloud_admin role are left out here.
An idle backend changed state recently: the latest state_change among idle backends counts as the last activity (the backend check). A connection that ran a query 20 seconds ago and went idle moves the clock to then. Connecting moves it once too, but a connection that then just sits idle does not keep moving it.
A logical replication walsender exists, an active logical replication subscription exists, or an autovacuum worker is running.

The control plane reads the last activity time and suspends the compute once the timeout has passed. This is the open-source code at commit f3a0e4f255; the version running in production may differ, which is one more reason to measure.

So the code predicts that an idle connection should not keep a compute awake, and an open transaction should. The rest of this post checks those predictions and puts numbers on them.

How we measured

The-DevOps-Daily/neon-keep-awake on GitHub

One Neon project in aws-eu-central-1 running Lakebase Postgres (PostgreSQL 17), every compute fixed at 0.25 CU so CU-hours are easy to read off. Each health-check configuration gets its own branch, and each branch has its own compute, so configurations running side by side do not affect each other. The idle-client scenarios ran one after another on a single branch.

The compute's state comes from GET /projects/{id}/endpoints/{id}, which returns current_state and suspended_at. That is a control-plane call: it never touches the compute, so watching cannot keep it awake. The scripts read it every 2 seconds for the idle clients, every 3 seconds for the health checks, and every 15 seconds for the long unattended watch.

Awake time is an estimate. An awake period starts when a check that found the compute suspended began connecting, and ends at the suspended_at that Neon reports, which is in whole seconds. We checked it against Neon's own record: the project's operations log lists every start_compute and suspend_compute. For the health-check runs, the awake time from the log and from our polling differed by at most half a percentage point. That is a cross-check of the timing, not of the bill.

That check caught one bug of ours. A poll taken a moment after a check woke the compute can still show the previous suspend, and our first summary ended the awake period there. The fix is to ignore a poll whose suspended_at is older than the wake, and even that went wrong at first: Neon's timestamps are whole seconds (17:57:46Z) and ours have milliseconds (17:57:46.649Z), and compared as strings 46Z sorts after 46.649Z. The bug cost two runs 3 to 4 seconds of awake time each, small but in the direction that makes the bill look smaller. The summary now compares timestamps as dates. A review of the draft found a second one: the runner could fire one extra check right at the end of a window. The summary now counts only checks scheduled inside the window, which took one check off two of the runs.

The scripts are split by stage, and each writes JSON lines stamped with a hash of the code that wrote it:

neon-keep-awake

$ npm run setup
project odd-poetry-65586042, main branch br-icy-recipe-b91wrumq, compute ep-weathered-shadow-b91petoo
# stage 1: idle clients, 60 s timeout
$ npm run idle-clients
none #1: suspended at 2026-10-18T17:52:15Z (wake 694 ms)
none #2: suspended at 2026-10-18T17:53:30Z (wake 947 ms)
none #3: suspended at 2026-10-18T17:54:46Z (wake 907 ms)
idle-direct #1: suspended at 2026-10-18T17:56:16Z (wake 815 ms)
idle-direct #2: suspended at 2026-10-18T17:57:46Z (wake 805 ms)
idle-direct #3: suspended at 2026-10-18T17:59:16Z (wake 793 ms)
idle-pooled #1: suspended at 2026-10-18T18:00:45Z (wake 774 ms)
idle-pooled #2: suspended at 2026-10-18T18:02:17Z (wake 787 ms)
idle-pooled #3: suspended at 2026-10-18T18:03:46Z (wake 804 ms)
idle-in-transaction-direct #1: still active after 360s (wake 815 ms)
idle-in-transaction-direct #2: still active after 360s (wake 1020 ms)
idle-in-transaction-pooled #1: still active after 360s (wake 940 ms)
idle-in-transaction-pooled #2: still active after 360s (wake 792 ms)
idle-in-transaction-with-timeout #1: suspended at 2026-10-18T18:34:31Z (wake 1080 ms)
idle-in-transaction-with-timeout #2: suspended at 2026-10-18T18:36:16Z (wake 895 ms)
listen-direct #1: suspended at 2026-10-18T18:37:46Z (wake 876 ms)
listen-direct #2: suspended at 2026-10-18T18:39:16Z (wake 296 ms)
monitor-postgres-db-20s #1: still active after 360s (wake 263 ms)

The wake figure in that log is the time to connect and run SELECT 1 at the start of each run. Most runs started right after a suspend, so it is usually a cold start. The last two started on a warm compute: the previous LISTEN run had just woken it to send its test notification, which is why they are near 300 ms.

Idle connections do not keep it awake, and they do not survive the suspend

Stage 1 sets the suspend timeout to 60 seconds, wakes the compute, leaves one kind of client open, and times the suspend from our last query.

What was left open	Runs	Suspended	Seconds from the last query to the suspend
Nothing	3	3 of 3	75, 68, 66
One idle direct connection	3	3 of 3	82, 81, 83
One idle pooled connection (PgBouncer)	3	3 of 3	81, 83, 81
A `LISTEN` connection	2	2 of 2	83, 79

An idle connection did not stop the suspend. The suspend did come later: the median was 82 seconds with an idle direct connection and 81 with a pooled one, against 68 with nothing open. With three runs per row, all on the same compute one after another, that is an observation we cannot explain, not an established effect. Either way, a compute with a 60-second timeout went to sleep in under a minute and a half with connections still open.

Look at what happened to those connections. Five seconds after each suspend, the script ran SELECT 1 on the connection it had held:

idle direct: 'error' event: Connection terminated unexpectedly
              next query: Client has encountered a connection error and is not queryable
idle pooled: 'error' event: Connection terminated unexpectedly
              next query: Client has encountered a connection error and is not queryable

Every held connection was dead, 6 out of 6, pooled ones included, and the LISTEN connections in the last row died the same way. The pooler did not keep its client connections open across the suspend, so a pooled connection string does not protect your app from one.

For most apps the fix is to make sure a dropped connection is handled, not fatal. With node-postgres, use Pool, not a single long-lived Client, and handle the pool's error event. Without a listener, an idle client that loses its socket throws an unhandled error and can crash the process:

import pg from 'pg';

const pool = new pg.Pool({
  connectionString: process.env.DATABASE_URL,
  // Close idle clients before the compute's suspend timeout closes them for you.
  idleTimeoutMillis: 30_000,
});

// An idle client whose socket closed (for example, the compute suspended).
// The pool removes it; this listener stops it from crashing the process.
pool.on('error', (err) => console.warn('idle client dropped:', err.message));

// Retry once on a dropped connection, but only when the caller says the
// statement is safe to run twice.
export async function query(sql, params, { retry = false } = {}) {
  try {
    return await pool.query(sql, params);
  } catch (err) {
    if (retry && /Connection terminated/.test(err.message)) return pool.query(sql, params);
    throw err;
  }
}

The retry is off by default for a reason. When a connection drops, you cannot always tell whether the statement ran before it did, so only retry statements with no side effects or with an idempotency key. A plain read qualifies; an INSERT without a key does not, and neither does a SELECT that calls a function with side effects. The error check also catches only this one message, so treat it as a starting point, not complete coverage.

LISTEN does not keep it awake either

A connection waiting on LISTEN sits in the idle state, so the monitor treats it like any idle connection. Both runs suspended on schedule. After each suspend the script opened a new connection, which woke the compute, and sent NOTIFY keep_awake, 'after-suspend'. The listener never got it, in either run, because its connection had been closed by the suspend.

If you use LISTEN/NOTIFY for cache invalidation or live updates on a compute that scales to zero, the listener must reconnect and run LISTEN again, and it must not assume it saw every notification sent while it was gone. Read the current state after each reconnect.

An open transaction keeps it awake for as long as it stays open

The monitor counts any backend that is not idle as activity, and a session that ran BEGIN and then stopped is idle in transaction. That is the classic leaked transaction: a request handler that checks out a client, starts a transaction, and returns on an error path without COMMIT or ROLLBACK.

What was left open	Runs	Suspended within 6 minutes	After the transaction ended
`BEGIN` on a direct connection, then nothing	2	0 of 2	suspended 65 and 72 s later
`BEGIN` on a pooled connection, then nothing	2	0 of 2	suspended 68 and 72 s later
The same, with `idle_in_transaction_session_timeout = '30s'`	2	2 of 2, 102 and 98 s after the last query

Through the pooler it behaves the same way. In transaction mode, PgBouncer pins a server connection to the client for as long as the transaction is open, so Postgres still sees a backend idle in transaction.

With the timeout set, Postgres ended the leaked session after 30 seconds, and the compute suspended about 70 seconds after that, the same delay as with nothing open. That setting is the one line worth adding to every Neon project that scales to zero, at the database or role level so it applies to every connection:

-- Ends sessions that hold a transaction open with nothing running.
ALTER DATABASE neondb SET idle_in_transaction_session_timeout = '60s';

We checked this form too, with 30s: the database owner role that Neon creates is allowed to run it, a new connection reported 30s without setting anything itself, and a transaction leaked on that connection no longer kept the compute up. It suspended 101 seconds after the last query (scripts/07-database-timeout.mjs).

Pick a value longer than any legitimate pause inside your transactions. A leaked transaction is a bug anyway: it holds locks and stops vacuum from cleaning up. On a compute that scales to zero, it also bills you for every hour it stays open.

A monitoring agent keeps it awake

Monitoring agents such as postgres_exporter keep a connection open and run queries on a timer. We ran the same pattern: one connection to the postgres database, reading pg_stat_database every 20 seconds.

The compute stayed awake for the whole 6-minute window and suspended 63 seconds after the agent stopped. One run only, but it matches the code: each query moves the connection's state_change, and the backend check counts that whichever database the query runs in. Only the pg_stat_database totals leave the postgres database out.

So a database scrape every 15 or 30 seconds keeps the compute awake around the clock. If you want monitoring and scale to zero together, scrape less often than the suspend timeout and accept gaps, or read compute state from the Neon API, which does not wake the compute. That is how we watched compute state for this post without adding database activity.

Health checks: the interval decides everything

Uptime monitors, load balancer probes and Kubernetes readiness probes often run a query to prove the database is reachable. Stage 2 runs exactly that: every N seconds, open a new pooled connection, run SELECT 1, close it. Each configuration gets its own compute, starts from a suspended state, and runs for 12 to 60 minutes while the script records when the compute is awake.

Suspend timeout	Check every	Minutes	Checks	Checks that woke it	Awake	CU-hours per 30 days at 0.25 CU
60 s	30 s	12	24	1	100%	180
60 s	90 s	15	10	10	82.2%	148
60 s	120 s	16	8	8	70.2%	126
60 s	180 s	18	6	6	40.5%	73
300 s	60 s	30	30	1	100%	180
300 s	300 s	40	8	1	100%	180
300 s	360 s	42	7	7	90.6%	163
300 s	420 s	42	6	6	74.7%	135
300 s	600 s	60	6	6	52.1%	94

"Checks that woke it" counts the checks for which the last API reading before the check showed the compute suspended. When that number is 1, only the very first check of the run had to wake the compute, and every later check found it already awake.

Share of time the compute was awake, by health-check interval

	Value	Series
60 s timeout, check every 30 s	100%	60 s suspend timeout
60 s timeout, check every 90 s	82.2%	60 s suspend timeout
60 s timeout, check every 120 s	70.2%	60 s suspend timeout
60 s timeout, check every 180 s	40.5%	60 s suspend timeout
5 min timeout, check every 1 min	100%	5 min suspend timeout (default)
5 min timeout, check every 5 min	100%	5 min suspend timeout (default)
5 min timeout, check every 6 min	90.6%	5 min suspend timeout (default)
5 min timeout, check every 7 min	74.7%	5 min suspend timeout (default)
5 min timeout, check every 10 min	52.1%	5 min suspend timeout (default)

A new pooled connection and SELECT 1 every N seconds, one 0.25 CU compute per row, 12 to 60 minutes each, one run per row. Estimated by polling the Neon API; matches Neon's operations log within half a percentage point. Data: results/summary.json in The-DevOps-Daily/neon-keep-awake.

The reason a 5-minute check beats a 5-minute timeout is the time between a wake and the suspend that follows it. Where the compute suspended between checks, we can estimate that cycle, from the moment the check started connecting to Neon's suspended_at:

With the 60-second timeout, a compute woken by a single check suspended 71 to 87 seconds later (24 cycles, median 74 s).
With the 5-minute timeout, it suspended 312 to 327 seconds later (19 cycles on three computes, median 314 s).

So in our runs, with the default timeout, a check that came within 312 seconds of the previous one always found the compute still up and started the clock again, and a check every 300 seconds kept it up for the whole 40-minute run. That is what we observed, not a guaranteed grace period: the suspend came 12 to 27 seconds after the timeout here, and Neon does not document that gap.

Make the interval a little longer and the compute does get to sleep between checks, but not for long. Every check pays for a full cycle of about 5 minutes and 15 seconds, so every 6 minutes it was awake 90.6% of the time and every 7 minutes 74.7%. A check every 10 minutes woke the compute 6 times out of 6, and each wake kept it up 312 to 313 seconds for a query that took about a second. That is 52% of the hour, for a database that did nothing else.

What a wake costs your users

Letting the compute sleep has a price too: the first request after a quiet stretch waits for the compute to start.

	Median	Max	Checks
Check that found the compute awake	308 to 339 ms per run		59 checks, 3 runs
Check that had to wake it	847 to 1,134 ms per run	1,878 ms	46 wakes, 9 runs
Plain TCP connect to the region (no TLS, no query)	48 ms		10

These are end-to-end times for the whole check: open a new TLS connection, authenticate, run the query and close. That is why even a warm check takes about 300 ms from a client 48 ms away. In three of the runs only the first check woke the compute, so their wake figure is a single sample. Comparing across runs rather than pair by pair, waking the compute added roughly 0.5 to 0.8 seconds, and the slowest wake took 1.9 seconds. None of the 105 checks failed. For context, our Neon free tier benchmarks measured cold starts with a median of 568 ms and a p95 of 1.06 s from a client in the same region, with a different method.

For a health check, an extra second does not matter. For the first user request of the morning it might, and that is the real trade-off you are making when you decide how long the compute may sleep.

What it costs

Neon bills compute in CU-hours: compute size times the hours it runs. A compute kept awake by a light pattern like these sits at its minimum size, so the cost is the minimum CU times the hours awake (real load can scale it up, which costs more). With list prices from Neon's pricing page ($0.106 per CU-hour on Launch, $0.222 on Scale) and the Free plan's 100 CU-hours per project per month, for one 0.25 CU compute over 30 days with nothing else using it:

What keeps it awake	Awake	CU-hours per 30 days at 0.25 CU	Launch	Scale	Free plan
Health check every 1 or 5 min, default timeout	100%	180	$19.08	$39.96	allowance used up after 400 hours, about day 17
Health check every 6 min, default timeout	90.6%	163	$17.28	$36.19	used up around day 19
Health check every 10 min, default timeout	52.1%	94	$9.94	$20.82	fits, with 6 CU-hours left for real traffic
Leaked transaction, monitoring agent every 20 s	100% while it lasts	up to 180	up to $19.08	up to $39.96	as above
Idle connections, `LISTEN`	suspended normally	no sustained cost observed	none seen	none seen	none seen

On the Free plan, Neon's plans page says that when a project runs out of CU-hours, its compute is suspended until the next billing period. A 5-minute health check on a Free project, starting the month with the full allowance and nothing else using it, does not cost money; it takes the database down for the last 13 days of a 30-day month.

Two things make these numbers bigger in practice:

Minimum size. We fixed every compute at 0.25 CU. If your compute's autoscaling minimum is 1 CU, multiply by four: an always-awake 1 CU compute is 720 CU-hours a month, $76 on Launch and $160 on Scale.
Branches. Every branch with a compute has its own suspend timer. A preview environment per pull request, each with its own health check, keeps each preview branch's compute awake. Ten open previews with a 5-minute check are ten always-on computes.

Does Neon wake it on its own?

A May 2026 thread in Neon's GitHub discussions, neondatabase/neon#12900, describes Launch projects at 0.25 CU that each used about 6 CU-hours a day whatever their traffic, which is a compute awake around the clock. One reply blamed check_availability, an operation that Neon's control plane runs to check that a compute can start on a branch. We looked for it in two places.

Our test project. Stage 3 watched the main branch's compute, which nothing ever connected to, for 3.9 hours after it suspended, reading its state every 15 seconds. It never woke. The project's operations log, read at the end, shows no check_availability and no start_compute for it after the first suspend, and Neon's usage counter for that branch read 368 active seconds, about the six minutes the compute ran after we created the project.
An older project of ours. Its main compute suspended on 23 September, and we stopped using the project that evening. Its operations log shows one check_availability for that compute on 24 September, 929 ms long, and no start_compute for it afterwards. The compute's suspended_at still reads 23 September. Two weeks later Neon archived the idle branch.

That matches Neon's own description. The operations page says the check verifies that a compute can start, and a 2023 changelog entry says the availability checker runs on 0.25 CU computes and can use pre-started ones. So in these two records, no availability check restarted a suspended compute. That does not rule out other behavior on other projects, and we cannot tell what kept the computes in that thread awake. The patterns that kept ours awake were open transactions, frequent monitoring queries and frequent health checks.

What to change

Keep the database out of liveness checks. A liveness probe should check that the process is up, not that Postgres answers. If you must check the database, do it well below the suspend timeout's rate: with the default 5-minute timeout, every 10 minutes still kept the compute awake half the time, so think in hours, not minutes.
Check compute state through the API instead of with a query. GET /projects/{project_id}/endpoints/{endpoint_id} returns current_state without waking anything. A monitor that alerts on the API's view of the compute, and only queries the database when a user is actually using it, keeps scale to zero working.
Set idle_in_transaction_session_timeout at the database level. It ends leaked transactions, which otherwise keep the compute awake for as long as they stay open, and they are a bug anyway.
Scrape metrics less often than the timeout, or not from the database. A monitoring agent that queries every 15 to 30 seconds turns scale to zero off.
Expect connections to drop. Idle and LISTEN connections, direct or pooled, were closed when the compute suspended. Use a pool, handle its error event, retry only statements that are safe to repeat, and re-run LISTEN after a reconnect.
If you want it awake, say so. On paid plans you can raise the suspend timeout or turn scale to zero off. For a production database with steady traffic, that is often the right call. The compute bill is the same either way, but a setting you chose is explicit and predictable, and a health check that happens to keep the compute up is neither.

What we could not conclude

Sample sizes are small. Three runs for each idle-connection row, two for the transaction and LISTEN rows, one for the monitoring agent, one for the database-level timeout check, and one run per health-check row. The results agree with the activity-monitor code, which is why we trust them, but the later suspend we saw with idle connections (median 81 to 82 seconds against 68) is not explained, and those scenarios ran one after another on the same compute.
Open transactions were watched for 6 minutes. The code says such a session counts as activity for as long as it stays open, and all four runs stayed awake for the whole window; we did not run one for hours.
One region, one compute size. Everything ran in aws-eu-central-1 at 0.25 CU. The time from a wake to the suspend (71 to 87 seconds with a 60-second timeout, 312 to 327 seconds with 5 minutes) may differ in other regions or under load. It also differed between computes in the same project: with the 5-minute timeout, one compute's cycles took 325 to 327 seconds every time while the others took 312 to 314. We do not know why.
Cost is computed, not billed. Neon's per-branch usage counters (active_time_seconds, compute_time_seconds) read 0 for the short runs when we deleted their branches, and for the longer runs they were well behind our own measurement (for example 988 active seconds against our 1,877), so they seem to update with a delay. We could not check CU-hours against Neon's meter. We checked the awake time against Neon's operations log instead, and converted with list prices.
Health checks were on a precise schedule, and the gap after the timeout is not documented. A real uptime service has some jitter, and Neon may suspend sooner after the timeout on other computes or in other regions. Either could let a 5-minute check miss now and then; we did not test that.
We did not test logical replication or autovacuum. The code says both keep a compute awake, and Neon's docs say so for replication subscribers.

Summary

Scale to zero on Neon worked the way the compute's own activity monitor says it should. Idle connections, pooled or direct, and LISTEN connections did not hold a compute up; they were cut when it suspended. Activity did hold it up: a leaked transaction, a monitoring agent, a health check every 5 minutes with the default timeout. That last one is the expensive surprise. It costs a 0.25 CU compute 180 CU-hours a month, which is about $19 on Launch, $40 on Scale, or almost two weeks of downtime on Free.

Run the scripts against your own project to see which of these you have:

git clone https://github.com/The-DevOps-Daily/neon-keep-awake
cd neon-keep-awake && cp .env.example .env # add a Neon API key
npm install && npm run setup && npm run health-checks

Originally published at devops-daily.com.