DEV Community: Libme

pip "externally-managed-environment" Error: venv, pipx, or uv?

Libme — Sun, 01 Nov 2026 17:19:18 +0000

If pip install fails with error: externally-managed-environment, your operating system's package manager owns that Python, and pip is refusing to write into it (PEP 668). The fix is almost never --break-system-packages: use a virtual environment for project dependencies, pipx or uv tool for command-line applications, and reserve the override flag for throwaway containers.

What does the error actually say, and why does it happen?

On a recent Debian, Ubuntu, or Homebrew-installed Python, this is what you get:

$ pip install requests
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    ...
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at your own risk, by passing the --break-system-packages flag.

The mechanism is a marker file. Distributions that ship PEP 668 support place a file named EXTERNALLY-MANAGED in the interpreter's standard library directory, and pip checks for it before any install into that environment. You can see it yourself:

python3 -c "import sysconfig, pathlib; p = pathlib.Path(sysconfig.get_path('stdlib')) / 'EXTERNALLY-MANAGED'; print(p, p.exists())"

The reason it exists is real: apt and pip both write into the same site-packages, and when pip upgrades a library that an OS tool depends on, the OS tool can break in ways that are miserable to diagnose. As of late 2026 this behavior is the default on current Debian and Ubuntu releases, and Homebrew's Python applies it too. It is not a pip bug.

Takeaway: the error means "this interpreter belongs to your OS", and the right response is to stop installing into it, not to silence the check.

Which dead ends should you skip?

These are the ones I see suggested most often, and why they cost you later:

sudo pip install ... does not bypass the check, and if it did, you would be writing into the exact directory the protection exists for.
Deleting the EXTERNALLY-MANAGED file works until the next Python package update puts it back, and in the meantime you have turned off the guard for every user on the machine.
pip install --user is blocked by the same marker on most setups, so it does not help.
--break-system-packages works, and on a disposable machine that is fine. On a laptop or server you care about, it reintroduces the exact apt-versus-pip conflicts PEP 668 prevents.

Takeaway: every shortcut here trades a one-minute fix now for an unexplainable breakage of an OS tool later.

Which tool fits which job?

The right choice depends on what you are installing, not on personal taste.

Situation	Best fit	Why
Dependencies for a project you are developing	`venv` (or `uv venv`)	One isolated environment per project, reproducible from a requirements file
A CLI tool you just want on your PATH (e.g. a formatter, `httpie`)	`pipx` or `uv tool install`	Each app gets its own hidden environment, the command lands on PATH
Many projects, slow installs, lockfile wanted	`uv`	Fast resolver and installer, handles Python versions too
Docker image or CI job that is thrown away	`venv`, or the override flag	The OS-protection argument is weaker in a disposable root filesystem
A library packaged by your distro	`apt install python3-<name>`	Managed and patched by the OS, but often older

Takeaway: libraries go in a project venv, applications go in an isolated tool environment, and neither goes in the system interpreter.

How do you set up a venv correctly (including the Debian gotcha)?

The standard library route needs no extra tools:

python3 -m venv .venv
source .venv/bin/activate
pip install requests

On Debian and Ubuntu you may hit a second error first: The virtual environment was not created successfully because ensurepip is not available. Debian splits venv into its own package, so run sudo apt install python3-venv and retry.

Do not skip source .venv/bin/activate. If you forget, pip resolves to the system one and you are back at the original error. When in doubt, call the venv's interpreter explicitly: .venv/bin/python -m pip install requests.

If you want the managed version of this workflow, uv is the tool that creates the environment and installs from a requirements file in one fast step:

uv venv
uv pip install -r requirements.txt

A genuine drawback: uv is a separate binary you have to install and trust, and its uv pip interface follows pip closely but not identically, so edge cases in older requirements files can resolve differently than they did under pip. Test before you swap it into CI.

Takeaway: the venv is only protecting you if it is active or invoked by path; most "it still fails" reports are an unactivated shell.

How do you install command-line tools without polluting anything?

For applications, pipx creates one isolated environment per tool and exposes only its executables:

sudo apt install pipx   # or: brew install pipx
pipx ensurepath
pipx install httpie

If you want the one that handles isolated tool installs without a separate package from your distro, uv tool install httpie does the same job with the same one-environment-per-app model.

The limitation of both: they are for applications. If you pipx install a library and then import it in your own script, it will not be importable, because the library lives in its hidden environment. That is by design, and it is the most common confusion.

Takeaway: if you will type its name in a terminal, use pipx or uv tool; if you will import it, use a venv.

When is `--break-system-packages` acceptable?

In a Dockerfile or CI runner that is rebuilt from scratch every time, there is no OS tool left to protect, and a venv adds a layer for little benefit. Setting ENV PIP_BREAK_SYSTEM_PACKAGES=1 in the image is a reasonable, explicit choice there. Even so, a venv in the image keeps the same code path you use locally, which makes "works on my machine" bugs rarer. Choose the override flag knowingly, not because a forum answer said so.

Takeaway: the override flag is a tool for disposable filesystems, not for your daily laptop.

FAQ

How do I fix "error: externally-managed-environment" on Ubuntu or Debian?

Create a virtual environment with python3 -m venv .venv, activate it, and run pip inside it. If venv creation fails, install python3-venv with apt first.

Is `pip install --break-system-packages` safe?

It is safe on a throwaway container or CI runner. On a machine where apt manages Python tools, it can overwrite files that OS utilities depend on and cause breakage that is hard to trace.

Should I use pipx or uv to install Python command-line tools?

Either works: both give each application its own isolated environment and put its command on your PATH. Choose uv tool if you already use uv for projects, otherwise pipx is available from most package managers.

Bottom line

Use a venv (or uv venv) for anything you import, pipx or uv tool install for anything you run from a terminal, and apt install python3-<name> only when you want the distro-maintained version. Treat --break-system-packages as a Docker-and-CI setting. If you are on a team, put the one-line environment setup in the README so nobody rediscovers this error on day one.

Git Hooks Not Running for Teammates? Husky vs Lefthook vs pre-commit

Libme — Sat, 31 Oct 2026 17:04:47 +0000

Git hooks don't run for teammates because .git/hooks/ is never committed, so every clone starts with no hooks at all. Husky, lefthook, and pre-commit all solve this the same way: the hook definitions live in the repo, and an install step points Git at them. Pick husky if your repo is already a Node project, lefthook if you want one fast binary with no runtime dependency, and pre-commit if your team is Python-heavy or wants a large catalog of ready-made hooks.

Why do hooks work on my machine but not on a teammate's?

The symptom is boring and easy to miss: a teammate pushes code that your pre-commit lint would have rejected, and nobody sees an error, because nothing ran. There is no failure to find.

Git only executes hooks from the directory in core.hooksPath, which defaults to .git/hooks. That directory is inside .git, so it is not part of the repository content. Anything you put there stays on your machine. You can check what a clone is actually using:

git config --get core.hooksPath   # empty output means the default .git/hooks
ls .git/hooks | grep -v '\.sample$'

If the second command prints nothing, no hooks are installed in that clone. A fresh clone always looks like this.

The first fix people try is committing a scripts/hooks/ folder and asking everyone to symlink it. That works until someone forgets, which is the whole problem. The tools below automate the one step that humans skip.

Takeaway: hooks are local state, so any hook setup that depends on a person remembering to run a command will eventually be missing on someone's machine.

How do husky, lefthook, and pre-commit install themselves?

All three need an install step to run once per clone. They differ in how automatic they can make it.

	Husky	Lefthook	pre-commit
Runtime	Node	Single Go binary	Python
Config	Shell scripts in `.husky/`	`lefthook.yml`	`.pre-commit-config.yaml`
Auto-install trigger	`prepare` script in `package.json`	`lefthook install` (a `prepare` script or a manual step)	`pre-commit install` (manual or a setup script)
Staged-files filtering	Add a tool such as lint-staged	Built in (`{staged_files}`, `glob`)	Built in (hooks receive matching files)
Parallel execution	Not built in	Built in (`parallel: true`)	Not by default for different hooks; it runs hooks serially
Hook catalog	Bring your own commands	Bring your own commands	Large community catalog of hook repos

Husky's trick is that npm install runs the prepare script automatically. As of its v9 line, setup is npx husky init, which creates a .husky/ directory and a prepare entry that runs husky. That command sets core.hooksPath to a path inside .husky/.

{
  "scripts": {
    "prepare": "husky"
  }
}

# .husky/pre-commit
npx lint-staged

Lefthook looks like this:

# lefthook.yml
pre-commit:
  parallel: true
  commands:
    lint:
      glob: "*.{js,ts,tsx}"
      run: npx eslint {staged_files}
    format-check:
      glob: "*.{js,ts,tsx,json,md}"
      run: npx prettier --check {staged_files}

And pre-commit:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.6.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-merge-conflict

Check the rev against the project's latest tag before you copy it; pre-commit autoupdate rewrites it for you.

Takeaway: the real difference between these tools is not the hook syntax, it's what triggers the install on a fresh clone.

What goes wrong even after you adopt one?

These are the failure modes I'd check first when "we have husky, but it still isn't running."

The install step never ran. Husky depends on prepare. If someone installs with npm ci --ignore-scripts, or a package manager is configured to skip lifecycle scripts, prepare doesn't run and no hooks get wired up. The same applies to Docker images and CI, where this is usually what you want, but it also applies to a laptop with a hardened npm config. Verify with git config --get core.hooksPath.

Another tool already owns core.hooksPath. pre-commit refuses to install when it's set, with a message along the lines of "Cowardly refusing to install hooks with core.hooksPath set." If you're migrating from husky to pre-commit, unset it first:

git config --unset-all core.hooksPath
pre-commit install

Git GUIs and PATH problems. A hook that calls npx or node can fail under a GUI client that launches with a different PATH than your shell, especially with version managers like nvm. Lefthook's single binary sidesteps part of this, though the commands it runs still need their own tools on PATH.

Someone passed --no-verify. git commit --no-verify skips pre-commit and commit-msg hooks by design. This is not a bug and you can't prevent it locally.

Takeaway: if hooks stop running, check core.hooksPath and whether the install script executed, in that order, before blaming the hook tool.

Should hooks be the only enforcement?

No. Hooks are a convenience for fast feedback, not a gate. Anyone can skip them, and a contributor on a fork has never run your install step. Run the same checks in CI so the merge button is the actual enforcement.

pre-commit makes this easy: pre-commit run --all-files in a CI job runs the same hook set you run locally. With husky or lefthook, call the same underlying commands (lint, format check) directly from your CI workflow rather than invoking the hook runner. Keep the local hook fast, ideally only the staged files, and let CI do the slow, complete checks.

A drawback worth stating for each: husky needs a Node toolchain and adds nothing for non-JS repos; lefthook has a smaller ecosystem of shared recipes, so you write more commands yourself; pre-commit creates its own isolated environments for hooks, which makes the first run slow and adds cache management to CI.

Takeaway: treat local hooks as a speed-up and CI as the rule, because only CI applies to everyone.

FAQ

Why are my git hooks not running after cloning a repo?
Hooks live in .git/hooks, which is not versioned, so a fresh clone has none. You need a tool or script that installs them, such as npm install triggering husky's prepare, lefthook install, or pre-commit install.

How do I check whether a hook is installed?
Run git config --get core.hooksPath and list the target directory (.git/hooks if the config is empty). A hook file must exist there without the .sample suffix and be executable.

Can I enforce git hooks for everyone on the team?
Not locally, since --no-verify and skipped install steps bypass them. Run the same checks in CI and require them to pass before merging.

Bottom line

If your repo already has a package.json, husky plus a staged-files runner is the least friction, because npm install wires it up for free. If you want parallel hooks and no language runtime, lefthook is the one to try. If your team writes Python or wants ready-made hooks for YAML, secrets, and merge conflicts, pre-commit is the strongest fit. Whichever you choose, mirror the checks in CI.

GitHub Actions Cache Never Hits on Pull Requests: Branch Scope, Immutable Keys, and restore-keys

Libme — Fri, 30 Oct 2026 21:37:17 +0000

If actions/cache reports "Cache not found" on every pull request even though you can see caches in the repo's Actions tab, the cause is almost always one of three things: the cache was saved on a branch your PR cannot read, the key is identical to an existing one (caches are immutable, so the new content is never saved), or your restore-keys prefix does not match anything. All three are fixable in a few lines of YAML, and you can diagnose which one you have from the job log alone.

What does the failure actually look like?

The symptoms I see most often in logs (wording as of late 2026; the exact text may change between releases):

Cache not found for input keys: Linux-npm-3f1a9c..., Linux-npm-

and, in the post-job step:

Warning: Failed to save: Unable to reserve cache with key Linux-npm-3f1a9c..., another job may be creating this cache.

The first line means a restore found nothing. The second means a save was refused because that exact key already exists. People often see both in the same repo and conclude the feature is flaky. It is not. It follows three deterministic rules.

Takeaway: a cache miss is never random; it is one of three rules, and the log tells you which.

Why can't my PR see the cache that main built?

Caches are scoped to a branch (more precisely, to a git ref). According to GitHub's documentation, a workflow run can restore caches created on its own ref, on the base branch of a pull request, and on the default branch. It cannot read caches created by a sibling branch or a different PR.

That has two practical consequences:

If your workflow only runs on pull_request and never on pushes to main, then nothing ever populates the default-branch cache. Every PR starts cold, forever, and each PR writes a cache that no other PR can read.
Two feature branches never share caches with each other. They both fall back to main's cache, if one exists.

The fix is to run the same cache-saving job on pushes to the default branch:

on:
  push:
    branches: [main]
  pull_request:

A related trap: pull requests from forks run with a restricted token, and the cache they write is scoped to the PR merge ref, so don't expect fork PRs to warm anything for others.

Takeaway: if main never runs the cache step, no PR can ever get a warm start from it.

Why does the cache never update when my lockfile changes... or doesn't?

Caches are immutable. Once a key is saved, it cannot be overwritten. This bites in two opposite ways.

Key too static. A key like deps-${{ runner.os }} is saved once, then every later run restores it (an exact hit) and the post-step refuses to save new content. Your cache slowly goes stale and installs get slower, while the log says "Cache hit".

Key too volatile. A key containing ${{ github.sha }} is unique on every commit, so every run is a miss on the primary key and writes a fresh multi-hundred-MB entry. That floods the repository's cache quota and evicts the useful entries.

The standard pattern is a key derived from the file that determines the content, with a prefix fallback:

- uses: actions/cache@v4
  with:
    path: ~/.npm
    key: ${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-npm-

When the lockfile changes, the primary key misses, the prefix in restore-keys restores the most recent older cache, npm ci fills in the difference, and the post-step saves a new entry under the new key.

Takeaway: hash the lockfile, not the commit, and let restore-keys carry you across lockfile changes.

How do restore-keys and cache-hit actually behave?

Two details trip people up.

First, restore-keys are tried in order, and each is treated as a prefix. The first prefix that matches picks the most recently created cache among the matches. Put the most specific prefix first.

Second, the cache-hit output is 'true' only for an exact match on the primary key. A restore via restore-keys leaves it 'false' even though files were restored. If you gate npm ci on cache-hit != 'true' you will still run installs after a partial restore, which is usually what you want. If you gate it the other way, you will skip installs after a partial restore and ship a stale node_modules. Gating on cache-hit for something like node_modules is risky for exactly this reason; caching the package download directory (~/.npm) and always running npm ci is the safer shape.

Situation	Primary key	restore-keys	Result
Lockfile unchanged, same branch lineage	exact match	n/a	`cache-hit: true`, no save
Lockfile changed	miss	prefix match on older cache	`cache-hit: false`, files restored, new entry saved
First PR in a repo, main never cached	miss	no match	cold install, entry saved to PR ref only
Key includes `github.sha`	always miss	prefix match	works, but one new entry per run
Same key already exists	exact match	n/a	"Unable to reserve cache" warning on save

Takeaway: cache-hit means "exact key", not "something was restored", so don't use it to skip installs.

What about eviction and size limits?

As of late 2026, GitHub's documented defaults are: caches not accessed for 7 days are removed, and the repository total is capped (10 GB by default); when you exceed it, the oldest caches by last access are evicted first. Check the current numbers in the GitHub docs, since these have been adjustable. The practical effect: if you generate one big cache per PR, they can push out the default-branch cache that every PR depended on, and the next PR goes cold for no visible reason.

You can inspect what exists with the GitHub CLI:

gh cache list --limit 50
gh cache list --sort size_in_bytes --order desc

If the list is full of refs/pull/*/merge entries and the refs/heads/main entry is missing or old, you have found your problem.

To stop PRs from writing caches at all while still reading main's, split restore and save:

- uses: actions/cache/restore@v4
  with:
    path: ~/.npm
    key: ${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-npm-

- run: npm ci

- uses: actions/cache/save@v4
  if: github.ref == 'refs/heads/main'
  with:
    path: ~/.npm
    key: ${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }}

Only main saves, so PR caches never crowd out the shared one. The tradeoff: a PR that changes dependencies pays the full install every time until it merges. For most teams that is the right trade.

Takeaway: one writer (the default branch) and many readers (PRs) is the cache topology that stays warm.

When is a built-in cache not enough?

The built-in cache is fine for package downloads. It is a poor fit for Docker layers (large, and the 10 GB quota fills fast) and for build outputs that are expensive to restore compared with rebuilding. For Docker, BuildKit's registry or GitHub Actions cache exporter is the usual route. If you would rather not manage cache quotas at all, a hosted runner service with persistent disks, such as Depot or Blacksmith, is the kind of product that takes this problem off your plate, at the cost of another vendor and bill to evaluate. I would only reach for that after measuring that cache restore time actually dominates your pipeline.

FAQ

Why does GitHub Actions say "Cache not found" when the cache exists?
Caches are scoped by branch. A PR can read its own ref, its base branch, and the default branch, but not caches saved by other branches or PRs.

Can I overwrite an existing GitHub Actions cache?
No. Caches are immutable per key. Change the key (for example by hashing your lockfile) or delete the old entry with gh cache delete before saving again.

Does cache-hit be true when restore-keys matched?
No. cache-hit is 'true' only when the primary key matched exactly; a restore-keys match restores files but reports 'false'.

Bottom line

If your cache never hits on PRs, check three things in order: does main run and save the cache, is your key derived from the lockfile rather than a static string or a commit SHA, and do your restore-keys prefixes match the primary key's prefix. For most small teams, a single saver on the default branch plus restore-only PRs is the simplest setup that stays fast. Reach for external cache services only once you have measured that restore time, not install logic, is the bottleneck.

Docker Compose depends_on Doesn't Wait for the Database: Fix "Connection Refused" on Startup

Libme — Thu, 29 Oct 2026 22:03:53 +0000

Short answer: plain depends_on only waits until the dependency's container has started, not until the process inside is ready. To make the app wait for Postgres, give the database a healthcheck and use depends_on with condition: service_healthy. Then add a connection retry in the app anyway, because a healthcheck narrows the race but cannot close it.

What does the failure look like?

You run docker compose up on a fresh checkout. The database container starts, the API container starts a moment later, and the API dies:

api-1  | psycopg2.OperationalError: connection to server at "db" (172.19.0.2), port 5432 failed: Connection refused
api-1  |     Is the server running on that host and accepting TCP/IP connections?
db-1   | PostgreSQL init process complete; ready for start up.

Run it again and it works, because the data volume now exists and Postgres boots in a second instead of ten. That "works on the second try" pattern is the signature of a startup race. Teammates on a clean clone hit it, and CI hits it every time.

The compose file that causes it usually looks like this:

services:
  db:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: example
  api:
    build: .
    depends_on:
      - db

Short-form depends_on controls start order. Compose starts db, sees the container is running, and starts api. It never asks whether Postgres is listening.

Takeaway: "container running" and "service ready" are different states, and short-form depends_on only knows the first.

How do you make Compose wait for Postgres?

Define a healthcheck on the database and make the dependency conditional on it:

services:
  db:
    image: postgres:16
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: example
      POSTGRES_DB: app
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U app -d app"]
      interval: 3s
      timeout: 3s
      retries: 20
      start_period: 10s

  api:
    build: .
    depends_on:
      db:
        condition: service_healthy

Compose now holds api in "created" state until db reports healthy. If the database never becomes healthy within the retries, docker compose up exits with dependency failed to start: container db is unhealthy, which is a far better error than a stack trace from your app.

Takeaway: the long-form depends_on with condition: service_healthy turns "start order" into "readiness order".

Why does the healthcheck pass too early on a fresh volume?

This is the part most tutorials skip, and it bit me. The official postgres image, on the very first start with an empty data directory, runs its init scripts against a temporary server. As I understand the entrypoint script, that temporary server is configured to listen only on the Unix socket, not on TCP. When it finishes, it shuts down and the real server starts.

A healthcheck of just pg_isready -U app talks to the local Unix socket by default. During the init phase it can report "accepting connections" even though no TCP listener exists yet, and the server is about to restart. The result is a database that looks healthy, an app that connects over TCP to db:5432, and then either Connection refused or the database system is shutting down.

Adding -h 127.0.0.1 forces the check over TCP, which only succeeds once the final server is up. That one flag is why the example above uses it. If you use the socket form and only see failures on first boot (never on later boots with an existing volume), this is the cause.

Healthcheck command	Passes during init temp server?	Safe to gate the app on?
`pg_isready -U app`	Can, via Unix socket	No, flaky on first boot
`pg_isready -h 127.0.0.1 -U app -d app`	No, needs TCP listener	Yes
`psql -h 127.0.0.1 -U app -d app -c 'select 1'`	No, and proves auth works	Yes, slightly heavier

Takeaway: check readiness over the same transport your app will use, or the healthcheck can lie on first boot.

What about migrations that must finish before the app starts?

Waiting for the database is half of the problem. If a migration job must complete before the API serves traffic, Compose has a condition for that too:

services:
  migrate:
    build: .
    command: ["python", "-m", "alembic", "upgrade", "head"]
    depends_on:
      db:
        condition: service_healthy

  api:
    build: .
    depends_on:
      migrate:
        condition: service_completed_successfully
      db:
        condition: service_healthy

service_completed_successfully waits for the dependency to exit with code 0. If the migration fails, the API never starts. That is the behavior you want in a dev stack and in CI. (Alembic is just the example here; any one-shot command works.)

Takeaway: model one-shot setup work as its own service and gate on service_completed_successfully, instead of chaining shell commands in the app's entrypoint.

Do you still need retries in the app?

Yes. The healthcheck only governs the initial docker compose up. It does nothing when Postgres restarts later, when you run the app outside Compose, or in production on an orchestrator that has its own readiness semantics. An app that exits on its first failed connect is fragile everywhere except your laptop.

A minimal retry loop in Python:

import time
import psycopg2

def connect_with_retry(dsn: str, attempts: int = 10, base_delay: float = 0.5):
    for attempt in range(attempts):
        try:
            return psycopg2.connect(dsn, connect_timeout=3)
        except psycopg2.OperationalError:
            if attempt == attempts - 1:
                raise
            time.sleep(min(base_delay * 2 ** attempt, 5))

Keep the loop bounded and let the final error surface, so a genuinely broken config fails loudly instead of retrying forever. Pair it with restart: on-failure on the service if you want Compose to give the app another chance.

Which approach should you pick?

Approach	Handles first-boot race	Handles later DB restarts	Main drawback
Short `depends_on`	No	No	Only start order
`service_healthy` + healthcheck	Yes	No	Compose-only; healthcheck must be written correctly
`wait-for-it` style script in entrypoint	Mostly	No	Extra script to maintain; TCP-open is not the same as ready
App-level retry with backoff	Yes	Yes	Needs a code change in every service

If you live in Docker Compose for local development and CI, Docker Compose's service_healthy condition is the cleanest first fix because it needs no code changes. The durable answer is both layers: the healthcheck for a clean startup log, and retries in the app for everything the healthcheck cannot see. A drawback of leaning only on the healthcheck: it adds seconds to every cold start, and a too-tight retries value will mark a slow CI runner's database as unhealthy.

FAQ

Why does docker compose depends_on not wait for the database?
Short-form depends_on only waits for the dependency container to start, not for the process inside to be ready. Use condition: service_healthy with a healthcheck on the database.

What is the right pg_isready healthcheck for the Postgres Docker image?
Use pg_isready -h 127.0.0.1 -U <user> -d <db>. Forcing TCP avoids passing during the image's first-boot init phase, when only a temporary socket-only server is running.

Does depends_on condition service_healthy work with docker compose v2?
Yes. It is part of the Compose Specification supported by Docker Compose v2, as of late 2026, and it is not available in the legacy v1 file format with version: "2"/"3" semantics.

Bottom line

If your stack runs in Docker Compose, add a TCP-based pg_isready healthcheck and gate dependents on service_healthy, then put one-shot migrations in their own service with service_completed_successfully. Keep a bounded retry with backoff in the app so restarts and non-Compose environments behave too. Skip wait-for-it scripts unless you cannot use healthchecks, since they only prove a port is open.

httpx.PoolTimeout: Why Your Async Python Client Runs Out of Connections (and How to Fix It)

Libme — Wed, 28 Oct 2026 15:35:07 +0000

httpx.PoolTimeout means a request waited for a free connection in the client's pool and never got one before the pool timeout expired. The server is usually fine. The cause is almost always one of three things: you fire more concurrent requests than the pool allows, you leak responses by never closing them, or you create a client per request and defeat pooling altogether. The fixes are a semaphore, async with on streams, and one shared client.

What does PoolTimeout actually mean?

The traceback looks like this:

httpx.PoolTimeout

Often it has no message at all, which is why people go hunting in the network layer. It is different from ConnectTimeout (TCP handshake too slow) and ReadTimeout (server too slow to send bytes). PoolTimeout is purely local: the client has no idle connection to hand out and is already at its connection ceiling.

As of httpx 0.2x, the defaults are a 5 second timeout for each of connect, read, write, and pool, and Limits(max_connections=100, max_keepalive_connections=20). Check your installed version's docs, since defaults are the kind of thing that gets tuned between releases.

Takeaway: PoolTimeout is a queueing symptom on your side of the wire, not a slow-server symptom.

Why do I hit it? Three causes, in order of likelihood

1. Unbounded fan-out. asyncio.gather over thousands of coroutines sharing one client means thousands of requests competing for 100 slots. Everything past the first 100 waits, and anything waiting longer than 5 seconds raises. With a slow upstream, this happens fast.

2. Leaked responses. A streamed response holds its connection until the body is fully read or the response is closed. If you open a stream, hit an exception or an early return, and never close it, that connection is gone from the pool permanently. After 100 leaks, every request times out, even though traffic looks low. This is the one that bites in production, because it only shows up hours after deploy.

3. A client per request. Creating httpx.AsyncClient() inside a handler gives you a fresh pool each time, a fresh TLS handshake each time, and sockets that linger until garbage collection. It rarely raises PoolTimeout, but it causes the slow, flaky behavior people blame on the network.

How do I reproduce the leak?

This script leaks on purpose against any reachable URL. It never closes the streamed responses, then tries a normal request:

import asyncio
import httpx

URL = "https://example.com/"

async def main():
    limits = httpx.Limits(max_connections=5, max_keepalive_connections=5)
    timeout = httpx.Timeout(5.0, pool=2.0)
    async with httpx.AsyncClient(limits=limits, timeout=timeout) as client:
        leaked = []
        for _ in range(5):
            req = client.build_request("GET", URL)
            # stream=True returns before the body is read; we never close it
            leaked.append(await client.send(req, stream=True))

        try:
            await client.get(URL)
        except httpx.PoolTimeout:
            print("PoolTimeout: all 5 connections are held by unclosed responses")

        for r in leaked:
            await r.aclose()

asyncio.run(main())

The tell in real systems is that the error rate does not track traffic. If PoolTimeout climbs steadily and never recovers until restart, suspect a leak, not load.

Takeaway: a leak looks like a pool that shrinks over time; overload looks like a pool that is full only during bursts.

How do I fix it?

Cap concurrency below the pool size with a semaphore, and always close streams with a context manager:

import asyncio
import httpx

async def fetch_all(urls: list[str], concurrency: int = 50) -> list[int]:
    limits = httpx.Limits(
        max_connections=concurrency,
        max_keepalive_connections=concurrency,
    )
    timeout = httpx.Timeout(10.0, pool=30.0)
    sem = asyncio.Semaphore(concurrency)

    async with httpx.AsyncClient(limits=limits, timeout=timeout) as client:

        async def one(url: str) -> int:
            async with sem:
                async with client.stream("GET", url) as resp:
                    async for _ in resp.aiter_bytes():
                        pass
                    return resp.status_code

        return await asyncio.gather(*(one(u) for u in urls))

Two details matter. The semaphore is acquired before the request starts, so waiting for a slot is not counted against any httpx timeout. And async with client.stream(...) guarantees the connection is released even if the loop body raises.

Resist the urge to just raise max_connections to 10,000. That moves the queue from your process to the upstream, and many APIs will answer with 429s or reset connections instead. Pair this with the retry handling you would use anyway for rate limits.

httpx vs aiohttp vs requests: how do their pools compare?

Client	Pool knob	Default behavior when full	Gotcha
httpx (async)	`Limits(max_connections=...)`	Waits, then raises `PoolTimeout`	Unclosed streams hold slots forever
aiohttp	`TCPConnector(limit=...)`	Waits indefinitely unless you set a total timeout	No error at all, just a silent hang
requests	`HTTPAdapter(pool_maxsize=...)`	Opens extra connections and discards them, logging "Connection pool is full, discarding connection"	Only a warning, so thread pools degrade quietly

I prefer httpx's behavior here. A loud PoolTimeout is easier to alert on than aiohttp's silent wait. If you are choosing a client today and need HTTP/2 or a sync and async API from one library, httpx is the one that gives you both. The real drawback is that its per-request overhead is higher than aiohttp's in high-throughput fan-out. I have not benchmarked that myself, so measure with your own workload before switching for speed.

Takeaway: all three clients cap connections by default; they differ in whether you find out loudly (httpx), silently (aiohttp), or as a log line (requests).

How do I confirm the fix?

Log pool pressure instead of guessing. Count in-flight requests with a counter around the semaphore, and alert when it sits at the cap for long periods. If in-flight stays at the cap while throughput is flat, the upstream is the bottleneck. If in-flight creeps up while traffic is steady, you still have a leak, so search for every client.send(..., stream=True) and every early return inside a stream block.

FAQ

What is the default connection limit in httpx?

As of recent 0.2x releases, max_connections is 100 and max_keepalive_connections is 20, with a 5 second pool timeout. Verify against the docs for your installed version.

Should I increase the pool timeout to fix PoolTimeout?

Only if requests are legitimately bursty and short-lived. If the pool is exhausted by leaks, a longer timeout just makes each failure slower.

Is it safe to share one httpx.AsyncClient across tasks?

Yes. It is designed to be shared and reused for the life of your application. Create it once at startup, close it at shutdown.

Bottom line

If you see PoolTimeout, check for leaked streams first, then bound your fan-out with a semaphore sized at or below max_connections. Use one long-lived client per upstream host group. Raise limits only after confirming the upstream can absorb the extra concurrency.

Git "detected dubious ownership in repository": Fix It in Docker and CI Without Trusting Everything

Libme — Tue, 27 Oct 2026 22:06:54 +0000

The error means the user running Git does not own the repository directory, and since Git 2.35.2 Git refuses to run there as a security measure. The quick fix is git config --global --add safe.directory <path>, but in containers and CI the better fix is usually to make the UIDs match, and a blanket safe.directory '*' is only acceptable on throwaway machines.

What does the error actually look like?

fatal: detected dubious ownership in repository at '/workspace'
To add an exception for this directory, call:

    git config --global --add safe.directory /workspace

You tend to meet it in four places: a bind-mounted source directory inside Docker, a CI job that runs inside a container image, a repo checked out by one user and used by another (a deploy user versus root, say), and network or Windows-shared drives. In every case the cause is identical: the owner of the .git directory's parent differs from the effective user running git.

The check exists because of CVE-2022-24765. A malicious repo placed in a shared location such as C:\ or /tmp could carry a config that executes code when someone else ran a harmless-looking git status there. Git now refuses unless you say that directory is trusted. Takeaway: the error is a security check working as designed, not a corrupted repository.

Why does `git config --global` seem to do nothing in my container?

This is the dead end that cost me the most time. Three things commonly go wrong:

The setting is read only from trusted scopes. safe.directory is honored in system, global and command-line config. A safe.directory line inside the repository's own .git/config is ignored on purpose, otherwise the untrusted repo could vouch for itself.
--global writes to $HOME/.gitconfig. If your container runs as an arbitrary UID with no home directory, or with a read-only filesystem, the write fails or lands somewhere Git never reads.
The path must match what Git sees. Trusting /app does not help if the process runs in /app/ through a symlink, or if the mount point differs between build and run steps.

Takeaway: when safe.directory "doesn't work", check which config file you actually wrote to with git config --show-origin --get-all safe.directory.

Which fix should I use?

Situation	Fix	Trade-off
Local Docker bind mount	Run the container as your host UID/GID	Cleanest; no Git config needed
CI container job, ephemeral runner	Trust the one workspace path	Fine, the runner is disposable
Shared build box or long-lived server	Trust specific paths in system config	Wildcards widen the attack surface
Read-only or random-UID container	Pass config via environment variables	No file write required
Quick throwaway debugging	`safe.directory '*'`	Trusts every repo this user will ever touch

Match the UID instead of trusting the directory

For a bind mount, the root cause is ownership, so fix ownership:

docker run --rm \
  --user "$(id -u):$(id -g)" \
  -v "$PWD":/workspace -w /workspace \
  -e HOME=/tmp \
  alpine/git status

Setting HOME matters: an arbitrary UID usually has no home, and Git will complain about a missing $HOME/.gitconfig location or about tools that cache there. On Linux hosts this is the standard answer. On Docker Desktop for Mac, file ownership is translated by the file-sharing layer, so you may never see this error locally and then hit it for the first time in Linux CI. That mismatch is why the failure feels random.

Trust one explicit path in a throwaway job

In a CI container where the workspace is mounted by the runner as a different user than the job runs as, trust that single path:

jobs:
  build:
    runs-on: ubuntu-latest
    container: node:22
    steps:
      - uses: actions/checkout@v4
      - name: Allow Git in the workspace
        run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
      - run: git describe --tags --always

As of late 2026, actions/checkout registers the workspace as safe for its own operations, but later steps that run git as a different user, or in a different path, can still hit the error. Trusting $GITHUB_WORKSPACE explicitly is a one-line, auditable fix. Takeaway: in disposable CI, trusting exactly one named path costs nothing and tells the next reader why it is there.

Pass the setting through the environment

When you cannot write a config file at all, Git 2.31 and later reads config from environment variables:

export GIT_CONFIG_COUNT=1
export GIT_CONFIG_KEY_0=safe.directory
export GIT_CONFIG_VALUE_0=/workspace
git status

This is the approach I use for containers running under platforms that assign a random UID. Nothing is persisted, so nothing leaks into the next job that reuses the image. The same effect works per command with git -c safe.directory=/workspace status.

Is `safe.directory '*'` ever okay?

Newer Git versions accept * to mean "trust every directory". On a single-purpose CI runner that is destroyed after each job, the risk is small because there is no second user whose repos you could be tricked into running hooks from. On a developer laptop or a shared build server it removes the very protection the check provides, since any repo that lands in a world-writable location becomes trusted. I would not bake '*' into a base image that people also use interactively. Takeaway: '*' is a property of the machine's lifetime, not of how annoying the error is.

What about sudo and root?

If you run sudo git ... in a repo owned by your normal user, recent Git versions consult SUDO_UID and let it through when it matches the owner, so the common "sudo deploy" case often works. Plain su or a service running as root against a repo owned by a deploy user does not set that variable, and you will see the error. For deployment scripts, run Git as the owning user (sudo -u deploy git pull) rather than widening trust for root.

FAQ

How do I fix "fatal: detected dubious ownership in repository" in Docker?

Run the container with --user "$(id -u):$(id -g)" so the process owns the mounted files. If that is not possible, add the exact mount path with git config --global --add safe.directory /path, or pass it through GIT_CONFIG_COUNT, GIT_CONFIG_KEY_0 and GIT_CONFIG_VALUE_0.

Why does safe.directory in .git/config not work?

Git deliberately ignores safe.directory in repository-local config, because an untrusted repo could otherwise mark itself as safe. Put it in the global, system or command-line config instead.

Is it safe to set safe.directory to `*`?

Only on short-lived machines where you are the only user. Elsewhere it makes Git trust every repository on the system, which defeats the check added for CVE-2022-24765.

Bottom line

If the cause is a bind mount, fix ownership with a matching UID and skip Git config entirely. In disposable CI, trust the one workspace path and leave a comment explaining why. For random-UID platforms with read-only filesystems, use the GIT_CONFIG_* environment variables, and keep '*' for machines you will delete in ten minutes.

Is Paying for Bigger CI Runners Worth It? Break-Even Math Against Caching and Test Sharding

Libme — Mon, 26 Oct 2026 15:29:42 +0000

Short answer: pay for larger runners only after you have removed the minutes that should not exist (redundant runs, cold dependency installs) and only for the steps that actually scale with cores. For most small teams, cancelling superseded runs and caching come first, sharding second, and bigger machines last, because a machine with twice the cores almost never finishes a pipeline in half the time.

This post gives you a way to decide with your own numbers instead of a vendor's "up to Nx faster" claim. Pricing models vary by provider and change often, so I describe the shape of the costs rather than quoting rates. Check your provider's current price page for the per-minute multiplier before plugging numbers in.

Where do CI minutes actually go?

Before comparing options, split one representative run into phases. On GitHub Actions the per-step durations are visible in the run's log view; every other CI system has an equivalent. I group them into four buckets:

Setup (checkout, toolchain, dependency install): usually serial, usually cacheable.
Build/compile: partly parallel, depends on your toolchain.
Tests: the most parallelizable bucket if tests are independent.
Waste: runs nobody needed, such as a push-after-push where the first run is already obsolete.

What tripped me up the first time I did this exercise was the waste bucket. It does not show up inside any single run, only across runs. Listing the last few dozen runs and counting how many were cancelled-or-superseded by a newer commit on the same branch is more informative than any single profile.

Takeaway: you cannot choose between caching, sharding, and bigger runners until you know which bucket your minutes live in.

What should I fix before paying for anything?

Two changes cost nothing and remove minutes without touching speed.

First, cancel runs that a newer push has made obsolete:

concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}

Keeping cancel-in-progress off for main is deliberate: you generally want every merge commit to finish its checks, while feature branches only care about the latest push.

Second, stop cold-installing dependencies. Most official setup actions for Node, Python, and others have a built-in cache option keyed on your lockfile:

- uses: actions/setup-node@v4
  with:
    node-version: 22
    cache: npm
- run: npm ci

Caching helps only if the install is a meaningful share of the run. If it is 20 seconds out of 10 minutes, move on. Also note the limitation: caches are evicted under storage limits and are scoped by branch, so a cache restored on a feature branch comes from that branch or the default branch, not from a sibling. A cache that mysteriously misses is usually a key problem, not an outage.

Takeaway: cancelling superseded runs and caching lockfile-keyed dependencies are free, and they change what the later options are worth.

Does a bigger runner make CI proportionally faster?

No, and the reason is Amdahl's law: if a fraction p of the run can use extra cores and the rest is serial, the speedup from n times the compute is 1 / ((1 - p) + p / n). Checkout, installs, a single-threaded bundler step, and a database migration are all serial. A pipeline that is 60% parallelizable and runs on a machine with four times the cores speeds up by roughly 1.8x, not 4x. Meanwhile the per-minute price of a larger runner typically scales with its size, so you pay a multiple of the rate for a smaller multiple of the speed.

Here is a script that turns your phase measurements into a break-even check. It uses only the standard library:

def run_cost(minutes, rate_multiplier):
    """Billed cost in 'standard-runner-minutes'."""
    return minutes * rate_multiplier

def speedup(parallel_fraction, cores_multiplier):
    return 1 / ((1 - parallel_fraction) + parallel_fraction / cores_multiplier)

def compare(base_minutes, parallel_fraction, cores_multiplier, rate_multiplier):
    new_minutes = base_minutes / speedup(parallel_fraction, cores_multiplier)
    return {
        "new_minutes": round(new_minutes, 1),
        "billed_before": run_cost(base_minutes, 1),
        "billed_after": round(run_cost(new_minutes, rate_multiplier), 1),
    }

# Replace these with measurements from your own pipeline:
print(compare(base_minutes=12, parallel_fraction=0.6,
              cores_multiplier=4, rate_multiplier=4))

With those placeholder inputs (they are illustrative, not measured), the run gets shorter but the billed cost goes up. The only thing you are buying is developer waiting time. That can still be a good trade, which is the next question.

Takeaway: a larger runner is a purchase of wall-clock time at a worse unit price, so the only valid justification is that wall-clock time is worth more than the extra billed minutes.

When is the faster pipeline worth the extra spend?

Convert waiting time into a number. Take the number of runs per day that a human actually waits on, multiply by minutes saved per run, and compare with the added billed cost. If five engineers each wait on the pipeline several times a day, shaving a few minutes per run is real focus time. If most runs are bot-triggered or nobody watches them, the saved minutes are worth close to nothing.

Option	Cuts billed minutes	Cuts wall-clock time	Main drawback
Cancel superseded runs	Yes	No	Needs care on `main` and release branches
Dependency/build caching	Yes	Yes	Key mistakes cause silent misses or stale builds
Test sharding (matrix)	No, usually slightly up	Yes	Setup cost repeats per shard; uneven shards
Larger runner	No, usually up	Yes, sub-linearly	Higher unit price for serial steps
Self-hosted runner	Shifts cost to your infra	Depends on hardware	You own patching, isolation, and scaling

Sharding is the middle path, and it is worth showing because the tooling is built in. Jest and Playwright both support splitting a suite with a --shard flag:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx jest --shard=${{ matrix.shard }}/4

The catch: each shard repeats checkout and install, so four shards cost more billed minutes than one run, and the slowest shard sets the finish time. Shard counts beyond what your install overhead justifies give diminishing returns quickly.

If you decide the managed option is the right call for the serial-heavy build step, GitHub Actions larger runners are the version of this that needs no infrastructure work from you, with the tradeoff that you pay the higher rate for every minute including the serial ones.

Takeaway: shard the parallel part and keep the serial part on cheap machines, rather than upgrading the whole job.

What about self-hosted runners?

Self-hosted runners trade a bill for an operations job. You pay for the hardware (or cloud instances) and for the time to patch, isolate untrusted code from forks, and scale them. Billing rules for self-hosted runners on hosted CI platforms have been under revision, so verify the current policy before building a business case on "self-hosted is free." For a team under roughly a handful of engineers, the maintenance time usually exceeds the saving unless builds are very heavy or need special hardware.

Takeaway: self-hosting is a hardware-and-ops decision, not a discount.

FAQ

Are larger GitHub Actions runners worth it?
Only for jobs that are mostly parallel, such as compiling or running a big independent test suite, and only when engineers are actually waiting on the result. For serial-heavy pipelines you pay a higher per-minute rate for little speedup.

Why is my CI slower after adding more shards?
Each shard repeats checkout and dependency install, and the pipeline finishes when the slowest shard does. If shards are uneven or setup is a large share of the run, more shards add overhead faster than they remove test time.

How do I reduce CI minutes without slowing builds down?
Cancel superseded runs on feature branches, cache dependencies keyed on your lockfile, and skip jobs whose paths did not change. These remove work instead of buying speed.

Bottom line

Measure one representative run by phase, then count how many runs in the last month were obsolete. Fix cancellation and caching first because they lower the bill and the wait. Shard tests when the test phase dominates and the install is small. Buy larger runners only for steps you have shown scale with cores, and only when someone's waiting time is worth more than the extra billed minutes.

Redis vs Valkey vs Dragonfly: Which In-Memory Store Should a New Project Use After the License Changes?

Libme — Sun, 25 Oct 2026 21:30:35 +0000

For most new projects, the answer is to speak the Redis protocol and keep the server swappable: Valkey is the low-risk default if you want a permissively licensed fork, Redis itself is fine if you use a managed offering or its AGPL option fits your situation, and Dragonfly is worth testing only if a single node's throughput or memory efficiency is your actual bottleneck. The license matters mostly if you redistribute the server or resell it as a service. For an app that merely connects to a cache, the client code is identical in all three cases.

This post covers what changed, how to test compatibility in an afternoon, and where each option has real limits. Details are as of late 2026; check the projects' own license files before making a legal decision.

What actually changed with Redis licensing?

In 2024, Redis moved from the BSD license to a dual source-available arrangement (RSALv2 and SSPL). That prompted the Linux Foundation to host Valkey, a fork of the last BSD-licensed Redis code (the 7.2 line), with backing from several large cloud vendors. In 2025, Redis added AGPLv3 as a licensing option for newer versions, which is OSI-approved open source again, though AGPL carries its own obligations.

What this means in practice:

If your app only connects to the server over the network, none of these licenses restrict you from running it internally.
If you ship the server inside a product you distribute, or offer it as a managed service to others, the license is a real constraint, and you should read it with a lawyer rather than a blog post.
If your company has a blanket "no AGPL / no SSPL" policy, that policy decides the question before technical merit does.

Takeaway: the license change is a procurement and distribution question, not a code-compatibility question for the application developer.

How do Redis, Valkey, and Dragonfly differ?

	Redis	Valkey	Dragonfly
License (as of late 2026)	Source-available options plus AGPLv3	BSD-style, Linux Foundation project	Source-available (BSL-style)
Lineage	Original	Fork of Redis 7.2	Independent rewrite in C++
Protocol	Redis (RESP)	Redis (RESP)	Redis (RESP), most commands
Threading model	Mostly single-threaded command execution, I/O threads available	Same base, with ongoing I/O threading work	Multi-threaded, shared-nothing design
Module ecosystem	Largest (Redis modules, vendor features)	Smaller, growing	Not module-compatible
Managed offerings	Redis's own cloud and others	Offered by major clouds as a managed option	Vendor cloud, or self-host
Main drawback	License complexity	Feature set trails newer Redis-only additions	Compatibility is "most", not "all"; younger project

I would not trust any single row without checking it yourself, particularly the managed-offering row, which changes quickly. The point of the table is the shape of the trade-off: Valkey minimizes surprise, Redis maximizes features, Dragonfly changes the performance model.

Takeaway: Valkey and Redis differ mainly in governance and newer vendor-specific features, while Dragonfly differs in architecture.

How do you test compatibility before committing?

Do not read compatibility pages; run your own command surface against each server. The usual trap is that your app uses a command, Lua script, or data type you forgot about. Here is a small script that records every command your test suite issues, then replays that set against a candidate server.

Start the candidates locally (swap the image for the one you are evaluating):

docker run -d --name cache-valkey -p 6380:6379 valkey/valkey:latest
docker run -d --name cache-redis  -p 6381:6379 redis:latest

Then run your real integration tests against each port. For a fast sanity check of the specific features you rely on:

import redis

def check(port: int) -> dict:
    r = redis.Redis(port=port, decode_responses=True)
    results = {}

    r.flushall()
    r.set("k", "v", ex=5, nx=True)
    results["set_nx_ex"] = r.get("k") == "v"

    r.xadd("stream", {"a": "1"})
    results["streams"] = r.xlen("stream") == 1

    script = "return redis.call('INCR', KEYS[1])"
    results["lua_eval"] = r.eval(script, 1, "counter") == 1

    r.zadd("z", {"a": 1, "b": 2})
    results["sorted_set"] = r.zrange("z", 0, -1) == ["a", "b"]

    # Blocking pop with a short timeout, a common compatibility gap
    results["blpop_timeout"] = r.blpop("empty", timeout=1) is None
    return results

for name, port in [("valkey", 6380), ("redis", 6381)]:
    print(name, check(port))

Extend it with the commands your code uses. To find those, run MONITOR against a staging instance during a test run, or grep your codebase and your libraries (queue libraries and session stores often use Lua scripts and streams you never wrote yourself).

What tripped me up in this kind of exercise: libraries that detect the server by version string. A client that checks INFO server for a specific redis_version can behave differently against a fork that reports its own version. Test with your actual client library versions, not just redis-cli.

Takeaway: compatibility is a property of your command set plus your libraries, so replay both rather than trusting a feature list.

When is Dragonfly worth the extra evaluation?

Dragonfly's pitch is that one multi-threaded process can use all cores on a large machine, where a single-threaded Redis-style process uses roughly one core for commands and you scale out with clustering. If you are running a big instance and sharding purely to get past one core, that is the situation where it deserves a benchmark. I have not run a controlled benchmark here and will not quote numbers; vendor benchmarks are measured on their chosen workloads, so run your own traffic shape.

If you are at a few thousand operations per second on a small instance, none of this matters. The cache is not your bottleneck, and the extra variable of a newer, less widely deployed server costs more in operational unknowns than it saves.

Limitations to weigh: Dragonfly does not support Redis modules, a few commands and behaviors differ, and the community and hosting options are smaller than for Redis or Valkey. If the managed service you want only offers Redis or Valkey, that constraint probably decides it. If you want the one with the strongest performance story on big single nodes and you are willing to self-host and test, Dragonfly is the one to put on your shortlist.

Takeaway: choose Dragonfly because you measured a single-node scaling problem, not because the architecture sounds modern.

What is the safest way to keep your options open?

Treat the cache as a protocol, not a product:

Connect through a standard client and a single CACHE_URL environment variable (redis:// works for all three).
Avoid vendor-specific modules unless the feature is worth being locked in for. If you need something like JSON documents or search inside the store, you are choosing a vendor, not a protocol.
Keep Lua scripts and Streams usage in one module so a future migration has a single place to test.
Run your integration suite in CI against two servers, so drift shows up in a pull request instead of a migration weekend.

# docker-compose.ci.yml
services:
  cache-a:
    image: valkey/valkey:latest
  cache-b:
    image: redis:latest

If you want the managed version of the open-source fork without running servers yourself, check whether your existing cloud provider's managed Valkey offering covers your region and version needs before looking elsewhere. If you want Redis's own feature set and support, Redis Cloud is the managed route from the vendor itself, with the usual limitation that you are tied to its roadmap and pricing model.

Takeaway: a CACHE_URL variable and a two-server CI matrix make this decision reversible.

FAQ

Is Valkey a drop-in replacement for Redis?
For Redis 7.2-era features, largely yes: the same protocol, commands, and clients work. Features that Redis added after the fork, or Redis-specific modules, may be missing, so replay your own command set before switching.

Can I still use Redis for free after the license change?
Yes for running it internally as a cache or queue backend. The license restrictions bite mainly if you redistribute the server or offer it as a service, and the AGPL option adds its own source-sharing obligations.

Is Dragonfly faster than Redis?
On large multi-core machines it can be, because it uses multiple threads for command execution, but the gain depends entirely on your workload. Benchmark your own traffic before deciding.

Bottom line

Default to Valkey or managed Redis, whichever your cloud and legal team already accept, and write your application against the protocol so you can change your mind. Pick Redis when you need its newer or module-based features. Evaluate Dragonfly only when you have a measured single-node throughput or memory problem and can accept self-hosting and a smaller ecosystem. Everyone else should spend the afternoon on a two-server CI matrix, not on picking a winner.

FastAPI Latency Spikes Under Load: How One Blocking Call in an async Endpoint Stalls Every Request

Libme — Sat, 24 Oct 2026 22:08:17 +0000

TL;DR: An async def endpoint runs directly on the event loop. If it calls anything that blocks (time.sleep, requests.get, a sync database driver, heavy CPU work), the loop cannot serve any other request until that call returns. The fix is to either declare the endpoint with plain def so the framework runs it in a thread pool, push the blocking call into a thread with asyncio.to_thread, or switch to a genuinely async client. The symptom that sends people searching is odd: health checks time out and p99 explodes while CPU looks fine.

What does the symptom look like?

The pattern I see over and over: an endpoint is fast in isolation, and a load test or a traffic burst makes unrelated endpoints slow, including /health. Uptime probes fail, the load balancer marks the instance unhealthy, and the instance restarts under load, which makes things worse.

Here is a minimal reproduction. Python 3.11+, with fastapi, uvicorn and httpx installed:

# app.py
import time
from fastapi import FastAPI

app = FastAPI()

@app.get("/health")
async def health():
    return {"ok": True}

@app.get("/slow-async")
async def slow_async():
    time.sleep(0.5)  # blocks the event loop
    return {"done": True}

@app.get("/slow-sync")
def slow_sync():
    time.sleep(0.5)  # runs in a worker thread
    return {"done": True}

Run it with uvicorn app:app and fire ten concurrent requests at each slow route. Against /slow-async, the ten requests are served one after another, so the last one waits roughly ten times 0.5 seconds. That is arithmetic, not a benchmark: a single-threaded loop can only do one blocking call at a time. During that window /health also waits, because it is queued behind them on the same loop.

# hammer.py
import asyncio, time, sys
import httpx

async def main(path: str, n: int = 10):
    async with httpx.AsyncClient(base_url="http://127.0.0.1:8000", timeout=30) as c:
        start = time.perf_counter()
        # fire n slow requests, then probe /health while they are in flight
        slow = [asyncio.create_task(c.get(path)) for _ in range(n)]
        await asyncio.sleep(0.1)
        h0 = time.perf_counter()
        await c.get("/health")
        print(f"/health took {time.perf_counter() - h0:.2f}s while {path} was busy")
        await asyncio.gather(*slow)
        print(f"all {n} done in {time.perf_counter() - start:.2f}s")

asyncio.run(main(sys.argv[1]))

Run python hammer.py /slow-async and then python hammer.py /slow-sync. The first shows /health stuck for seconds; the second answers it almost immediately.

Takeaway: async def is a promise that you never block, and the framework cannot enforce it for you.

Why does `def` work when `async def` doesn't?

FastAPI (via Starlette and AnyIO) treats the two kinds differently. A plain def endpoint is dispatched to a worker thread pool, so a blocking call occupies a thread, not the loop. An async def endpoint is awaited on the loop itself. As of late 2026 the AnyIO thread limiter defaults to 40 concurrent threads, so a def endpoint is not free of limits either: the 41st concurrent blocking request queues.

The common mistake is the well-meaning "everything should be async" refactor. Converting def to async def without converting the calls inside it makes the endpoint worse, because you moved it off the thread pool onto the loop.

Takeaway: converting def to async def is a performance regression unless every call inside it was converted too.

How do I find the blocking call?

Do not guess. Two tools find it quickly.

1. asyncio debug mode. Start the server with PYTHONASYNCIODEBUG=1 uvicorn app:app. The loop logs any callback that holds it longer than slow_callback_duration (0.1 s by default), with the task and the coroutine involved:

Executing <Task pending name='Task-7' coro=<slow_async() running at app.py:13> ...> took 0.501 seconds

That line names the endpoint. Debug mode adds overhead, so use it in staging or a local repro, not on a hot production box.

2. A sampling profiler. If you cannot reproduce locally, py-spy dump --pid <pid> shows where each thread currently is. When the loop thread sits in time.sleep, socket.recv, or a driver's execute, that is your culprit. The honest limitation of py-spy: it needs permission to attach to the process (ptrace on Linux, root on macOS), which is often locked down in containers.

The usual suspects, in the order I find them: requests inside an async handler, a sync ORM session or psycopg2 call, open().read() on large files, json.loads on multi-megabyte payloads, password hashing (bcrypt/argon2 are deliberately slow), and image or PDF processing.

Takeaway: the slow-callback log turns "something is blocking" into a file and line number in one run.

Which fix should I use?

Situation	Fix	Trade-off
Handler is mostly sync code (sync ORM, `requests`)	Change `async def` to `def`	Limited by the thread pool size; no `await` inside
One blocking call inside an otherwise async handler	`await asyncio.to_thread(fn, ...)`	Same thread pool limits; code must be thread-safe
HTTP calls to other services	Use an async client such as `httpx.AsyncClient`	Needs one shared client, not one per request
CPU-bound work (hashing, image resize, big parsing)	Process pool or a job queue	Serialization cost; more moving parts

For the one-call case:

import asyncio
import bcrypt
from fastapi import FastAPI

app = FastAPI()

def _hash(password: bytes) -> bytes:
    return bcrypt.hashpw(password, bcrypt.gensalt())

@app.post("/signup")
async def signup(password: str):
    hashed = await asyncio.to_thread(_hash, password.encode())
    return {"hash_len": len(hashed)}

For outbound HTTP, create the client once at startup rather than per request:

from contextlib import asynccontextmanager
import httpx
from fastapi import FastAPI, Request

@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.http = httpx.AsyncClient(timeout=httpx.Timeout(5.0))
    yield
    await app.state.http.aclose()

app = FastAPI(lifespan=lifespan)

@app.get("/proxy")
async def proxy(request: Request):
    r = await request.app.state.http.get("https://example.com")
    return {"status": r.status_code}

If you want an HTTP client that works in both sync and async code with the same API, httpx is the one I reach for; its drawback is that it is a bit slower than lower-level clients such as aiohttp on very high request rates, and the default timeouts differ from requests, so set them explicitly.

CPU-bound work is the case threads do not rescue: threads share the GIL, so heavy pure-Python computation in to_thread still starves other Python threads (C extensions that release the GIL, such as bcrypt, are fine). Use ProcessPoolExecutor via loop.run_in_executor or hand it to a worker queue.

Takeaway: match the fix to the kind of blocking: threads for I/O and GIL-releasing libraries, processes or queues for pure-Python CPU work.

Can running more workers hide the problem?

Yes, and that is why it survives so long. uvicorn --workers 4 (or Gunicorn with Uvicorn workers) gives you four independent loops, so one stuck loop no longer stalls everything. But four workers each blocked by one slow request is just a smaller version of the same outage. More workers are good capacity planning and bad bug-hiding; fix the block first, then size the workers.

If you want the guard rail rather than the fix, Uvicorn's --timeout-keep-alive and your proxy's upstream timeout will at least turn a silent stall into a visible 504, which is easier to alert on.

Takeaway: extra workers raise the number of requests a blocking call can hurt at once; they do not remove the block.

FAQ

Why is my FastAPI app slow when I use async def?
Almost always because something inside the handler blocks the event loop, such as requests, a sync database driver, or time.sleep. Run with PYTHONASYNCIODEBUG=1 to see which coroutine holds the loop.

Should I use def or async def in FastAPI?
Use async def only when every I/O call inside it is awaited. Use plain def when you depend on sync libraries; FastAPI runs it in a thread pool.

Does asyncio.to_thread make blocking code non-blocking?
It moves the call off the event loop onto a worker thread, so other requests keep being served. It does not speed up the call, and pure-Python CPU work still contends for the GIL.

Bottom line

If you use sync libraries, write plain def endpoints and stop worrying. If you are committed to async def, treat every library call as guilty until proven awaitable, and keep asyncio debug mode in your staging setup so a new blocking dependency gets caught in review rather than at 2 a.m. For pure CPU work, move it out of the web process entirely.

GitHub Push Rejected: "File Exceeds 100 MB" (GH001) — Fix It With filter-repo or LFS

Libme — Fri, 23 Oct 2026 22:26:52 +0000

Deleting the big file in a new commit does not fix this error, because Git still carries the file in an earlier commit and GitHub rejects the whole push. You have to either rewrite history so the blob never existed (git filter-repo), or move the file to Git LFS and rewrite history so old commits point to LFS instead (git lfs migrate). Pick filter-repo if the file should not be in the repo at all, LFS if it should be versioned but is legitimately large.

What does the error actually look like?

remote: error: Trace: 3f1c...
remote: error: See https://gh.io/lfs for more information.
remote: error: File data/embeddings.bin is 143.21 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
 ! [remote rejected] main -> main (pre-receive hook declined)
error: failed to push some refs to 'github.com:you/project.git'

As of late 2026, GitHub prints a warning for files over 50 MB and blocks pushes containing files over 100 MB. The check runs on every blob in the commits you are pushing, not just the tip of your branch. That is the part that surprises people.

Takeaway: GH001 is about every commit in the push, so the fix has to change history, not just the working tree.

Why doesn't `git rm` and a new commit fix it?

I have watched this dead-end happen more than once: the developer runs git rm data/embeddings.bin, commits, pushes, and gets the identical error. The file is deleted from the latest snapshot, but the commit that added it is still an ancestor of your branch, and it is part of what you are pushing.

Other dead ends that look plausible:

git commit --amend only works if the big file is in the very last commit.
Adding it to .gitignore after the fact does nothing for a file already tracked or already committed.
git reset --soft and recommit works for unpushed local history and is actually fine for a small number of commits, but gets tedious fast and loses your commit messages.

First, find out which commit and which path is the problem:

# list the 10 largest blobs anywhere in history
git rev-list --objects --all \
  | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
  | awk '$1 == "blob" {print $3, $4}' \
  | sort -rn \
  | head -10

The first column is bytes, the second is the path. If the same path shows up in multiple versions, every version is in history and every one over the limit will be rejected.

Takeaway: before touching anything, list the largest blobs in history so you know whether you are removing one file or twenty.

Option A: Remove the file entirely with git filter-repo

Use this when the file was committed by accident (a dataset, a build artifact, a database dump, a model checkpoint) and does not belong in the repository.

git filter-repo is the tool the Git project itself recommends over the older git filter-branch. Install it with pip install git-filter-repo or your package manager. Work on a fresh clone, because filter-repo deliberately refuses to run on a repo that looks like it has remotes and local work you could lose.

git clone git@github.com:you/project.git project-clean
cd project-clean

# remove one path from all history
git filter-repo --path data/embeddings.bin --invert-paths

# or: remove every blob over a size threshold, whatever its path
git filter-repo --strip-blobs-bigger-than 50M

filter-repo removes the origin remote as a safety measure, so re-add it and force-push:

git remote add origin git@github.com:you/project.git
git push --force-with-lease origin main

Two honest drawbacks. First, this rewrites every commit hash from the first affected commit onward, so every collaborator must re-clone or hard-reset, and open pull requests will break. Second, if the file contained a secret, rewriting history on GitHub does not make the secret safe: assume it leaked and rotate it. Cached views of old commits can persist on the platform even after a force push.

If you want a single-purpose tool with a simpler interface, BFG Repo-Cleaner does the same job, but it is less flexible with path filtering than filter-repo.

Takeaway: use git filter-repo --invert-paths when the file was a mistake, and treat every collaborator clone as needing a reset afterward.

Option B: Keep the file but move it to Git LFS

Use this when the file is a legitimate part of the project, like design assets, test fixtures, or a small model, and you want it versioned.

git lfs install
# rewrite ALL history so matching files become LFS pointers
git lfs migrate import --include="*.bin,data/*.parquet" --everything
git push --force-with-lease origin main

--everything rewrites every branch and tag; without it only the current branch is migrated and the other refs still contain the big blobs. Check the result with git lfs ls-files, and confirm .gitattributes now contains the tracking rules, since that file has to be committed for teammates to get the same behavior.

The drawback is that LFS is a separate storage and bandwidth quota on GitHub. The free allowance is limited, large files in CI re-download on every checkout unless you cache them, and forks have their own quirks with LFS objects. Check GitHub's current LFS billing page before committing a multi-gigabyte dataset to it.

Takeaway: LFS keeps big files versioned, but you are trading a hard push error for a storage and bandwidth quota you now have to watch.

Which fix should you use?

Situation	Best fix	History rewritten?	Ongoing cost
Accidental commit of dump, build output, `.env`, `node_modules`	`git filter-repo --invert-paths`	Yes	None
Several oversized blobs, unsure which	`filter-repo --strip-blobs-bigger-than`	Yes	None
Large assets that must be versioned	`git lfs migrate import`	Yes	LFS quota
Large file only in unpushed local commits	`git reset --soft` and recommit	Local only	None
Dataset or model that changes often	Object storage (S3, R2) plus a download script	No	Storage fees

The last row is the one I would choose most often for ML-ish repos: store the artifact in a bucket and keep a small script plus a checksum in Git. It keeps clones fast and leaves no history problem to fix later.

How do you stop this from happening again?

Add the pattern to .gitignore before the file exists, and add a pre-commit check so the failure happens on your laptop instead of at push time:

#!/usr/bin/env bash
# .git/hooks/pre-commit — block staged files over 50 MB
limit=$((50 * 1024 * 1024))
for f in $(git diff --cached --name-only --diff-filter=AM); do
  size=$(wc -c < "$f")
  if [ "$size" -gt "$limit" ]; then
    echo "Refusing to commit $f ($size bytes, limit $limit)" >&2
    exit 1
  fi
done

Make it executable with chmod +x .git/hooks/pre-commit. Hooks in .git/ are not shared, so for a team use a framework such as pre-commit, which has a built-in large-file check.

FAQ

Why does GitHub reject my push after I deleted the large file?
Because the commit that added the file is still in your branch history, and GitHub scans every commit being pushed. You must rewrite history with git filter-repo or git lfs migrate to remove the blob from those commits.

What is GitHub's maximum file size for a regular Git push?
GitHub blocks files larger than 100 MB and warns above 50 MB, as of late 2026. Larger files need Git LFS or external storage.

Is it safe to force-push after running git filter-repo?
It is safe for your repo if you use --force-with-lease and coordinate with collaborators, who will need to re-clone. It is not a fix for leaked secrets, which must be rotated regardless.

Bottom line

If the file should never have been committed, run git filter-repo on a fresh clone and force-push once. If you genuinely need large files versioned alongside code, Git LFS is the standard route on GitHub, with a quota to monitor. For datasets and model weights that change often, keep them in object storage and let Git hold only the script that fetches them.

Sentry vs GlitchTip vs Self-Hosted Sentry: Error Tracking for a Small Team When the Event Quota Keeps Running Out

Libme — Thu, 22 Oct 2026 21:31:45 +0000

For most small teams, hosted Sentry is the right default, and the thing that decides whether it stays affordable is not the vendor but how many duplicate events your app emits. GlitchTip is a credible cheaper alternative if you only need error tracking and are happy with the Sentry SDKs. Self-hosting full Sentry is the option I would pick last, because the cost moves from a bill to your on-call time.

Why does the error quota run out in the first week?

The usual story: you add error tracking, everything looks calm, then a downstream API goes down for twenty minutes. A retry loop or a queue consumer throws the same exception on every attempt, and a plan priced by event volume absorbs thousands of identical events. The dashboard shows one issue with a huge count. The quota is gone and the useful errors that arrive afterwards are dropped or throttled.

Hosted tools price by events (or similar units) per month, and exact tiers change, so check the current pricing page rather than trusting a blog post. What does not change is the shape of the problem: volume is driven by your worst incident, not your normal day. A single noisy failure mode can cost more quota than a month of ordinary errors.

Takeaway: error-tracking cost is set by your noisiest incident, so cap duplicates in the client before comparing vendors.

Fix the volume first: client-side dedupe and sampling

Before you switch tools, drop repeats at the source. The Sentry JavaScript/Node SDK exposes beforeSend, ignoreErrors and sampleRate, and GlitchTip accepts events from the same SDKs, so this work carries over if you move.

import * as Sentry from "@sentry/node";

const WINDOW_MS = 60_000;
const MAX_PER_WINDOW = 5;
const seen = new Map<string, { count: number; start: number }>();

function fingerprintOf(event: Sentry.ErrorEvent): string {
  const ex = event.exception?.values?.[0];
  return `${ex?.type ?? "unknown"}:${ex?.value ?? ""}`;
}

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  // Network noise you cannot act on
  ignoreErrors: ["ECONNRESET", "AbortError"],
  beforeSend(event) {
    const key = fingerprintOf(event);
    const now = Date.now();
    const entry = seen.get(key);
    if (!entry || now - entry.start > WINDOW_MS) {
      seen.set(key, { count: 1, start: now });
      return event;
    }
    entry.count += 1;
    // Returning null drops the event before it counts against quota
    return entry.count > MAX_PER_WINDOW ? null : event;
  },
});

Two caveats from using this pattern. The Map grows with distinct messages, so if your error text embeds IDs you should key on the type and a normalized message, or add periodic cleanup. And a per-process limiter is per process: ten replicas each send their own five events per minute. That is still a large reduction from unbounded.

Hosted Sentry also offers server-side controls such as spike protection and inbound filters (as of late 2026 these are plan-dependent in places, so verify on your tier). Server-side filtering is a safety net, but dropping events in the client means they never reach the quota at all.

Takeaway: a ten-line beforeSend limiter removes the failure mode that makes every error-tracking plan look too small.

Sentry vs GlitchTip vs self-hosted Sentry: what actually differs?

	Hosted Sentry	GlitchTip (hosted or self-hosted)	Self-hosted Sentry
SDK	Native Sentry SDKs	Sentry-compatible SDKs, change the DSN	Native Sentry SDKs
Scope	Errors, performance, session replay, profiling, more	Mostly error tracking, plus basic uptime and performance features	Broad, close to hosted feature set
Ops burden	None	Low: a small app plus Postgres and a queue	High: many services to run and upgrade
Cost driver	Event volume and add-ons	Hosted: plan tier; self-hosted: your server	Server size and your time
Main drawback	Quota anxiety, add-ons priced separately	Smaller feature set, smaller ecosystem	Upgrades and resource needs

GlitchTip is open source, and because it speaks the Sentry ingestion protocol you can point an existing SDK at it by swapping the DSN. In my testing of that swap on a small Node service, errors, stack traces and breadcrumbs arrived without code changes. What you give up is breadth: if you rely on session replay, deep performance tracing or the newer AI-assisted triage features, expect gaps, and verify against GlitchTip's current docs before committing.

Self-hosted Sentry is the full product, but it runs as a large set of containers (databases, queues, ingestion workers and more) and its own docs describe a substantial minimum resource footprint. Upgrades are guided by an install script and are not a casual weekend task. For a team of two to five people, I have seen the hidden cost land as "nobody wants to own the upgrade", which is how an error tracker quietly stops working.

Takeaway: the SDK is portable across all three, so the real choice is how much of the platform you want to operate.

When is hosted Sentry worth paying for?

If you want one place for errors, performance traces and release health with no infrastructure, hosted Sentry is the product that covers all of that out of the box, and a paid plan is justified once errors are tied to revenue and someone is on call. The drawbacks are real: pricing is usage-based, so a bad incident can push you into overage or throttling, and optional features are billed separately.

If you only need "tell me when production throws and show me the stack trace", GlitchTip is the one that does that with the least overhead, and it is cheap or free to run on a server you already have. Its limitation is that anything beyond core error tracking is thinner than the original.

Self-hosted Sentry makes sense mainly when data residency or policy forbids a third-party SaaS and you have someone who will own upgrades.

Takeaway: pay for hosted Sentry when breadth matters, pick GlitchTip when errors are all you need, and self-host full Sentry only under a compliance constraint.

FAQ

Is GlitchTip compatible with the Sentry SDK?
Yes, GlitchTip accepts events from Sentry client SDKs, so you usually only change the DSN. Features beyond error ingestion, like replay, may not be available.

How do I stop one error from using my whole Sentry quota?
Drop repeats in the SDK with beforeSend and ignoreErrors, and use sampleRate for high-volume noise. Server-side spike protection helps, but client-side filtering keeps events from being counted at all.

Is self-hosting Sentry cheaper than the hosted plan?
Only if your time is free. The server cost is modest compared with the upgrade and maintenance effort of running a multi-service stack.

Bottom line

Start with hosted Sentry and a beforeSend limiter; most teams never need to leave. Move to GlitchTip if your bill is driven by volume and you only use error tracking. Self-host full Sentry only when policy forces it and an owner is named. Whatever you choose, test an incident: throw the same error 10,000 times in staging and see what the dashboard and your quota do.

Docker "toomanyrequests: You have reached your pull rate limit" in CI: Log In, Mirror, or Run a Pull-Through Cache?

Libme — Tue, 20 Oct 2026 21:10:08 +0000

If your CI pipeline suddenly fails with toomanyrequests on docker pull or FROM, Docker Hub is throttling the IP your runners share, not your code. The fastest fix is to authenticate the pull with docker login; the durable fix is to stop pulling the same base images from Docker Hub on every run, using a registry mirror or a pull-through cache. Which one you need depends on how many pulls you make and who owns the IP.

What does the error actually look like?

The text varies slightly by Docker version, but the shape is stable:

ERROR: failed to solve: ubuntu:24.04: failed to resolve source metadata for
docker.io/library/ubuntu:24.04: unexpected status from HEAD request to
https://registry-1.docker.io/v2/library/ubuntu/manifests/24.04: 429 Too Many Requests

toomanyrequests: You have reached your pull rate limit. You may increase the limit
by authenticating and upgrading: https://www.docker.com/increase-rate-limit

What tripped me up the first time: the job had passed for weeks and nothing in the repo changed. The limit is counted per source, not per project. Unauthenticated pulls are counted against the client IP, and hosted CI runners share IP ranges with many other tenants. Your pipeline can fail because a stranger's pipeline pulled too much. Once you authenticate, the count moves to your account.

Takeaway: a rate-limit failure that appears with no code change usually means the shared IP's quota ran out, not that your pipeline got heavier.

How do I check how many pulls I have left?

Docker documents a special repository, ratelimitpreview/test, that returns your remaining quota in response headers without spending a pull. Run this from the machine (or CI step) that is failing:

TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)

curl -sI -H "Authorization: Bearer $TOKEN" \
  https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest \
  | grep -i ratelimit

You will see ratelimit-limit and ratelimit-remaining, each with a window (for example ;w=21600 for six hours). To see the authenticated numbers, pass your credentials to the token request with curl -u "$USER:$PAT". Comparing the two outputs is the quickest way to confirm that logging in actually changes your bucket.

The exact limits and windows have changed over the years and Docker has revised the free-tier numbers more than once, so read the current figures on Docker's pull-usage documentation rather than trusting any blog post, including this one (as of October 2026, I would not hardcode a number anywhere).

Takeaway: read ratelimit-remaining before changing anything, so you know whether you are fixing the right problem.

Fix 1: authenticate every pull (five minutes)

In GitHub Actions, log in before anything touches the registry. Use a personal access token with read-only scope, not your account password:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - uses: docker/setup-buildx-action@v3
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: false

Two gotchas. First, docker/login-action makes the login visible to Buildx builds, but a plain docker pull inside a nested container or a services: image may use a different credential path. Second, secrets are empty in pull requests from forks (I wrote about that separately), so fork PRs fall back to anonymous pulls and can still hit the limit.

Takeaway: logging in is the cheapest fix, but it only helps jobs that can see the secret.

Fix 2: stop pulling from Docker Hub at all

Most of the pulls in a typical pipeline are a handful of official base images: node, python, postgres, redis. Two options remove them from Docker Hub's accounting.

Use a mirror of the official images. Amazon publishes copies of Docker's official images on its public registry, so FROM public.ecr.aws/docker/library/node:22-slim pulls the same library image from a different provider. The limitation is that only the official library images are mirrored there, not arbitrary third-party images, and you depend on that copy staying in sync (check the tag you pin actually exists before you rewrite every Dockerfile).

Run a pull-through cache. A cache sits between your runners and Docker Hub, stores each layer once, and serves it to every later job. The open source registry image can do this in a few lines:

# config.yml for the registry:2 image
version: 0.1
storage:
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
proxy:
  remoteurl: https://registry-1.docker.io
  username: your-dockerhub-user
  password: your-dockerhub-read-token

Then point the Docker daemon at it with "registry-mirrors": ["http://your-cache:5000"] in /etc/docker/daemon.json. Note that this mirror setting only applies to Docker Hub images, which is exactly the case you are fixing. Cloud registries offer managed versions of the same idea (Amazon ECR's pull-through cache can front Docker Hub when you give it credentials), which removes the server you would otherwise babysit.

The drawbacks are real. A self-run cache is another service that can go down and block every build, it needs disk cleanup, and on ephemeral hosted runners there is no private network to put it on. If your builds run on self-hosted runners in one network, a cache pays off quickly; on hosted runners, prefer the mirror or login.

Takeaway: a cache turns N pulls per day into one, but only if your runners can actually reach it.

Which option fits which setup?

Situation	Best fix	Main drawback
Hosted CI, few images, secrets available	`docker login` with a read-only token	Fork PRs still anonymous
Hosted CI, mostly official base images	Mirror via `public.ecr.aws/docker/library`	Official images only; tag sync
Self-hosted runners, many pipelines	Pull-through cache	You run and patch another service
Already on AWS with ECR	ECR pull-through cache	Setup of credentials and repo prefixes
Publishing your own images too	Push to GHCR or ECR, keep Hub as a pull source	Migration of image names

If you are choosing a place for your own images, GitHub's container registry is the lowest-friction choice when your code already lives on GitHub, since GITHUB_TOKEN can authenticate the push without another secret.

Takeaway: pick by where your runners live, not by which fix sounds most robust.

FAQ

Why does docker pull say toomanyrequests when I only pulled once?
Unauthenticated pulls are counted per IP address, so shared CI or office NAT addresses burn the quota across many users. Authenticating moves the count to your account.

Do HEAD requests count toward the Docker Hub rate limit?
Docker's documentation says manifest HEAD requests do not count as pulls, which is why the ratelimitpreview check above is free. The 429 in a build log can still appear on a HEAD when you are already over the limit.

How do I avoid Docker Hub rate limits without paying?
Authenticate with a free-account token, pin base images to a mirror such as the Amazon ECR public registry copy of the official images, and cache layers in CI so unchanged layers are not re-pulled.

Bottom line

If you are on hosted CI with a handful of images, add docker/login-action with a read-only token today and check ratelimit-remaining to confirm it worked. If most of your pulls are official images, move the FROM lines to a mirror and you are mostly done. Only build a pull-through cache when you run self-hosted runners at a scale where a mirror's gaps start to hurt, and plan for it being one more thing that can break your builds.

DEV Community: Libme

pip "externally-managed-environment" Error: venv, pipx, or uv?

What does the error actually say, and why does it happen?

Which dead ends should you skip?

Which tool fits which job?

How do you set up a venv correctly (including the Debian gotcha)?

How do you install command-line tools without polluting anything?

When is --break-system-packages acceptable?

FAQ

How do I fix "error: externally-managed-environment" on Ubuntu or Debian?

Is pip install --break-system-packages safe?

Should I use pipx or uv to install Python command-line tools?

Bottom line

Related reading

Git Hooks Not Running for Teammates? Husky vs Lefthook vs pre-commit

Why do hooks work on my machine but not on a teammate's?

How do husky, lefthook, and pre-commit install themselves?

What goes wrong even after you adopt one?

Should hooks be the only enforcement?

FAQ

Bottom line

Related reading

GitHub Actions Cache Never Hits on Pull Requests: Branch Scope, Immutable Keys, and restore-keys

What does the failure actually look like?

Why can't my PR see the cache that main built?

Why does the cache never update when my lockfile changes... or doesn't?

How do restore-keys and cache-hit actually behave?

What about eviction and size limits?

When is a built-in cache not enough?

FAQ

Bottom line

Related reading

Docker Compose depends_on Doesn't Wait for the Database: Fix "Connection Refused" on Startup

What does the failure look like?

How do you make Compose wait for Postgres?

Why does the healthcheck pass too early on a fresh volume?

What about migrations that must finish before the app starts?

Do you still need retries in the app?

Which approach should you pick?

FAQ

Bottom line

Related reading

httpx.PoolTimeout: Why Your Async Python Client Runs Out of Connections (and How to Fix It)

What does PoolTimeout actually mean?

Why do I hit it? Three causes, in order of likelihood

How do I reproduce the leak?

How do I fix it?

httpx vs aiohttp vs requests: how do their pools compare?

How do I confirm the fix?

FAQ

What is the default connection limit in httpx?

Should I increase the pool timeout to fix PoolTimeout?

Is it safe to share one httpx.AsyncClient across tasks?

Bottom line

Related reading

Git "detected dubious ownership in repository": Fix It in Docker and CI Without Trusting Everything

What does the error actually look like?

Why does git config --global seem to do nothing in my container?

Which fix should I use?

Match the UID instead of trusting the directory

Trust one explicit path in a throwaway job

Pass the setting through the environment

Is safe.directory '*' ever okay?

What about sudo and root?

FAQ

How do I fix "fatal: detected dubious ownership in repository" in Docker?

Why does safe.directory in .git/config not work?

Is it safe to set safe.directory to *?

Bottom line

Related reading

Is Paying for Bigger CI Runners Worth It? Break-Even Math Against Caching and Test Sharding

Where do CI minutes actually go?

What should I fix before paying for anything?

Does a bigger runner make CI proportionally faster?

When is the faster pipeline worth the extra spend?

What about self-hosted runners?

FAQ

Bottom line

Related reading

Redis vs Valkey vs Dragonfly: Which In-Memory Store Should a New Project Use After the License Changes?

When is `--break-system-packages` acceptable?

Is `pip install --break-system-packages` safe?

Why does `git config --global` seem to do nothing in my container?

Is `safe.directory '*'` ever okay?

Is it safe to set safe.directory to `*`?

Why does `def` work when `async def` doesn't?

Why doesn't `git rm` and a new commit fix it?