DEV Community: Temir Kussainov

I built a CLI that tells you which MCP servers your repo actually needs

Temir Kussainov — Tue, 20 Oct 2026 08:11:11 +0000

Every week someone posts another "top 10 MCP servers for Claude Code" list. I kept installing things from those lists, forgetting what I had, and once I noticed a project of mine was still running @modelcontextprotocol/server-postgres. That server was archived in May 2025, and Datadog showed its read-only mode can be bypassed with COMMIT; .... Nobody is going to fix it.

So I wrote stackhint. It looks at a repository, checks what is already configured in Claude Code and Codex, and suggests the MCP servers, plugins, skills and CLIs that fit. It explains why each one fits, and it can add a server to your config after showing you the exact change.

It runs locally, has no dependencies and sends nothing anywhere.

What it looks like

$ npx -y github:Protemir/stackhint recommend --task "orders query is slow"
Detected: PostgreSQL, C#, .NET, Redis, GitHub Actions, Docker, EF Core, Jira, OpenAPI, GitHub

1. Recommendation: Postgres MCP Pro  fit 67/100, confidence high
Why now: Service image postgres:17 (docker-compose.yml); your task mentions "query"; 9 database migrations.
Permissions: stackhint uses --access-mode=restricted (read-only transactions).
Actions:
  - stackhint install postgres-mcp-pro --host claude - shows the exact change and asks before writing
  - stackhint try postgres-mcp-pro --host claude - one session without installing
  - stackhint dismiss postgres-mcp-pro --for 14d - remind me later

2. Recommendation: Microsoft Learn MCP Server  fit 58/100, confidence high
Why now: Solution file (DotnetShop.sln); ASP.NET Core web SDK.

3. Recommendation: Atlassian Rovo MCP Server  fit 52/100, confidence high
Why now: Links to acme-demo.atlassian.net Jira (README.md).

Every line in "Why now" points at a file. If your README only says "we might try Kubernetes one day", you get no Kubernetes suggestion. It needs a manifest, a compose image, a dependency or a link.

How it decides

The analyzer reads the files that usually give a stack away: *.csproj, package.json, pyproject.toml, go.mod, compose files, Kubernetes and Terraform files, CI configs, .http files, the key names in .env.example, and links in the README. It also reads git remotes and recent commit subjects. Issue keys like SHOP-123 hint at Jira, and twelve "add migration" commits hint that a project skill would help. It never opens .env, private keys or secret settings files.

Then it reads your setup: ~/.claude.json, .mcp.json, enabled Claude Code plugins and the servers they bundle, ~/.codex/config.toml, Codex plugins, skills. If you already have GitHub's MCP server, or gh on your PATH, it won't push another GitHub integration on you.

The ranking is a plain formula: relevance from the evidence, times value, minus setup cost and risk. The numbers are my estimates, and the tool says so in its output. stackhint explain <id> prints the whole breakdown.

The registry is the actual work

The hard part wasn't the code. It was checking 68 entries by hand: the endpoint, the package name, the read-only flag, whether the project is still maintained. A few things I didn't expect:

Atlassian's recommended endpoint is now /v2/mcp, and the old /v1/sse stopped working in June.
kubernetes-mcp-server removed its --read-only flag on main. The next release will reject it, so stackhint pins 0.0.67, where the flag still works.
Microsoft deleted its MSSQL MCP sample as unsafe in June. The supported route is Data API builder.
CircleCI's npm package, Notion's local server, Semgrep's standalone repo and Cloudflare's per-product servers are all deprecated or archived.

stackhint knows the old ones too. If it finds one in your config, stackhint check tells you what to replace it with.

Installing without surprises

stackhint install prints a plan first: where it will write, the diff, what the server can access, which environment variables it needs (by name only) and how to undo it. It asks, saves a backup, writes atomically, parses the file back and restores the original if anything looks wrong. stackhint rollback <id> puts the file back byte for byte.

A few rules:

Read-only variants are the default whenever the vendor offers one.
Tokens are never written. Claude Code gets ${GITHUB_PERSONAL_ACCESS_TOKEN}, and Codex gets bearer_token_env_var.
For Codex it appends one [mcp_servers.x] table and leaves the rest of config.toml alone. codex mcp add rewrites the whole table and drops your comments.
For Claude Code's user and local scopes it calls claude mcp add, because Claude rewrites ~/.claude.json all the time.

Things I learned the hard way

My unit tests were green, then the first run against the real claude binary failed. claude mcp add takes --env with several values, so --env KUBECONFIG=... kubernetes -- npx ... swallows the server name. The fix is to put --env and --header before --transport. There is now a test that runs the real Claude Code and Codex CLIs in CI against configs stackhint wrote.

On native Windows both clients need cmd /c npx ... for npx-based servers, and the tool now writes it that way.

Before the release I put the code through three rounds of review. The worst find: git log inside a scanned repository reads that repository's own .git/config, and with log.showSignature plus a custom gpg.program it runs whatever program the repo points at. So "just read the commit history" could execute code from a repo you only wanted to look at. stackhint now switches all of that off, and a test with a fake signed commit checks it stays off.

Claude Code and Codex in one repo

The same GitHub repo works as a plugin marketplace for both clients. It ships a skill, so you can ask "which MCP servers would help here?", and two hooks. In the default balanced mode, the session-start hook passes up to two high-confidence suggestions to the agent, at most once a day per project. You can turn it off with stackhint config set mode manual.

/plugin marketplace add Protemir/stackhint
/plugin install stackhint@stackhint

codex plugin marketplace add Protemir/stackhint

There is also a read-only MCP server (stackhint mcp) if you prefer that. On purpose, it has no install tool.

Update: 0.2.0

The same day I released 0.2.0. Three more review passes found 34 problems, and a review of the fixes found 9 more; each one now has a test. The worst: installing into a config.toml saved in a non-UTF-8 encoding could corrupt it, and secrets from a team's .vscode/mcp.json could end up in scan --format json output.

The registry grew from 68 to 143 entries: servers for Next.js, Angular, Svelte, Expo, Flutter, Unity and Unreal, hosting like Netlify and Fly.io, observability like Datadog, and all 13 official LSP plugins. The independent check of the new entries caught a nice one: monday.com's MCP server ignores --read-only unless you pass a value, so "read-only" quietly meant full access. The scan is also about three times faster.

Try it

npx -y github:Protemir/stackhint recommend
npx -y github:Protemir/stackhint check

It's MIT and on GitHub: https://github.com/Protemir/stackhint

The registry is where help matters most. If your stack isn't covered, or a vendor changed an endpoint, open an issue or a PR with the source you checked.

Syncing a browser extension's settings without a server: three reviews, three ways to lose data

Temir Kussainov — Sun, 18 Oct 2026 10:57:04 +0000

Headerwise is a header editor for Chrome and Edge I wrote after ModHeader got pulled from the stores. People asked for one thing ModHeader charged for: having the same profiles on the work laptop and the home desktop.

I didn't want a server. The extension's manifest forbids network requests (connect-src 'none'), and I'd like to keep it that way. Luckily the browser already has sync: chrome.storage.sync goes through the Google (or Microsoft, in Edge) account the browser is signed in to, and the extension never talks to anything itself.

So the job looked small: copy the profiles into chrome.storage.sync, copy them back out on the other computer. It took four rounds of review to get right, and the first three each found a way to lose someone's profiles. Here's what broke, in case you're about to do the same thing.

The limits shape the design

chrome.storage.sync gives you 100 KB in total, 8 KB per item, 512 items, and 120 writes a minute. A profile with a few headers is a few hundred bytes, so one item per profile works, and it means two computers editing different profiles never touch the same item.

Some things shouldn't sync at all: whether a profile is switched on (turning on a production header at work shouldn't turn it on at home), "only in this tab", the timer, and secret values. Tokens and cookies are stripped before anything leaves the computer, and each computer keeps its own.

Version 1: push and pull. Lost edits.

Push: write every profile and an index of ids. Pull: replace local profiles with what's in sync; anything I'd seen before that is gone from the index was deleted elsewhere.

The review found the problem fast. A write to chrome.storage.sync fires onChanged on the same computer too, so a pull follows every push. If you type again between the push and that pull, the pull puts back the version you just pushed. Your text jumps back while you type.

Worse, "gone from the index" isn't the same as "deleted". The browser keeps the last write of each key. If computer B writes its index before it has seen the profile A just created, A's profile is missing from the index, and A dutifully deletes it.

Version 2: a three-way merge. Lost edits, differently.

The fix for the first problem is old: keep a base. Each computer remembers every profile as it was when it and sync last agreed. Then a round can tell who changed what:

local equals base, sync differs: changed elsewhere, take it in;
sync equals base, local differs: changed here, send it;
both differ: changed on both, keep both versions (the other one as a copy, so nothing disappears silently).

And deletions became explicit marks instead of "missing from the index". Profiles exist as items; the order is just an order.

The second review modelled each computer with its own copy of sync storage and a last-write-wins server, which is how the browser actually behaves, and found that "local equals base" doesn't mean what I thought. A edits a profile and sends it, so A's base is A's version. B, which hadn't seen it yet, sends its own edit. Last write wins: B's. On A, local equals base, sync differs, so "changed elsewhere": A takes B's version and A's edit is gone, with no copy.

Version 3: remember the parent. Old copies win.

So every item got a parent: a fingerprint of the version it was edited from. If what arrives wasn't built on what I last agreed on, both versions are kept.

The third review found that this keeps far too much. A computer that was asleep misses versions all the time. A goes v0 → v1 → v2, B wakes up and sees only v2, whose parent is v1. B's base is v0, so "not built on what I have", and B kept its v0 under the id, sent it, and turned v2 into a copy. Every computer ended up back on v0. With three computers it was worse: each one saw a parent mismatch and resent its own version, forever, until the browser's write quota ran out.

Version 4: only send what changed here

The rule that finally held: a computer sends a profile only if it changed on that computer. If it changed nothing, it takes the newest version, whatever it missed in between. That alone makes every round settle, because writes only follow edits.

The parent check stays, but only for the one case it's for: this computer wrote the version it last agreed on, and something replaced it without having seen it. Then (and only then) both are kept. The base remembers which versions it sent itself for exactly this.

The fourth review ran all 18 timelines from the first three, plus new ones aimed at this rule, and found nothing in the merge. It did find that the popup could drop a token pasted in the same quarter second a change arrived, which is a different story.

What I'd tell myself at the start

Model the browser honestly in tests: separate copies of sync per computer, last write wins per key, rounds that don't see each other. My first tests shared one store and ran rounds in a tidy order, and everything passed.
"Missing" is not "deleted". Write deletions down.
A base tells you who changed what; a parent tells you who saw what. You need both, and only the computer that wrote a version may defend it.
Every round should be able to run again and change nothing. If it can't, it will eventually run forever.

The code is MIT and the merge is one file, src/core/sync.ts, with the timelines as tests in tests/sync.test.ts: https://github.com/Protemir/headerwise. Headerwise itself is free in the Chrome Web Store: https://chromewebstore.google.com/detail/jedoaeaapdofoldmkacjbpojbnpapkna

If you've done sync over chrome.storage.sync differently, I'd like to hear how.

ModHeader in Selenium stopped working. Here's what sets HTTP headers in tests in 2026

Temir Kussainov — Mon, 12 Oct 2026 10:00:59 +0000

Updated October 18, 2026: Chrome for Testing 155 stopped loading extensions from --load-extension, so the Selenium examples below now install the extension over WebDriver BiDi.

If your browser tests set headers through ModHeader (chrome-modheader on npm, modheader_selenium for Python and Java), they probably broke this year, and not only because ModHeader was pulled from the stores in July. Two other things happened:

The extension bundled in chrome-modheader is ModHeader 4.3.2, a Manifest V2 build. Chrome turned off MV2 extensions, so it never loads, and the package hasn't been updated since.
Google Chrome 137 and newer ignore --load-extension, which is how ChromeDriver installs any extension you pass with addExtensions. Chrome for Testing followed in version 155 (October 2026), so Selenium now installs extensions over WebDriver BiDi instead, below.

The package still gets about 150k downloads a month, mostly from CI, so a lot of pipelines are installing something that does nothing. Here's what works now, from the simplest option up.

Request headers on every request: no extension needed

Playwright has it built in:

const context = await browser.newContext({ extraHTTPHeaders: { 'X-Env': 'staging' } });

Selenium 4 with Chrome can ask the DevTools protocol:

driver.execute_cdp_cmd('Network.enable', {})
driver.execute_cdp_cmd('Network.setExtraHTTPHeaders', {'headers': {'X-Env': 'staging'}})

Both add the headers to every request from that context or tab, which covers most "log in as a test user" and "route to staging" cases. They don't change response headers, can't remove a header the browser sends, and don't limit headers to some URLs (in Playwright, page.route() can, at the cost of handling each request in your test).

Response headers, removing headers, some URLs only

That's where an extension still earns its place: CORS or CSP response headers, stripping Referer or Cookie, a header only for api.example.com. A proxy like mitmproxy can do all of that too, but then you manage certificates, and on a cloud grid that's often not an option.

Disclosure: I maintain Headerwise, an open source (MIT) MV3 header extension, and it has an automation build for this. It has a fixed extension id, and your test sets headers by opening one URL:

chrome-extension://ogjbgamdhnjnagcdgboifgmhlgddcdce/automation.html?X-Env=staging&res:Access-Control-Allow-Origin=*

The page waits until Chrome has the rules and then reports ready. Playwright, with its own Chromium:

const ext = '/path/to/headerwise-automation';
const context = await chromium.launchPersistentContext('', {
  channel: 'chromium',
  args: [`--disable-extensions-except=${ext}`, `--load-extension=${ext}`],
});
const page = await context.newPage();
await page.goto('chrome-extension://ogjbgamdhnjnagcdgboifgmhlgddcdce/automation.html?X-Env=staging');
await page.waitForFunction(() => document.documentElement.dataset.status === 'ready');

Selenium installs it over WebDriver BiDi (checked with Selenium 4.51; the path is the unzipped folder):

options = webdriver.ChromeOptions()
options.enable_bidi = True
options.enable_webextensions = True
driver = webdriver.Chrome(options=options)
driver.webextension.install(path='/path/to/headerwise-automation')

If your tests are written for ModHeader

The automation build also answers ModHeader's own webdriver.modheader.com/add, /clear and /load URLs (nothing is sent to modheader.com: the browser redirects them to the extension before any request leaves) and titles the page "Done", like ModHeader did. So a test like this only changes how the extension gets in:

const { getAddHeaderUrl } = require('chrome-modheader');
const options = new chrome.Options().addArguments('--enable-unsafe-extension-debugging', '--remote-debugging-pipe'); // was addExtensions(getExtension())
options.enableBidi();
const driver = await new Builder().forBrowser('chrome').setChromeOptions(options).build();
await (await driver.getBidi()).send({ method: 'webExtension.install',
  params: { extensionData: { type: 'path', path: '/path/to/headerwise-automation' } } });
await driver.get(getAddHeaderUrl('X-Env', 'staging'));
await driver.wait(until.titleIs('Done'), 10000);

On a grid that takes the extension as a base64 .crx in the capabilities, ask for Chrome for Testing 154 for now (BrowserStack calls it chromefortesting): newer versions don't install a packed extension that way, and BiDi installs only a folder on the machine that runs the browser.

Setup for Selenium, Playwright and Puppeteer: https://protemir.github.io/headerwise/automation.html. CI runs those snippets on every change, including one test written for ModHeader with only the extension swapped.

Which one to pick

Same request headers everywhere: extraHTTPHeaders or CDP. Nothing to install.
Response headers, removals, per-URL rules, or a lot of existing ModHeader URLs in your tests: an extension (Headerwise or whatever you trust).
Firefox, or traffic outside the browser too: a proxy.

If ModHeader itself got switched off in your everyday browser too, I wrote up how to get your profiles back. And if something in your test setup doesn't fit any of these, tell me in the comments.

ModHeader got pulled from the Chrome Web Store. Here's how to get your profiles back

Temir Kussainov — Mon, 12 Oct 2026 09:58:17 +0000

If you opened Chrome recently and ModHeader was gone, or sitting there turned off, you're not alone. Microsoft pulled it from Edge Add-ons on July 3, 2026 and Google followed on July 10. Version 7.0.18 shipped a dormant collector for the domains you visit, plus install and update pings to a third-party server (The Hacker News has the details). The collector reportedly never switched on, but the listings are gone either way.

The annoying part: once Chrome turns an extension off, you can't open it, so the Export button is gone too. If you had a dozen profiles with tokens, URL filters and staging headers, that hurts.

Your profiles are still on your disk, though. Chrome keeps every extension's chrome.storage.local in a LevelDB folder under your browser profile, and ModHeader keeps everything under one key called profiles.

Where the folder is

ModHeader's id in the Chrome Web Store was idgpnmonknjnojddfkpgkljpfnnfcklj (on Edge Add-ons it was opgbiafapkbbnbnjcdomjaghbckfkglc).

Windows:

%LOCALAPPDATA%\Google\Chrome\User Data\Default\Local Extension Settings\idgpnmonknjnojddfkpgkljpfnnfcklj

macOS:

~/Library/Application Support/Google/Chrome/Default/Local Extension Settings/idgpnmonknjnojddfkpgkljpfnnfcklj

Linux:

~/.config/google-chrome/Default/Local Extension Settings/idgpnmonknjnojddfkpgkljpfnnfcklj

If you use several Chrome profiles, open chrome://version in the one where you had ModHeader and use its "Profile Path" instead of Default. If the folder isn't there, Chrome may have deleted it when it removed the extension, and then there's nothing left to recover. So do this soon, and don't click "Remove" on ModHeader before you have your profiles.

Option 1: a command line tool

Requestly published a small CLI that finds the folder and writes the same JSON file ModHeader's own export would have: npx @requestly/modheader-export-backup. You need Node 20 or newer. The file works with any tool that imports ModHeader JSON.

Option 2: read it in the browser

This is partly why I built Headerwise, a free and open source header editor. Its "Move from ModHeader" page lets you pick that folder and reads the LevelDB files right in the page (I wrote a small LevelDB and snappy reader in TypeScript for it), with nothing uploaded. Headers, URL filters, site, type and method filters, URL replacements and cookie rules come over, and the profile you had selected stays on. Chrome Web Store, source.

Option 3: by hand

If you'd rather not run anything: copy the folder somewhere safe first, then open the copy with any LevelDB viewer and read the value of the profiles key. It's plain JSON.

Whatever you pick

Exports often hold Authorization tokens and session cookies. Treat the file like a password: don't commit it, and rotate anything sensitive you kept in ModHeader if you're not sure what happened to it.

If you used ModHeader in Selenium or Playwright tests, that broke separately: its test package ships a Manifest V2 build that current Chrome won't load. Here's what works now. And if your setup needs something none of these handle, tell me in the comments.

Why ClickHouse® fills the disk in self-hosted Langfuse and SigNoz, and how to fix it

Temir Kussainov — Sat, 26 Sep 2026 04:40:18 +0000

Also in Español · Português · Русский · 日本語 · 한국어 · 中文.

Your Langfuse or SigNoz server is running out of disk, but your traces are small. In most public reports the space goes to ClickHouse® itself: to its own log tables, such as system.trace_log and system.text_log, which have no size limit by default. ClickStack has the same problem.

This guide shows how to confirm it with one read-only query, free the space now, and stop it from coming back without hitting the known traps. You don't need any extra tools.

Tested on stock clickhouse/clickhouse-server images 24.8, 25.12 and 26.9 (the SigNoz commands on 25.5 and 25.12), with the ClickHouse service and config taken from Langfuse's and SigNoz's compose files.

TL;DR

Check. Run one read-only query on system.parts. If system.*_log tables are bigger than your data, ClickHouse's own logs are the problem.
Free space now. TRUNCATE the big logs. For a table over 50 GB, add SETTINGS max_table_size_to_drop = 0.
Stop it coming back. Add a config.d file with a TTL for each log (for opentelemetry_span_log the TTL goes inside <engine>), then restart ClickHouse.
Clean up. Drop the *_log_0, *_log_1 copies that the restart leaves behind.
Still full? Check Docker's container logs, deleted rows and stuck parts. On Langfuse, update Langfuse before you move ClickHouse to 26.8+.

1. What is using the disk? Check ClickHouse disk usage by table

Save this query as size.sql. It lists the biggest tables by size on disk:

SELECT database, table,
       formatReadableSize(sum(bytes_on_disk)) AS size,
       sum(rows) AS rows,
       count() AS parts
FROM system.parts
WHERE active
GROUP BY database, table
ORDER BY sum(bytes_on_disk) DESC
LIMIT 15;

Langfuse (docker compose). Run this in the folder with Langfuse's docker-compose.yml. Its compose file already sets CLICKHOUSE_USER and CLICKHOUSE_PASSWORD inside the container:

docker compose exec -T clickhouse sh -c \
  'clickhouse-client --user "$CLICKHOUSE_USER" --password "$CLICKHOUSE_PASSWORD" --readonly=1 --format PrettyCompact' < size.sql

SigNoz (docker compose). The container is usually called signoz-clickhouse (check with docker ps). SigNoz's users.xml leaves the default user without a password:

docker exec -i signoz-clickhouse clickhouse-client --readonly=1 --format PrettyCompact < size.sql

--readonly=1 makes the session read-only: ClickHouse refuses any change.

How to read it. Compare the system rows with your own data (default for Langfuse, signoz_* for SigNoz). If system.trace_log or system.text_log is at the top, go on to section 2.

2. How do I free the disk space right now?

Open a client with write access:

# Langfuse
docker compose exec clickhouse sh -c 'clickhouse-client --user "$CLICKHOUSE_USER" --password "$CLICKHOUSE_PASSWORD"'
# SigNoz
docker exec -it signoz-clickhouse clickhouse-client

Empty the big logs:

TRUNCATE TABLE system.trace_log;
TRUNCATE TABLE system.text_log;
TRUNCATE TABLE system.metric_log;
TRUNCATE TABLE system.query_log;

This deletes only ClickHouse's own diagnostics, not your traces, and needs no restart.

Langfuse reads system.query_log to follow some of its own queries (#13123). So empty it, but don't switch it off.

TRUNCATE fails with Code 359? Tables over 50 GB

ClickHouse won't drop or truncate a table bigger than max_table_size_to_drop (default 50,000,000,000 bytes, about 46.6 GiB). The limit applies to TRUNCATE too, and a bigger table fails with Code: 359 ... (TABLE_SIZE_EXCEEDS_MAX_DROP_SIZE_LIMIT).

Big logs do hit it: the text_log in Langfuse discussion #15024 was 59.20 GiB. There are two ways around it.

Option A. Lift the limit for one statement:

-- ClickHouse 23.12+: lift the limit for this one statement
TRUNCATE TABLE system.text_log SETTINGS max_table_size_to_drop = 0;

Option B. Create the one-time flag file, then run the plain TRUNCATE:

docker compose exec clickhouse sh -c 'touch /var/lib/clickhouse/flags/force_drop_table && chmod 666 /var/lib/clickhouse/flags/force_drop_table'

(SigNoz: docker exec signoz-clickhouse sh -c '...' with the same command.) ClickHouse deletes the flag after the first DROP or TRUNCATE that needs it, so create it again before the next big table.

The space is back now, but the logs start growing again right away. Section 4 stops that. Section 3 explains why it happens.

3. Why are trace_log and text_log so big?

ClickHouse writes its own diagnostics into tables in the system database: query_log, trace_log, text_log, metric_log and more. The documentation says it plainly: "By default, table growth is unlimited."

Recent default configs set a TTL only for a few small logs (since 25.9, processors_profile_log keeps 30 days), not for the big ones. Two defaults make the big ones grow fast:

the query profiler is on and writes stack samples of running queries into trace_log;
text_log stores the server log at trace level.

Public examples:

Langfuse #13123: system.trace_log was 66.86 GiB, while the Langfuse tables listed there (traces, observations, scores) took about 51 MiB together. The maintainers decided not to override ClickHouse's defaults and added a FAQ page instead.
SigNoz #12050: more than 80 GB of system logs (including a trace_log_0 copy) next to less than 500 MB of telemetry. Truncating the system.* tables freed about 80 GB. SigNoz's new Foundry installer sets TTLs; older docker compose installs don't have them.
ClickStack Helm chart PR #275: a 10 Gi volume hit 100% after 10 days, with 5.2 GB of trace-level server logs and a 3.7 GB system.text_log against ~70 MB of telemetry. The PR, merged in September 2026, adds a 7-day TTL.

4. How do I set a TTL on ClickHouse system logs?

A TTL makes ClickHouse delete old log rows by itself. Set it after the TRUNCATE from section 2, because of how ClickHouse applies it.

When you add a TTL and restart, ClickHouse doesn't shrink the old table. It renames it to trace_log_0 with all its rows and creates a new one with the TTL. If you truncated first, these copies are tiny, and step 4 drops them.

Run the SQL below in the same write-access client as in section 2.

Step 1: which logs have no TTL?

SELECT name,
       formatReadableSize(total_bytes) AS size,
       extract(engine_full, 'TTL [^S]*') AS ttl
FROM system.tables
WHERE database = 'system' AND engine LIKE '%MergeTree' AND match(name, '_log(_[0-9]+)?$')
ORDER BY total_bytes DESC;

An empty ttl means the log is kept forever. Names like trace_log_0 are old copies; step 4 removes them.

Step 2: add a TTL file in config.d

Save this as clickhouse-ttl.xml next to docker-compose.yml:

<clickhouse>
    <!-- Keep 7 days of ClickHouse's own logs. List only logs your server has. -->
    <query_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></query_log>
    <trace_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></trace_log>
    <text_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></text_log>
    <metric_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></metric_log>
    <asynchronous_metric_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></asynchronous_metric_log>
    <part_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></part_log>

    <!-- The stock config defines this log with <engine>, so a separate <ttl>
         stops the server from starting (ClickHouse#88366). TTL goes inside. -->
    <opentelemetry_span_log>
        <engine>ENGINE = MergeTree PARTITION BY toYYYYMM(finish_date) ORDER BY (finish_date, finish_time_us) TTL finish_date + INTERVAL 7 DAY DELETE</engine>
    </opentelemetry_span_log>
</clickhouse>

Mount it into the clickhouse service (keep your existing volume lines):

  clickhouse:
    volumes:
      - ./clickhouse-ttl.xml:/etc/clickhouse-server/config.d/clickhouse-ttl.xml:ro

Three traps in this step:

opentelemetry_span_log takes its TTL inside <engine>. The stock config defines this log with <engine>, and per the docs that conflicts with <ttl>. The server then exits with If 'engine' is specified for system table, TTL parameters should be specified directly inside 'engine' (ClickHouse #88366). Copy PARTITION BY and ORDER BY from SELECT engine_full FROM system.tables WHERE database = 'system' AND name = 'opentelemetry_span_log' (SigNoz's config also sorts by trace_id). Or switch the log off with <opentelemetry_span_log remove="1"/> (Langfuse's scaling docs). remove doesn't delete the existing table, so drop it yourself.
List only the logs your server has. A section in the config is what turns a log on (the stock config.xml disables session_log by commenting it out). SigNoz's own config.xml doesn't enable text_log, so delete that line there. Also add processors_profile_log, which has no TTL in that file.
Profile settings go into users.d, not config.d. If you also turn off the query profiler (query_profiler_real_time_period_ns and query_profiler_cpu_time_period_ns = 0, as in Langfuse's scaling docs), put that <profiles> block into users.d/. In config.d/ it is silently ignored (trigger.dev #4343).

Step 3: restart ClickHouse

The TTL from config.d takes effect only when the server restarts (Altinity KB). A config reload is not enough. With compose:

docker compose up -d clickhouse     # recreates the container with the new mount
docker compose ps clickhouse

If ClickHouse keeps restarting, the reason is in its error log inside the container. docker compose logs may not show it:

docker cp "$(docker compose ps -aq clickhouse)":/var/log/clickhouse-server/clickhouse-server.err.log .
tail -n 20 clickhouse-server.err.log

Step 4: drop the old trace_log_0 copies

After the restart, every changed log has a copy like trace_log_0 (then _1, _2 after later changes). The copy keeps all the old rows and has no TTL (ClickStack PR #275 upgrade note, Sentry snuba #7311).

This query writes the DROP statements for you. Read them, then paste them into the client:

SELECT 'DROP TABLE system.' || name || ' SETTINGS max_table_size_to_drop = 0;'
FROM system.tables
WHERE database = 'system' AND match(name, '_log_[0-9]+$');

ClickHouse no longer writes to these tables. DROP can't be undone.

Step 5: check the result

Run the step 1 query again. Every big log should show a TTL, and no _N names should be left.

ClickHouse removes expired rows during merges, by default at most every 4 hours (merge_with_ttl_timeout), so the size shrinks over the next hours.

A script can do the checking for you: diskvet (my project) runs the step 1 check read-only and prints the TRUNCATE commands, the TTL file and the DROP list for your tables, with the flag for tables over the drop limit (see section 7).

5. Disk still full? Where else the space goes

Compare df -h with the total size of all parts (SELECT formatReadableSize(sum(bytes_on_disk)) FROM system.parts). If the disk holds much more than that, the space is outside ClickHouse's tables.

Are Docker container logs filling the disk?

Docker's default json-file log driver has no size limit (max-size defaults to -1, see the Docker docs). In Langfuse #16339 the ClickHouse container's log reached 89.1 GiB and filled a 244 GiB root disk.

Find the biggest ones:

sudo sh -c 'du -h /var/lib/docker/containers/*/*-json.log | sort -h | tail -5'
docker ps -a --no-trunc --format '{{.ID}} {{.Names}}'   # the folder name is the container ID

To fix it, add a logging: block to every service in docker-compose.yml. In #16339 the Langfuse maintainers recommended doing it for all containers, not only ClickHouse. SigNoz's compose file already sets 50m × 3.

    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"

Then run docker compose up -d --force-recreate: log options apply only to newly created containers. Recreating also removes the old log file; in #16339 that freed 81 GiB.

Two notes:

If your Docker daemon uses another logging driver, set retention there instead, because driver: json-file overrides it.
Langfuse's README (PR #16363) describes the daemon-wide way: set max-size and max-file as daemon defaults, then restart Docker and recreate the containers. It also advises against truncating or rotating these files with external tools.

Are ClickHouse's own log files big?

ClickHouse also writes plain log files to /var/log/clickhouse-server (a volume in Langfuse's compose). By default they are at trace level, rotated at 1000M, with up to 10 old files kept (logger docs).

ClickStack PR #275 uses the values below. Save them as another config.d file, mount it like the TTL file and restart ClickHouse (section 4, step 3):

<clickhouse>
    <logger><level>information</level><size>100M</size><count>10</count></logger>
</clickhouse>

Why do deleted rows still take space?

Langfuse's data retention removes old rows with a lightweight DELETE. ClickHouse only marks such rows; the space comes back when the part is merged.

Background merges skip parts over max_bytes_to_merge_at_max_space_in_pool (150 GiB), and old monthly partitions are rarely merged. In Langfuse discussion #13969 a May partition of ~802 GiB was ~85% deleted rows.

Find the partitions with deleted rows:

SELECT database, table, partition_id, count() AS parts,
       formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts
WHERE active AND has_lightweight_delete
GROUP BY database, table, partition_id
ORDER BY sum(bytes_on_disk) DESC;

Then apply the delete mask, one partition at a time (APPLY DELETED MASK):

ALTER TABLE default.observations APPLY DELETED MASK IN PARTITION ID '202605';

Careful: this is a heavyweight mutation. It rewrites the affected parts, so it needs free space for the new copies and loads the disk. Run it off-peak and watch it with SELECT * FROM system.mutations WHERE NOT is_done. In #13969 it freed ~360 GiB per replica in ~17 minutes.

Since v3.179.0 the Langfuse worker can do this on a schedule: LANGFUSE_CLICKHOUSE_DELETED_MASK_CLEANER_ENABLED=true (off by default; see .env.prod.example, PR #14035).

Are old parts stuck on disk? (inactive and detached parts)

SELECT database, table, count() AS parts,
       formatReadableSize(sum(bytes_on_disk)) AS size,
       min(remove_time) AS oldest, max(refcount) AS max_refcount
FROM system.parts
WHERE NOT active
GROUP BY database, table
ORDER BY sum(bytes_on_disk) DESC;

Inactive parts are left over from merges. ClickHouse deletes them after old_parts_lifetime (480 s), once no query uses them (refcount = 1).

If they pile up (112.55 GiB of them in Langfuse discussion #15024) and oldest is hours ago, look for long queries in system.processes and stop them with KILL QUERY WHERE query_id = '...'.

Detached parts stay on disk until you remove them:

SELECT database, table, reason, count() AS parts,
       formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.detached_parts
GROUP BY database, table, reason;

Check reason first. Remove a part with ALTER TABLE <db>.<table> DROP DETACHED PART '<name>' SETTINGS allow_drop_detached = 1, not by deleting folders.

6. Upgrading ClickHouse? Check this first

Langfuse + ClickHouse 26.8 or newer: update Langfuse first. ClickHouse 26.8 changed the default of input_format_read_datetime_number_as_raw_value from 1 to 0. Older Langfuse sends DateTime64 values as millisecond numbers, and on 26.8+ they end up stored as 9999-12-31 23:59:59 (Langfuse #16858: 0.59% to 21.43% of writes per project). The fix is PR #16892 (backported to v3 in PR #16957), released in Langfuse v4.28.0 and v3.225.7. Langfuse partitions by month, so such rows land in partition 999912. Check your version and look for them:

  SELECT version();
  SELECT database, table, partition_id, count() AS parts
  FROM system.parts
  WHERE active AND startsWith(partition_id, '9999')
  GROUP BY database, table, partition_id;

After any upgrade, run the step 1 query from section 4 again. When a release changes a log's schema, ClickHouse renames the current table and creates a new one, so new _N copies appear. Drop them as in step 4.

7. Check everything at once

diskvet (my project) is an open-source (Apache-2.0) read-only script that runs most of the checks above and prints a report with the fix commands. It can't see Docker's log files, but it shows how much of the disk is outside ClickHouse's parts.

On the machine with Langfuse's docker-compose.yml (read checks.sql before you run it):

curl -fsSLO https://github.com/Protemir/diskvet/releases/latest/download/diskvet.sh
curl -fsSLO https://github.com/Protemir/diskvet/releases/latest/download/checks.sql
sh diskvet.sh report --docker auto > report.md

For SigNoz, use --docker signoz-clickhouse.

It reads only metadata from system.* tables (system.tables, system.parts, system.disks, system.detached_parts, system.mutations, system.part_log and a few more), with readonly=2 and resource limits. It never reads rows of your tables, system.query_log or query texts, and sends nothing anywhere. Nothing runs by itself: you read each fix and run it.

Sources

ClickHouse documentation and source:

System tables overview (unlimited growth, <engine> vs <ttl>, renaming on schema change): https://clickhouse.com/docs/reference/system-tables/overview
Default config.xml (logs without TTL, opentelemetry_span_log engine, text_log and logger at trace, session_log commented out): https://github.com/ClickHouse/ClickHouse/blob/master/programs/server/config.xml
max_table_size_to_drop, server setting and force_drop_table: https://clickhouse.com/docs/reference/settings/server-settings/settings/max-table#max_table_size_to_drop
max_table_size_to_drop, query setting: https://clickhouse.com/docs/reference/settings/session-settings/max#max_table_size_to_drop
Query-level override since 23.12: https://github.com/ClickHouse/ClickHouse/pull/57452
The flag is removed after use (checkCanBeDropped): https://github.com/ClickHouse/ClickHouse/blob/master/src/Interpreters/Context.cpp
TRUNCATE: https://clickhouse.com/docs/reference/statements/truncate
opentelemetry_span_log TTL error: https://github.com/ClickHouse/ClickHouse/issues/88366
MergeTree settings: merge_with_ttl_timeout (14400 s) https://clickhouse.com/docs/reference/settings/merge-tree-settings/merge-with#merge_with_ttl_timeout, old_parts_lifetime (480 s) https://clickhouse.com/docs/reference/settings/merge-tree-settings/other#old_parts_lifetime, max_bytes_to_merge_at_max_space_in_pool (150 GiB) https://clickhouse.com/docs/reference/settings/merge-tree-settings/max-bytes#max_bytes_to_merge_at_max_space_in_pool
Server log rotation (logger: level, size, count): https://clickhouse.com/docs/reference/settings/server-settings/settings/other#logger
Lightweight DELETE: https://clickhouse.com/docs/reference/statements/delete
APPLY DELETED MASK: https://clickhouse.com/docs/reference/statements/alter/apply-deleted-mask
system.parts: https://clickhouse.com/docs/reference/system-tables/parts
system.detached_parts: https://clickhouse.com/docs/reference/system-tables/detached_parts
DROP DETACHED PART: https://clickhouse.com/docs/reference/statements/alter/partition
allow_drop_detached: https://clickhouse.com/docs/reference/settings/session-settings/allow#allow_drop_detached
system.mutations: https://clickhouse.com/docs/reference/system-tables/mutations
KILL QUERY: https://clickhouse.com/docs/reference/statements/kill
Altinity KB, "System tables ate my disk" (restart needed): https://kb.altinity.com/altinity-kb-setup-and-maintenance/altinity-kb-system-tables-eat-my-disk/

Docker:

json-file logging driver (max-size defaults to -1; existing containers keep old settings): https://docs.docker.com/engine/logging/drivers/json-file/

Langfuse:

FAQ, reduce ClickHouse disk size: https://langfuse.com/faq/all/reduce-clickhouse-disk-size
Scaling docs, ClickHouse system log tables: https://langfuse.com/self-hosting/configuration/scaling
#13123, system tables grow unbounded: https://github.com/langfuse/langfuse/issues/13123
#16339, unbounded Docker log: https://github.com/langfuse/langfuse/issues/16339
PR #16363, Docker log rotation in the README (daemon defaults, no external truncation): https://github.com/langfuse/langfuse/pull/16363
Discussion #13969, lightweight-deleted rows: https://github.com/orgs/langfuse/discussions/13969
Discussion #15024, inactive parts and 59 GiB text_log: https://github.com/orgs/langfuse/discussions/15024
PR #14035, deleted-mask cleaner (released in v3.179.0): https://github.com/langfuse/langfuse/pull/14035, https://github.com/langfuse/langfuse/releases/tag/v3.179.0
#16858, DateTime64 on ClickHouse 26.8+: https://github.com/langfuse/langfuse/issues/16858
PR #16892 and v3 backport PR #16957: https://github.com/langfuse/langfuse/pull/16892, https://github.com/langfuse/langfuse/pull/16957
Releases with the fix: https://github.com/langfuse/langfuse/releases/tag/v4.28.0, https://github.com/langfuse/langfuse/releases/tag/v3.225.7

SigNoz, ClickStack and others:

SigNoz #12050, 80+ GB of system logs: https://github.com/SigNoz/signoz/issues/12050
SigNoz v0.129.0 ClickHouse config.xml and users.xml: https://github.com/SigNoz/signoz/blob/v0.129.0/deploy/common/clickhouse/config.xml, https://github.com/SigNoz/signoz/blob/v0.129.0/deploy/common/clickhouse/users.xml
SigNoz v0.129.0 docker-compose.yaml (container signoz-clickhouse, max-size: 50m, max-file: "3"): https://github.com/SigNoz/signoz/blob/v0.129.0/deploy/docker/docker-compose.yaml
ClickStack Helm chart PR #275: https://github.com/ClickHouse/ClickStack-helm-charts/pull/275
Sentry snuba #7311, *_log_N copies without TTL: https://github.com/getsentry/snuba/issues/7311
trigger.dev #4343, profile settings in config.d ignored: https://github.com/triggerdotdev/trigger.dev/issues/4343

Written and tested by an AI agent: every command was run on the ClickHouse versions listed at the top. Also on diskvet.dev.

ClickHouse is a registered trademark of ClickHouse, Inc. diskvet is not affiliated with ClickHouse, Inc. Langfuse, SigNoz and ClickStack are trademarks of their respective owners.