DEV Community: Vladimir Chemeris

Two Modbus Bugs That Cancel Out: How 230.4 V Reads as 230.92 V

Vladimir Chemeris — Fri, 30 Oct 2026 16:25:33 +0000

I build Modbus TCP FieldKit, an iPhone and iPad app for Modbus TCP. You won't need it here: the scripts below run on a laptop with Python and pymodbus.

Picture a meter manual with the line Voltage L1: 40021, FLOAT32 and no mention of word order. You drop the leading 4, pass 21 as the address, decode with your library's default and print:

volts: 230.92
hertz: 49.75

The meter's display reads 230.4 V and 49.98 Hz. Half a volt looks like calibration drift, so you move on to the next register.

That output carries two bugs. You would catch either one alone in seconds, because it gives you a number like -3.2e+26. Together they give you a reading close enough to pass a glance at the display.

Reproducing it

I wrote a small simulated meter with a layout you will meet on real devices: low word first. Install pymodbus and run this in one terminal:

# meter.py
import asyncio, struct
from pymodbus.server import ModbusTcpServer
from pymodbus.simulator import DataType, SimData, SimDevice

def cdab(value):
    high, low = struct.unpack(">HH", struct.pack(">f", value))
    return [low, high]

registers = [0] * 20 + cdab(230.4) + cdab(49.98) + [0] * 10
device = SimDevice(1, simdata=[SimData(0, values=registers, datatype=DataType.REGISTERS)])

async def main():
    server = ModbusTcpServer(context=[device], address=("127.0.0.1", 15020))
    await server.serve_forever()

asyncio.run(main())

Then read it the way the manual suggests:

# read_meter.py
from pymodbus.client import ModbusTcpClient

client = ModbusTcpClient("127.0.0.1", port=15020)
client.connect()

def read_float(address):
    regs = client.read_holding_registers(address, count=2, device_id=1).registers
    return client.convert_from_registers(regs, client.DATATYPE.FLOAT32)

print("volts:", round(read_float(21), 2))
print("hertz:", round(read_float(23), 2))

I ran both on 30 October 2026 with pymodbus 3.16.1 and Python 3.15 on macOS.

The two bugs

Off by one. 40021 names the 21st holding register, and the frame carries it as address 20. Section 4.4 of the Modbus specification numbers data model elements from 1 and PDU addresses from 0. I wrote up the prefixes and 6-digit references in this guide to register addressing.

Word order. The specification defines 16-bit registers and leaves multi-register values to the vendor. My meter stores the low word first, and convert_from_registers in pymodbus defaults to word_order="big", high word first. The four word orders go by the names ABCD, CDAB, BADC and DCBA.

Fixing both gives the right value:

# fixed.py
from pymodbus.client import ModbusTcpClient

client = ModbusTcpClient("127.0.0.1", port=15020)
client.connect()
regs = client.read_holding_registers(20, count=2, device_id=1).registers
volts = client.convert_from_registers(regs, client.DATATYPE.FLOAT32, word_order="little")
print("volts:", round(volts, 2))

volts: 230.4

The arithmetic behind the cancellation

address     20    21    22    23
register  6666  4366  EB85  4247
          |  volts  |  |   Hz   |

IEEE 754 encodes 230.4 as 0x43666666. The meter writes the low word 6666 first, then the high word 4366. A read from 21 grabs 4366 EB85. Decoding high word first puts the voltage's real high word in the right slot and fills the low half from the frequency.

The high word of a float32 holds the sign, the exponent and the top seven mantissa bits. At this magnitude 0x4366 pins the value between 230.0 and 231.0, and the low word picks a spot inside that volt. 0xEB85 lands at .92.

You can see how loud each bug is alone by decoding each window under all four orders:

# grid.py
import struct
from pymodbus.client import ModbusTcpClient

ORDERS = {"ABCD": (0, 1, 2, 3), "CDAB": (2, 3, 0, 1),
          "BADC": (1, 0, 3, 2), "DCBA": (3, 2, 1, 0)}

def decode(pair, order):
    raw = struct.pack(">HH", *pair)
    return struct.unpack(">f", bytes(raw[i] for i in ORDERS[order]))[0]

client = ModbusTcpClient("127.0.0.1", port=15020)
client.connect()
start = 19
regs = client.read_holding_registers(start, count=6, device_id=1).registers

print("addr  " + "".join(f"{o:>12}" for o in ORDERS))
for i in range(len(regs) - 1):
    row = [f"{decode(regs[i:i + 2], o):12.5g}" for o in ORDERS]
    print(f"{start + i:<6}" + "".join(row))

addr          ABCD        CDAB        BADC        DCBA
19      3.6734e-41  2.7154e+23  3.6734e-41  2.7154e+23
20      2.7185e+23       230.4  2.7201e+23  2.3069e+23
21          230.92 -3.2221e+26  2.3083e+23 -2.2137e-35
22      -3.222e+26       49.98 -2.2125e-35       49798
23           49.75  2.3776e-41       49664  2.5562e-41

Bug one alone sits at row 21 under CDAB: -3.2221e+26. Bug two alone sits at row 20 under ABCD: 2.7185e+23. Both together give 230.92 at row 21 under ABCD.

Rows 22 and 23 repeat the pattern for frequency with 49.98 and 49.75. You get two diagonals of sane-looking readings, so a range check can't choose between them.

The tell: decimals that follow the wrong quantity

I rewrote the simulated readings from a second client and read them back with the buggy code:

  true V  true Hz   read V  read Hz
  230.40    49.98   230.92    49.75
  231.70    49.98   231.92    49.75
  229.10    49.98   229.92    49.75
  230.40    50.01   230.04    50.00
  230.40    49.90   230.60    49.75

The integer part of the voltage tracks the real voltage. The fraction stays at .92 while the voltage moves, then jumps to .04 and .60 after I change the frequency. The frequency reading snaps to steps of 0.25, because a zero register follows it in my map. On a real meter, the contents of the next register set that fraction.

Catching it on a device

Read a window around the documented address and print the grid. Expect two plausible diagonals.
Choose between them with a value whose decimals you can check: a display with one decimal place, or a setpoint you control. 230.92 and 230.4 differ in the first decimal.
Change the quantity and read again. The fraction has to move with it.
Store the address and word order next to the device model, and add a test with an exact expected value.

For a one-off check, paste two registers into this float converter, which decodes all four orders in the browser. The address converter does the 40021 arithmetic.

FieldKit stores format, word order, scale and unit with each register in a saved device profile, so the next meter of that model starts from the right settings. If you do this next to a panel with a phone, the client page shows what it reads and writes.

Have you hit a plausible but wrong reading on real hardware? Tell me which device in the comments.

Inside a 9.8 GB Xcode Simulator: Apps Took 350 MB

Vladimir Chemeris — Tue, 27 Oct 2026 14:27:40 +0000

I build StorageRadar, a disk space app for macOS. Everything below uses built-in commands, so you can repeat it without the app.

Last week I measured where 51 GB of simulator storage went on my Mac. One QA simulator held half of the Devices folder, and a reader asked what was inside it. Short answer: not my app.

One simulator, 9.8 GB

The QA simulator is an iPhone 18 Pro on iOS 27.0. I measured it on 26 October 2026 while it was booted:

du -k -d 4 ~/Library/Developer/CoreSimulator/Devices/<UDID>/data | sort -rn | head -20

Folder inside `data`	Size
`Library/Application Support/PRBPosterExtensionDataStore`	2.88 GB
`var/db/diagnostics`	2.07 GB
`private/var/MobileAsset`	2.03 GB
`Library/Caches/com.apple.coresymbolicationd`	1.13 GB
`var/db/uuidtext`	0.49 GB
`Containers`, all app containers	0.34 GB

Everything my app and its tests wrote fits in Containers: 0.34 GB of 9.81.

Ten simulators side by side

I have 53 simulators. Forty-three hold about 20 MB each, the size of a device that never ran an app. The other ten:

App containers stayed between 0.07 and 0.34 GB on every device.
MobileAsset showed up everywhere, from 0.17 to 2.03 GB.
The poster store was big only on the two booted iOS 27.0 devices: 2.88 and 1.65 GB. On iOS 18.6 and 26.5 it stayed under 0.05 GB. Two devices prove nothing about iOS 27.0, but the gap is wide.

What the folders are

PRBPosterExtensionDataStore. The name, along with GalleryCache and SnapshotCache.cachedb, is a string in the private PosterBoard.framework of the iOS 27.0 runtime. Its subfolders carry the IDs of Apple's poster extensions: WeatherPoster, EmojiPoster, PhotosPosterProvider and others. Weather and photos are among the Lock Screen wallpapers Apple lists. On my QA device, 2,911 .atx snapshot files of about 1 MB each made up 2.9 GB. Apple doesn't document the folder, and I don't know what makes it grow.

var/db/diagnostics and var/db/uuidtext. The simulator's own copy of the unified log. man log describes a log store as a database like the one in /var/db/diagnostics, with uuidtext next to it.

private/var/MobileAsset. Asset sets the simulator downloads itself. On the QA device the biggest were named after Siri Understanding (805 MB), linguistic data (409 MB) and Siri text-to-speech (289 MB). Someone on the Apple forums reported this folder growing by tens of gigabytes on iOS 26.5. No answer there yet.

com.apple.coresymbolicationd. man coresymbolicationd calls it a "daemon to manage debug symbols cache". 1.13 GB on the QA device, close to zero elsewhere.

What I do about it

I don't delete files inside these folders. None of them is documented, so I can't predict what a half-deleted log store or poster store does to a simulator. I work on whole devices:

xcrun simctl erase <UDID> wipes contents and settings and keeps the device.
xcrun simctl delete <UDID> removes the device.
xcrun simctl delete unavailable removes devices the current Xcode SDK no longer supports.

An erase drops your installed apps and logins along with the system data, and the caches can come back once the device runs again. It buys space; it doesn't stop the growth.

The full version with every number, the measurement method and an FAQ is on my blog.

StorageRadar's Disk Map drills into folders like these, and its Dev Cleanup lists CoreSimulator under Review before cleaning, next to DerivedData and package caches: storageradar.app.

Where 51 GB of Xcode simulator storage went on my Mac

Vladimir Chemeris — Sun, 25 Oct 2026 06:23:39 +0000

I build StorageRadar, a disk space app for macOS. Every step below uses built-in commands, so you can follow along without it.

My Mac runs macOS 27.0.1 and Xcode 27.1, and I test several iOS apps on it. I measured what the simulators take on 25 October 2026.

Two places, two owners

Simulator storage lives in two places, and different tools manage them.

What	Where	On my Mac
Runtimes, one per iOS version	system-managed disk images	31.3 GB for 4 runtimes
Device data, one folder per simulator	`~/Library/Developer/CoreSimulator/Devices`	20 GB for 53 simulators

A runtime is the iOS system image. Device data is what each simulated iPhone or iPad keeps: installed apps, their containers, logs and settings.

Runtimes: about 8 GB each

xcrun simctl runtime list prints every runtime and the total:

$ xcrun simctl runtime list
== Disk Images ==
-- iOS --
iOS 27.1 (24A94232) - … (Ready)
iOS 27.0 (24A434) - … (Ready)
iOS 26.5 (23F77) - … (Ready)
iOS 18.6 (22G86) - … (Ready)

Total Disk Images: 4 (31.0G)

Add -j and you get each runtime's size and the last time it ran. Mine range from 7.87 GB (iOS 27.1) to 8.84 GB (iOS 18.6).

Remove runtimes with simctl, not Finder. The images sit in protected system storage, and simctl shuts down booted simulators and unmounts the image first. Start with a dry run:

xcrun simctl runtime delete --notUsedSinceDays 30 --dry-run

You can also pass --unusable, --outdated, or one identifier from the list. Xcode's Settings window lists the same runtimes.

All four of mine ran in the last day, so I keep them. If you dropped an iOS version from your deployment targets, its runtime is the biggest single thing you can remove.

The du number that misleads

du -sh /Library/Developer/CoreSimulator/Volumes reported 51 GB on my Mac. That folder holds mount points:

$ mount | grep CoreSimulator/Volumes
/dev/disk5s1 on /Library/Developer/CoreSimulator/Volumes/iOS_23F77 …

du walks into the mounted images and counts the files inside them. The images take 31 GB on disk, and 51 GB is the size of their contents. Leave Volumes alone and use simctl runtime delete.

Device data: one simulator held half

The Devices folder took 20 GB, and the split surprised me:

du -sh ~/Library/Developer/CoreSimulator/Devices/* | sort -h | tail -5

one QA simulator: 9.6 GB
the next four: 3.3, 2.5, 2.1 and 1.5 GB
47 of 53 simulators: under 200 MB each

The folder names are UDIDs. xcrun simctl list devices maps them to names.

Two commands handle the heavy ones:

xcrun simctl erase <UDID> wipes the simulator's apps and data and keeps the device.
xcrun simctl delete <UDID> removes the device.

xcrun simctl delete unavailable removes simulators your current Xcode can no longer run. I had none, but the list grows after Xcode upgrades and runtime deletions. Erasing a simulator leaves its runtime in place.

The rest of ~/Library/Developer

Two more folders sat next to the simulators:

DerivedData, 6.2 GB. Xcode rebuilds it, so you can delete it and pay with one slower build.
Archives, 3.9 GB. Keep the archives of builds you shipped: they hold the dSYMs that symbolicate crash reports.

My checklist now

xcrun simctl runtime list for runtimes I no longer test on.
du on Devices to find the one or two heavy simulators, then erase them.
xcrun simctl delete unavailable after each Xcode update.

I keep a longer guide with every path and what rebuilds it: Xcode Simulator taking up space on Mac. To test yourself on ten developer folders, from DerivedData to Docker data, try this quiz.

StorageRadar's Dev Cleanup lists ~/Library/Developer/CoreSimulator with a Caution label and DerivedData as Safe, next to npm, Docker and Homebrew caches, and moves files to the Trash after you confirm: storageradar.app.

Why Your MQTT TLS Connection Fails, Stage by Stage

Vladimir Chemeris — Tue, 08 Sep 2026 16:34:03 +0000

I build MQTT Commander, an MQTT client for iPhone and iPad, so I read a lot of other people's failed TLS handshakes. The same half-dozen causes come up every time, and you can pin most of them down before you touch the broker config.

Plain MQTT on port 1883 either connects or it doesn't. Add TLS and five more things sit between you and a working session: the certificate chain, hostname verification, ALPN, client certificates and broker-side authorization. The MQTT client that reports the failure sees a closed socket and little else.

Three layers, in order

Before the first MQTT packet moves, three things have to succeed one after another.

TCP. The client resolves the broker hostname, opens a socket on 8883 (MQTT over TLS) or 443 / 8084 (MQTT over WebSocket and TLS), and completes the handshake. A firewall, a wrong port or a broker that isn't running fails here, and the fix is on the network side even when the error text mentions TLS. On iOS a LAN broker that never answers has one more possible cause: the app has no Local Network permission in Settings.

TLS handshake. Client and server agree on a cipher suite, exchange certificates and set up the encrypted channel, and most secure-MQTT failures land somewhere in that exchange.

MQTT CONNECT. Only now does the protocol itself start. The client sends CONNECT, the broker answers with CONNACK. A non-zero reason code at this point means the broker refused you on purpose: bad credentials, a client ID it won't accept, an ACL that denies you. TLS finished before any of that.

Find out which of the three failed before you change anything.

Where TLS breaks

Most MQTT clients collapse a TLS failure into "connection failed". To see the real verdict, ask OpenSSL:

openssl s_client -connect broker.example.com:8883 \
  -servername broker.example.com -verify_hostname broker.example.com

The verify error lines below are what it prints for each cause. Other TLS stacks word them differently, but the categories are the same.

An expired or incomplete chain

Every certificate in the chain expires, intermediates included. An expired intermediate breaks the connection while the leaf certificate is still valid, and the leaf is the one you check by hand.

verify error:num=10:certificate has expired
verify error:num=20:unable to get local issuer certificate
verify error:num=21:unable to verify the first certificate

Codes 20 and 21 mean the chain stops short of a root you trust. Either the broker sends only its leaf certificate and skips the intermediate, or the root is a private CA you haven't imported yet (next section). The first case you fix on the broker: every other client hits the same wall. Code 9, certificate is not yet valid, usually means the device clock is wrong rather than the certificate.

A private CA the device doesn't trust

Mosquitto on a Raspberry Pi, an internal staging broker, an AWS IoT custom endpoint: a private CA signs all of these, and no phone trusts that CA out of the box.

verify error:num=19:self-signed certificate in certificate chain
verify error:num=18:self-signed certificate

Code 19 is a private root; code 18 is a broker that serves a self-signed leaf with no CA at all. Import the root as PEM (for code 18, the broker's certificate is its own root) and compare its SHA-256 fingerprint with the one your CA published before you trust it. Whoever holds the key of an unverified root can impersonate your broker.

The hostname isn't in the certificate

TLS clients check that the hostname you connected to appears in the certificate's SAN (Subject Alternative Name); a name that only appears in CN no longer counts on iOS. The match is exact, so broker.example.com and mqtt.example.com are different names to TLS even when they point at the same box.

verify error:num=62:hostname mismatch

Connect using a name that is in the SAN. If you have to reach the broker by an address that isn't in the certificate, send the right server name during the handshake with an SNI override instead of editing your hosts file.

Missing or wrong ALPN

Some brokers multiplex MQTT onto a shared port and use ALPN to route it. AWS IoT Core takes MQTT with certificate authentication on port 443 only when the client advertises x-amzn-mqtt-ca. Leave it out and the handshake dies with a generic alert, or the socket closes right after it, and nothing in the message says ALPN. One common form of it:

error:0A000438:SSL routines::tlsv1 alert internal error

Standard MQTT over TLS on 8883 needs no ALPN entry in most setups. On 443, ALPN is the first thing to check. The AWS IoT guide on my site lists the exact values for that endpoint.

A TLS version the client won't negotiate

iOS negotiates TLS 1.2 or 1.3 and refuses 1.0 and 1.1. A broker built on an old library that tops out at 1.1, which still happens on legacy gateways and old embedded images, produces:

error:0A000102:SSL routines:ssl_choose_client_version:unsupported protocol

The client is refusing what the broker offered, so the fix is on the broker.

Mutual TLS

Standard TLS authenticates the broker to you. mTLS authenticates you back, which is why AWS IoT device provisioning and industrial fleets use it: each device carries its own certificate instead of a shared password.

You need three things: a client certificate, its private key, and the CA that signed both ends. They arrive as a passphrase-protected .p12 / .pfx bundle, or as separate PEM files.

Three failures cover most mTLS trouble.

The passphrase is wrong. Import fails before anything reaches the network, and OpenSSL calls it mac verify failure. Passphrases are case-sensitive. If you have lost it, re-export the bundle from the CA rather than guessing.

The certificate and key don't match. Import the client identity as a single .p12 bundle and the pair stays together by construction. With separate PEM files, a key from the wrong export surfaces as:

key values mismatch

The handshake succeeds and the broker hangs up anyway. The broker accepted your certificate and then refused the session on authorization grounds. In AWS IoT Core, look for a policy that doesn't grant iot:Connect for the client ID you present. With MQTT 5 the CONNACK reason code says so: 0x87, not authorized. With MQTT 3.1.1 the broker often closes the socket without a CONNACK at all.

Reading CONNACK

If TLS completed and the session still dies, the reason code names the cause:

MQTT 5	MQTT 3.1.1	Meaning
`0x87`	`5`	Not authorized: an ACL or IoT policy denies `Connect`
`0x86`	`4`	Bad username or password
`0x85`	`2`	Client identifier not valid

These three cover most post-handshake refusals.

The checklist

Port matches the transport: 8883 for MQTT over TLS, 8084 for MQTT over WSS, 443 where ALPN does the routing
Chain is complete and nothing in it has expired, intermediates included
Private CA root is imported, and you verified its fingerprint before trusting it
Connection hostname appears in the certificate's SAN
ALPN is set if the broker needs it, such as x-amzn-mqtt-ca for AWS IoT on 443
For mTLS: certificate, key and CA all present, passphrase correct
Client key matches the certificate's public key
Broker policy grants Connect for this client ID

Work down it in order and stop at the first item that fails. You can't test anything below it until you fix that one. The site version of this checklist sits next to a table of what each diagnostic stage looks like when it fails.

On iOS I do this with MQTT Commander. Its Connection Doctor runs the connection stage by stage, from DNS and TCP through the TLS handshake, chain, expiry, hostname and ALPN/SNI to CONNACK, authentication and a write test against the broker ACL, stops at the first stage that fails and says what to change. The free download shows the transport and MQTT stages plus whichever stage fails, TLS included; the one-time Pro unlock lists each TLS check on its own line and adds a shareable report that carries stage names and statuses only, with no hosts, topics or credentials. The Certificate Manager is free on every tier: it imports a CA as PEM and a client identity as .p12 or .pfx, shows the SHA-256 fingerprint, and flags an expired certificate before you connect.

ClickHouse "Too many parts": what to check before you run OPTIMIZE FINAL

Vladimir Chemeris — Tue, 11 Aug 2026 13:59:02 +0000

Disclosure: I build ProbeDeck, an iOS app for monitoring and operating ClickHouse clusters. The SQL
below works in any ClickHouse client. You do not need ProbeDeck to use it.

An insert fails with this message:

Too many parts (N). Merges are processing significantly slower than inserts

The tempting response is OPTIMIZE TABLE ... FINAL. Hold that command. ClickHouse raises this error
when a partition accumulates active parts faster than the server can merge them. A forced merge can
consume the same CPU, disk bandwidth, and free space that the background merge process needs.

Use this order during an incident:

Count active parts by partition and by table.
Read the live delay, throw, and total-part thresholds from the cluster.
Sample active merges twice to see whether the backlog shrinks.
Check disk headroom and, for replicated tables, replica health.
Reduce part creation at the writer or partitioning layer.
Consider a forced merge after you control incoming pressure.

Find the affected partition

parts_to_throw_insert applies to active parts in one partition. A table-wide count can hide the
shape of the problem. Run this first:

The system tables below report data from the node that serves your query. A load balancer can route
you to a healthy replica while another replica holds the backlog. Connect to each node, or list the
configured cluster names with SELECT DISTINCT cluster FROM system.clusters and replace a local
table such as system.parts with clusterAllReplicas('my_cluster', system.parts). Keep
hostName() in the result so you can see which node produced each row.

SELECT
    hostName() AS host,
    database,
    table,
    partition,
    count() AS active_parts,
    countIf(level = 0) AS level0_parts,
    max(level) AS highest_level,
    sum(rows) AS total_rows,
    formatReadableSize(sum(bytes_on_disk)) AS size_on_disk
FROM system.parts
WHERE active
  AND database NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema')
GROUP BY host, database, table, partition
ORDER BY active_parts DESC
LIMIT 20;

If one partition sits far above the rest, inspect the hot partition and its partitioning key. If you
see level0_parts grow between samples, inspect the writer for frequent small inserts. If the count
is high but spread across healthy partitions, compare the table total with max_parts_in_total
below.

Do not copy a threshold from a blog post. Read the settings on the cluster you are debugging:

SELECT
    hostName() AS host,
    name,
    value,
    changed
FROM system.merge_tree_settings
WHERE name IN (
    'parts_to_delay_insert',
    'parts_to_throw_insert',
    'max_parts_in_total',
    'max_avg_part_size_for_too_many_parts'
)
ORDER BY name;

system.merge_tree_settings gives you the server-level value. A table can override it in its own
SETTINGS clause. Check the table definition before you treat the result as the effective limit:

SHOW CREATE TABLE analytics.events;

The writer can also pass settings with the insert query. Run the checks through the same user and
connection path as the writer, then inspect its client configuration. Treat an app or dashboard
threshold as a warning heuristic unless it reads the effective setting.

ClickHouse's current guide to OPTIMIZE FINAL uses 150 active parts as an early-warning heuristic.
Your cluster limit comes from its effective settings. Investigate when two samples show growth or
level-0 parts dominate. A count near parts_to_delay_insert means ClickHouse may throttle inserts
soon; parts_to_throw_insert rejects them. max_avg_part_size_for_too_many_parts can disable the
delay and throw checks when the average part size in the affected partition exceeds its value. The
total table limit still applies.

max_parts_in_total protects the table across all partitions. Check the matching table total:

SELECT
    hostName() AS host,
    database,
    table,
    count() AS active_parts_total
FROM system.parts
WHERE active
  AND database NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema')
GROUP BY host, database, table
ORDER BY active_parts_total DESC
LIMIT 20;

Check whether merges are making progress

Next, inspect the active merge workload:

SELECT
    hostName() AS host,
    database,
    table,
    partition,
    elapsed,
    round(progress * 100, 1) AS progress_pct,
    num_parts,
    is_mutation,
    merge_type,
    formatReadableSize(total_size_bytes_compressed) AS source_size,
    formatReadableSize(memory_usage) AS memory
FROM system.merges
ORDER BY elapsed DESC;

system.merges shows work in progress, not a queue. An empty result does not prove that merges have
stopped. Read it together with two system.parts samples:

Parts are high and merge progress changes between samples: the cluster is working through a backlog. Reduce incoming pressure and keep measuring.
Parts keep rising while no merge appears for the affected table: inspect disk space, merge-pool pressure, disabled merges, replication state, and recent configuration changes.
Merges run, but new level-0 parts arrive faster: fix the writer. More merge threads will not cure a producer that creates parts without bound.

If your server writes system.part_log, estimate how completed merges changed the part count over
the same window:

SELECT
    hostName() AS host,
    countIf(event_type = 'NewPart') AS new_parts,
    countIf(event_type = 'MergeParts') AS merge_outputs,
    sumIf(length(merged_from), event_type = 'MergeParts') AS merge_inputs,
    new_parts + merge_outputs - merge_inputs AS estimated_net_change
FROM system.part_log
WHERE event_time >= now() - INTERVAL 10 MINUTE
  AND database = 'analytics'
  AND table = 'events'
  AND event_type IN ('NewPart', 'MergeParts')
GROUP BY host;

One MergeParts event can consume several source parts, so raw event counts use different units.
The estimate above ignores downloads, removals, and events that cross the time-window boundary.
Use repeated system.parts snapshots as the primary signal. Replace the example identifiers and
save two or three samples a few minutes apart.

Stalled merges also need disk and replica checks. On self-managed clusters, inspect every server
that stores the affected table:

SELECT
    hostName() AS host,
    name,
    path,
    formatReadableSize(free_space) AS free,
    formatReadableSize(unreserved_space) AS unreserved,
    formatReadableSize(total_space) AS total,
    formatReadableSize(keep_free_space) AS reserved
FROM system.disks
ORDER BY name;

No fixed free-space percentage fits every merge because candidate parts and storage policies differ.
Treat disk as the bottleneck when unreserved_space keeps falling while merges stall, or when the
server log reports space-reservation failures. Check disk saturation in your host monitoring. For
ReplicatedMergeTree tables, inspect replica state as well:

SELECT
    hostName() AS host,
    database,
    table,
    is_readonly,
    is_session_expired,
    queue_size,
    inserts_in_queue,
    merges_in_queue,
    queue_oldest_time,
    absolute_delay
FROM system.replicas
WHERE is_readonly OR is_session_expired OR queue_size > 0
ORDER BY queue_size DESC;

A nonzero queue can be normal. Sample it twice. A read-only replica, an expired session, a growing
queue, or rising absolute_delay points to replication trouble and adds pressure to ClickHouse
Keeper.

Check the writer before tuning the server

Each synchronous insert can create at least one part for every partition touched by its block.
ClickHouse recommends batching at least 1,000 rows per insert, with 10,000 to 100,000 rows as the
better range for many workloads. The right batch size still depends on row width, latency, memory,
and the number of partitions touched. For synchronous ingestion, start near one insert per second
and adjust from measurements.

Check these failure modes:

A producer sends one row or a tiny batch per request.
One insert touches many partitions.
Several writers use incompatible async-insert settings, so ClickHouse cannot combine their data into one buffer.
Materialized views multiply one source insert into parts in several target tables.
Mutations, TTL work, or disk pressure compete with regular merges.

Check partition granularity before you tune merge pools. Run SHOW CREATE TABLE and count the
active partitions:

SELECT
    hostName() AS host,
    database,
    table,
    uniqExact(partition) AS active_partitions,
    count() AS active_parts
FROM system.parts
WHERE active
  AND database NOT IN ('system', 'INFORMATION_SCHEMA', 'information_schema')
GROUP BY host, database, table
ORDER BY active_partitions DESC
LIMIT 20;

A large partition count alone does not prove a bad key. Look for a key that opens a partition per
ID, per tenant, or per timestamp truncated to an hour or less. ClickHouse cannot merge parts across
partition boundaries. Fixing that design requires a replacement table with a coarser partition key
and a controlled data migration.

ClickHouse 26.3 LTS enables async inserts by default. On older versions, enable them on the ingest
path when client-side batching is not practical. On 26.3 and later, verify that the client did not
override the setting. Check it using the same user and connection settings as the writer:

SELECT hostName() AS host, name, value, changed
FROM system.settings
WHERE name IN ('async_insert', 'wait_for_async_insert', 'compatibility')
ORDER BY name;

An older pinned compatibility value can preserve the previous async-insert default after a server
upgrade. Trust the async_insert value returned through the writer's connection instead of the
server version alone.

Keep wait_for_async_insert = 1 when the client needs acknowledgement after the buffer reaches
durable storage. With 0, the server can acknowledge data while it still lives in memory, and the
client will not receive flush errors in the same way.

`OPTIMIZE FINAL` does not fix ingestion

OPTIMIZE TABLE ... FINAL rewrites the parts that exist now. It does not change the rate at which
the writer creates new ones. When ingestion resumes, the exception can return. The forced merge can
also create a large part and consume resources on a cluster that already lacks merge capacity.

Reserve a forced merge for cases where you understand the affected table and partition, have enough
disk and I/O headroom, and have reduced the source of new parts. Raising parts_to_throw_insert has
the same problem: it moves the safety boundary without repairing ingestion.

The durable fixes are upstream:

Batch more rows per insert.
Reduce insert frequency.
Use async inserts where they fit the durability and latency requirements.
Revisit a high-cardinality partition key through a planned migration.
Tune merge resources after measurements show a server-side bottleneck.

During an incident, I run the same checks from the ProbeDeck guide to ClickHouse Too many parts.
ProbeDeck shows raw maximum active-part counts and current merges from an iPhone or iPad. Compare
those counts with the effective settings on your cluster. Monitoring is free. Credentials stay in
the iOS Keychain, and the app has no backend between the device and your cluster.

ClickHouse is a registered trademark of ClickHouse, Inc. ProbeDeck is not affiliated with,
endorsed by, or sponsored by ClickHouse, Inc.

Why iperf3 Says “Bad File Descriptor” and What to Check First

Vladimir Chemeris — Fri, 07 Aug 2026 13:58:34 +0000

Disclosure: I develop an iPerf3 app for Apple platforms and Android. The examples below use the upstream iperf3 CLI and apply to compatible clients and servers.

Suppose iperf3 prints this line:

iperf3: error - unable to write to the control socket: Bad file descriptor

Keep the whole line. Bad file descriptor comes from your operating system. It does not identify the operation that failed. The text before it names the iperf3 stage.

In this case, iperf3 tried to write to its control socket. The operating system rejected the descriptor passed to that write with EBADF.

One line, two sources

iperf3 builds many error strings from two parts:

iperf3 operation: unable to write to the control socket
OS errno: Bad file descriptor

The iperf3 message answers: What was the program doing?

The errno text answers: Why did that system call fail?

The implementation is visible in src/iperf_error.c. For IECTRLWRITE, iperf3 starts with unable to write to the control socket. It then appends strerror(errno) when the failing call supplied an errno.

Some messages have no appended errno. control socket has closed unexpectedly is one example. iperf3 uses it after a control read returns no state byte. On the client, an empty read points to end-of-file from the peer. The server can reach the same message after its timed read returns without data.

What `EBADF` tells you

Bad file descriptor is the text for EBADF. The process passed an invalid descriptor to a system call.

That establishes one local fact: the descriptor was invalid inside the process that printed the error. It does not tell you why the descriptor reached that state. Possible triggers include cleanup after another failure, a descriptor-lifetime bug, or code running after a connection teardown.

A firewall or the remote peer can start a failure sequence by closing a connection. The final EBADF still describes only the local system call that failed at the end of that sequence.

Similar messages point to different stages

Read the iperf3 part before interpreting the suffix:

Message	Failed stage	First check
`unable to write to the control socket: <errno>`	A write to the control socket failed.	Check the previous state exchange and the appended errno.
`unable to receive cookie at server: <errno>`	The server accepted a data-stream socket but could not read its cookie.	Compare the client log and the server errno.
`select failed: <errno>`	The kernel rejected something in a `select()` descriptor set.	Use `--debug` to find the last completed stage.
`control socket has closed unexpectedly`	A control read returned no state byte. No errno is appended.	Compare both logs at the same timestamp.

This table explains why searching only for bad file descriptor produces weak answers. The same suffix can appear after several iperf3 operations.

A debugging sequence that keeps the evidence

Start with the process that printed the line. Then collect the matching evidence from the other side.

Save the complete client and server commands.
Save the full output from both processes.
Record iperf3 --version on both machines. iperf2 and iperf3 do not interoperate.
Classify the failed stage from the iperf3 message.
Interpret errno only after you know that stage.
Repeat the test with debug logs:

iperf3 -s --debug >server.log 2>&1
iperf3 -c 192.168.1.10 --debug >client.log 2>&1

Match the last completed state on both sides. If one log stops first, check whether that host slept, the process exited, or the network path changed.

Run a short and a long test when the failure appears late:

iperf3 -c 192.168.1.10 -t 30
iperf3 -c 192.168.1.10 -t 300

A short run that passes and a long run that fails gives you a time-dependent lead. Check idle firewall state, device sleep, Wi-Fi roaming, and process restarts. Treat each as a testable candidate.

If the logs disagree about which side stopped first, capture port 5201 and compare packet timestamps with both logs:

sudo tcpdump -ni any tcp port 5201

Use a real interface name on systems without the any pseudo-interface.

Other suffixes use the same method

The same reading order works for Connection refused, Permission denied, and Address already in use.

For example, unable to connect to server: Connection refused identifies a failed connect attempt. The suffix says the destination rejected that TCP connection. Check the server process, address, and port before changing unrelated socket settings.

The full iperf3 error reference covers busy servers, listener failures, temporary-file permissions, socket buffers, mobile sleep, and long-test failures.

If you want to run the same tests from an iPhone, iPad, or Mac, the iPerf3 app for Apple platforms can act as a client or server.

Measuring real LAN speed from a phone with iperf3

Vladimir Chemeris — Wed, 05 Aug 2026 16:49:27 +0000

Disclosure before we start: I build the native iPerf3 apps for iPhone, iPad, Mac, and Android mentioned near the end. Everything in this post also works with the free iperf3 CLI on two computers. The phone enters the picture when one endpoint has to move around the building.

Speedtest answers one question: how fast is the path from your device to the ISP's nearest test server. Inside a home or office a different question comes up: how fast is the link between the phone and the NAS, between two mesh nodes, between the office switch and the upstairs access point. Traffic between two local devices never leaves the LAN, so an internet speed test cannot see it.

iperf3 measures that link. ESnet maintains it as a from-scratch redesign of the original iperf, and common Linux, BSD, and macOS package repositories carry it.

How iperf3 measures

iperf3 is a client-server tool. One device listens (iperf3 -s), the other connects to it, pushes traffic for a fixed duration, and counts what arrived. Because you control both endpoints, the result describes one specific link instead of a whole path across the internet.

The server side installs in one line on Linux and macOS:

# Debian/Ubuntu
sudo apt install iperf3 && iperf3 -s

# macOS
brew install iperf3 && iperf3 -s

ESnet does not publish or support native Windows binaries. If you need a Windows endpoint, check the upstream installation notes before choosing a third-party build.

Some router and NAS platforms make iperf3 available through their package systems. Availability depends on the model and firmware, so check the package source before treating the appliance as a permanent server.

A five-minute test

Start the server on a wired device and note its LAN IP, for example 192.168.1.10.
From the client, start with the default ten-second test. Use sixty seconds when you need to see changes caused by interference, roaming, or sustained load:

iperf3 -c 192.168.1.10 -t 60

Read the receiver line at the end. That number is the throughput the link sustained.

A healthy wired 1GbE TCP link often sustains about 930 to 950 Mbit/s after protocol overhead. Wi-Fi varies with channel width, spatial streams, interference, access-point configuration, and the phone's radio. Compare the same phone and settings across locations instead of judging a result from the Wi-Fi generation label alone. If a wired link reports far below its expected rate, check the negotiated link speed, cable, switch port, USB adapter, and CPU load on both endpoints.

Now walk. Run the same test from the far bedroom, the garage, the conference room. The room-to-room gap is the thing a signal-strength bar hides.

Which output fields to read

Throughput is the headline number: application data delivered per second, usually shown in Mbit/s or Gbit/s.

Retransmits (TCP) count segments the sender had to transmit again. A clean wired LAN should produce few retransmits. A rising count on repeated runs gives you a reason to isolate the cable, port, queueing, or wireless hop; the counter alone does not identify the cause.

Jitter and packet loss appear in UDP mode. Set the target below the link's measured TCP capacity, then raise it in steps. For example:

iperf3 -c 192.168.1.10 -u -b 100M -t 60

Packet loss and jitter affect calls and games before raw throughput becomes the obvious problem. Your acceptable limits depend on the application, codec, and traffic pattern.

Traps that produce wrong numbers

Both endpoints on Wi-Fi. You measure the weaker of two radio links plus their contention for airtime. Put the server on Ethernet and test the wireless hop you care about.
USB Ethernet adapters. USB 2.0 adapters often top out around 300 to 350 Mbit/s in practice. Check the adapter and negotiated link speed before blaming the network.
Short tests. Connection setup, TCP ramp-up, and brief interference can dominate a short result. Use longer runs when you care about stability under sustained load.
A sleeping phone. Some phones reduce background network activity or suspend the test when the screen locks. Keep the app in the foreground and the screen on during a run.

Running iperf3 on the phone itself

The phone is often the endpoint that matters: it is the device the complaint is about, and it is the device you carry from room to room. Options for a native endpoint:

iPhone / iPad / Mac: iPerf3 Client & Server on the App Store. One purchase ($6.99, one-time) covers iPhone, iPad, and Mac. Client and server modes, TCP/UDP, live charts, history, CSV/JSON export, Apple Shortcuts automation.
Android: iPerf3 TCP/UDP Client & Server on Google Play ($3.99, one-time). Same idea: both modes, advanced parameters, export.

Both apps use the iperf3 protocol, so a phone in server mode can accept tests from compatible iperf3 CLI versions and vice versa. If two endpoints fail to negotiate, compare their iperf3 versions before debugging the network.

Two phones make a portable end-to-end test kit: put one in server mode and use the other as the client. Because both endpoints share Wi-Fi airtime, this setup cannot isolate mesh backhaul performance. Use a wired server when you need to test one wireless hop or compare rooms.

Where to go deeper

iperf3 upstream project (source, releases, and official documentation)
How to test LAN speed, step by step
Why Speedtest and iperf3 answer different questions

Questions about a weird result are welcome in the comments; I read all of them.