DEV Community: Live Miracles

SRS, MediaMTX and Restreamer reject bonded SRT. Here is a relay that fixes it

Nandisha — Sun, 01 Nov 2026 10:41:29 +0000

SRT bonding sends one live stream over several network links at once, for example two uplinks or two SIM cards. If one link degrades or dies, the receiver still has the data from the other, so the stream does not drop. libsrt implements it as a socket group.

The catch is the receiving end. A server has to accept the group, and most open source media servers cannot. We run SRS in Bunryu, so we wrote a small relay, Goryu, that accepts the bonded group and forwards ordinary SRT to the server behind it. This post shows how to put it in front of SRS, MediaMTX or any SRT server, and what we actually checked.

The fix: terminate the group before the server

Goryu listens on one port (i.e. 10081) with group connections enabled. libsrt merges the legs and drops the duplicate packets before Goryu reads anything, so Goryu sees one clean MPEG-TS stream. It automatically forwards that stream to your server's port (i.e. 10080) as a normal single-link SRT caller, with the same streamid. The server never knows the input was bonded. An ordinary single-link SRT sender works on the same port.

Try it with MediaMTX

Start MediaMTX, then Goryu with a config that points at MediaMTX's SRT port (8890):

docker run -d --network host bluenviron/mediamtx

{
    "input_host": "0.0.0.0",
    "input_port": 10081,
    "output_host": "127.0.0.1",
    "output_port": 8890,
    "status_port": 8081,
    "passphrase": ""
}

curl -fsSL -o goryu.tar.gz https://github.com/live-miracles/goryu/releases/latest/download/goryu-linux-x86_64.tar.gz
mkdir goryu && tar -xzf goryu.tar.gz -C goryu
LD_LIBRARY_PATH=goryu/lib goryu/bin/goryu goryu.json

Now send a bonded stream to srt://<host>:10081 with streamid=publish:stream, and play it back from MediaMTX:

ffplay 'srt://127.0.0.1:8890?streamid=read:stream'

The streamid is the one thing to get right, because Goryu copies it unchanged to the server. These are the publish ids we used:

Passphrases

SRT encrypts each hop on its own. By default Goryu uses one passphrase on both: set passphrase. So similiarly to streamid the encoder will use the same passphrase as when directly connecting to the server. You can optionally give the server side its own passphrase with output_passphrase: an empty string for a server without a passphrase, or a different one.

{
    "input_host": "0.0.0.0",
    "input_port": 10081,
    "output_host": "127.0.0.1",
    "output_port": 8890,
    "status_port": 8081,
    "passphrase": "a-long-secret-for-encoders",
    "output_passphrase": ""
}

Broadcast or main/backup

libsrt 1.5 offers two group types, and the sender picks one. The receiver, Goryu, treats them the same: it merges what arrives and forwards one stream.

Broadcast puts the whole stream on every link. If a link fails the others already have every packet, so nothing has to switch. The price is bandwidth: two links cost twice the data.
Main/backup gives each link a weight. The heaviest link carries the stream and the others sit idle, sending only keepalives. If the main link becomes unstable the sender activates a standby. It costs one link's worth of data, which matters on metered cellular, but the switchover is its weak spot: packets sent while the main link was failing, before it was judged unstable, can be late or lost.

In Goryu's status API every leg of a broadcast group is running. In a main/backup group the standby is idle, with a receive rate of about zero.

We sent a main/backup group over loopback at 4 Mbps, with weights 10 and 5. The main leg was running at 4.4 Mbps and the backup idle at 0.000 Mbps. When we closed the main leg the backup became running, about 2 MB arrived in the next four seconds (roughly 4 Mbps), and the downstream never reconnected.

One catch: at a trickle (0.12 Mbps) libsrt kept both legs active, so a main/backup group needs real traffic before it settles on a main leg.

Our hardwere encoder AJA Bridge Live supports only the Broadcast mode, so we are using it with goryu. We have not tested Main/backup mode with real hardware encoders yet.

What we saw when a leg dropped

We sent a stream through Goryu with two legs and closed the second leg nine seconds in, watching the status API:

Time	Legs	Bytes forwarded
8 s	2	472,444
12 s	1	708,008
16 s	1	933,044

The leg left the legs[] list, the byte counter kept growing, and outputConnected stayed true. The server behind Goryu saw an unbroken stream. Each leg reports its own round-trip time, loss, retransmissions and buffer, which is what you want when you are deciding which uplink is the weak one.

With a real encoder, over the internet

This is how we run it: Goryu in front of SRS (known as srt-bonding-relay until this month) has carried our live events for the past few months, a few days every week.

We pointed an AJA BRIDGE LIVE at the Goryu relay (v3.3.0) on our Bunryu server, a Google Cloud VM in Mumbai, which forwards to the SRS on the same host. AJA lists SRT bonding over the public internet as a feature from firmware v1.16 (release notes), and it needs the Annual Maintenance Option. Four minutes after the relay started, its status API showed this for the two legs:

Per leg	Leg A	Leg B
Round-trip time	44.0 ms	40.8 ms
Receive rate	22.6 Mbps	22.6 Mbps
Packets received	491,703	491,737
Packets lost on this leg	48	157
Retransmissions	8	208
Dropped for arriving late	0	0
Receive buffer	242 ms	239 ms

Two full copies. In broadcast mode each leg carries the whole stream, so both show about 22.6 Mbps. The two legs came from distinct public addresses, with 240 ms of latency.
Real loss, handled. Each leg lost a small share of its packets (about 0.01% and 0.03%) and asked for retransmissions, and nothing arrived too late. Goryu reports loss per leg, not for the merged stream, so this shows how each link behaved, not what the merged stream lost.
Downstream. Zero retry failures and no error on the stream.

The relay's journal covers five days. It records 18 bonded sessions and no plain ones, with no downstream retries and no failed connects. The longest session ran 21 hours 51 minutes and forwarded 215.8 GB, about 22 Mbps on average, and it ended only because the service was stopped for maintenance. These sessions are our regular use of the relay. The snapshot above and the cable pull were taken on a test feed, not during a live event.

Pulling one cable

With the stream running, we unplugged one of the encoder's two LAN cables for about 45 seconds and plugged it back in, while recording the relay's status once a second. These are the samples around the event, on the VM's clock:

Time	Legs	Bytes forwarded	What changed
14:22:37	2	2,713,522,252	both legs running
14:22:41	1	2,725,683,408	one leg gone from `legs[]`
14:23:22	1	2,837,954,000	still one leg
14:23:27	2	2,852,886,652	the leg is back, with no restart

The stream never paused. Between 14:22:37 and 14:23:27 the relay forwarded 139.4 MB in 50 seconds, which is 22.3 Mbps. In steady state before the pull it was 21.9 Mbps. Over the whole event outputConnected stayed true, retryFailures stayed 0 and no error was recorded, so SRS saw one unbroken connection.
The leg came back by itself. Nobody touched Goryu or the encoder after the cable went back in. The status showed the leg absent for about 46 seconds, which matches the 45-second pull.
One oddity. The rejoined leg reports 7,977 lost packets from the moment it appears. That is roughly the first 3.7 seconds of the stream at this bitrate, so it is probably the gap before the leg joined being counted as loss. We have not confirmed that, and the merged stream did not slow down.
Nothing in the log, in v3.3.0. The journal has no line for the leg dropping or for it coming back. The status API shows it live, but afterwards only the counters remain. The next release, v3.4.0, logs both: one line per change with the leg's address and how many legs remain, such as Bonded leg left streamid=... leg=<ip>:<port> legs=1, and Bonded leg joined ... when it returns. A leg that flaps is limited to 12 lines per 30 seconds and then summarised, so a bad link cannot flood the journal. Bunryu writes the same changes to the pipeline's log.

Takeaways

Bonding needs both ends. The receiving server is as much a part of it as the encoder, and the open source ones mostly do not accept groups yet.
A relay is a small, replaceable piece. Goryu is about 2,800 lines of C and a config of six fields (a seventh is optional), so you can read all of it before trusting it with a live stream.
Check the sender first. The hard part today is finding something that sends a libsrt group, not receiving it. Of the hardware we know of, the AJA BRIDGE LIVE is the one we have seen do it.

Goryu and its documentation are open source (MIT).

50 inputs and 500 outputs in one dashboard

Nandisha — Sun, 01 Nov 2026 10:33:40 +0000

A long live event can mean dozens of inputs, each sent to a handful of destinations. Past a few streams, the question stops being "does it forward?" and becomes "can I still see what is happening without the screen turning into noise?"

Bunryu was tested with up to 50 inputs and 500 outputs on one server. Here is what the dashboard showed and how the control plane performed.

How we tested

We started the isolated test stack that ships with the repo, created 50 pipelines, and gave each one 10 outputs: 500 in total, the supported maximum. Every pipeline had a live input: a looped 720p clip published over RTMP. All 500 outputs were running.

One thing to know before looking at the pictures: the outputs are simulated. Each one is a stand-in for FFmpeg that reports progress and sends nothing over the network. The control plane, SRS, the inputs and the dashboard are real, so this shows what the dashboard looks like and what it costs the control plane. It does not show FFmpeg's own CPU and RAM, which our sizing page covers separately. The machine was an 8-core laptop shared with other work.

The overview

The left column lists every pipeline, organized into groups (here Main Stage, Halls, Studios, Chapels and Plaza). Each row has a status dot, its output count, its name, how long it has been up and its input protocol. The totals at the top say "50 inputs, 500 outputs" with a green badge for each. To the right are the host's CPU, RAM and network charts and a table of what each process costs.

Below that are two lists: every input, then every output.

When something goes wrong

The useful test of a big dashboard is whether a problem shows up without hunting for it. We took two of the 50 inputs offline and watched.

Within a few seconds:

The totals changed to 48 live inputs and 480 healthy outputs, with a yellow 20 for the outputs that belong to the two dead inputs.
The two pipelines turned yellow in their groups in the sidebar.
The Issues tab narrowed the output list to those 20 outputs.

Each row says why: "Output is running but input is not live." The outputs were not failing; they were waiting for an input that was no longer live, and that is what the dashboard told us.

One pipeline at a time

Opening a pipeline shows its input, its publish URLs and its 10 outputs on one screen, each with status, uptime, bitrate and memory. A group of ten outputs fits without scrolling, which is the level you work at when you are adjusting one destination.

What it costs

Measure	Result
Control plane with one dashboard open	about 17% of one core, 251 MB of RAM
`/api/health`, fetched every 5 s by each dashboard	296 KB, 18 KB gzipped, about 30 ms to produce
Dashboard first render (headless Chrome, locally)	0.7 to 0.8 s (one of four runs took 1.3 s)
Overview page length	about 28,000 px, roughly 32 screens

These are the control plane's costs. A real 500-output server also runs 500 FFmpeg processes. Our sizing page puts that at about 30 vCPUs and 17 to 30 GB of RAM at 8 Mbps per output.

One faulty audio embedder disrupted a mega event

Nandisha — Sun, 01 Nov 2026 10:31:00 +0000

Both audio meters were moving. The dry run had gone well. We had two fiber paths from the event to the streaming server. Yet when the event went live, viewers across more than 100 streams started reporting poor-quality audio that kept breaking up.

Switching to the backup feed did not help. Switching to a third feed finally did, about an hour into the live event. By then, many viewers had left.

This is how one suspected faulty audio embedder affected all those streams, why our monitoring missed it, and what we changed in Live Gallery afterwards.

Our monitoring heard only one channel

We were monitoring the streams in Live Gallery with a mono speaker setup that, in our arrangement, let us hear only the left channel. We could also see separate volume bars for left and right. Both showed audio, and what we heard sounded acceptable, so we thought the feeds were healthy.

But listening to the left channel alone is different from listening to a mono mix of left and right. We had checked that audio was present in each channel. We had not checked whether the channels worked together.

That distinction became painful during the event. Some phones played the audio acceptably; others sounded bad or lost much of it. Headphones could sound fine too. At the time, that inconsistency made the reports harder to explain. In retrospect, it was a useful clue: the result depended on how the playback device handled the two channels.

Why two active meters can hide bad audio

When two channels carry similar sound with opposite polarity, each channel can sound fine on its own. Their volume meters can look normal too. Trouble appears when a playback system adds them together.

A mono mix commonly combines the channels as (L + R) / 2. If the right channel is an inverted copy of the left, the positive parts of one meet the negative parts of the other, and the sound cancels. With partial cancellation, voices can become thin, quiet, or unstable rather than disappearing completely.

Stereo playback can keep the channels separate, so headphones may conceal the problem. Devices that combine the channels can expose it.

Analysis of the recording later showed negative correlation between the channels and a loss of level when they were combined. The strength varied through the affected portion. That points to a stereo compatibility problem; it does not, by itself, tell us which physical channel or connection was at fault.

This 33.3-millisecond sample from just before the feed switch shows the problem clearly: when one channel rises, the other falls. Cancellation was not equally strong throughout the incident.

For comparison, this sample after the switch to the clean feed shows the channels rising and falling together. Their correlation is close to +1. The key difference is their shared direction, not the height of the waveform: the two images show different moments of audio. Healthy stereo does not need to match this closely, but these channels reinforce rather than cancel each other when combined.

The backup had the same upstream fault

Our first response was to switch fiber paths. We had two paths from the venue to the streaming server, but both carried audio from the same suspected faulty embedder. Changing the transport path could not remove a problem already present in the source.

The third feed was different. It took video from the video mixer and audio from the audio mixer through a separate embedding path. We normally used this clean feed to make social media shorts because it did not include the event overlays and lower thirds.

Switching to that feed cleared the audio problem about an hour after the live broadcast started. It also exposed the gap in our backup plan: the two fiber paths were separate, but a critical part of their source chain was shared.

What we added to Live Gallery

Live Gallery is our open-source application for monitoring many live streams simultaneously. After this incident, we added an audio phase mismatch alert and a choice of mono or stereo listening mode.

The phase check looks at the relationship between left and right, including their correlation and how much level is lost when they are combined. It evaluates sustained evidence across multiple samples rather than treating one negative reading as a fault. We want to avoid nuisance warnings: normal stereo material can contain differences between channels, and negative correlation alone is not proof of broken equipment.

The mono listening option combines left and right so we can hear cancellation during a check. It closes the specific gap in our earlier setup, where hearing only the left channel gave us confidence that the full stereo feed was safe.

Our screenshot of Live Gallery 1.6.1 shows the phase warning directly over the playing recording, even while the channel meters show audio activity. This demonstrates the monitoring added after the incident, not an alert available during the original event.

What these changes cannot guarantee

The alert is a warning about possible cancellation, not a diagnosis of an embedder. It cannot identify a loose cable, determine which channel was inverted, or promise that every viewer's device will sound good. Quiet passages, intermittent faults, and less pronounced cancellation may also escape detection. We prefer some missed detections to constant false alarms that operators learn to ignore.

Mono listening and phase warnings give us additional ways to notice this failure. They do not replace listening, investigating early hardware trouble, or checking which components our backup feeds share.

Why it took so long to fix

The first reports of bad audio were confusing because the feed sounded fine on our monitoring setup. We only confirmed that something was wrong after checking the streams on our own phones.

Even after confirming the issue, we did not understand its cause. In all our years of regularly running live events, this was the first time we had encountered this particular phase mismatch problem. We initially suspected a fault in our encoder or other hardware, especially because switching to the backup feed had not helped.

Only later did we consider trying the clean feed. Switching to it would mean the editors would lose their feed, and viewers would experience a few seconds of buffering. We were not even sure it would fix the issue. Ultimately, though, it was the right choice.

Using only the left channel in the streaming encoder would also have fixed the cancellation problem. At the time, however, we did not yet understand the root cause, so we tried the clean feed first.

Takeaways for the next event

Check the feed in stereo and in a combined mono mix during the dry run. Left-only listening is a separate check.
Trace backup feeds all the way upstream. Two transport paths can still share one faulty source device.
Have a mature monitoring system that can detect various issues.

The clean feed saved this event's remaining broadcast. The more lasting change was learning to test what our viewers might actually hear, including what happens when two apparently healthy channels become one.

One leaking FFmpeg process took down six live streams

Nandisha — Thu, 22 Oct 2026 05:10:13 +0000

A live event cannot be restarted. When we built Bunryu, aself-hosted server that forwards one live stream to many destinations, we assumed things would go wrong and tried to keep each failure small. In July 2026 a single FFmpeg process showed us where that was not true yet.

The question

Someone looked at the dashboard and asked: "Output 1 has an error, output 2 doesn't. Why?"

Output 1's last error was exit=SIGKILL. That is not an application error, so we went to the host logs.

What happened

Six streams were running on one VM, with three pipelines and six outputs. This is the chain we pieced together from the logs:

All six encoders dropped and reconnected repeatedly for about eleven minutes.
When the streams settled, every output's FFmpeg process restarted and resynchronised. The audio at the reconnect boundary was corrupted.
FFmpeg's AAC decoder misread the corrupted audio as a bogus 22.2-channel layout and leaked memory instead of failing cleanly.
One process grew to about 1.6 GB and the kernel's OOM killer killed it.
FFmpeg children share a cgroup with the control-plane service, so systemd saw the whole service as failed. It restarted the service, and the restart cascaded to SRS and the SRT relay.
All six encoders reconnected within the same two seconds, which recreated the corrupt-resync conditions.
Fourteen minutes later a second output hit the same bug and was killed. That second kill was the error on the dashboard.

Output 2 had no error only because it was still on its way up. When we looked, it was at 644 MB and
climbing. "No error" did not mean "healthy".

The root cause was a loop: flaky input, a decoder leak on resync, an OOM kill, a restart of the whole
stack, and a fresh mass reconnect. Isolation between outputs did not help, because the process
topology made the kernel kill a shared fate.

What we changed

Drop damaged packets before they reach the decoder. Outputs now run with
-fflags +discardcorrupt -err_detect crccheck+bitstream. This reduces the chance of the misparse, but it does not remove it, so we did not stop there.

A memory watchdog that acts first. Every five seconds the app checks each output's resident memory. A healthy copy output sits at about 65 to 90 MB. If one crosses 200 MB, we kill it, record why, and let the normal retry loop start a fresh one. 200 MB is far below the 1.5 GB where the kernel steps in, so the kernel never needs to, and the cascade never starts. Transcoding outputs have higher limits because they legitimately use more.

Retries that don't pile up. An output retries indefinitely, waiting 1, 2, 4, 8, then 16 seconds after repeated quick failures. A run that stayed up for a minute resets the sequence, so a drop late in a long event is retried after about a second. Each wait is shortened by up to a quarter at random, so outputs that fail together do not respawn in lockstep.

Evidence on every failure. Each output keeps an error history with the exit status, the reason and the tail of FFmpeg's stderr. That history, together with the host logs, is how we could explain a SIGKILL after the fact.

What we did not fix

The shared cgroup is still there. The watchdog keeps the kernel from ever triggering the cascade, but if a process grew faster than the watchdog checks, or something else ran the host out of memory, the same chain could start again. We would rather say that than claim the cascade is gone.

We also test these paths on purpose: a destination that freezes, one that accepts a connection and never reads, one that refuses connections, a CPU-saturated host, and a soak run with 50 healthy outputs. No test makes a server perfect. The goal is that when something fails it stays small, fixes itself where that is safe, and leaves enough evidence to explain it afterwards.

Takeaways

Check how failures propagate, not only whether components are isolated. Separate processes still shared a cgroup.
Put your own watchdog below the kernel's threshold. An OOM kill is the last resort, not the plan.
"No error" is not "healthy". Look at resource trends, not only error flags.
Write down the timeline. A real incident is the best test case you will ever get.

The project and its reliability documentation are open source.