DEV Community: Athreya aka Maneshwar

PHP2Shell: Your Code Was Fine, The Engine Was Not

Athreya aka Maneshwar — Thu, 22 Oct 2026 20:20:24 +0000

Hello, I'm Maneshwar, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

Imagine you do everything right.

You validate your inputs. You use prepared statements. You escape your output. You have a linter, a code review process, and a slightly smug Slack channel called #security-wins.

Then someone sends your server one weird HTTP request, and it runs their code instead of yours.

Not because of anything you wrote.

Because of something PHP did while cleaning up after itself.

On October 21, 2026, the PHP project shipped a security release for two critical bugs, CVE-2026-109618 and CVE-2026-109619.

The internet has already nicknamed the pair PHP2Shell, which is a great name and a terrible Wednesday.

Let's walk through what actually broke, why it is a bigger deal than "yet another CVE", and also where the scary headlines are a little ahead of the facts.

Why this one is different

PHP is not a niche language. W3Techs says 69.3% of websites whose server-side language it can identify run PHP.

WordPress, Laravel, Drupal, Magento, that one invoice portal from 2014 that nobody dares touch.

Most PHP vulnerabilities live in an application or a plugin. You patch the plugin, you move on.

These two live in the engine.

The engine is the thing underneath your code. It is the C program that parses your PHP, allocates your memory, and tears down after every request.

You can write the most boring, correct PHP of your life, and the floor below it can still have a hole in it.

[DIAGRAM PLACEHOLDER] the stack, bottom to top: the web server (Apache or nginx), the SAPI (mod_php or PHP-FPM), the PHP engine, then your application code on top. Highlight the engine layer in red with the caption "both bugs live here, below everything you wrote". Show that an attacker's request has to pass through the engine before your code even sees it.

Both bugs are about the same unglamorous thing: memory management around a request that goes wrong.

To see why, you need to know how PHP spends its life.

PHP's life is one request long

PHP's model is famously "shared nothing".

A request comes in, PHP sets up a fresh little world for it, runs your script, then throws the whole world away.

The worker process survives, but the request's state is supposed to die with the request.

That is why PHP is so forgiving. Leak some memory in a script? Nobody cares, the request ends and the memory gets reclaimed.

Here is the happy path:

flowchart LR
    A([Request arrives]):::start --> B[Request startup<br/>build per-request state]:::engine
    B --> C[Run your script]:::app
    C --> D[Request shutdown<br/>cleanup and free]:::engine
    D --> E([Worker waits for the next request]):::start
    E --> A

    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef engine fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef app fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a

Notice that the purple boxes, startup and shutdown, are the engine's job.

Your code only owns the teal one in the middle.

Both vulnerabilities are about what happens when something goes wrong between those purple boxes, and the engine's cleanup code assumes a state that no longer holds.

PHP has a hard ceiling on how much memory one request may use, memory_limit. Hit it, and PHP gives up on the request on the spot.

"Gives up on the spot" is where the trouble starts.

Bug 1: cleaning up a house that was never built

CVE-2026-109618, the out-of-memory one.

Here is the scenario, paraphrasing the PHP Foundation's write-up and Rapid7's analysis.

PHP is in the middle of request startup. It is building that per-request state, piece by piece.

A memory allocation fails. PHP abandons the request abruptly.

But the state is only half built. Some fields are set, some are whatever bytes happened to be sitting in that memory.

Then the cleanup code runs, and it does what cleanup code does: it trusts the state it is handed.

It follows pointers that were never initialized. It frees things that were never allocated. It treats garbage as a structure.

The mental model I like: a demolition crew is sent to tear down a building.

The building was never finished, and the blueprint is a napkin with random scribbles on it.

The crew follows the napkin anyway, and demolishes whatever the scribbles point at.

Here is the shape of the bug as a deliberately simplified sketch. This is not PHP's real source, just the pattern:

// simplified illustration of the bug class, not actual PHP code
int request_startup(request *r) {
    r->headers = alloc_table();     // ok
    r->uploads = alloc_table();     // out of memory! bail out...
    if (!r->uploads) return FAILURE;
    r->hooks   = alloc_table();     // never reached, r->hooks is junk
    return SUCCESS;
}

void request_shutdown(request *r) {
    free_table(r->headers);
    free_table(r->uploads);
    free_table(r->hooks);           // frees a pointer nobody ever set
}

The mistake is not in either function alone. It is in the assumption that shutdown only ever runs on a fully started request.

Now here is the part that makes this one nasty.

How do you make a server run out of memory on purpose? You do not need to hack anything, you just ask nicely and badly.

Rapid7 notes that crafted multipart requests can exhaust memory even when their overall size looks like it fits inside the configured limits.

And on Apache with mod_php, the demonstrated attack happens before any of your application code runs.

That is the whole reason the "zero authentication" line is true. There is no login to bypass. Your code never got the chance to say no.

Memory corruption you can trigger with a request, in a place your app cannot guard, is how you end up with "9.2 critical" in a headline.

Bug 2: the ghost of requests past

CVE-2026-109619, the stale state one.

Same family, different flavour.

Remember that a worker process handles request after request. Some bookkeeping state is global to the worker, and is supposed to be reset between requests.

Now picture this sequence:

sequenceDiagram
    participant A as Request A
    participant W as PHP worker
    participant B as Request B
    A->>W: runs, global state points at memory M
    W->>W: fatal error interrupts cleanup
    W-->>W: M is freed, but the pointer to M is not cleared
    B->>W: arrives on the same worker
    W->>W: reads the old pointer before init resets it
    Note over W: use-after-free: B is reading memory it does not own

A fatal error, like running out of memory or a timeout, can interrupt the cleanup in the middle.

The memory gets freed, but a pointer to it survives. The next request lands on the same worker, and something reads that pointer before normal initialization has a chance to wipe it.

That is a classic use-after-free. The name is the best part, because the thing you are using after it is free is not free at all, it is very expensive.

Think of a hotel room that was demolished but the key card still opens the door.

Whoever is standing there next gets to decide what is inside.

Rapid7 describes two concrete paths to reach it.

One is the PHP 8.5 fatal error backtrace feature, which is on by default and was confirmed on mod_php. The other is a session upload progress path that works on PHP-FPM, and needs an application operation to leave state behind first.

Note the pattern. Neither bug is about a missing bounds check in some parser. They are both about error paths.

The happy path of PHP has been tested for decades. The path where a request dies at the worst possible moment is the one that gets less love.

So can anyone hack 70% of the internet?

Short answer: no, and it is worth being precise, because the headline version of this story sells more than it delivers.

What is solid:

Both bugs are real and rated 9.2 (CVSS 4.0). Both can lead to code execution as the web worker's user, with no authentication.

RCE was demonstrated by researchers, against real setups, not just crashes.

mod_php is the highest risk. Rapid7 notes that for the OOM bug, mod_php runs cleanup even when startup failed. PHP-FPM kills the worker when startup fails, which blocks that specific sequence.

What is more nuanced:

"Any PHP server" is too strong. Rapid7 reviewed multiple public proof of concepts, and each one works only for a specific combination of OS, PHP version and web server. None is broadly reliable yet.

The attack complexity is rated High. Memory corruption exploits are fiddly. That is a speed bump, not a wall.

Reports of exploitation are mixed. watchTowr says it has seen exploitation in the wild through its honeypots, while Rapid7 has seen attempts but not confirmed in-the-wild compromise.

Whichever way that lands, Rapid7 also warns that a more portable technique could show up quickly.

And the exposed surface is not small. Palo Alto's Unit 42 brief cites telemetry that counts over 22 million potentially vulnerable internet-facing instances. That is a vendor number, so season to taste, but it is the right order of magnitude for "a lot".

If I had to summarize it: not the end of the web, but also not a drill.

The part that is actually uncomfortable

Here is the story behind the patch, and it is the most human part of the whole thing.

flowchart LR
    A[2024<br/>Quarkslab audit,<br/>first signs of the OOM issue]:::early --> B[Fix stalls<br/>competing patches,<br/>each with trade-offs]:::slow
    B --> C[Aug 2026<br/>Calif, OpenAI and Wiz<br/>report it, RCE shown]:::report
    C --> D[Oct 20<br/>CVE IDs leak<br/>2 days early]:::bad
    D --> E[Oct 21<br/>Chainguard images<br/>break the embargo]:::bad
    E --> F[Oct 21, 14:00 UTC<br/>patched releases ship]:::good

    classDef early fill:#6ea8ff,stroke:#3a6fcc,color:#1a1a1a
    classDef slow fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef report fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef bad fill:#ff9a5c,stroke:#cc6a2a,color:#1a1a1a
    classDef good fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a

Per the PHP Foundation, an external audit by Quarkslab back in 2024 led a core developer to spot the first signs of the OOM problem while fixing something else.

No exploit path was known at the time.

Then the fix stalled. Competing patches each had a cost: breaking compatibility, opening new denial of service angles, or touching lots of third-party extensions.

The Foundation's own words are refreshingly blunt: the delay was "a structural failure, caused by both the security team's decision process and its lack of resources."

Then in August 2026 the bug was reported again, with a working unconditional RCE. The CVE IDs leaked online before the release. The embargo then cracked a second time when a vendor published images that already contained the fix.

I find this oddly reassuring and deeply annoying at the same time.

Reassuring, because an open source project wrote that sentence about itself in public, and is changing its process: majority decisions, an ecosystem security team, more staff.

Annoying, because "everyone who maintains critical infrastructure is under-resourced" is not a new lesson, and PHP is carrying a very large share of the web on it.

The AI angle, and where I would push back

A lot of the commentary around this release points at one thing: AI changes who can exploit this.

The argument goes like this. Turning a subtle memory corruption bug into a working attack used to need rare skills. Now somebody with a laptop and an open-weights model can point an agent at the internet, hand it the advisory and say "go find the ones that are still vulnerable".

I think the direction is right and the details are overstated.

What is right: the cost of going from advisory to attempt is falling fast. The patch itself is a diff. A diff tells a model exactly where to look. And scanning is cheap when the scanner does not get tired.

What is overstated: the hard part of this bug is not finding targets, it is the exploit reliability. Rapid7 says the public PoCs are fragile. Memory corruption is a craft, and models are not magic yet at crafting heap layouts for your particular distro.

And there is a side of the story that gets less airtime: the defenders have the same tools.

The Foundation says it used AI-assisted scanning and validation as part of its security work, with the line that "a person makes every call and works on every patch."

So the realistic summary is not "attackers got superpowers". It is this:

The window between "patch published" and "someone is trying it on your server" is shrinking.

Your patching speed is now a security control, not an IT chore.

What to do about it

Boring answer first: update.

The fixed releases are PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35.

If you are on 8.1 or older, there are no routine upstream releases. Rapid7 notes limited source backports exist for 7.4, 8.0 and 8.1, so lean on your distribution or an LTS vendor, or finally take the upgrade you have been postponing since 2022.

Two gotchas people will trip on:

A patched package on disk is not a patched process in memory. Running workers keep the old code until they restart. Qualys specifically calls this out, so restart or gracefully reload your PHP services after upgrading.

Containers do not save you from stale base images. If your image was built last week, it is still carrying last week's PHP.

Quick check:

php -v                       # compare against 8.5.12 / 8.4.27 / 8.3.36 / 8.2.35

# then make sure the running workers are the new ones
sudo systemctl reload php8.3-fpm     # or: apachectl graceful

If you truly cannot patch yet, the advisories list temporary mitigations. They reduce exposure, they do not fix it:

; php.ini or FPM pool config, NOT ini_set() or .user.ini (too late for these)
session.upload_progress.enabled = 0   ; closes the known session path
fatal_error_backtraces = 0            ; PHP 8.5 only
post_max_size = 8M                    ; and cap multipart size at the frontend too

You will see memory_limit=-1 suggested too, and here is where I have to confess the Roll Safe meme is doing real work:

It genuinely appears in the advisory guidance, set administratively so apps cannot override it. But Rapid7 is clear that an allocation failure can still reach the vulnerable path, so it narrows the window and does not close it.

Also, a server with no memory limit has its own personality. It is called "an outage".

And two things not to reach for. Rapid7 warns that RLIMIT_AS and RLIMIT_DATA can themselves trigger the vulnerable allocation failure path, and container or cgroup memory limits are an incomplete answer and can kill the whole worker pool.

Do not hide the evidence either. Silencing errors with display_errors or @ does not prevent stale state from being read, and a quiet log is not proof you are fine.

Here is the whole decision as a flowchart, for the person who has a patch window and a manager:

flowchart TD
    A([Do you run PHP on a web server?]):::start --> B{Is it on 8.2.35, 8.3.36,<br/>8.4.27 or 8.5.12 or newer?}:::decision
    B -- yes --> C{Did you restart or reload<br/>workers after upgrading?}:::decision
    C -- yes --> D([You are patched. Go touch grass.]):::good
    C -- no --> E[Restart workers now]:::action
    B -- no --> F{Is it 8.1 or older?}:::decision
    F -- yes --> G[Distro or LTS backport,<br/>or plan the upgrade]:::action
    F -- no --> H[Upgrade to a fixed release]:::action
    G --> I[Apply interim mitigations<br/>until it lands]:::warn
    H --> E

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef good fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef action fill:#6ea8ff,stroke:#3a6fcc,color:#1a1a1a
    classDef warn fill:#ff9a5c,stroke:#cc6a2a,color:#1a1a1a

The thing worth taking away

The patch is the urgent lesson. The durable one is quieter.

Your security posture includes layers you did not write and never read.

Your code can be perfect and the runtime under it can still ship a bug that was first noticed two years ago.

That is not a reason to panic about PHP specifically. Every runtime has error paths nobody loves, and the next bug of this kind will be in a different one.

It is a reason to treat runtime and base image updates as first class work, to know which version is actually running in production, and to keep the gap between "advisory" and "deployed" as small as you can.

Cleanup code, it turns out, is also code.

Maybe even the code most worth reviewing.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Risk-Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is built for engineering teams that ship often, but also care about keeping production stable for customers. It approves some 40-50% of low-risk changes automatically, and it gives your team the tools to handle the rest:

Understand risky changes faster.

Fix problems in those changes, with human attention directed to the work that needs it.

Enforce your engineering standards.

Learn from production outcomes, such as incidents, rollbacks, and queue wait times.

Improve the review process over time and make it more efficient.

Connect to the major engineering tools that your team already uses.

Auto-Approve: Ship the Safe 50% Without a Human

Auto-Approve Upto 50% of Your Pull Requests, Safely. Risk scoring decides which MRs/PRs are safe enough to ship without a human, so your team only spends review time on risky and critical changes.
demo.mp4
LiveReview auto-approving a low-risk change while…

View on GitHub

Click below to try LiveReview with your codebase:

Processes vs Threads, as the Linux Kernel Sees Them

Athreya aka Maneshwar — Tue, 20 Oct 2026 20:59:17 +0000

"What is the difference between a process and a thread?"

You have been asked this. You have probably asked it.

The standard answer goes: a process is heavy, a thread is light, and threads share memory.

That answer is true.

It is also the kind of true that you can recite for ten years without ever understanding what is going on underneath.

So let's skip the textbook and look at both from the seat where the decisions actually get made.

The kernel's seat.

Start with one long bar of RAM

Picture your machine's memory as a single long bar. Say 16 GB.

All of it belongs to the kernel. No program is allowed to touch it directly.

When you launch an app, the kernel creates a process to represent it, and carves out a slice of the bar as that process's private memory.

It writes down who owns which slice in a notebook, and it guards those boundaries.

Now watch it with real software.

You open a photo editor. The kernel makes a process for it, and the editor fills its slice with a 5 MB photo.

You also have Spotify open. That is a second process, with a second slice further down the bar.

The editor cannot read Spotify's memory. Spotify cannot read your photo.

If the editor reaches outside its slice, the kernel stops it on the spot.

And if Spotify crashes, your photo does not even flinch.

That wall is the entire reason one program cannot read or corrupt another's data.

A small confession about the bar

The bar is a teaching picture, and it is a little too tidy.

Real kernels do not hand out literal chunks of physical RAM. Every process gets its own virtual address space, and the CPU's memory management unit translates those addresses through page tables that the kernel controls.

Two processes can both use "address 0x1000" and land on completely different physical memory.

Touch an address that is not mapped for you, and the hardware raises a fault, and the kernel typically answers with a SIGSEGV.

The kernel's own memory docs cover the concepts if you want the full tour.

The wall in the picture is a very real wall. It is just made of page tables instead of drywall.

Now the editor wants to be fast

Brightening a photo one pixel at a time on a single core is slow.

So the editor asks the kernel for help: "give me four threads".

Here is the part that matters.

The kernel does not hand each thread more RAM.

It creates four lightweight tasks and points all of them at the same slice the process already owns.

Then it schedules those four tasks onto four cores, and they run at the same moment.

Each thread takes a quarter of the photo and brightens its own region, reading and writing the pixels directly where they sit.

Nothing was copied.

When the last thread finishes, the whole image is already brighter.

Now imagine doing the same job with four processes instead.

Each one needs its own slice. So the photo gets copied into all four, which is four times the memory.

Then somebody has to collect the four finished pieces and stitch them back together, which means sending data between processes with pipes, sockets or shared memory.

That is the entire pitch for threads in one sentence: they skip the copy, and they skip the stitching.

A fairness note for processes

I have been a bit mean to processes, so let me correct the record.

Modern fork() does not actually copy the whole photo up front. On Linux it is implemented with copy-on-write pages.

sequenceDiagram
    participant P as parent
    participant K as kernel
    participant C as child

    P->>K: fork()
    K-->>C: new task, page tables copied, pages not copied
    Note over P,C: both see the same 5 MB, marked read-only
    C->>K: write to one pixel
    K->>K: page fault, copy just that one 4 KB page
    K-->>C: retry the write on the private copy
    Note over P,C: every untouched page is still shared

So the child starts almost for free, and only pays for pages it actually changes.

The picture above is the worst case, where every process rewrites everything. That is the case that happens when a job really does touch the whole photo.

The real difference is not "copy vs no copy" so much as "shared by default" vs "private by default".

And that, as we are about to see, is where all the trouble comes from.

The kernel's dirty secret: they are the same thing

Here is the sentence that rewired my head.

To the Linux kernel, a process and a thread are nearly the same object.

Both are just a task that the scheduler puts onto a CPU.

The scheduler does not schedule "processes". It schedules tasks. A four-thread process is four tasks that happen to be friends.

The only real difference is memory:

make a new task with a fresh slice, and you have a process
make a new task pointed at an existing slice, and you have a thread

On Linux that choice is one function: clone(), and one flag: CLONE_VM.

Do not take my word for it. Here is a tiny program that does the same write twice, once from a child process and once from a thread:

int brightness = 0;

void *worker(void *arg) { brightness = 100; return NULL; }

int main(void) {
    pid_t child = fork();
    if (child == 0) { brightness = 100; _exit(0); }   // a process writes
    waitpid(child, NULL, 0);
    printf("after a child PROCESS wrote 100: brightness = %d\n", brightness);

    pthread_t t;
    pthread_create(&t, NULL, worker, NULL);            // a thread writes
    pthread_join(t, NULL);
    printf("after a THREAD wrote 100:        brightness = %d\n", brightness);
}

I compiled it and ran it on Ubuntu 24.04:

after a child PROCESS wrote 100: brightness = 0
after a THREAD wrote 100:        brightness = 100

Same line of code in both. The process wrote into its own copy and the parent never saw it. The thread wrote into the copy.

Then I ran the same binary under strace -f -e trace=clone,clone3 to see what glibc really asked the kernel for (trimmed a little):

clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD)
clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|...})

The first line is fork(). No CLONE_VM.

The second is pthread_create(). There it is, CLONE_VM, plus friends.

Those friends decide what else gets shared:

CLONE_FILES: the open file descriptors
CLONE_SIGHAND: the signal handlers
CLONE_THREAD: the new task joins the same thread group, which is why getpid() returns the same number for every thread and the kernel hands each one its own thread id instead

(In my run the main thread and the worker both reported the same pid, and the worker had its own tid. The man page for gettid covers the details.)

And what is private to each thread? Its own stack, its own registers, and its own thread-local storage. Everything else, the heap and the globals and the open files, is shared.

A fun side effect of all this: containers are the same trick. Namespaces are created with CLONE_NEW* flags on the same clone() call. Linux has one very flexible "make a task" function, and the whole zoo of processes, threads and containers is just different flag combinations.

Danger one: two threads, one pixel

Now for the bill.

The kernel let all four threads into the same memory. That is the whole point. It is also the whole problem.

If two threads write the same pixel at the same time, you get a race condition.

Here is how it goes wrong. Both threads add 10 to a pixel that holds 50:

Both read 50. Both compute 60. Both write 60.

You expected 70. You got 60.

There is no crash. There is no error. One brighten operation simply evaporated.

I wanted a real number instead of a hand-wavy one, so I ran four threads, each bumping one shared counter a million times:

for (int i = 0; i < 1000000; i++) {
    unsafe++;                      // shared, no protection
    pthread_mutex_lock(&lock);
    safe++;                        // shared, behind a mutex
    pthread_mutex_unlock(&lock);
}

expected: 4000000
no lock:  3966879
mutex:    4000000

Over thirty thousand increments went missing, and the program exited happily. A second run lost a different amount.

That is what makes races miserable. They are not wrong in a consistent way. They are wrong in a different way every time, which is a spectacular property for a bug to have when you are trying to reproduce it.

The textbook fix is a lock. Only one thread at a time may touch the contended spot, and the rest wait their turn.

That works. It also costs you, because threads that wait are threads that are not working.

The better fix is to need no lock at all.

Look again at the photo example. Each thread was given its own quarter. Nobody writes anyone else's pixels, so nobody has to wait for anybody.

The best concurrency code is the kind where the threads rarely touch the same data. Partition first, lock only the seams.

Danger two: shared memory means shared fate

The second bill is nastier because it is not subtle.

All four threads live in one address space. So when one of them does something the kernel considers fatal, the kernel has no way of killing "just that thread".

A segmentation fault on any thread, and the default action for that signal is to terminate the whole process.

I did not want to just claim that, so I tried it. One thread ticks along. Another sleeps half a second, then writes through a null pointer:

steady thread: tick 1
steady thread: tick 2
steady thread: tick 3
Segmentation fault (core dumped)       # exit code 139

The steady thread had two more ticks to go. It never got them.

Three perfectly healthy threads, executed for the crime of sharing an address space with a guilty one.

The same logic applies to quieter failures. A buffer overflow in one thread can scribble over memory that another thread is using, and the crash shows up somewhere completely unrelated, an hour later.

Meanwhile Spotify, a separate process, is still playing. That wall from the first picture just paid for itself.

So which one do I pick?

You can see this trade-off baked into real software.

Chrome gives each site its own process, so one rogue tab cannot take the browser down with it. Isolation is worth the memory.

PostgreSQL uses a process per connection, so a crashing backend does not take the whole database with it. Isolation is worth the memory there too.

On the other side, anything that crunches one big shared dataset on many cores, like our photo brightener, wants threads, because copying and stitching would eat the speedup.

(And if you are in Python, the GIL means threads will not run your Python bytecode in parallel in the standard build, which is why so many people reach for processes there. That is a language story, not a kernel one.)

Here is the decision as a flowchart:

flowchart TD
    A[you need concurrency] --> B{must one failure<br/>stay contained?}
    B -->|yes: untrusted code,<br/>plugins, separate tenants| P[processes]
    B -->|no| C{do the workers need to<br/>share lots of data fast?}
    C -->|no| P2[processes,<br/>simpler and safer]
    C -->|yes| D{can you partition the data so<br/>threads rarely touch the same bytes?}
    D -->|yes| T[threads,<br/>almost no locks]
    D -->|no| T2[threads plus locks,<br/>and budget for the debugging]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef proc fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef thr fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a

    class B,C,D decision
    class A start
    class P,P2 proc
    class T,T2 thr

The 30 second interview answer

If somebody asks you the question tomorrow, here is the whole thing:

The kernel gives every process its own private memory, and the wall between processes is why one app cannot corrupt another.
A thread is a task the kernel points at memory that already exists, so threads share one address space and nothing gets copied.
On Linux both are made by clone(). The difference is basically one flag, CLONE_VM.
Sharing is the speed, and it is also the two dangers: races when threads touch the same data, and shared fate when one thread crashes.

"Threads are lightweight" was never wrong.

It just left out what you are paying with.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

How Do You Make a Microservice Only Your Other Services Can Call?

Athreya aka Maneshwar — Mon, 19 Oct 2026 16:50:30 +0000

Here is a classic microservices interview question.

"One of your services should only accept calls from other internal services. How do you secure it?"

Most people answer in one line.

"Put it in a private network, no public IP."

That answer is correct.

It is also about 25% of the answer, and the other 75% is where the interview is actually won.

So let's take the question apart the way an attacker would, one layer at a time.

The setup: a payments service with a secret

Make it concrete.

A user places an order. The order service calls the payments service to charge the card.

That is the only legitimate way payments is ever used.

No user should talk to it directly. Nothing on the internet should even know it exists.

So the goal is not "make payments secure".

The goal is narrower and much more useful: make payments answer exactly one caller, and prove who that caller is.

Hold on to that sentence, because every layer below is an attempt to make it true.

Layer 1: no public address

Start with the thing everyone says.

In Kubernetes, a Service defaults to type ClusterIP, which gets a virtual IP that is only routable inside the cluster.

So you do not give payments a LoadBalancer or a NodePort, and you make sure no Ingress rule points at it.

From the outside, the service simply does not exist. There is nothing to scan and nothing to brute force.

That is a genuinely good first move, and it kills the laziest class of attack, which is "somebody found the URL".

Now look at what it does not do.

Inside the cluster, the network is flat by default.

Every pod can open a connection to every other pod, and there is no door between them.

Your marketing website runs in the same cluster. It has a CMS, three plugins nobody updated, and a contact form written during a hackathon.

An attacker finds one bug in it.

Congratulations, they are now inside your private network, and payments will happily answer them, because from payments' point of view the request came from a neighbour.

"Private" is a statement about strangers.

It says nothing about tenants.

Layer 2: a firewall for pods

The fix for a flat network is to stop it being flat.

Kubernetes has NetworkPolicy for exactly this. Think of it as a firewall rule that follows pods around.

The pattern is two policies. First close everything, then name exactly who gets in:

# 1. the namespace starts closed
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: shop
spec:
  podSelector: {}
  policyTypes: [Ingress]
---
# 2. payments accepts order pods, on its own port, and nothing else
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-from-order
  namespace: shop
spec:
  podSelector:
    matchLabels: { app: payments }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels: { app: order }
      ports:
        - protocol: TCP
          port: 8080

Now the compromised marketing pod can still open a socket, but its packets are dropped on the way in.

One trap here, and it is a nasty one.

A NetworkPolicy is only enforced if your network plugin enforces it.

The Kubernetes docs are blunt about this: if the plugin does not implement policies, creating the object has no effect.

Calico and Cilium enforce them. Some setups do not, or need a switch flipped on.

So you can write a lovely allow list, kubectl apply it, get a calm green "created", and be protected by absolutely nothing.

Test that it actually drops traffic. We will do that at the end.

The blind spot

Even when it is enforced, a network policy has a fundamental limit.

It decides based on where a packet came from, not who actually sent it.

It is checking the address on the envelope, and says nothing about the person who wrote the letter.

In practice that breaks down in three ways:

The label is the identity, and labels are cheap.
Anyone who can create a pod with app: order in that namespace walks straight through the allow list. A policy trusts the label, and nothing verifies the label.

It is blind above the network layer.
It knows "TCP, port 8080". It has no idea whether the request is POST /charge or DELETE /everything.

It does not encrypt anything.
Inside the cluster, the bytes are readable by anyone who can sniff the node's traffic.

Pod IPs also get recycled constantly, so an address that was the order service a minute ago can belong to a different pod now.

That is why a good policy matches on labels instead of raw IPs.

But the deeper problem is the one above: there is no proof attached to the claim.

A network policy is great at shrinking the blast radius.

It is not great at proving identity.

For that, you need ID cards.

Layer 3: mutual TLS

Regular TLS is what your browser does. The server shows a certificate, and you check it.

In mutual TLS, the client shows one too, and the server checks it before any data moves.

So when order calls payments:

order proves "I am the order service"
payments proves "I am the payments service"
both are verified against the same trusted authority, and only then does the request go through

An attacker sitting inside the network can reach the port. They cannot forge a certificate signed by your authority, so the handshake fails and no request is ever read.

That one change moves you from "trust the address" to "trust the proof".

Who hands out the certificates?

This is the practical question that makes most people wave their hands.

Every service needs its own certificate, issued by one authority that you run inside the cluster.

The identity inside it is usually a SPIFFE style name, and in a mesh it is derived from the pod's Kubernetes ServiceAccount, something like spiffe://cluster.local/ns/shop/sa/order.

Two details matter more than they look:

The certificates are short-lived.
Hours, not years. A stolen cert is a very temporary problem, and nobody has a spreadsheet of expiry dates to forget about.

The identity is the ServiceAccount.
If every deployment in your cluster runs as the default ServiceAccount, they all share one identity, and your fancy mTLS can no longer tell them apart. Give each service its own.

You do not write any of this

Nobody hand-rolls certificate rotation inside every service.

You use a service mesh, such as Istio or Linkerd.

The mesh injects a small proxy, a sidecar, next to every pod. That proxy fetches the certificate, presents it, verifies the peer's, rotates it before it expires, and encrypts the traffic.

Your application keeps speaking plain HTTP to localhost. It never sees a certificate.

(Sidecars are not the only shape. Istio also has an ambient mode that moves the proxying into a per-node component, so the idea survives, the extra container goes away.)

Here is one call, start to finish, as the time-ordered version of that picture:

sequenceDiagram
    participant CA as Cluster CA
    participant OP as order sidecar
    participant PP as payments sidecar
    participant P as payments app

    OP->>CA: prove who I am (ServiceAccount token)
    CA-->>OP: short-lived cert for sa/order
    PP->>CA: prove who I am (ServiceAccount token)
    CA-->>PP: short-lived cert for sa/payments
    Note over OP,PP: later, the order app makes a call
    OP->>PP: TLS handshake, here is my cert
    PP-->>OP: and here is mine
    Note over OP,PP: both verify against the CA root
    OP->>PP: POST /charge, encrypted
    PP->>P: plain request on localhost
    P-->>OP: 200, back through both proxies

The gotcha that gets teams

Installing a mesh does not mean you are enforcing mTLS.

Istio's default mode is PERMISSIVE, which accepts both mTLS and plain text, so that you can migrate without breaking things.

That is a sensible default for rollout.

It is also a state in which an attacker's plain curl still works, while your dashboard cheerfully says "mesh enabled".

You have to flip payments to STRICT on purpose. The config for that is in the next section.

Layer 4: authenticated is not the same as allowed

Now every caller has a verified identity.

So the next question is the one most people forget: should this caller be allowed to do this thing?

The order service may call payments.

It has no business calling the refund service, because refunds belong to the support service.

Here is why this matters. Say the marketing site also runs in the mesh. It has a sidecar, so it has a perfectly valid certificate.

mTLS says: "yes, that really is the marketing service."

It does not say: "and the marketing service may charge cards."

That second sentence is an authorization policy, and the same mesh enforces it:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: shop
spec:
  mtls:
    mode: STRICT          # the default is PERMISSIVE, which still accepts plaintext
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: payments-allow-order
  namespace: shop
spec:
  selector:
    matchLabels:
      app: payments
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/shop/sa/order"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/charge"]

Read the rules block out loud.

"The identity sa/order may POST to /charge on payments."

Once an ALLOW policy exists for a workload, anything that does not match a rule is denied. You get default-deny for free, at the level of identity and URL rather than IP and port.

The proxy next to payments checks this on every single request.

This is least privilege, applied between services instead of between people.

If someone steals the order service's identity, they can charge cards. They still cannot issue refunds, because that was never in its rules.

Linkerd has the same idea under different names, a Server plus an AuthorizationPolicy that binds identities to routes. The shape transfers, the YAML does not.

Proving it, because "created" is not "enforced"

Every one of these layers fails differently, and that is a feature.

Run one request from a pod you are not supposed to trust, and the way it fails tells you which gate caught it:

kubectl exec deploy/marketing -- \
  curl -sS -m 3 -X POST http://payments.shop:8080/charge

# NetworkPolicy dropped it   -> curl: (28) Connection timed out after 3001 milliseconds
# mTLS STRICT refused it     -> curl: (56) Recv failure: Connection reset by peer
# AuthorizationPolicy said no -> RBAC: access denied   (HTTP 403)

Those are the usual signatures. The exact text shifts a little between versions and tools, but the character is stable: a hang, a reset, a polite 403.

flowchart TD
    A[a call to payments fails] --> B{what does the<br/>client actually see?}
    B -->|name does not resolve,<br/>or no route at all| C[ClusterIP: you are outside the cluster]
    B -->|hangs, then times out| D[NetworkPolicy dropped the packets]
    B -->|connection reset| E[mTLS: no valid cert,<br/>or plaintext sent to a STRICT peer]
    B -->|HTTP 403 RBAC access denied| F[AuthorizationPolicy: valid identity,<br/>not allowed to do this]
    B -->|HTTP 200| G[all four gates said yes]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef block fill:#ff6b6b,stroke:#c94a4a,color:#1a1a1a
    classDef ok fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a

    class B decision
    class A start
    class C,D,E,F block
    class G ok

One honest caveat on that chart.

A timeout is ambiguous. "No route" and "dropped by policy" can both look like a hang from the client side.

That is why you test from inside a pod that should be blocked, and from a pod that should be allowed. Both directions. If the blocked one succeeds, your network plugin is not enforcing anything.

Also note the marketing pod, if it is in the mesh, will sail past gate three and be stopped at gate four. A request that dies at the last gate is not a failure of the earlier ones, it is the reason the last one exists.

Putting it together

Here are the four gates in one picture:

Each one stops a different mistake:

ClusterIP stops the internet. It sees: no route.
NetworkPolicy stops other pods. It sees: a timeout.
mTLS stops pods with no identity. It sees: a reset.
AuthorizationPolicy stops the wrong service. It sees: a 403.

The attacker has to beat all four.

You only need one of them to hold.

That is the actual interview answer: defense in depth, where no single layer has to be perfect, because each is covering for a hole in the one before it.

What is still left (the part that gets you the senior grade)

Four gates is a strong answer. It is not the end of the story, and it helps to say so.

mTLS authenticates services, not users.
If the order service is compromised, or has an SSRF bug, it can ask payments to charge any card, and every gate above will wave it through, because it really is the order service.

This is the classic confused deputy problem.

The usual answer is to carry the end user's identity downstream, for example a signed token that payments validates with something like Istio's RequestAuthentication, plus sanity checks such as "does this order id belong to this user".

Stolen credentials still work until they expire.
Short-lived certs shrink that window. They do not remove it.

You can do all of this without a mesh.
Cloud providers have their own service-to-service identity, such as IAM based auth on AWS or Google Cloud. SPIFFE and its runtime, SPIRE, work outside Kubernetes. A mesh is the convenient way to get the same ideas, it is not the only way.

The mesh is now critical infrastructure.
If your certificate authority goes down for long enough, certs expire and services stop trusting each other. Monitor it like you monitor your database.

The 30 second version

If you get asked this in an interview, here is the whole thing in one breath:

Give the service a ClusterIP and no public address, so the internet cannot reach it.
Add a default-deny NetworkPolicy with an allow list, so only the right pods can connect.
Turn on mutual TLS, in STRICT mode, so callers prove who they are with short-lived certificates.
Add an authorization policy, so even a verified caller can only do what it is supposed to do.
Then mention that none of this authenticates the end user, and you will have said something most candidates do not.

"Put it in a private network" was never wrong.

It was just the first gate of four.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

One Rust Unicode Bug, Three Coding Models: A Reproducible Comparison

Athreya aka Maneshwar — Sun, 18 Oct 2026 05:10:27 +0000

Coding-model leaderboards reveal little about how a model handles a real bug. A Rust function can work on ASCII and still panic on valid Unicode because it slices UTF-8 at the wrong byte boundary. The fix must respect character boundaries and preserve the &str return type. This article uses one reproducible bug, one prompt, and the same regression tests to compare three models on the quality of their patches, not the confidence of their explanations.

A one-line Rust slice can panic on valid UTF-8. That makes it a useful coding-model challenge. The fix must preserve the function signature, handle character boundaries, and pass regression tests. This comparison uses the same bug for three models and judges their patches using compiler output rather than leaderboard scores.

The bug

The function below is supposed to return the first n characters of a string. It works on plain ASCII and panics when n cuts through a multibyte UTF-8 character.

pub fn prefix(input: &str, n: usize) -> &str {
    if n >= input.len() {
        return input;
    }
    &input[..n]
}

#[cfg(test)]
mod tests {
    use super::prefix;

    #[test]
    fn ascii() {
        assert_eq!(prefix("rustacean", 4), "rust");
    }

    #[test]
    fn unicode() {
        assert_eq!(prefix("café", 4), "café");
    }

    #[test]
    fn beyond_length() {
        assert_eq!(prefix("é", 10), "é");
    }
}

The original implementation counts bytes, not characters. In UTF-8, the final é in café occupies two bytes. Slicing at byte index four cuts through that character and panics. Save the code as src/lib.rs in a new Cargo library project. Run cargo test to reproduce the failing Unicode case.

Three models, one task

The three candidates are DeepSeek V4 Pro, Claude Fable 5.1, and Grok Build 0.1. These are the target model pages for the comparison. Model availability, exact API identifiers, and supported parameters must be checked against the live catalog before execution.

Model	API identifier	Input	Version
DeepSeek V4 Pro	Confirm from catalog	Same Rust file and prompt	Record API version
Claude Fable 5.1	Confirm from catalog	Same Rust file and prompt	Record API version
Grok Build 0.1	Confirm from catalog	Same Rust file and prompt	Record API version

Use the same request body, system instructions, maximum output allowance, and test environment for every model. Record the exact model ID returned by the API. A zero temperature reduces randomness but does not guarantee identical answers on repeated calls.

The prompt

Use the following prompt unchanged for every model.

Fix the Rust function below. It must return the first n Unicode scalar values without panicking. Keep the return type &str. Explain the bug briefly, return a complete src/lib.rs, and add a regression test for an emoji and a combining character. Do not add external dependencies.

Append the entire original src/lib.rs after that instruction. Do not tell one model about another model's failure. Do not silently repair a response before testing.

What to inspect in each model's patch

DeepSeek V4 Pro

Capture the raw response, its reported input and output tokens, the elapsed request time, and the first cargo test output. A correct patch should iterate over char boundaries rather than slice at an arbitrary byte index. If the returned code does not compile, retain the error and count the next prompt as another attempt. Add a screenshot of the response and terminal output from the actual run.

Claude Fable 5.1

Use the identical source and prompt. Record whether the model preserves the borrowed &str return type and whether it handles a request longer than the character count. Check the combining-character case carefully. Unicode scalar values are not the same as grapheme clusters. A request for one scalar value from an e followed by a combining accent may return only the e. That is correct under this task's definition.

Grok Build 0.1

Repeat the procedure without editing the prompt. Review whether the model changes the API, introduces a dependency, or returns a String instead of &str. These changes matter even when the visible output looks correct. Keep the first attempt and any follow-up attempts as separate records.

What a correct fix looks like

One valid approach uses char_indices to find the byte boundary after the requested number of scalar values.

pub fn prefix(input: &str, n: usize) -> &str {
    let end = input
        .char_indices()
        .nth(n)
        .map(|(index, _)| index)
        .unwrap_or(input.len());
    &input[..end]
}

This reference implementation is a correctness oracle for the exercise, not an attributed output from any model. The test suite should include ASCII, multibyte characters, empty strings, n equal to zero, n greater than the character count, emoji, and decomposed accents. The reference code should be kept out of model prompts.

Comparison scorecard

Model	First attempt passes	Attempts	Wall time	Input tokens	Output tokens	Notes
DeepSeek V4 Pro	Enter from cargo test	Enter from logs	Enter from logs	Enter from API	Enter from API	Compiler and behavior notes
Claude Fable 5.1	Enter from cargo test	Enter from logs	Enter from logs	Enter from API	Enter from API	Compiler and behavior notes
Grok Build 0.1	Enter from cargo test	Enter from logs	Enter from logs	Enter from API	Enter from API	Compiler and behavior notes

The scorecard becomes useful once each row is populated from the first response, Cargo output, and API logs. Until then, it is a repeatable evaluation worksheet rather than a model ranking.

What to pick for each coding task

For short patches, first-pass compilation and correct edge cases matter more than a polished explanation. For larger refactors, examine how often a model changes unrelated code. For code review, test whether it identifies the byte-versus-character distinction without prompting. For terminal automation, compare how predictably each model follows the required output format.

The best LLM for coding is task dependent. One Rust bug cannot establish which model performs best across a whole repository, language, or toolchain. It can reveal whether a model handles a concrete failure mode under controlled conditions.

How to repeat the comparison

Create a Cargo library with cargo new unicode-prefix --lib. Replace src/lib.rs with the buggy example and run cargo test. Send the same prompt and file to each confirmed model identifier. Save the raw JSON response, request timestamp, token usage fields, and model version. Extract the proposed code into a clean working tree for each attempt. Run cargo fmt --check, cargo test, and cargo clippy. Record failures without modifying the first answer.

Keep the repository organized as follows:

baseline/src/lib.rs
oracle/src/lib.rs
prompts/task.txt
runs/<model>/response.json
runs/<model>/cargo-test.txt

Publish a repository link only after the files are uploaded. The reproducible artifact is the code and logs, not a ranking graphic.

A shareable comparison includes the repository, unedited first responses, and Cargo logs. Readers can inspect both the proposed fixes and the failures.

We Rendered a HQ Demo Video on a Free Farm Run by Strangers

Athreya aka Maneshwar — Wed, 14 Oct 2026 11:08:09 +0000

Last post, my four-laptop render farm lost a race against my own laptop.

I promised you the ending.

Here it is.

It involves a free farm run by strangers and a Java file called sheepit-autoupdate.jar.

The lesson I was too slow to learn

If you missed it, last time we turned four laptops into a Flamenco render farm.

Athreya aka Maneshwar

Oct 12

We Built a Blender Render Farm Out of Four Laptops

#blender #networking #tailscale #devops

16 min read

It worked.

And then the farm spent almost all of its time downloading a 115 MB .blend over my 5 Mbit/s home upload, while my laptop finished the job alone.

The takeaway was one sentence:

A render farm is a file distribution problem wearing a GPU costume.

So the question for round two was not "where do I find more GPUs".

It was "who can I hand a file to once, and have it spread out without my upload being the highway".

Meet the flock

SheepIt is a free, community-powered render farm for Blender.

The deal is beautifully simple.

You install a small client on your machine.

While your machine is idle, it renders other people's frames.

That earns you points.

You spend the points to have the community render yours.

Everyone lends a machine, everyone gets frames.

I did not build a farm.

I joined a flock.

The client is a single Java jar (sheepit-autoupdate.jar, which updates itself, hence the name), and the upload side is a web page where you drop a zip.

That is the entire infrastructure I had to run.

No Samba, no tailnet, no NAT archaeology.

The file goes to one place, and it is their problem to hand it to a hundred machines.

Points: the currency of the flock

SheepIt runs on points.

You earn them by rendering other people's frames.

You spend them when you order your own.

The core loop needs no money, just render time in and render time out.

I registered on October 10th, the same Saturday the laptop farm gave up, and left the client running.

Here is our profile a few days later:

Let me read that back to you.

Our laptops spent 2 days and 7 hours rendering strangers' frames.

That is 6,981 frames for other people.

And in return I ordered 18,651 frames across 17 projects.

I ordered almost three times as many frames as I rendered, and the balance still says 1,261,589 points.

I cannot give you SheepIt's exact exchange rate, so treat this as an observation, not a formula.

As far as I can tell, a light 360p frame is cheap to order, a heavy 1080p ray traced frame costs more, and a stronger GPU earns faster.

What I can show you is the shape of it:

The points line only goes up.

Four days of render streaks, and I never ran dry.

The frames line flattens after the first couple of days, which is my guess for the laptop getting busier with my own work.

That is the part I did not expect.

I thought I was borrowing a farm.

I was joining a barter economy where the same laptop that edits Miles during the day renders somebody else's sheep at night.

Everyone is the customer and the supplier at the same time.

The catch: strangers cannot run your scripts

There is exactly one thing SheepIt asks of you, and it is the whole engineering problem.

Your .blend has to run on a machine you have never seen, unattended, with none of your project around it.

My scenes were not built for that.

They are built by Python scripts that load a city, retarget animations, key thousands of values and bake the result into a .blend.

On my laptop that file leans on a whole project folder:

Absolute paths to textures, like /home/lovestaco/pers/lr_demo/....

Image sequences for the animated screens in the city (the bugs board, the queue counters).

Sound strips for the voice-over.

Render settings that my own scripts apply at render time.

A stranger's machine has none of that.

It cannot run my scripts, so settings I apply from code simply never happen.

So I wrote a packer, sheepit_pack.py, whose job is to turn a built scene into a hermetic one:

Pack every image into the .blend.

Textures travel inside the file now.

Copy image sequences next to it with relative paths.

Blender cannot pack a sequence, so the folder rides along in the zip and the paths become //assets/....

Drop the sound strips.

The voice-over and sound effects are mixed later with ffmpeg anyway, so the farm never needs them.

Bake the settings into the file.

1920x1080 at 100%, 64 samples, ray tracing at full resolution, motion blur on, PNG output.

Then zip it with the .blend at the root.

My zips came out between 130 and 240 MB.

SheepIt takes up to 2 GB, so that is a comfortable fit.

Trust nothing, verify everything

Here is the thing about sending your scene to strangers.

If it is broken, you find out hours later, when a stranger's machine has already burned your points on a hundred identical black frames.

So every package goes through a gate before it leaves my laptop.

flowchart LR
    A[Built .blend] --> B[Pack<br/>images, relative paths, baked settings]
    B --> C[Zip<br/>blend at the root]
    C --> D[Extract into a clean folder]
    D --> E{Verify<br/>paths, settings, no sound}
    E -->|fail| X[Fix and repack]
    E -->|pass| F[Render 1 or 2 test frames<br/>from the extracted copy]
    F --> G[Upload the zip]
    G --> H[Community renders]
    H --> I[Download the frames zip]
    I --> J{Frame check<br/>count, gaps, size, black frames}
    J -->|fail| Y[Re-queue those frames]
    J -->|pass| K[Assemble with audio and subtitles]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef mine fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef farm fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef bad fill:#ff9a5c,stroke:#c4642a,color:#1a1a1a
    class E,J decision
    class A,B,C,D,F,K mine
    class G,H,I farm
    class X,Y bad

The important part is the extraction.

I do not test the original .blend.

I unzip the package into a fresh temporary folder, the way the farm will see it, and run the checks there.

Every external file must exist and be a relative path.

No sound strips may be left.

The resolution, samples, ray tracing, motion blur and PNG format must all match what I think I baked.

Then I render one or two real frames from that extracted copy, with the baked settings.

If those frames look right, the package is right.

On my GTX 1650 those test frames take between 7 and 35 seconds at 1080p.

Which is a quick way to tell the difference between "this will take a day" and "this will take a week" before anyone else spends a point on it.

I run it as one command per part, and it stops with a loud error at the first problem.

It also never deletes anything.

An old package is moved aside, not overwritten.

The bug where the file was fine and the check was wrong

The first time I ran the new packager for part 1, it died at the zip step with ERROR: street_part_1_final.blend is not at the zip root.

It was at the zip root.

I looked at the zip.

The file was right there, first entry.

The check looked like this, with the fix underneath:

set -euo pipefail

# broken: dies on SIGPIPE even though the file is in the zip
unzip -Z1 "$ZIP" | grep -qx "street_part_1_final.blend" || die "not at the zip root"

# fixed: capture first, then grep
LIST=$(unzip -Z1 "$ZIP")
grep -qx "street_part_1_final.blend" <<<"$LIST" || die "not at the zip root"

grep -q is polite about one thing: the moment it finds a match, it exits.

That leaves unzip writing the rest of the file list into a pipe nobody is reading, so it gets SIGPIPE.

And pipefail says that if any stage of the pipe fails, the whole pipe failed.

So the file was found, and the check reported it missing.

The fix is to stop piping.

If you ever write producer | grep -q under set -o pipefail, remember this.

It works until the producer has more output than the pipe buffer, and then it fails on the one day it matters.

Render only what the film keeps

This part is satisfying.

The film is not one long take.

Parts of the Blender scenes are cut from the final video (we removed a mural scene, the LED ticker and a subway lightbox), and the 2D product boards in part 3 are not rendered in Blender at all.

If you upload a scene and ask for "frame 1 to the end", the community renders all of that for you, for free, and you throw half of it away.

Rude to the sheep.

So each package gets uploaded with exactly the frame ranges the film needs.

Part 0 is one range, 458 frames.

Part 1 is three ranges, because three scenes are cut.

That skips about 535 frames that never reach the final video.

Part 2 is one range, 2,441 frames.

Part 3 is the same zip uploaded twice, frames 1 to 664 and 2150 to 3661.

The 2D boards sitting between them come from a Python script on my laptop in a couple of minutes.

The whole film is 8,229 frames, or 4 minutes 34 seconds.

Last post I said 7,300.

The film grew.

Films do that.

Assembling what comes back

Each part comes home as a zip of PNG frames.

A 1080p part is between 2 and 4 GB.

I do not trust anything that arrives from a farm of strangers.

So before a single frame goes into a video, a check script looks at the zip:

Is the zip intact?

Is the frame count exactly what the build says it should be, numbered from the first frame to the last, with no gaps and no extras?

Does every PNG decode, at exactly the expected size?

Is any frame entirely black?

An all-black frame is the farm's way of telling you that a worker had a bad day.

If everything passes, the frames are hard-linked into a numbered folder (no copies, so a 4 GB set costs nothing extra), ffmpeg encodes the base video, and my assembly script mixes in the voice-over, sound effects, music, subtitles and the frame-number burn-in.

That last one is for the team.

When someone says "the thing at f 1201 is off", I can find it.

Stale frames beat missing frames

My render scripts skip frames that already exist, which is wonderful when a long render gets interrupted.

It is terrible after you change the scene.

Stale frames from the old version sit in the folder, the renderer says "already done", and your new film has the old shot spliced in.

The rule I follow now: when the scene changes, the old frame folder gets moved aside (not deleted, never wiped), and the render starts into a clean one.

A draft at home, the real thing at the farm

Here is the workflow that has been settled for the project, and it is the actual answer to the render bill from last post.

Drafts are local.

360p, EEVEE, no ray tracing, no motion blur.

About 0.7 seconds per frame, so part 2's 2,441 frames take roughly 30 minutes.

The whole film is about two hours.

Make chai, come back, watch Miles.

Finals are on the farm.

1080p, 64 samples, ray traced reflections, motion blur.

A day of laptop time, spread across a flock that does not need me.

And between them sits the thing that makes drafts useful: a review copy the team can actually open.

Did Miles get his 1080p?

Yes.

Two of the four parts have already come home at 1080p from the flock, a 2 GB zip of PNGs for the opening and 3.7 GB for part 2.

Then the team's notes arrived and the film changed under them.

So there is a fresh set of packages sitting verified and ready for the community, each one with a test frame already rendered from the extracted copy.

What I would tell last week's me

Do not build a farm when a community already runs one.

Four laptops taught me a lot about networking.

A flock taught me that I had been solving the wrong problem.

Make your scene hermetic before you make it fast.

If the file only works on your machine, it works on one machine.

Verify the package, not the original.

Extract it into a clean folder and render a frame from that.

Keep a cheap draft loop.

It is the thing that lets a teammate say "this word is wrong" without it costing you a day.

And a thank you to the strangers whose GPUs are going to render the final.

You are not a sheep.

You are the shepherd.

I would love to hear from you in the comments.

Have you ever lent your machine to a community farm, or used a community compute network, for a project?

Follow along, because next time I will show you the final cut.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

We Built a Blender Render Farm Out of Four Laptops

Athreya aka Maneshwar — Mon, 12 Oct 2026 17:06:37 +0000

We built a render farm out of four laptops on a Saturday evening.

It had a job queue, a web dashboard, a private network spanning three cities' worth of ISPs, and a shared drive that every machine could see.

It worked.

And then it lost a race against my own laptop, rendering alone.

This is the story of that farm, the networking rabbit hole it dropped us into, and why a 130 MB file taking seven minutes to cross the internet ended the whole experiment.

Where we left Miles

If you have been following along, you know the setup.

In part one I rigged a Spider-Man model in Blender without ever having opened Blender before.

Athreya aka Maneshwar

Oct 3

I Made Spider-Man Swing Without Animating a Single Frame

#blender #mcp #ai #beginners

13 min read

In part two Miles moonwalked off a billboard and a Python test caught it.

Athreya aka Maneshwar

Oct 5

Spider-Man Moonwalked Off a Billboard and Only My Test Suite Noticed

#blender #python #gamedev #debugging

14 min read

Since then, the film has grown into a roughly four minute LiveReview explainer in three parts.

That is about 7,300 frames at 30 fps.

And every one of them has to be rendered by something.

360p is a lie we tell ourselves

For review drafts, I render at 360p with EEVEE, Blender's real-time engine, at 8 samples with ray tracing and motion blur switched off.

On my laptop's GTX 1650 that comes out to about one second per frame.

A single part renders in 20 to 50 minutes.

Go make chai, come back, watch Miles swing.

Totally fine.

Then you want the real thing.

1080p, ray traced reflections on the wet street, proper sample counts.

So I asked Claude to stop guessing and time actual frames from our scene.

The 1080p EEVEE versions we actually wanted sit between 10 and 22 hours for the whole film.

Cycles, the path tracer, came in at 76 to 90 seconds per frame.

That is a week.

Of my laptop doing nothing else.

With 6 GB of disk left.

"Is it not even using the GPU?"

That is the first question anyone asks, and it is a fair one.

So we sampled nvidia-smi during a render.

Blender was on the card, using about 1.5 GB of the 4 GB VRAM, at roughly 44% utilization.

At 360p the GPU finishes its part of the frame quickly and then sits there waiting for the CPU, which is busy posing the characters, evaluating the crowd and writing the JPEG.

So the next obvious move was to run more Blender processes in parallel.

Two workers were only 1.3x faster than one.

Three workers were slower than one, because three Blender processes fighting over eight cores and one GPU is not parallelism, it is a custody battle.

The only free speed-up we found locally was rendering each frame range as a single animation call instead of calling render() once per frame, which skips redoing scene setup every time.

That bought about 22% on one worker.

Nice, but nobody turns a 22 hour render into an overnight one with a 22% discount.

The conclusion was pretty clear.

We did not need a faster script. We needed more GPUs.

And there were three other laptops in the team, mostly idle on a Saturday evening.

Enter Flamenco

Flamenco is Blender's own open source render farm software.

It is a lot smaller than the phrase "render farm software" suggests.

There are exactly two programs:

Flamenco Manager is the brain.

It holds the job queue, serves a web dashboard, and splits each job into tasks.

Flamenco Worker is the muscle.

It runs on each rendering machine, polls the manager for work, launches Blender in the background, and reports progress.

Both are single Go binaries you download, extract and run.

No database server, no Kubernetes, no YAML archaeology.

When you submit a job, say "render frames 2370 to 2429 of this .blend", the manager cuts the range into chunks of N frames.

Each chunk becomes a task.

Every idle worker grabs the next task, renders it, and asks for another, so a fast machine naturally ends up doing more of the work than a slow one.

While a task runs, the worker keeps pinging the manager with a "may I keep running?" check, which is how cancelling a job actually reaches a laptop in another city.

If a task fails, it gets retried on a different worker.

After three failures on the same job, that worker gets blocklisted for the job so it stops eating chunks.

Here is the farm we built:

flowchart LR
    ME[My laptop<br/>Manager + Worker]
    SMB[(spidey share<br/>Samba)]
    TS{{Tailscale<br/>private network}}
    SH[Windows desktop<br/>later WSL]
    LI[pop-os laptop]
    RU[ASUS TUF laptop]

    ME --- SMB
    ME <-->|jobs + status| TS
    TS <--> SH
    TS <--> LI
    TS <--> RU
    SH -.->|reads .blend, writes frames| SMB
    LI -.->|reads .blend, writes frames| SMB
    RU -.->|reads .blend, writes frames| SMB

    classDef host fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef net fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef store fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef peer fill:#ff9a5c,stroke:#c4642a,color:#1a1a1a
    class ME host
    class TS net
    class SMB store
    class SH,LI,RU peer

Notice the dotted lines.

That is the part that would end up mattering.

The one assumption Flamenco makes

Flamenco does not move your files around.

The shared storage docs are upfront about it: the manager and every worker must be able to see the same files, on a NAS or a file server or some other shared drive.

The job just says "open this path", and it is your problem to make sure that path exists everywhere.

On four laptops scattered across home connections, that sounds like a strange requirement.

To be fair to Flamenco, a studio render farm lives on one fast LAN with one NAS, and this is a completely reasonable design there.

Our "LAN" was four laptops on four home connections.

Which brings us to the networking.

Tailscale: a LAN made out of the internet

My first worry was the obvious one: does this mean my machine has to push everything over my internet connection?

It does not.

Tailscale just creates a private network so all our laptops can talk to each other.

Every machine runs the Tailscale client and logs into the same account (a "tailnet").

Each machine gets a stable private address in the 100.x.y.z range, and every pair of machines talks over an encrypted WireGuard tunnel.

Tailscale's servers only do the coordination: who is in the network, what their public keys are, where they might be reachable.

Your actual traffic is supposed to go machine to machine.

"Supposed to" is doing a lot of work in that sentence, and we will get there.

On my laptop, Claude set up a Samba share called spidey and kept it bound to loopback only, so it was never exposed to my Wi-Fi or the internet.

Then Tailscale itself forwarded the SMB port, so only machines on the tailnet could reach it:

# on my laptop: Samba listens on 127.0.0.1 only,
# Tailscale forwards tailnet traffic on 445 to it
tailscale serve --bg --tcp 445 tcp://127.0.0.1:445

# on each Linux worker: mount it at the same path everywhere
sudo mount -t cifs //100.76.212.8/spidey /mnt/spidey \
  -o credentials=/root/.spidey_smb,uid=$(id -u),gid=$(id -g),vers=3.0

The first milestone was gloriously low-tech.

Someone created a file from Windows.

I created one from Linux.

Both files showed up on both machines.

We celebrated a text file like it was a product launch.

The path boss fight

Having a shared folder is step one.

Getting Blender on three operating systems to agree on what is in it is the actual game.

Paths baked into the .blend

Claude inspected street_part2_b.blend and found 31 external asset references.

25 of them were absolute paths starting with /home/lovestaco/pers/lr_demo/....

The Windows desktop has no /home/lovestaco.

Neither does the pop-os laptop.

The fix was to copy the project into the share and convert every reference to a Blender relative path (//assets/...), resolved from wherever the .blend itself lives.

After that, all three street files had zero absolute paths left.

Paths in the job itself

The .blend location and output folder in a Flamenco job are a separate problem, and Flamenco does solve this one with two-way variables.

The manager stores job paths with a placeholder, and each worker swaps in its own platform's value:

# flamenco-manager.yaml
variables:
  spidey:
    is_twoway: true
    values:
      - platform: linux
        value: /mnt/spidey
      - platform: windows
        value: D:\spidey

Two gotchas we learned the hard way.

On Windows, Flamenco wants a drive letter, not a \\server\share UNC path.

And on Linux, every machine has to use the same mount point, because there is only one Linux value.

That is why everyone mounted at /mnt/spidey, and why the wizard rejected ~/spidey outright (it does not expand ~).

Creating D:\spidey with mklink also failed, which turned out to be because mklink is a builtin of the old Command Prompt, not of PowerShell.

cmd /c mklink ... from an admin window fixed it.

Where is Blender?

Then the Windows worker printed this:

There was an error finding Blender on this system.

Blender was on PATH.

The shell had been restarted.

It still could not find it.

The Blender variable docs explain why: on Windows the worker looks up which program is associated with .blend files, rather than trusting PATH.

The fix was gloriously petty.

Create an empty junk .blend file, use it to set the file association, and the worker immediately reported Blender 5.2.2 LTS.

By 8:15 pm we had four workers awake on the dashboard:

image pending: Flamenco web dashboard showing four workers awake and the farm status idle

My laptop, a Windows desktop, a pop-os laptop and an ASUS TUF gaming laptop.

All on Blender 5.2.2.

An actual render farm.

On a Saturday.

The first job

We kept it small on purpose.

60 frames of part 2b at 360p draft quality, split into three chunks of 20.

Three workers picked up a chunk each within seconds.

Frames started landing in the shared folder.

And the farm genuinely worked.

When the job finished, Claude compared frames rendered by three different machines and they were identical: same lighting, same colours, same Miles.

That part is honestly very cool.

Then we looked at how long everything took.

My worker read the 115 MB .blend off its own disk in a few seconds and rendered at about 0.4 s per frame.

The two relayed laptops spent over three minutes just opening the file before rendering a single frame.

A 20 frame chunk at 360p renders in seconds.

So the farm was spending almost all its time downloading and almost none of it rendering.

"Missing DNA block"

The Windows worker did something more dramatic.

It spent 82 seconds reading the file and then Blender died with:

ERROR Failed to read blend file 'D:\spidey...\street_part2_b_360p.blend': Missing DNA block

This error is a great little window into how .blend files work.

A .blend is basically a memory dump.

It is a header followed by a long run of data blocks, each one a chunk of Blender's in-memory structs written straight to disk.

To read those structs back, Blender needs to know their layout, which fields, which types, which offsets, and that description lives in a special block called DNA1 (the "SDNA").

That is how a file saved years ago, by a Blender with differently shaped structs, can still be opened today.

The catch is that DNA1 gets written near the end of the file.

So if a read dies partway through, Blender never reaches the block that explains everything before it, and you get "Missing DNA block".

We checked the obvious thing first.

The copy on Windows had the right size, 114,783,416 bytes, and once its SHA-256 finally finished (that alone took ages over the link), the hash matched mine exactly.

The file was fine.

The read was not.

The chunk size argument

This sparked the best debate of the night.

Claude suggested bigger chunks, so each worker would pay the slow load less often.

The obvious pushback: wouldn't smaller chunks download faster, in more pieces, spread across more machines?

The intuition makes sense if chunks map to pieces of the file, like a torrent.

They do not.

A chunk is a range of frames, but every chunk opens the entire scene.

Smaller chunks mean more full reads, not smaller ones.

The next question was a better one though: once the file has been read, isn't it done? Shouldn't only the first read be slow, since the OS caches the file?

So we ran a second job with 10 frame chunks and watched the ASUS laptop take its second chunk.

The first read took 3 min 02 s.

The second took 31 seconds.

Caching was real.

Not instant, but real.

On Windows, though, the cached retry failed with the same "Missing DNA block", so that SMB client and Blender were clearly not getting along.

Moving that worker into WSL got past the error, and it promptly rendered at about 50 seconds per frame, which smells a lot like EEVEE running without access to the GPU.

A side quest: whose GPU is it anyway

Even after loading, the remote workers were slow per frame.

The pop-os laptop took about 2.3 s per frame against my 0.4 s, and its GPU sampling step alone was over 3x slower than mine.

Its lspci explained it: an NVIDIA GTX 1650 Mobile, and an AMD Radeon Vega built into the Ryzen processor.

Two graphics chips, like most gaming laptops.

And glxinfo said OpenGL was running on the AMD integrated chip by default, because that is what laptops do to save battery.

EEVEE was happily rendering Miles on the weak chip while the GTX 1650 sat there.

The fix is PRIME render offload, starting the worker with __NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia so Blender lands on the NVIDIA card.

Worth knowing if you ever farm out renders to other people's laptops.

Their GPU is not necessarily the GPU Blender gets.

Direct vs DERP (and what on earth is "blr")

Back to the real problem, the reads.

tailscale status on my machine had one telling word at the end of each peer's line.

The WSL worker was connected direct.

The pop-os and ASUS laptops were going through relay "blr".

Here is what those two words actually mean.

Direct

Tailscale's first choice is a direct WireGuard tunnel over UDP between the two machines.

The hard part is that almost nobody's laptop has a public IP.

You are behind your home router's NAT, and quite often behind your ISP's NAT too.

So Tailscale does NAT traversal, also known as hole punching.

Each machine discovers what its public IP and port look like from the outside, the coordination server shares those with the peer, and both sides fire UDP packets at each other at the same time.

Each router sees an outgoing packet, creates a mapping for the reply, and if the timing and ports line up, the two outgoing "holes" meet in the middle.

Tailscale's How NAT traversal works is one of the best networking posts on the internet if you want the full version.

DERP

When hole punching fails, Tailscale falls back to DERP, short for Designated Encrypted Relay for Packets.

DERP servers are relays that Tailscale runs around the world.

Your packets go to the relay over HTTPS and it forwards them to the other machine.

They stay WireGuard-encrypted end to end, so the relay cannot read them, it just shuffles opaque bytes.

DERP regions are named after airport-style city codes.

"blr" is Bengaluru.

So "relay blr" just means "that laptop's packets to me were taking a detour through Tailscale's Bengaluru relay".

Direct is faster for the reasons you would guess.

A relay adds an extra hop, it is a shared server rather than your own path, and it runs over TCP, so a congested relay hurts you in ways a direct UDP path does not.

Pings from the pop-os laptop through blr ranged from 58 ms to 415 ms, and one test run stalled after receiving about 159 MB, with even a plain ls on the share hanging for two minutes.

Why it could not go direct

tailscale netcheck on the pop-os laptop said:

* UDP: true
* IPv6: no, but OS has support
* MappingVariesByDestIP: true
* PortMapping: UPnP
* Nearest DERP: Bengaluru (blr: 48.6ms)

The important line is MappingVariesByDestIP: true.

That is a "hard" NAT.

Its router assigns a different public port for every destination, so the port my laptop learns about is not the port that router will actually accept traffic on from me.

Hole punching needs at least one side to be predictable.

Mine was the easy kind, but my router offered no UPnP or NAT-PMP to open a port automatically, and that side had no IPv6 to sidestep the whole mess.

Here is roughly how Tailscale decides:

flowchart TD
    A[Two peers want to talk] --> B{Either side has<br/>a public, open port?}
    B -->|yes| D[Direct WireGuard over UDP]
    B -->|no| C{Both NATs map ports<br/>predictably?}
    C -->|yes| E{Hole punch succeeds?}
    C -->|no, hard NAT| F{IPv6 on both sides?}
    E -->|yes| D
    E -->|no| G[Relay through nearest DERP]
    F -->|yes| D
    F -->|no| G

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef good fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef bad fill:#ff9a5c,stroke:#c4642a,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    class B,C,E,F decision
    class D good
    class G bad
    class A start

The suggested fix was forwarding UDP 41641 on my router so the hard NAT side would have an open door to knock on.

The twist: direct did not save us either

Then we noticed something awkward.

The WSL worker was on a direct connection.

And it still took about six minutes to load the file.

So we measured my connection.

0.65 MB/s upload.

About 5 Mbit/s.

Every worker in this farm was reading the .blend from my laptop.

Which means every byte, direct or relayed, had to squeeze out through my upload.

At 0.65 MB/s, one 115 MB read takes about three minutes.

Two workers reading at once, about six.

That worker took in 136 MB in roughly six minutes, about 0.38 MB/s, which is exactly what half of my upload looks like.

Fixing the hard NAT would have shortened the detour.

It would not have widened the pipe.

I switched to a faster Wi-Fi network for one more run, which bumped my upload to about 2.15 MB/s, and the remote first loads dropped to about a minute and a half.

Better.

But in that same run my laptop alone finished four of the six chunks while the remote workers were still loading.

Why we pulled the plug

For a remote worker to help, two things have to be true.

It has to get the scene fast, and it has to render fast once it has it.

Ours had neither.

The scene came through my home upload, partly via a relay, and the laptops that did load it were either on the wrong GPU, in WSL with no GPU, or tripping over SMB on Windows.

Every fix we found was real, and every one of them was its own project.

Copy the project to every laptop ahead of time and lose the "one shared folder" simplicity that made the setup appealing.

Fight every home router.

Fix every laptop's GPU setup.

Make sure every laptop stays awake, plugged in and on Wi-Fi for ten hours.

So at 9:52 pm we shut it all down.

Flamenco manager and worker stopped, Samba stopped and disabled, the Tailscale forward turned off, Tailscale itself stopped.

Nothing deleted, just switched off.

A few things I would tell past me before this Saturday:

A render farm is a file distribution problem wearing a GPU costume.

The render was the easy part.

Getting the same 130 MB to every machine, reliably, was the whole job.

Measure your upload before you build anything.

Five minutes with a speed test would have predicted most of this evening.

"Direct" is about the route, not the speed.

A perfect peer-to-peer tunnel is still capped by the slowest pipe on either end.

Check which GPU Blender is actually using.

Dual-GPU laptops will quietly hand you the integrated chip.

Flamenco is fine.

It assumes a studio LAN with a NAS, and we gave it four home connections and a laptop pretending to be a NAS.

So did Miles ever get his 1080p?

We really wanted this one to work.

Four laptops, one Saturday, our own little render farm.

It is a great story, it just is not a fast one.

But the film still needs rendering, and we did eventually crack it, by spreading the work across multiple providers instead of multiple laptops.

That is the next post.

Follow along so you do not miss how we finally got Miles rendered.

And I would love to hear from you in the comments.

Have you ever tried to farm out a render, or any big compute job, across machines you did not own?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

The Stack I'd Need for Claude to Direct a Whole YouTube Video in Blender

Athreya aka Maneshwar — Fri, 09 Oct 2026 18:10:15 +0000

Two posts ago, I had never opened Blender.

Since then, Miles Morales has swung on a web, yanked a DEV board into frame, and moonwalked 6.5 metres off the edge of a billboard without anyone noticing.

Every one of those shots was me and Claude Code, pair programming a 3D scene one prompt at a time.

It works.

It is also slow, because I am still the one holding the whole movie in my head.

So this post is about the next question, the one I keep circling back to at 1 a.m.:

What would it actually take for me to type one paragraph and get a finished YouTube video out the other end?

Not "Claude helps me in Blender".

Claude directs the thing.

I went digging through every MCP server, Claude Code skill, mocap source and audio model I could find, checked the GitHub stars so we know what is battle tested and what is a weekend project, and wrote down the stack I would build.

Spoiler: Blender is maybe a third of it.

The job description: director, not animator

The single most useful thing I learned in part one was not about rigging.

It was about where the LLM is good and where it is hopeless.

Claude was great at writing bpy code, relinking broken textures, sequencing clips on a timeline, and stitching root motion between them.

Claude was never once asked to invent how a human body lands from a jump.

That came from Mixamo, which came from a real performer in a mocap suit.

And that split is the whole architecture.

An LLM is a brilliant writer of structure and code, a decent critic of a still image, and a terrible source of biomechanics.

So every part of a video should go to whoever is actually good at it:

If it moves like a human, fetch it.

If it is code, write it.

If it is a sound, order it from a model that does nothing but sound.

Here is the whole stack with that rule applied:

flowchart TD
    ME[One paragraph from me] --> DIR[Claude Code<br/>director skill]
    DIR --> SL[Shot list<br/>YAML]
    SL --> BR[Blender bridge<br/>one MCP server]
    SL --> MO[Motion library<br/>Mixamo, CMU, Rokoko]
    SL --> AS[Assets<br/>Poly Haven, Sketchfab, image to 3D]
    SL --> AU[Audio<br/>TTS, SFX, music]
    AU --> LS[Rhubarb<br/>mouth shapes]
    MO --> BR
    AS --> BR
    LS --> BR
    BR --> BL[Blender, headless<br/>EEVEE drafts, frame sequences]
    BL --> QA{Checks pass?<br/>eyes + asserts}
    QA -->|no| DIR
    QA -->|yes| FF[FFmpeg<br/>frames + stems]
    AU --> FF
    FF --> YT[YouTube-ready MP4]

    classDef human fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef brain fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef fetch fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef order fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a
    classDef engine fill:#6ea8ff,stroke:#2f62c4,color:#1a1a1a
    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a

    class ME,YT human
    class DIR,SL brain
    class MO,AS,LS fetch
    class AU order
    class BR,BL,FF engine
    class QA decision

Let's walk it bottom up, starting with the piece everyone argues about.

Layer 1: the Blender bridge (pick exactly one)

This is the MCP server that lets Claude reach into a running Blender and press buttons by writing Python.

There are now a lot of them.

Here are the four I looked at seriously, with stars as of this week:

Bridge	What it is good at
mcp-for-blender ⭐ (~30.2k) (formerly `blender-mcp`)	The one I used in parts 1 and 2. Code execution, viewport screenshots, plus Poly Haven, Sketchfab, Poly Pizza, Hyper3D Rodin, Hunyuan3D and Tripo built in
Official Blender MCP ⭐ by Blender Lab	Understanding a scene: explaining node setups, finding what uses a material, cleanup. Blender's page lists 5.1 or newer
carlosh7/blender-mcp ⭐ (8 stars)	248 tools across 14 categories, 371 commits, 479 passing tests, MIT
claude-blender ⭐ (5 stars)	Undo checkpoints, camera auto-framing, local TripoSR and Shap-E generation

The official one is official in the sense that Blender developers built it, and Anthropic announced it as a Claude connector this year.

Read its own page though, and it is pitched as a way to understand complex setups, not to churn out a film.

It also carries the most honest warning I have seen on any MCP page: the server "will execute LLM generated code in Blender without any guards in place", and Blender suggests a VM or a machine without sensitive data on it.

That applies to every server in the table, by the way.

They all ship some flavour of "run this Python", and that is the tool doing the real work.

Which brings me to the 248 tool one.

It is clearly a lot of careful work, with real tests and CI.

But it made me realise that tool count is not a feature.

Every tool an MCP server exposes comes with a schema: a name, a description, a JSON shape for its arguments.

Many clients put all of those in the context window on every single turn, whether you use them or not.

Some, Claude Code included, can defer them and search on demand, which fixes the space problem.

It does not fix the picking problem.

If there are five slightly different ways to move a cube, the model has to choose one, and every choice is a chance to choose wrong.

One execute_blender_code and a model that writes good bpy already covers all 248, because every button in Blender is a thin skin over bpy anyway.

So my pick is boring: mcp-for-blender as the one bridge, installed with claude mcp add blender uvx mcp-for-blender, and the official one kept around for "why is this scene slow" archaeology.

And I am stealing the best idea from the 5 star repo without installing it.

Checkpoints before risky edits are a great idea.

In part two the .blend file became a build artifact generated by scripts, so my checkpoints are just git commit.

Layer 2: skills, not more tools

If tools are expensive, how do you teach Claude cinematography?

With Claude Code skills.

A skill is a folder in .claude/skills/ with a SKILL.md inside: a one line description that is always visible, and a body of instructions and scripts that only loads when the task needs it.

It is the lazy import of prompting.

blender-skills (~143 stars) is a good example of the idea.

It has camera moves as skills, turntable, slow-zoom, dolly-rotate, crane-shot, perfect-loop, plus image to 3D through Meshy and a toolkit that does Mixamo retargeting.

It is aimed at product shots, so I would not install it wholesale for a superhero short.

I would copy its shape.

The thing I actually want is a small library of film vocabulary, each word backed by tested code:

establishing_wide, push_in, pull_out, orbit, crane_up, tracking, over_the_shoulder, handheld_shake
rainy_night, golden_hour, villain_red, neon_street, soft_closeup_key

Then "start wide, track him to the ledge, push in slowly as he says the line" stops being a creative writing exercise for the model.

It becomes three skill calls with numbers.

Same idea applies to lighting.

A preset that is right once is right forever, and nobody has to rediscover three point lighting at 2 a.m. because Miles's suit rendered as a black blob again.

Layer 3: motion, fetched not generated

This is where most "AI made my animation" demos fall apart, and it is the layer I am most opinionated about.

Do not let the LLM keyframe a human.

It will produce something that technically moves and spiritually does not.

Real motion has weight shifts, overshoot, a little settle at the end of a landing.

Mocap has all of that for free, because a person did it.

The sources I would wire in, in order of effort:

Mixamo. Free, auto-rigs a T-posed character from five markers, thousands of clips. I downloaded 78 of them one click at a time in part one, and I would do it again.

CMU Motion Capture Database. About 2,500 free clips, with a BVH mirror on GitHub (~203 stars). Needs a retargeter, which part one built and part two debugged.

Rokoko. Their Blender add-on is free and does retargeting. Their free plan includes Rokoko Vision, single camera AI mocap. That last one is the exciting bit: the motion Mixamo does not have, I can act out in my living room with a webcam.

And then the controls.

In part one I dragged Miles's hips down and his whole body sank through the floor like a ghost, because the rig had bones but no IK.

Two fixes, depending on budget:

Mixamo Rig is a free, GPL extension that builds a proper IK control rig on top of a Mixamo skeleton and bakes animation in and out of it. Check the version range on the listing before you build on it.

Auto-Rig Pro by Artell is the paid, serious option, about $50, used in shipped games like Manor Lords, with rigging, retargeting between skeletons and facial rigs in one place.

If I spend money anywhere in this stack, it is here.

Rigging is the one job where a mistake quietly poisons every shot that comes after it.

Layer 4: the face, or the close-up problem

Here is a thing I did not appreciate until I tried to frame a close-up.

In a wide shot, nobody looks at the face.

Then the camera pushes in for the line, and the face is the only thing on screen.

A mouth flapping open and closed on a timer reads instantly as fake.

So does a character who never blinks, or whose eyes are locked dead ahead like a mannequin's.

The good news is that most of facial performance can be fetched too, and the most important piece is wonderfully small.

Rhubarb Lip Sync (~2.6k stars) is a command line tool that listens to a voice recording and writes out a timeline of mouth shapes.

It uses nine shapes, A to H plus X for rest, the same small set classic cartoon studios used, and you can hand it the script with --dialogFile so it knows the words instead of guessing.

It says it is built for 2D, but the output is just JSON with timestamps, and a 3D face can use it just as well.

There is a Blender add-on for it (~228 stars), but it has not been touched since 2023.

The glue is short enough that I would rather have Claude own it:

# rhubarb -f json -d line.txt -o line.json line.wav
import bpy, json

cues = json.load(open("line.json"))["mouthCues"]
keys = bpy.data.objects["Miles_Head"].data.shape_keys.key_blocks
fps, start = bpy.context.scene.render.fps, 120   # line begins on frame 120

# Rhubarb's letters -> this model's shape key names
VISEME = {"A": "MBP", "B": "EE", "C": "EH", "D": "AA", "E": "OH",
          "F": "OO", "G": "FV", "H": "L", "X": "rest"}

for cue in cues:
    frame = start + round(cue["start"] * fps)
    for shape in VISEME.values():
        keys[shape].value = 1.0 if shape == VISEME[cue["value"]] else 0.0
        keys[shape].keyframe_insert("value", frame=frame)

That is the whole trick: audio in, a keyframe per mouth change out.

Blender's default Bezier interpolation eases between the keys, so the mouth glides instead of snapping.

If your model drives its mouth with face bones instead of shape keys, like the 143 bone face rig on my Miles, the same loop poses bones instead.

Blinks and eye darts are even cheaper.

A blink every two to six seconds, randomised, and a tiny eye flick toward whatever the character is reacting to, is a dozen lines of procedural keyframing, and it is the difference between a person and a wax figure.

Notice the dependency hiding here, too.

Lip sync needs the audio to exist first.

Which means the voice is not post production.

It is pre production.

Layer 5: assets, mostly fetched

Same rule again: generate only what nobody has made yet.

Poly Haven. Thousands of CC0 HDRIs, textures and models. CC0 means no attribution and no strings. mcp-for-blender searches and imports from it directly, and an HDRI alone fixes half of "this scene looks fake".

Sketchfab. Huge, and also where my Miles came from. Read every license, because they vary model to model.

Image or text to 3D, for the props nobody has made:

Hunyuan3D-2 (~15k stars) and Hunyuan3D-2.1 (~4.1k stars, adds PBR materials), from Tencent, runnable locally if your GPU is braver than my 4 GB GTX 1650
TripoSR (~7k stars), fast single image reconstruction
Hosted options like Hyper3D Rodin and Tripo, already wired into mcp-for-blender

One honest caveat from someone who just learned what skin weights are.

Generated meshes are usually a dense soup of triangles.

That is fine for a mailbox in the background.

It is not fine for anything that has to bend, because rigging wants clean edge loops around elbows and knees, and a triangle soup deforms like wet cardboard.

So: generated props, sourced or hand built heroes.

Layer 6: audio is half the movie

Every engineer I know who tries video makes the same mistake I did.

They spend three days on render settings and zero minutes on sound.

Put a whoosh on a jump, a thwip on a web shot and a low thud on the landing, and the exact same frames suddenly look like they cost ten times more.

The trick is to treat audio as separate stems, never one mixed track:

Dialogue. ElevenLabs for character voices. Generate it first, because the face needs it.

SFX and ambience. ElevenLabs sound effects turns a text description into a sound, up to 30 seconds per generation, with a seamless loop mode for rain and city hum, and 48 kHz WAV for one shots.

Music. Its own model, its own stem. stable-audio-tools (~3.9k stars) is Stability's open code for their audio models if you want local control. Check the license of whatever you pick before it goes on a monetised channel.

Separate stems are what make the edit programmable.

Music too loud under the line? Change one number.

Want the music to dip automatically whenever someone talks? That is FFmpeg's sidechaincompress filter, keyed off the dialogue stem.

None of that is possible once everything is baked into one WAV.

Layer 7: render like a build system, assemble with FFmpeg

Part two already turned the scene into a build artifact, so this layer is mostly "keep doing that, at movie scale".

Three rules:

Render headless. blender -b with no UI, driven by scripts, so a render is a command, not a person clicking.

Render frame sequences, not MP4s. If Blender crashes on frame 1,400 of a movie file, you lose the movie. If it crashes on frame 1,400 of a PNG sequence, you lose one frame and resume. Blender's own docs recommend exactly this.

Draft at 360p, final once. Part two's --fast mode, with ray tracing, motion blur and extra samples switched off, roughly halved frame time. Everything is reviewed in drafts.

Then FFmpeg (~64.9k stars, and it is only a mirror) glues frames and stems into a video:

# render shot 30 as a PNG sequence (flags before -a, Blender reads them in order)
blender -b build/sh030.blend -o //frames/sh030_#### -F PNG -s 1 -e 96 -a

# frames + three stems -> one mp4, music ducked under the dialogue
ffmpeg -framerate 30 -i frames/sh030_%04d.png \
  -i audio/dialogue.wav -i audio/sfx.wav -i audio/music.wav \
  -filter_complex "[1:a]asplit[d][key];[3:a][key]sidechaincompress=threshold=0.05:ratio=8[m];\
[d][2:a][m]amix=inputs=3:normalize=0[a]" \
  -map 0:v -map "[a]" -c:v libx264 -pix_fmt yuv420p -crf 18 -c:a aac sh030.mp4

asplit makes two copies of the dialogue, one to hear and one to act as the key that pushes the music down.

normalize=0 stops amix from quietly turning everything down to make room, which is a fun one to debug by ear.

-pix_fmt yuv420p is there because some players will happily show a black screen without it.

I would still do the final colour pass and titles in DaVinci Resolve for anything going on my channel, but a machine should be able to produce the rough cut alone.

Layer 8: two reviewers, because one of them is blind

The loop that made parts one and two work was Claude grabbing a screenshot, looking at it and fixing what it saw.

That loop is great.

It is also how a moonwalk got shipped.

Vision models review a still picture.

Motion bugs live between the frames.

The 22 cm pop in the middle of a crossfade, the single frame where Miles hovered at a cut, the stance wider than the ledge he stood on, none of those survive a look at frame 1,201.

They all fail an assertion that measures where the feet are on every frame.

So every shot gets both reviewers:

Eyes: Claude looks at a contact sheet of draft frames and judges framing, lighting, mood and readability.
Asserts: scripts measure contact, bounds, continuity at cuts and missing textures, and fail loudly.

If the asserts fail, nobody wastes time looking.

The missing piece: a director skill

Everything above is a department.

What turns departments into a studio is one skill that takes my paragraph and writes the plan everyone else works from.

That plan should be data, not prose:

title: rooftop_siren
fps: 30
character: original_hero_v2       # not Miles, see above
shots:
  - id: SH010
    seconds: 4
    set: rooftop_night_rain
    camera: [establishing_wide, push_in]
    motion: { clip: "Standing Idle" }
    audio: { ambience: "rain on a rooftop, distant traffic, wind" }
  - id: SH020
    seconds: 2.5
    camera: [extreme_closeup]
    motion: { clip: "Look Around", turn: 30 }
    dialogue: { voice: hero, line: "You hear that?" }
    face: { lipsync: rhubarb, blinks: auto, look_at: siren }
  - id: SH030
    seconds: 3.2
    camera: [tracking, low_angle]
    motion: { clip: "Jump Down", speed: 1.2 }
    audio: { sfx: ["cloth whoosh", "web thwip", "heavy landing on concrete"] }

Look at what is in there.

Every value is a skill name, a library lookup or a prompt for an audio model.

Nothing asks the LLM to invent a joint angle, and nothing asks it to remember the whole movie at once.

The director writes this once.

Then each shot runs through the same loop, and a bad shot is re-run on its own instead of the whole film:

flowchart LR
    S[Shot from the YAML] --> A[Generate dialogue<br/>and SFX]
    A --> L[Rhubarb<br/>mouth cues]
    S --> M[Fetch motion<br/>and set]
    L --> B[Build shot<br/>bpy scripts]
    M --> B
    B --> D[360p draft]
    D --> T{Asserts pass?}
    T -->|no| B
    T -->|yes| V{Claude likes<br/>the contact sheet?}
    V -->|no, fix camera or light| B
    V -->|yes| F[Final frames]
    F --> X[FFmpeg mux]

    classDef src fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef order fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a
    classDef fetch fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef build fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef out fill:#6ea8ff,stroke:#2f62c4,color:#1a1a1a

    class S src
    class A order
    class L,M fetch
    class B,D build
    class T,V decision
    class F,X out

It is CI for a movie.

Each shot is a job, the YAML is the config, the asserts are the tests, and Claude is the reviewer who also writes the fixes.

The order I am building it in

Not the order of the diagram, the order of the dependencies:

Bridge + skills. One MCP server, a camera and lighting skill library. Without reliable shots, nothing else matters.
Motion library. Mixamo and CMU behind one retargeter, Mixamo Rig for IK, Rokoko Vision for the clips nobody has.
Audio stems. Voice, SFX, music, all separate. This comes before the face, because the face reads the audio.
Face. Rhubarb glue, procedural blinks and eye targets, then close-ups stop being scary.
Asserts. Part two's checks, generalised to every shot.
Director skill. Only once every department works on its own. A director with no crew just writes very confident YAML.

And the question I will be honest about: will this make a Pixar short?

No.

What it can make is something I could not make at all a month ago: a 30 second scene with a character who moves like a person, talks with a mouth that matches the words, sounds like it happened somewhere, and was assembled by a machine while I wrote the next paragraph.

That is a wrap for part three.

If you have built any piece of this, especially the face, tell me what broke. I clearly have a lot of breaking ahead of me.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by tests…

View on GitHub

Click below to try LiveReview with your codebase:

Spider-Man Moonwalked Off a Billboard and Only My Test Suite Noticed

Athreya aka Maneshwar — Mon, 05 Oct 2026 20:26:54 +0000

Miles Morales moonwalked 6.5 metres past the edge of a billboard.

Not on the billboard.

Past it.

On nothing.

I had watched that shot more than once and did not notice.

The viewport did not notice either.

A 44 line Python script noticed, and it has been catching my mistakes ever since.

This is part two of my accidental Blender education.

Athreya aka Maneshwar

Oct 3

I Made Spider-Man Swing Without Animating a Single Frame

#blender #mcp #ai #beginners

13 min read

In part one I went from never having opened Blender to rigging a Spider-Man model, pulling animations from Mixamo and retargeting old Carnegie Mellon mocap onto him.

This one is about what happens next, when you try to turn that into an actual film and every mistake is suddenly a bug report.

The film, and why it is really a build system

The goal is a roughly 60 second explainer for LiveReview.

A masked Miles web-swings through a city at sunset, and every "slide" is part of the street.

A wall board, a rooftop billboard, an LED ticker, bus stop and subway lightboxes, a big street screen, and a runaway train for the finale.

Here is the twist that shapes everything below.

Nothing in the scene is hand edited.

The whole thing is generated headless by Python scripts against Blender 5.2, rendered with EEVEE on a 4 GB GTX 1650.

build/street.blend is not a file I open and tweak.

It is a build artifact, like a dist/ folder.

So every fix is a code change plus a rebuild, and the workflow ended up looking a lot more like CI than like art class:

flowchart LR
    S[Python scripts] -->|blender -b| B[street.blend]
    B --> C{support check clean?}
    C -->|no| S
    C -->|yes| D[--fast draft render]
    D --> T[review copy with timecode]
    T --> R{looks right?}
    R -->|no, f1201 is off| S
    R -->|yes| G[git commit]
    G --> F[full quality render]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef code fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef build fill:#6ea8ff,stroke:#2f62c4,color:#1a1a1a
    classDef render fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a
    classDef done fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a

    class C,R decision
    class S,G code
    class B build
    class D,T render
    class F done

Almost every lesson in this post fell out of that loop.

"Can't you just use the GPU to render faster?"

This was my first question once renders started taking a while.

The answer was a little embarrassing.

It already was.

EEVEE is a GPU rasterizer, basically a game engine renderer.

While it rendered, the card sat at about 80% utilisation and 2.2 GB of VRAM.

There was no idle GPU waiting to be switched on.

And the other renderer, Cycles, is a path tracer, which on a GTX 1650 would have been slower, not faster.

So the real question was never "CPU or GPU".

It was "what is each frame paying for?"

Three settings turned out to dominate:

Setting	What it costs	Draft value
Screen-space ray tracing	reflections on the wet street, the single biggest per-frame cost	off
Motion blur	renders extra time steps for every frame	off
Samples	anti-aliasing and soft shadows, linear in render time	8 instead of 16 to 24

So the render script grew a --fast flag that flips exactly those three:

fast = "--fast" in args          # draft: no ray tracing, no motion blur, 8 samples
...
if fast:
    sc.eevee.taa_render_samples = samples or 8
    if hasattr(sc.eevee, "use_raytracing"):
        sc.eevee.use_raytracing = False
    sc.render.use_motion_blur = False

At 360p that took a frame from about 0.6 s to about 0.33 s.

Roughly 2x.

The full 58 second film is 1,760 frames, so a review render went from around 18 minutes to around 10.

What you give up is real but harmless.

The street looks flatter without the ray-traced reflections, and fast motion looks crisper without blur.

Nobody reviewing "does Miles land on the ledge" cares about either.

The finals still get everything.

A few other levers came along for the ride:

A 270p option (480x270) for the quickest "is the blocking right" passes.
Contact sheets. A strip of stills at 25% resolution and 8 samples, before committing to any video at all.
Partial re-renders. 05_render.py -- full 33 <from_frame> re-renders only from a given frame onward and reuses the earlier frames, for when only the ending changed.

The lesson is the same one you learn profiling a slow API.

Before you ask for a bigger machine, find out what the work actually is.

Most of the time you are paying for something nobody looks at.

"How do I go back? Is there a snapshot button?"

Blender has more undo than you would expect, in layers:

Ctrl+Z and Ctrl+Shift+Z for in-session undo and redo, plus Edit → Undo History to jump back several steps at once.
File → Revert to throw away unsaved changes and reload from disk.
.blend1 backups. Every save keeps the previous version next to the file, and Preferences → Save & Load → Save Versions sets how many (docs).
File → Recover → Auto Save for when Blender crashes, which it will.

All useful, and all beside the point here.

Because when the .blend is generated, it is not the source of truth.

The scripts are.

So the real snapshot button is git:

git log --oneline -- miles/scripts/04_street.py    # pick a commit
git stash                                          # park current work
git checkout <commit> -- miles/                    # that version's scripts and assets
blender -b build/character.blend --python scripts/04_street.py   # rebuild that scene
git checkout HEAD -- miles/ && git stash pop       # and come back

If the scene is generated from code, version the code, not the binary.

I commit after every review round now, so any version I ever showed someone can be rebuilt exactly.

The old rendered videos stick around too (street_v1, v4a and friends), because sometimes you just want to play two versions side by side without rebuilding anything.

A reader comment that found a real bug

Part one got a comment from @lewisywliu that was genuinely better than some docs I have read.

Two tips, paraphrased:

BVH files define their own euler rotation order per joint, and the importer does not always match it, so wrists and feet can end up subtly wrong even when the hips look fine. If a pose reads almost right but the hands are off, check euler order before touching offsets.

Consecutive clips with different hip heights, like land to idle or crouch to stand, need the vertical offset blended too, otherwise the character floats or sinks for a few frames while the horizontal path snaps correctly.

The first one turned out to be a near miss.

The second one was a bug I already had and had not seen.

Tip 1: euler order, or why the hands go wrong first

A BVH file, which is what the CMU mocap library ships as, stores each joint's rotation as three angles.

But it also stores the order those three rotations are applied in, and that order is per joint.

One joint can say Z then X then Y, the next can say Y then X then Z.

That matters because rotations do not commute.

Rotate 30 degrees around Z and then 45 around X, and you end up facing somewhere different from 45 around X and then 30 around Z.

Same numbers, different pose.

Now put that in a skeleton.

Every bone inherits its parent's rotation, so a small error at the shoulder moves the elbow, which moves the wrist, which moves the hand.

The error compounds down the chain.

The hips are at the top of the tree, so they look perfect.

The hands are at the bottom, so they are the first thing that looks "almost right but weird".

I checked the importer call, half expecting the worst.

It already passed rotate_mode="NATIVE" to bpy.ops.import_anim.bvh, which means "use each joint's own order from the file".

So we were safe.

But it is now the first thing on my list if a CMU clip ever shows odd wrists or ankles, before anyone reaches for an offset hack.

Tip 2: hip height across a blend, which bit me

Quick refresher on how the clips are chained.

Each clip is mostly "in place", so to make Miles actually travel, each clip's movement is added on top of wherever the last one ended.

A running sum.

That is root motion, the same idea game engines use.

Everyone remembers to blend the horizontal part of that path.

The second tip was that the vertical part needs blending too.

(Game engines usually call that Y. Blender is Z-up, because of course it is.)

On flat ground I never saw a problem.

On raised surfaces I had exactly this bug, and I only found it once I measured it.

The probe was simple: on every frame through each blend, take the gap between the lowest toe and the surface under him.

The target is about +5 cm, because the toe bone sits a little above the sole of the shoe.

Blend, on a raised surface	Before	After
Crouch → web shot, on a board's top ledge	sinks ~10 cm for 3 frames	+0.03 to +0.07 m
Crouch → moonwalk, on a billboard catwalk	pops up 22 cm, then settles	+0.02 to +0.10 m
Crouch → taunt, on a rooftop	drifts up 10 cm	+0.04 to +0.07 m

A 22 cm pop is a Spider-Man who briefly levitates for no reason.

Which, to be fair, is on brand, but not what I wanted.

The first fix did exactly what the tip says, and it did nothing.

I eased the root height smoothly across the blend window instead of letting it step.

The numbers barely moved.

The measurement showed why.

The hips were not the problem.

The crossfaded pose was.

Halfway between a crouch and a standing pose, the blended legs are half extended, so the feet rise on their own, no matter how gently the hips move.

A smooth curve for the hip height cannot follow what the legs are doing.

What worked was to stop predicting and start measuring.

For every frame in a raised shot, find the lowest toe, and move the root so that toe sits a fixed distance above the surface.

Then smooth it lightly so it does not jitter.

This is the real function, trimmed a little:

def contact(self, start, end, floor, toe=0.05, smooth=2, limit=0.35):
    """Keep the lowest toe `toe` above a surface at height `floor` for every frame."""
    frames = list(range(int(start), int(end) + 1))
    fc = anim.fcurve(self.root, "location", 2)
    base, low = [], []
    for f in frames:
        self.scene.frame_set(f)
        base.append(fc.evaluate(f) if fc else self.root.location.z)
        low.append(min((self.rig.matrix_world @ self.rig.pose.bones[b].head).z
                       for b in ("mixamorig:LeftToeBase", "mixamorig:RightToeBase")))
    corr = [max(-limit, min(limit, floor + toe - z)) for z in low]   # a hop in the clip stays a hop
    sm = [sum(corr[max(0, i - smooth):i + smooth + 1]) / len(corr[max(0, i - smooth):i + smooth + 1])
          for i in range(len(corr))]                                   # average over ±2 frames
    if fc:                                     # remove back to front: indices stay valid
        idx = [i for i, kp in enumerate(fc.keyframe_points) if start <= kp.co[0] <= end]
        for i in reversed(idx):
            fc.keyframe_points.remove(fc.keyframe_points[i])
    for f, b, c in zip(frames, base, sm):
        anim.key(self.root, "location", f, b + c, 2, "lin")          # a key on EVERY frame

That limit=0.35 is load bearing.

Without a cap, one clip that genuinely hops at the end got "corrected" by 8 metres.

The function saw feet in the air, decided the floor must be way off, and dragged the whole character to meet them.

The ±35 cm clamp means a real hop stays a hop.

And yes, that "remove back to front" comment is a scar.

Which brings us to the bugs.

Three bugs the measurement dug up

Once there was a probe printing a number per frame, the number started pointing at things I would never have seen in a viewport.

1. Miles, 18 metres up, for exactly one frame

An earlier floor-lock pass keyed the root height on every other frame, with linear interpolation in between.

That is a perfectly reasonable thing to do inside one shot.

It is a terrible thing to do across a hard cut.

At a cut from a rooftop to the street, the frame between two keys landed halfway between two shots.

Linear interpolation averaged "rooftop" and "street" and put Miles about 18 m up, in the sky above the street, for one frame.

At 30 fps that is a 33 millisecond glitch.

You would never catch it playing the video.

The probe flagged it instantly.

The fix: key every frame inside these ranges, so there is no in-between frame left to interpolate.

2. Hard cuts on fractional frames

Clips are scheduled in float frames, because blend lengths and speeds are computed.

So a "hard cut" could start at frame 1037.4.

Blender renders whole frames, which meant frame 1037 showed the old pose at the new position.

Half of one shot glued to half of another.

The fix: hard cuts snap to a whole frame with math.ceil, and a shot's last frame is ceil(end) - 1.

This is the animation version of an off-by-one, and it was hiding inside a float.

3. Deleting keyframes while iterating over them

This one is a classic, and I still walked straight into it.

The code collected references to the keyframes it wanted to replace, then looped over them calling keyframe_points.remove(kp, fast=True).

But removing a keyframe shifts every index after it.

So the next reference in the list now pointed at its neighbour, and the loop silently deleted the wrong key.

The first frame of two shots ended up with no height key at all.

It is the same bug as removing items from a Python list inside for x in items:, just wearing a 3D costume.

The fix is the one in the snippet above: collect indices, then remove from the back.

The whole lesson of this section is not about any one of those bugs.

It is that the first fix looked right in theory and did nothing in practice, and only a number per frame could tell me that.

Do not eyeball root motion. Measure it.

The probe became a test suite

Once the foot probe existed, it seemed silly to only run it when I suspected a problem.

So it became scripts/check_support.py, a permanent pre-render check.

It is about 40 lines and it answers one question, sampling every third frame of the film: is there something under his feet?

flowchart TD
    F[Next frame] --> A{airborne clip or upside down?}
    A -->|yes, swing, hang, flip| F
    A -->|no| T[Find the lowest toe]
    T --> R[Ray cast straight down]
    R --> M{hit Miles, a web, a spider or cash?}
    M -->|yes, step past it| R
    M -->|no| G{gap over 25 cm or sinking?}
    G -->|no| F
    G -->|yes| B[Flag the frame]
    B --> F

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef step fill:#6ea8ff,stroke:#2f62c4,color:#1a1a1a
    classDef bad fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a

    class A,M,G decision
    class T,R step
    class B bad
    class F start

It uses Blender's scene.ray_cast, ignores anything that is part of the character or the props (a web is not a floor, sadly), skips the moments where he is supposed to be in the air, and groups flagged frames into runs so the output reads like a test report.

I run it before every render, the same way you would not deploy without running the tests.

And it caught things a visual check flat out missed.

Moonwalking on thin air.

The opening of this post.

Miles moonwalked 6.5 m past the edge of a billboard's frame, and, for bonus points, in the opposite direction to the customers walking off the screen behind him.

The fix had two parts.

A steel catwalk under the screen, because real billboards have one for maintenance anyway.

And the moonwalk now goes with the customers, which is a better joke.

A 40 cm stance on a 25 cm board.

He stood on top of a wall board whose top was 25 cm deep.

His stance was 40 cm.

One foot was hanging off the front, standing on air.

The fix was a 1.25 m service ledge on top of the board.

A turn that swung him 2 m off a ledge.

This was my favourite, because the cause was pure geometry.

In-place clips leave the rig's root pivot nowhere near the hips.

In this case it was about 4 m away.

So when a blend rotated the root to turn him 150 degrees, his whole body orbited a point 4 m away, like a door swinging on a hinge in the next room.

Mid-turn, his hips were 2 m off the ledge.

The fix was to key the root per frame during a turning blend, so the hips travel in a straight line while the body rotates around them.

A turn is a rotation about some point.

It is worth checking which one.

Small workflow wins

None of these deserve a section, but all of them saved real time.

Burned-in review copies.

Every render also writes a <name>_tc.mp4 with the Blender frame number and "time / total" stamped in the corner.

When someone says "f1201 looks off", that maps straight to a frame in the scene.

No more "uh, around the middle, after the bus thing".

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

I Made Spider-Man Swing Without Animating a Single Frame

Athreya aka Maneshwar — Sat, 03 Oct 2026 18:05:25 +0000

Yes, that is Miles Morales.

He drops in on a web, lands a superhero crouch, fires a web off screen, and yanks a giant DEV board into frame.

I did not animate a single frame of that by hand.

I also had never opened Blender before this week.

This post is the story of how that happened, and the surprising amount of computer science hiding inside a "simple" 3D character.

Why Blender, of all things

I was trying to make a video.

Not a screen recording, an actual video with something moving in it.

My first stop was HyperFrames, which lets you write a video as HTML and render it out.

It is genuinely great at motion graphics.

Text that slides in, logos that pop, charts that draw themselves, all of that is lovely.

But the moment I wanted a character to do something, it hit a wall.

HTML can move rectangles around beautifully.

It cannot make a guy swing on a web.

So the next question was obvious, if a little scary: what do actual animators use?

The answer, for free and open source, is Blender.

And Blender has an MCP server, which meant I would not be learning it alone.

The MCP that drove the whole thing

The thing that made this possible for a total beginner is blender-mcp.

It connects Blender to an LLM client like Claude Code, so instead of hunting through 400 menus, you say "load this model and relink its textures" and it happens.

The architecture is worth understanding, because it is simpler than it sounds.

flowchart LR
    ME[Me typing a prompt] --> CC[Claude Code]
    CC -->|MCP over stdio| SRV[blender-mcp server]
    SRV -->|JSON over TCP socket| ADDON[Blender add-on]
    ADDON -->|runs Python| BPY[bpy API]
    BPY --> SCENE[Live Blender scene]
    SCENE -->|viewport screenshot| CC

    classDef human fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef agent fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef bridge fill:#6ea8ff,stroke:#2f62c4,color:#1a1a1a
    classDef blender fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a

    class ME human
    class CC agent
    class SRV,ADDON bridge
    class BPY,SCENE blender

There are two halves.

An add-on inside Blender opens a small socket server and waits for commands.

An MCP server outside Blender speaks the Model Context Protocol to the LLM on one side, and forwards JSON commands down that socket on the other.

The most powerful tool it exposes is execute_blender_code, which runs arbitrary Python inside Blender through bpy, Blender's Python API.

Every button in Blender is just a thin skin over bpy.

So an agent that can write bpy can, in principle, press every button.

The loop that makes it work is the screenshot tool coming back the other way.

The agent changes something, grabs the viewport, looks at it, and decides what to fix next.

That is the same edit, run, inspect loop you use on code, just pointed at a 3D scene.

One honest warning: that tool is arbitrary code execution by design. Treat it like you would treat any local automation with a shell, and do not point it at a machine full of secrets.

Step one: get a character

I am not a 3D modeler, and I was not about to become one by Friday.

So I grabbed a ready made Miles Morales model, the Spider-Verse Spider-Man, from Sketchfab.

Even "just load it" had lessons in it.

The file came with four copies of Miles standing in a row: masked and unmasked, posed and T-posed.

The textures pointed at the original author's Windows folders, like C:\Users\someone\Desktop\..., so the agent had to relink them to the files in the zip.

And the zip was missing the normal and specular maps.

In Blender, a texture node pointing at a missing file renders hot pink.

It is the 3D equivalent of a 404, and it is very, very visible.

The fix was to disconnect those slots, at the cost of some surface detail on the suit.

Then I opened the unmasked Miles and his face looked like a horror movie, covered in white lines and dots.

Turns out the face was fine.

That model ships with a 143 bone face rig (eyelids, brows, lips, jaw, cheeks, even a tongue), and the armature was set to "draw in front", so every one of those bones was painted on top of his skin.

One checkbox later, he was handsome again.

The concept that unlocks everything: rigging

Here is where I learned the word that the rest of this post hangs on.

A 3D character is, at its core, a mesh: a big list of vertices (Miles has about 27k faces) glued into triangles.

A mesh is a statue.

You cannot make a statue wave.

To move it, you need three more things.

An armature, which is a skeleton.

It is a tree of bones, each one a parent of the next: hips to spine to chest to shoulder to upper arm to forearm to hand to fingers.

Rotate the shoulder and everything below it in the tree comes along for the ride, exactly like a transform hierarchy in any game engine or a nested <div> with a CSS transform.

Skin weights, which bind the mesh to the skeleton.

Every vertex stores a small table saying how much each bone influences it.

A vertex on the forearm might be 100% forearm.

A vertex right at the elbow might be 50% upper_arm and 50% forearm, which is what lets the elbow bend smoothly instead of folding like a paper straw.

The math is called linear blend skinning, and it is just a weighted average of matrices:

final_position(v) = Σ  weight(v, bone) × bone_matrix × rest_position(v)
                   bones

A rest pose, which is the reference everything is measured from.

That is why every character you download stands in a T-pose, arms straight out.

It is not a dance move.

It is the zero point, the git init of the body, and every animation is stored as rotations relative to it.

Put those together and you have a rig.

And then the naive plan fell apart, because the model's rig had bones but no controls.

Forward vs inverse kinematics

I had seen YouTube videos where an animator grabs a character's hips, drags them down, and the knees bend while the feet stay glued to the floor.

When I dragged Miles's hips, his whole body floated down through the floor like a ghost.

The difference is one of the nicest ideas in this whole field.

Forward kinematics (FK) is what a raw skeleton does.

You set the angle of every joint, and the hand ends up wherever the chain of rotations puts it.

Easy to compute, miserable to animate. Want the hand on the doorknob? Good luck guessing the shoulder, elbow and wrist angles that land it there.

Inverse kinematics (IK) flips the question.

You say where the hand (or foot) should be, and a solver works backwards to find joint angles that put it there.

It is literally an optimisation problem solved every frame, and Blender ships an IK constraint that does it for you.

Pin the feet with IK targets, drag the hips down, and the solver has to bend the knees because the feet are not allowed to move.

That is the magic in those YouTube videos.

It is not magic, it is a constraint solver.

Building a full IK control rig by hand is a real skill though.

And I wanted Miles doing backflips this week, not next quarter.

Mixamo: rigging as a service

Enter Mixamo, Adobe's free auto-rigger and animation library.

The flow is: export your character as an FBX in T-pose, upload it, and Mixamo builds a skeleton and skin weights for you.

The only thing it asks of you is to place a few markers.

Chin, wrists, elbows, knees, groin.

Five landmarks, with symmetry on, so really just three clicks per side.

From those it infers the whole skeleton, which is a nice reminder of how much structure a human body has. If you know where the joints are, the rest is mostly proportions.

Placement matters more than you would think.

My first pass had the wrists sitting on the forearms, which would have made his hands bend in the wrong place every time he shot a web.

I picked Standard Skeleton (65), which includes finger bones, because a two handed pull without fingers looks like a man wearing oven mitts.

And then it just... worked.

The rigged character goes straight into Mixamo's library of motion captured animations.

These are real performers in mocap suits, recorded and cleaned up, so they have the weight shifts and tiny wobbles that hand keyed animation struggles to fake.

Jumps, climbs, hangs, throws, dances, falls, the lot.

There was one trade off, and it hurt a little.

Mixamo builds its own body skeleton.

It does not know or care about the beautiful 143 bone face rig the model came with.

"With skin" and "without skin"

When you download an animation, Mixamo asks whether you want it with skin or without skin.

This is a good little lesson in separating data from presentation.

With skin gives you the mesh, the skeleton and the motion.

Without skin gives you only the skeleton and the motion, a few hundred kilobytes of bone rotations over time.

Because every clip targets the same 65 bone skeleton with the same bone names, you only need the skin once.

Everything else is just keyframe data keyed by bone name, and Blender happily applies it to any armature whose bones match.

So I downloaded the first clip with skin and the rest without.

And I downloaded a lot of them.

Seventy eight files.

Clicked one at a time, because Mixamo has no multi select.

Some of them even share a name with a different animation inside, because the site just appends (1) to whatever you downloaded last.

They are not the same picture.

The agent ended up scanning every FBX to see what was actually inside before trusting a single filename.

From a pile of clips to a performance

Here is the whole pipeline, end to end:

flowchart TD
    A[Sketchfab model] --> B[Clean up in Blender<br/>relink textures, drop pink slots]
    B --> C[Export T-pose FBX]
    C --> D[Mixamo auto-rig<br/>5 markers]
    D --> E[Download clips<br/>1 with skin, rest without]
    E --> F[Import as Blender actions]
    F --> G{Clip looks right<br/>from this angle?}
    G -->|no| E
    G -->|yes| H[Chain clips on NLA<br/>crossfades + root motion]
    H --> I[Webs, board, camera]
    I --> J[360p preview render]
    J --> K{Good enough?}
    K -->|no| H
    K -->|yes| L[Final render]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef source fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef rig fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef scene fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef out fill:#ff9a5c,stroke:#c4602a,color:#1a1a1a

    class G,K decision
    class A,B,C source
    class D,E,F rig
    class H,I scene
    class J,L out

Each Mixamo clip lands in Blender as an action, a named bundle of animation curves (an f-curve per bone per rotation channel).

On its own, an action is a single move.

A performance is several of them in a row, and that is what Blender's Nonlinear Animation editor is for.

The NLA is basically a video editor timeline for animation.

Each action becomes a strip, strips sit on tracks, and overlapping strips can blend into each other.

That blend is what stops a cut from "hanging" to "landing" from snapping like a broken GIF.

The agent wrapped all of this into a tiny sequencer, so the shot is described as a list of moves instead of a pile of frame numbers:

perf = Performer(rig)
perf.then("Hanging Idle", length=40)
perf.then("Hard Landing", to=50, speed=1.3, blend=6)
perf.then("Standing 1H Magic Attack 02", frm=16, to=40, speed=1.25, blend=6)
perf.then("Pull Heavy Object Stop", to=30, speed=1.1, blend=5, face=65)
perf.then("Victory Idle", blend=8, face=10, length=51)
perf.build()   # NLA strips, crossfades, stitched root motion

blend=6 is a six frame crossfade into the next clip.

face=65 turns him 65 degrees so the yank points at the board.

And the "web shot" is, I promise, a clip literally named Magic Attack. Mixamo does not have a "thwip".

The interesting problem hiding in build() is root motion.

Some clips move the character through space (a landing drops him, a pull leans him back), and each clip starts from its own origin.

Play two of them back to back naively and Miles teleports back to the start between moves.

So the sequencer reads where the hips ended in one clip and offsets the next one to begin there.

It is the same idea as stitching GPS tracks together: each segment is relative, so you keep a running sum.

Because all timing derives from that list, swapping one clip re-times the webs, the board and the camera automatically.

The web strands, the board and the camera are not Mixamo at all, they are plain Blender objects keyframed by script to fire on the frame where his hand reaches out.

Iterate at 360p, like a sane person

The first full render was 16 seconds at 720p and took about eight minutes on my GTX 1650.

That is fine once.

It is not fine when the feedback is "make it snappier", "make the board bigger" and "his suit is too dark" in the same breath.

So previews dropped to 360p, roughly 4x faster, and 720p became a finals only thing.

The same rule you know from code applies: make the feedback loop fast first, then make the output good.

Two more things that cost real time:

The suit rendered as a flat black blob.

The model used a toon shader that collapses lighting into a couple of hard bands, which looks great in a cartoon and terrible when you want to see muscles.

Swapping in physically based shading, plus a light just for him and red and blue rim lights, brought his shape back.

Light grey or dark studio?

A near black suit on a near black background is a silhouette of nothing.

Lighter backgrounds read better. Dark backgrounds look cooler. The GIF at the top is the dark one, because I have priorities.

Escaping the download button

After clicking "Download" 78 times, I asked the obvious engineer question: can I just do this locally?

Mixamo's rigger and library only live on Adobe's servers.

But motion capture data is not an Adobe invention.

The CMU Graphics Lab Motion Capture Database has about 2,500 free clips, and there is a BVH mirror on GitHub with a text index of what each one is.

Cartwheels, salsa, climbing, martial arts, "pulling a rope", all of it.

The catch is that CMU's skeleton is not Mixamo's skeleton.

Moving motion from one skeleton to another is called retargeting, and it is a lovely little linear algebra problem.

You cannot just copy rotations across, because the two skeletons have different rest poses. A 30 degree rotation from "arms slightly down" is not the same pose as 30 degrees from "arms straight out".

So the retargeter converts each bone's rotation out of the source rest pose, into a shared world frame, and back into the target's rest pose.

It also scales the hip travel by leg length, so a tall performer's stride does not make Miles moonwalk.

The first try rendered... nothing.

Miles was not in frame. He was not anywhere.

The BVH rest pose put the hips exactly at the origin, so "rest hip height" was zero, and the scale factor was a division by it.

Miles had been launched into orbit by a divide by zero.

Fixed by measuring the floor from the lowest foot in the clip instead, and suddenly he was walking, dancing and cartwheeling from a dataset recorded at Carnegie Mellon years ago.

What I actually learned

I set out to make a guy swing on a web.

I came back with a list of concepts I did not expect from an art tool:

A character is a mesh deformed by a tree of bones, via per vertex weights.
Animation is rotations relative to a rest pose, which is why everything T-poses.
IK is a constraint solver, and it is the difference between a puppet and a corpse.
Motion clips are data keyed by bone name, so one skin can wear a thousand animations.
NLA strips are a timeline of reusable actions, and chaining them means stitching root motion.
Retargeting is a change of basis, and it will divide by zero if you let it.
Blender is a Python program with a UI on top, which is why an MCP bridge can drive it at all.

There is a ton more in there for a CS person: shader node graphs are dataflow programs, Cycles is a path tracer, physics sims are numerical integrators, geometry nodes are basically a functional language for meshes.

But one step at a time.

Next on the list is the face rig I so cruelly abandoned in that pool, and maybe making Miles actually talk.

If you have played with Blender, rigging or the MCP, tell me what I should pick up next.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by tests…

View on GitHub

Click below to try LiveReview with your codebase:

What Happens When Two Databases Both Say Yes

Athreya aka Maneshwar — Thu, 01 Oct 2026 19:22:55 +0000

Your database has one primary and two replicas, and for about two years that is the best decision you ever made.

Reads scale out across the replicas. Writes all go to the one primary.

There is exactly one copy of the truth, and it lives on a machine you could point at.

Then sales closes a deal in Sydney.

Physics sends you an invoice

Your primary is in Virginia. Your new users are in Sydney.

Every write from Sydney now has to cross the Pacific, get acknowledged, and come back.

That is roughly a 200 millisecond round trip, and no amount of clever indexing touches it, because the delay is not your database being slow.

It is light being inconveniently finite.

Two hundred milliseconds does not sound fatal until you notice how many writes hide inside one user action.

A session token. An analytics event. A "last seen" bump. Suddenly a page that feels instant in Boston feels like dialup in Sydney.

And then there is the other problem, the one that shows up on a Saturday.

If us-east-1 has a bad day, nobody writes anything. Not Sydney, not Boston, not the replicas sitting right there with a full copy of the data.

They have the bytes. They are just not allowed to touch them.

One leader means one place to be slow and one place to fail.

So you give each region its own leader

This is multi-leader replication, and the idea is exactly as blunt as it sounds.

Put a leader in each region. Let each one accept writes from the users nearest to it. Then have the leaders ship their changes to each other in the background.

Sydney writes land in Sydney at 8 milliseconds. Virginia writes land in Virginia at 8 milliseconds.

A few seconds later, each change has made its way to the other side of the world and both leaders hold the same data.

You just bought two genuinely good things.

Write latency collapses for everyone who is not sitting next to your original region. Writes are now local.

Losing a region is survivable. Sydney goes dark, Virginia keeps taking writes, and when Sydney comes back it catches up from the log.

Your write availability is no longer the uptime of a single building.

This is the same shape as single-leader replication, just with the "single" removed. Which is a sentence that should make you slightly nervous, and it should.

Because the thing that made single-leader so pleasant to reason about was never the leader. It was the one.

The bill arrives as a row number

Alan is in Sydney. He opens row 1,000,000 and sets x = 7.

At more or less the same moment, John is in Virginia. He opens the same row and sets x = 9.

Sydney accepts Alan's write and returns a 200. Virginia accepts John's write and returns a 200.

Neither leader hesitated, because at the moment each of them decided, the other write did not exist yet as far as they could tell.

Now the replication streams cross in the middle of the ocean, and both leaders simultaneously learn that they were lied to.

So what is x?

There is no answer to that question hiding in the data. The database has already promised both values to two different humans.

Somebody is about to be wrong and nobody has told them yet.

This is the problem with multi-leader replication. Not a problem. The problem. Everything else in this post is a response to it.

And notice what single-leader gave you for free here.

With one primary, these two writes would have arrived at the same machine, one would have gone second, and it would have overwritten the first while knowing it was overwriting something. A SELECT in between would have seen a consistent story.

That is not a feature anybody shipped. It is a side effect of there being one queue.

"At the same time" is the wrong mental model

It is worth being precise here, because this is where people's intuition quietly breaks.

Alan and John did not have to write at the same instant for this to happen. They could have been forty seconds apart.

Two writes are concurrent if neither one knew about the other. That is the entire definition.

It has nothing to do with wall clock time and everything to do with what information had reached whom.

Leslie Lamport worked this out in 1978 in Time, Clocks, and the Ordering of Events in a Distributed System, and it is still the single most useful thing you can read on the subject.

If Alan's write had reached Virginia before John started typing, John's write would be a normal overwrite. Not a conflict, just an edit.

Replication lag is what turns an edit into a conflict.

Which means your conflict rate is not a property of your users. It is a property of how long your replication takes, multiplied by how much your users overlap.

The default answer is "last write wins", and it is worse than it sounds

Most systems that offer multi-leader replication ship with a default conflict resolution policy, and that default is almost always last write wins.

DynamoDB global tables do it. Cassandra does it. It is the policy you get when nobody chose a policy.

Each write carries a timestamp. When two versions of a row collide, the one with the bigger timestamp survives. The other one is discarded.

Not flagged. Not queued for review. Discarded.

Here is the part that gets people. LWW is a guaranteed data loss mechanism, and that is not a criticism, that is its job description.

If two writes conflict and you keep exactly one, you have deliberately thrown away a write that the database told a user it had accepted.

Sometimes that is completely fine. If the field is "last known GPS position", dropping the older one is correct and you should feel nothing.

If the field is "account balance", you have just invented a very exciting kind of bug.

And then there is the detail that turns a tolerable policy into a haunted one.

Whose clock?

LWW compares timestamps generated by two different machines, and those machines do not agree on what time it is.

NTP keeps them roughly in sync, where "roughly" means tens of milliseconds on a good day and considerably worse on a VM whose host is under load or whose clock just stepped.

So if Sydney's clock has drifted 120 milliseconds ahead, Alan's earlier write gets stamped with a later timestamp.

LWW dutifully keeps it and throws away John's write, which actually happened second.

The system did exactly what it was configured to do. It also reverted a value, told the user it had succeeded, and left no trace anywhere that it happened.

This is the reason Google built TrueTime for Spanner, with GPS receivers and atomic clocks in every datacenter, and then had Spanner deliberately wait out the clock uncertainty interval before committing.

They did not solve clock skew. They bounded it, paid for the bound in hardware, and then paid for it again in commit latency.

If you are doing LWW on timestamps from now() across regions, you are making the same correctness claim Spanner makes, with none of the equipment.

What you can do instead of picking a loser

The good news is that LWW is a default, not a law.

Once you accept that conflicts are going to happen, you get to choose what happens next, and there are better options than a coin flip weighted by NTP drift.

Keep both and let the application decide. CouchDB does this properly: a conflicting document keeps every version as a sibling, picks one deterministically so reads do not break, and leaves the rest sitting there for you to resolve.

Riak called these siblings too. It is the most honest option, and it is the one that forces you to actually write resolution logic, which is why people avoid it.

Write a merge function. For a lot of real data, "pick a winner" is just the wrong operation. Two users adding different items to a shared cart is a union, not a race.

Two users editing different fields of the same profile row is two independent edits that a column-level merge handles without anyone losing anything.

Change the data type so conflicts cannot exist. This is the good one.

Replicate the operation, not the value

Here is the move that makes a whole class of conflicts evaporate.

A conflict happens because two leaders made competing claims about what a value is. Sydney said the cart has 4 items.

Virginia said the cart has 4 items. They both read 3 and both added 1.

Last write wins picks one of those two 4s, and the answer is 4. The customer added two things and the cart grew by one.

Now watch what happens if the leaders stop shipping the result and start shipping the operation.

Sydney does not say "the cart is 4". It says "Sydney has contributed 1". Virginia says "Virginia has contributed 1".

The value of the cart is the sum of everyone's contributions, and both leaders compute 5 without ever disagreeing about anything.

Nothing had to win, because the two writes were never competing claims in the first place.

That is a CRDT, and the counter version of it is so simple you can build one in plain SQL with no special database at all:

-- one row per (cart, replica). each leader only ever touches its own row,
-- so two leaders can never write the same row and never conflict.
CREATE TABLE cart_counter (
  cart_id   bigint  NOT NULL,
  replica   text    NOT NULL,   -- 'syd' | 'iad' | 'fra'
  delta     bigint  NOT NULL DEFAULT 0,
  PRIMARY KEY (cart_id, replica)
);

-- Sydney's leader, adding an item. it writes replica = 'syd', always.
INSERT INTO cart_counter (cart_id, replica, delta)
VALUES (1841, 'syd', 1)
ON CONFLICT (cart_id, replica)
DO UPDATE SET delta = cart_counter.delta + 1;

-- the actual value, which every region computes identically
SELECT sum(delta) FROM cart_counter WHERE cart_id = 1841;

The write set of each leader is disjoint by construction. Replication can arrive in any order, duplicated, delayed by a week, and every region still converges on the same number.

This is the same principle behind Automerge and Yjs, which is how collaborative editors let twelve people type into one paragraph without a central referee.

And it is worth noticing that a Google Doc or a Figma file is a multi-leader system. Every laptop is a leader.

The offline mode you shipped last quarter made your mobile app one too.

The catch is that CRDTs are not free. They carry per-replica metadata, which grows, and they are only convenient for data types somebody has already worked out the merge semantics for: counters, sets, registers, sequences, maps.

The moment your business rule is "the total must never exceed the credit limit", you are back to needing coordination, because that is a global invariant and CRDTs specifically do not give you those.

The conflicts nobody warns you about

So far this has all been about one row, one column. The conflicts that actually ruin a migration weekend are the structural ones.

Auto-increment primary keys. Two leaders, both confidently handing out id 4817 to different rows. The fix is for each leader to generate ids from a disjoint space: offset sequences, UUIDs, or a Snowflake-style id with the region baked into the bits. If you are turning on a second leader and have not solved this, solve it first.

Unique constraints. Two users in two regions both register alan@example.com. Each leader checks its local table, sees nothing, and allows it.

The constraint held locally and is violated globally, which is a fun error to discover during replication rather than during signup. Uniqueness is a global invariant and asynchronous replication fundamentally cannot enforce one.

Foreign keys and deletes. Sydney deletes a parent row. Virginia inserts a child pointing at it. Both are valid locally.

Together they produce an orphan, and now your replication stream is stuck trying to apply something your schema refuses.

Triggers and defaults that call now(). Any logic that is not a pure function of the row will produce different results on different leaders. Replicate the computed value, not the computation.

Notice the pattern. Everything on that list is a constraint that spans rows your leaders do not coordinate on.

Single-leader hid all of them behind one serialisation point, for free, and you never knew they were being handled.

The strategy that actually ships

Here is what teams running this in production actually do, and it is less glamorous than any of the above.

They make the conflict impossible.

Give every record exactly one home leader that owns its writes. All writes for a given customer, tenant, or account are routed to that customer's home region, no matter where the request physically landed.

Every leader still replicates everything to everywhere, so reads stay local and failover stays possible.

Now the two leaders have disjoint write sets. Sydney writes rows Virginia never writes. There is no conflict to resolve because the two leaders are never making claims about the same data.

Conflict resolution logic you never run is the only kind that never has a bug.

This is why multi-leader works beautifully when your data has a natural ownership boundary. Per-customer SaaS. Per-tenant workspaces. Per-region marketplaces.

It works much less beautifully for a global leaderboard or a shared inventory count, where every write touches data everybody else touches too.

The routing itself is usually unexciting, which is the point:

def route_write(account_id, request_region):
    home = accounts.home_region(account_id)   # cached, changes rarely

    if home == request_region:
        return local_leader()                 # the fast path, ~8ms

    # cross-region write: slower, but still exactly one leader owns this row,
    # so it is an ordinary overwrite rather than a conflict waiting to converge
    return leader_in(home)

Note what that else branch is buying you. A European user acting on a US-homed account pays the 100 millisecond hop.

You have traded some users' write latency for never having to explain to your CFO why a balance went backwards.

That is usually the right trade, and it is a trade, not a free lunch.

flowchart TD
    A[Write arrives in Frankfurt] --> B{Which region<br/>owns this account?}
    B -->|Frankfurt| C[Local leader, 8 ms]
    B -->|Virginia| D[Forward to Virginia, 100 ms]
    C --> E[Commit, ack the user]
    D --> E
    E --> F[Async replication to every other leader]
    F --> G[All regions can serve reads<br/>and take over on failover]
    B -->|Home region is down| H{Can you wait<br/>for it to come back?}
    H -->|Yes| I[Reject the write, stay conflict-free]
    H -->|No| J[Reassign the home, accept<br/>that you may now conflict]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef good fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef warn fill:#ff9a5c,stroke:#c25f20,color:#1a1a1a

    class A start
    class B,H decision
    class C,D,E,F,G good
    class I,J warn

The honest caveat: homes move

Ownership routing is clean right up until the moment a region fails, which is the exact scenario you built multi-leader for in the first place.

Sydney goes down. Its accounts have no home.

You either reject their writes, which defeats the point, or you reassign their home to Virginia, which means accepting writes for rows Sydney might also have accepted writes for in the seconds before it died but not yet replicated.

That window is where your conflicts live. Not during normal operation, during failover.

So the real design is: disjoint write sets almost always, plus a conflict resolution policy you have genuinely thought about for the minutes per year when they are not disjoint.

Teams that skip the second half discover they needed it during an incident, which is the worst possible time to be reading about version vectors.

Two smaller things that bite, worth knowing before they bite you.

Reading your own writes gets harder. A user writes to their home leader in Virginia, then their next request hits a read replica in Frankfurt that is 400 milliseconds behind.

Their change is gone. You fix this by pinning a user's session to one region for a while after a write, or by having the client carry the version it last saw and the read wait for it.

Replication topology matters. All-to-all is the usual choice and is the most robust, but it lets messages overtake each other, so an update can arrive before the insert it depends on.

Circular and star topologies are cheaper on connections and much worse on failure, because one dead node breaks the chain. MySQL's circular replication is famous for exactly this.

So should you turn this on?

The uncomfortable truth is that most teams reaching for multi-leader replication do not need it.

If your problem is read latency, you want read replicas, and you already have them. Multi-leader only pays off when writes are the thing crossing an ocean.

If your problem is regional failover, a single leader with a well-rehearsed promotion runbook gets you most of the way there, with a fraction of the semantics to reason about.

flowchart TD
    A[Users in multiple regions] --> B{Are WRITES the<br/>slow part?}
    B -->|No, reads are| C[Read replicas per region<br/>Keep the single leader]
    B -->|Yes| D{Does the data split<br/>cleanly by owner?}
    D -->|Yes, per tenant or account| E[Multi-leader with home regions<br/>Disjoint write sets]
    D -->|No, everyone writes everything| F{Can you model it<br/>as a CRDT?}
    F -->|Yes, counters sets text| G[CRDTs, converge without picking]
    F -->|No, you need real invariants| H[Distributed SQL<br/>Spanner, CockroachDB, YugabyteDB]
    E --> I[Still need a policy for<br/>the failover window]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef good fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef alt fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef warn fill:#ff9a5c,stroke:#c25f20,color:#1a1a1a

    class A start
    class B,D,F decision
    class C,E good
    class G,H alt
    class I warn

And if you genuinely need strong consistency across regions, that is what CockroachDB and Spanner are for.

They run consensus under the hood and charge you for it in write latency, openly, instead of handing you an asynchronous system that looks consistent until it is not.

That is the actual choice. Not "multi-leader or single-leader", but where you would like to pay.

Single-leader charges distant users on every write. Distributed SQL charges everyone a smaller amount on every write.

Multi-leader charges nobody anything on the happy path and sends you one enormous invoice on the day two regions disagree about a row.

Pick the bill you would rather receive at 3am.

The one-line version

Multi-leader replication does not create conflicts.

It reveals that your data model always had them, and that a single primary was quietly doing your reconciliation for free.

The moment you add a second leader, that work moves to you.

You can do it with a timestamp comparison and hope, or with disjoint write sets and intent.

Only one of those lets you sleep.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by tests…

View on GitHub

Click below to try LiveReview with your codebase:

Building a Custom ERP Module on the MERN Stack: Data Modeling, Transactions and Audit Trails

Athreya aka Maneshwar — Tue, 29 Sep 2026 21:25:00 +0000

ERP software handles money, stock, and approvals, so small bugs get expensive fast. A stock count that drifts by a few units or a rounding error in an invoice can take days to trace.

Off-the-shelf ERPs cover the general case well, but many companies eventually need a custom module for a workflow the big platforms handle awkwardly.

The MERN stack works well for this. You get flexible schemas for evolving business rules, one language across the whole stack, and real multi-document transactions in MongoDB.

This guide walks through a small purchasing and inventory module and focuses on 3 things: data modeling, consistent writes, and audit trails.

The Example Module

The module has suppliers, products, purchase orders (POs) with line items, and stock levels per warehouse.

The critical operation is receiving goods: the system must update PO line quantities, change the PO status, and increase stock, all at once. If only half of that happens, the numbers stop matching reality.

Data Modeling: Embed What Belongs Together

A simple rule works well for ERP data. Embed things that live and die with the parent, like PO line items.

Reference things with their own lifecycle, like suppliers and products.

const { Schema } = require('mongoose');

const lineItemSchema = new Schema({
  product: { type: Schema.Types.ObjectId, ref: 'Product', required: true },
  sku: { type: String, required: true },            // snapshot at order time
  unitPriceCents: { type: Number, required: true, min: 0 },
  quantityOrdered: { type: Number, required: true, min: 1 },
  quantityReceived: { type: Number, default: 0, min: 0 },
});

const purchaseOrderSchema = new Schema({
  number: { type: String, required: true, unique: true },
  supplier: { type: Schema.Types.ObjectId, ref: 'Supplier', required: true },
  warehouse: { type: Schema.Types.ObjectId, ref: 'Warehouse', required: true },
  status: {
    type: String,
    enum: ['draft', 'approved', 'partially_received', 'received', 'cancelled'],
    default: 'draft',
  },
  lines: [lineItemSchema],
}, { timestamps: true, optimisticConcurrency: true });

purchaseOrderSchema.index({ supplier: 1, status: 1, createdAt: -1 });

const stockLevelSchema = new Schema({
  product: { type: Schema.Types.ObjectId, ref: 'Product', required: true },
  warehouse: { type: Schema.Types.ObjectId, ref: 'Warehouse', required: true },
  onHand: { type: Number, default: 0, min: 0 },
});
stockLevelSchema.index({ product: 1, warehouse: 1 }, { unique: true });

A few choices here are worth copying:

Snapshot fixed values. SKU and price are copied into each line, so old orders keep what was actually agreed even if the product changes later.
Store money as integers. JavaScript floats can't represent 0.1 exactly. Cents avoid a whole category of rounding bugs. If you need fractional cents, use Decimal128.
Turn on optimistic concurrency. If 2 people edit the same PO at once, the second save fails instead of silently overwriting the first.
Use a unique compound index on stock. One row per product per warehouse, guaranteed by the database.

It also pays to design indexes around the questions people will actually ask.

Procurement teams usually want "open orders from supplier X, newest first", and the compound index on supplier, status, and createdAt answers that without a collection scan.

Look at the reports and list screens in your mockups before writing schemas, and add indexes for them from day one.

Transactions: Keep Related Writes Consistent

Receiving goods touches 2 collections. Without a transaction, a crash between the 2 writes leaves you with phantom inventory.

MongoDB transactions need a replica set; locally, a single-node one (mongod --replSet rs0) is enough.

The easiest API is session.withTransaction(), which commits for you and retries on transient errors:

async function receiveGoods({ poId, receipts }) {
  const session = await mongoose.startSession();
  try {
    await session.withTransaction(async () => {
      const po = await PurchaseOrder.findById(poId).session(session);
      if (!po || !['approved', 'partially_received'].includes(po.status)) {
        throw new Error('Order cannot receive goods');
      }

      for (const { lineId, quantity } of receipts) {
        const line = po.lines.id(lineId);
        if (line.quantityReceived + quantity > line.quantityOrdered) {
          throw new Error(`Over-receipt on ${line.sku}`);
        }
        line.quantityReceived += quantity;

        await StockLevel.updateOne(
          { product: line.product, warehouse: po.warehouse },
          { $inc: { onHand: quantity } },
          { upsert: true, session }
        );
      }

      const done = po.lines.every(l => l.quantityReceived === l.quantityOrdered);
      po.status = done ? 'received' : 'partially_received';
      await po.save({ session });
    });
  } finally {
    await session.endSession();
  }
}

3 rules keep this safe.

1) Pass the session to every read and write.
2) Keep side effects like emails or payment calls out of the callback, since withTransaction may run it more than once.
3) Keep transactions short to avoid lock contention.

For side effects that must happen, the outbox pattern works nicely. Inside the transaction, insert a record like { type: 'po.received', poId } into an outbox collection.

A background worker picks these up after the commit and sends the email or notifies the accounting system.

If the transaction rolls back, the outbox record disappears too, so nothing gets sent for a change that never happened.

Retry semantics and write conflicts rarely show up in tutorials, which is why companies building finance-adjacent modules often look for MERN stack developers who have already debugged these issues under real load.

Warehouse networks are also flaky, so give each receipt an idempotency key, store it under a unique index inside the same transaction, and duplicate requests will fail harmlessly.

Audit Trails: Who Changed What, and When

Auditors will ask "who approved this order?" and "why did the price change?". A Mongoose plugin combined with AsyncLocalStorage answers these automatically:

const { AsyncLocalStorage } = require('node:async_hooks');
const get = require('lodash/get');
const auditContext = new AsyncLocalStorage();

// Express middleware, placed after authentication
function withAuditContext(req, res, next) {
  auditContext.run({ userId: req.user.id, requestId: req.id }, next);
}

function auditPlugin(schema, { entity }) {
  schema.post('init', function () {
    this.$locals.before = this.toObject({ depopulate: true });
  });

  schema.pre('save', async function () {
    const ctx = auditContext.getStore() || {};
    const changes = Object.fromEntries(this.directModifiedPaths().map(p => [
      p, { from: get(this.$locals.before, p), to: this.get(p) },
    ]));

    await AuditLog.create([{
      entity,
      entityId: this._id,
      action: this.isNew ? 'create' : 'update',
      changes,
      userId: ctx.userId,
      requestId: ctx.requestId,
    }], { session: this.$session() });
  });
}

purchaseOrderSchema.plugin(auditPlugin, { entity: 'PurchaseOrder' });

The key detail is this.$session(). The audit entry joins the same transaction as the change it describes, so a rollback removes both.

Small details like this are easy to miss, and a review from experienced MERN developers at this stage usually costs far less than cleaning up inconsistent audit data after a compliance check.

2 more things to handle. Query-level updates like the $inc on stock skip save hooks, so for inventory use a StockMovement ledger with one row per movement and treat onHand as a cached total.

The ledger also lets you rebuild stock for any date, which accountants love at month-end. And make the log append-only by giving the app's database user only insert and find rights on the audit collection.

On the React side, a simple timeline under each record showing who changed which field and when goes a long way.

Once users can see the history themselves, many "who changed this?" support tickets simply stop coming.

Testing the Hard Parts

mongodb-memory-server includes MongoMemoryReplSet, so transactions work in CI without a real cluster. Focus your integration tests on the failure paths:

2 concurrent receipts on the same PO, where one should fail on the version check.
A thrown error halfway through receiveGoods, after which stock, PO and audit rows should all be unchanged.
The same request sent twice with one idempotency key, which should change stock only once.

Wrapping Up

Embed data that belongs to its parent, store money as integers, and index for the reports people actually use.

Wrap related writes in withTransaction, move side effects to an outbox and write audit entries inside the same transaction.

Get these right, and the rest of the module becomes ordinary CRUD work, which is exactly how ERP code should feel.

Two-Phase Commit Is a Trap: How Real Systems Do Distributed Transactions

Athreya aka Maneshwar — Mon, 28 Sep 2026 21:55:38 +0000

When you are first building an application, a transaction is the cheapest guarantee you will ever get.

A customer places an order. You charge their card, reserve the inventory, write a ledger entry for accounting.

You wrap the whole thing in BEGIN and COMMIT, and if any one of those writes falls over, the database quietly rewinds the other two.

It is as if the order never happened. You probably wrote that code in four minutes and never thought about it again.

What you were leaning on there are two of the ACID guarantees, doing unpaid labour on your behalf.

Atomicity. All three writes land, or none of them do. There is no universe where the card gets charged but the stock never gets reserved.

Isolation. While the transaction is in flight, nobody else can see the half-finished middle. Another request checking the customer's balance will not catch a charge for an order that has not finished processing.

You wrote the SQL. The database did the genuinely hard part, for free, and never asked for credit.

Then you split the database, and the guarantee quietly leaves

Traffic grows. Writes grow. That one box starts breathing heavily.

So you do what everyone does. You shard, or you break the monolith into services and give each one its own database.

The details vary. The outcome is identical: your data now lives on several independent machines that have never been introduced to each other.

That one payment flow is now three separate operations against three separate databases.

Card charged in the payments database. Stock reserved in the inventory database. Ledger entry recorded in accounting.

There is no BEGIN big enough to cover all three, because none of them knows the others exist.

So when the card charge commits and the inventory reservation then fails because the last unit just sold, nothing rolls back.

That charge is already durable, on a different machine, in a database that has no idea an order died somewhere else.

At a few hundred orders a day, you will see this maybe once and fix it by hand. At a few thousand transactions a second across distributed infrastructure, partial failure stops being an incident and starts being a Tuesday.

This whole shape of problem has a name: the distributed transaction. One logical operation spanning multiple independent databases or services, where every step needs to either succeed together or get cleaned up when something goes sideways.

The literature offers you two answers. The industry has very firmly picked one of them, and knowing why will save you a genuinely painful quarter.

The textbook answer: two-phase commit

Two-phase commit is the classic academic solution, formalised decades ago in the X/Open XA specification.

You introduce a new component, a coordinator, whose entire job in life is to make every participant agree on the outcome before anybody makes anything permanent.

It runs in two phases, which is where the name comes from.

Phase one, prepare. The coordinator asks every participant: can you commit this? Each database does the real work, durably records the change so a crash cannot lose it, locks the affected rows so nothing else can touch them, and then answers yes or no.

Phase two, commit. If anyone voted no, the coordinator tells everyone to abort and release their locks. If everyone voted yes, it tells everyone to commit, the locks come off, and you are done.

What you get out of this is the same strong consistency you had with one database. Every participant agrees before anything is finalised, so there is no visible window where the system is half-done.

On a whiteboard, this is exactly what you want. It is beautiful. It is also why almost nobody runs it across services.

Blocking is the whole problem

2PC is a blocking protocol, and blocking in a distributed system means you have made your availability the product of everybody else's uptime.

Picture the coordinator collecting all three yes votes. Then, in the gap between collecting them and broadcasting the decision, it dies.

Now look at the participants.

They cannot commit on their own, because the coordinator might have been about to say abort.

They cannot abort on their own either, because the coordinator might have been about to say commit, and one of their peers might have already gone through with it.

So they wait. Holding locks. Indefinitely.

And every other transaction in your system that wants to touch any of those rows is now queued behind a decision that nobody alive can make.

Crashes are not even the interesting failure. The boring one is worse.

A single slow participant stalls the entire transaction. If the ledger service takes ten seconds to answer the prepare message, the payment service and the inventory service both sit there holding locks for ten seconds, doing nothing productive whatsoever.

Your whole system now moves at the speed of its slowest participant, which in a microservice estate is whichever team deployed most recently.

Add a network partition and it gets worse still, because the coordinator cannot distinguish "the message did not arrive" from "the message arrived and the reply got lost". There is no safe default to pick.

Pat Helland's Life beyond Distributed Transactions made this argument in 2007 and it has aged extremely well. Distributed transactions across autonomous services do not work at internet scale, so build systems that do not need them.

Where 2PC actually does live

To be fair to the protocol, 2PC is alive and shipping. It is just not shipping where you would write it yourself.

It lives inside distributed databases. Google Spanner and YugabyteDB both run two-phase commit under the hood to coordinate writes across shards.

The difference is coupling. In there, the coordinator and the participants are components of one system, with one deployment, one failure model, one team, and a consensus protocol like Paxos or Raft underneath to make the coordinator itself fault-tolerant.

Crucially, the database is absorbing that complexity so that you, the caller, never see it.

The moment you try to rebuild that arrangement across independent services with different release trains, different on-call rotations and different opinions about timeouts, every assumption that makes it work in Spanner evaporates.

Sagas: give up atomicity, keep your weekends

When companies genuinely need to coordinate work across services, the pattern they reach for is the saga. Uber, Netflix, Amazon and DoorDash all run this in production.

Sagas start from a deliberately humbler assumption.

You do not actually need all-or-nothing atomicity spanning five services. You need a way to reliably end up in a consistent state, even when a step blows up along the way.

So instead of one big locked distributed transaction, you break the work into a chain of independent local transactions. Each service does its piece and commits to its own database, on its own terms, immediately.

When something fails halfway down the chain, there is no rollback available, because the earlier steps are already committed somewhere else. Instead you run a compensating action.

That is a business-level undo. A refund instead of a rollback. A cancellation instead of an abort.

What you trade away is strong consistency. What you get is eventual consistency.

The system can genuinely be inconsistent for a few seconds. The customer might see a charge land before they see the refund arrive.

But it always converges, and nothing is blocked while it converges. Every other transaction in the system keeps flowing the entire time, because no saga step ever holds a lock that a stranger on another machine is responsible for releasing.

That is the actual trade. Not "correct versus incorrect", but "briefly visible mess" versus "system-wide stall".

Choreography or orchestration

Something has to notice the failure and kick off the compensations. Who that something is turns out to be the design decision that matters most.

Choreography is the decentralised option. It is publish and subscribe: each service broadcasts an event when it finishes, and whoever cares picks it up.

Payments charges the card and publishes CardCharged. Inventory is listening, reserves the stock, publishes StockReserved. Ledger picks that up and records the entry. On failure, a service publishes a failure event and upstream services run their own compensations.

Orchestration is the centralised option. A dedicated orchestrator drives the flow, calling one service at a time and waiting for confirmation before moving on.

Choreography is lovely at three steps. It is miserable at eight.

Once a dozen services are publishing and reacting to each other, "where is order 1841 right now?" becomes an archaeology project. Which step failed? Which compensations already ran? Did the refund actually go through, or is it still retrying?

Nobody owns the answer, so you go and read twelve services' logs to reconstruct it.

Orchestration costs you a central component, and buys you the one thing choreography cannot give you: a single place that knows the state of every in-flight transaction.

Temporal, built by the people behind Uber's Cadence workflow engine, and AWS Step Functions both exist specifically to be that component.

And here is the part that matters most, the reason the orchestrator is not just 2PC wearing a new hat.

When a saga orchestrator crashes, nothing is holding locks.

It is durable. It restarts, reads its own state back from its own database, and resumes from exactly where it stopped. Meanwhile the rest of your system never noticed, because no rows were ever frozen waiting on it.

A dead 2PC coordinator freezes your data. A dead orchestrator just delays one workflow.

Compensating actions are where the honesty comes in

"Just undo the previous step" sounds tidy. In practice, it is where sagas get their reputation.

The card charge committed, the inventory reservation failed, so you refund. Fine. Except that refund is not the invisible cleanup a database rollback gives you.

The customer sees a charge appear on their card. Then, seconds later, a refund. Their bank probably sent them a push notification for each one. It is correct, and it still generates a support ticket.

Some actions are worse than visible. Some are simply not undoable.

You cannot unsend an email. You can send a follow-up saying "ignore that last one", which is a different thing pretending to be the same thing.

If you fired a webhook at a third party, you can send a cancellation, but you cannot make them process it in time, or at all.

Every step in your saga needs a compensating action defined up front, and you should be clear-eyed that some of them are inherently imperfect.

Then there is the recursive problem: compensations fail too.

What happens when the refund API is down at exactly the moment you need to refund? Now you need retries on your failure handling. And the instant you have retries, you need idempotency, otherwise a flaky network refunds the customer twice and you have invented a new bug that is much harder to explain in a postmortem.

The trick is to derive the idempotency key from the thing you are undoing, never from the attempt:

def compensate_charge(order_id, amount_cents):
    # keyed on the order, not on this attempt, so ten retries of this
    # compensation collapse into exactly one refund on the customer's card
    key = f"refund:order:{order_id}"

    return stripe.Refund.create(
        payment_intent=lookup_payment_intent(order_id),
        amount=amount_cents,
        idempotency_key=key,
    )

Generate a fresh UUID per attempt and you have an idempotency key that guarantees nothing.

The uncomfortable summary is that your unhappy path needs the same reliability engineering as your happy path. Most teams budget for one of those.

The dual write problem, quietly waiting behind all of this

Even with perfect compensation logic, there is one more failure mode that catches teams off guard, and it is the one I would bet money is already lurking in your codebase.

When the payment service finishes charging the card, it has to do two things. Save the result to its own database, and publish an event so the next step knows to run.

Those are two writes, to two different systems, with no shared transaction between them.

If the database write succeeds and the event publish fails, the next step never fires and the saga stalls silently. No error, no compensation, just an order sitting in charged forever until a customer emails you.

If the event publishes and the database write fails, downstream services are now reacting enthusiastically to something that never happened.

The fix is the transactional outbox, and it is one of those patterns that feels like cheating the first time you see it.

Stop treating the event as a separate write. Put it in the same database, in the same transaction, as the data.

BEGIN;

UPDATE payments
   SET status = 'charged'
 WHERE order_id = 1841;

INSERT INTO outbox (aggregate_id, event_type, payload)
VALUES (1841, 'CardCharged', '{"order_id":1841,"amount_cents":4999}');

COMMIT;

One local transaction. Both rows commit, or neither does. The dual write is gone, because there is no longer a second write to fail.

A separate background relay then reads that outbox table and publishes to the broker. It can tail the database's own transaction log via change data capture, which is what Debezium does, or it can just poll the table on an interval, which is unglamorous and works fine at most scales.

flowchart TD
    A[Card charged successfully] --> B[BEGIN local transaction]
    B --> C[Write the payment row]
    C --> D[Write the event into the outbox table]
    D --> E[COMMIT: both rows, or neither]
    E --> F{How does the relay<br/>pick it up?}
    F -->|CDC, tail the WAL| G[Publish to the message broker]
    F -->|or just poll the table| G
    G --> H[Next saga step runs]

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef db fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef out fill:#6ea8ff,stroke:#2f5fbf,color:#1a1a1a

    class A start
    class F decision
    class B,C,D,E db
    class G,H out

Note that this gives you at-least-once delivery, not exactly-once. The relay can crash after publishing and before marking the row as sent, so the event goes out twice.

Which brings us back to idempotency. Every consumer in your saga needs to handle seeing the same event twice, because it will.

So what should you actually build?

Before either of these patterns, ask the question that people skip because it feels like a cop-out.

Do you need a distributed transaction at all?

If you can draw your service boundaries so that data which changes together lives in the same database, do that. Move the inventory and ledger tables next to the payments table if they are always updated together.

A local transaction is simpler, faster and more reliable than any distributed thing you can build on top of it. This is always the best answer when you can get away with it, and it is enormously easier to get right up front than to retrofit two years and forty services later.

flowchart TD
    A[One operation spans several services] --> B{Can that data live<br/>in a single database?}
    B -->|Yes| C[Just use a local transaction]
    B -->|No| D{Is eventual consistency<br/>acceptable here?}
    D -->|No| E[Put it in a distributed SQL database<br/>Spanner or YugabyteDB]
    D -->|Yes| F{Many steps, branching,<br/>or tricky compensations?}
    F -->|No| G[Saga with choreography]
    F -->|Yes| H[Saga with orchestration]
    H --> I[Temporal or AWS Step Functions]
    G --> J[Transactional outbox<br/>+ idempotent consumers]
    H --> J

    classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
    classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
    classDef good fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
    classDef alt fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
    classDef tool fill:#6ea8ff,stroke:#2f5fbf,color:#1a1a1a

    class A start
    class B,D,F decision
    class C,H good
    class G,E alt
    class I,J tool

If you truly cannot keep it in one database, you are writing a saga. That part is not really up for debate anymore.

The remaining question is which flavour.

Choreography if the flow is short, the services are genuinely independent, and nobody needs a central view of where a transaction stands. An order placed event triggering an email and a push notification is a perfect fit, and wiring up an orchestrator for it would be theatre.

Orchestration for anything with branching logic, more than a handful of steps, or compensation logic you would rather define in one file than scatter across six repos. Most teams end up here eventually, and the tooling has got good enough that starting here is no longer expensive.

And if eventual consistency is genuinely unacceptable for some specific slice of your data, put that slice in a distributed database that handles strong consistency internally. That is a completely different proposition from hand-rolling 2PC across services you do not control.

The shape that keeps winning

Strip it all back and the production answer at basically every company operating at scale looks the same.

A saga, orchestrated by something durable. Idempotent operations at every step, so retries are always safe. A transactional outbox, so events are as reliable as the database writes they describe.

It means accepting that your system is briefly, visibly inconsistent sometimes.

That is not a compromise the industry stumbled into. It is one it chose, on purpose, after watching two-phase commit deadlock enough production databases to make the decision for everyone.

The database gave you atomicity for free. The moment you split it, the bill came due, and sagas are simply the cheapest way anyone has found to pay it.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by tests…

View on GitHub

Click below to try LiveReview with your codebase: