DEV Community: Runcrate

torch.compile vs eager for training a 468M Llama-style model on four GPUs: step time, memory and compile cost

Runcrate — Thu, 29 Oct 2026 03:24:55 +0000

Does torch.compile make a small transformer train faster? I rented four GPUs on Runcrate today (2026-10-29), an RTX 5090, an L40S, an RTX A6000 and an H100 PCIe, trained the same 468M-parameter Llama-style model in eager mode and compiled, with PyTorch's SDPA attention and with the plain "eager" attention, and deleted every machine when the numbers were saved.

Short version: with SDPA attention, torch.compile changed the step time by between 9% slower (RTX 5090) and 3% faster (L40S), cost 22 to 39 s of compile time on the first step, and cut peak memory by 23% on every GPU (17.3 to 13.3 GB). With the plain attention it was a different story: on the H100 it made the step 2.6x faster and fit the model in 39 GB instead of 56 GB, and on the 48 GB cards it was the only way the run fit at all. Even then, eager mode with SDPA beat compiled plain attention on every GPU.

Setup

GPUs (one per machine), on-demand price at launch time: RTX 5090 32 GB (Austin, US, $1.15 per hour), L40S 48 GB (Des Moines, US, $0.54), RTX A6000 48 GB (Montreal, CA, $0.65), H100 PCIe 80 GB (US Midwest, $3.86).
Model: LlamaForCausalLM from a config, random init: hidden 1024, 24 layers, 16 heads, FFN 4096, vocab 32,000, 468.2M parameters, weights in bf16.
Training step: batch 4 x 2,048 tokens (8,192 tokens per step) of random token ids, loss.backward(), fused AdamW (lr 1e-4), zero_grad(set_to_none=True).
Four setups, each in its own process: eager or torch.compile(model) (default mode), times attn_implementation="sdpa" or "eager" (the naive matmul-softmax-matmul attention in transformers).
Measured: first step time (includes compilation), then 4 more warm-up steps, then the mean of 20 timed steps, tokens per second, and peak memory from torch.cuda.max_memory_allocated.
Software: torch 2.8.0+cu128, transformers 4.57.1, Python 3.12, the same on all four machines.

The script

import json, sys, time, torch
from transformers import LlamaConfig, LlamaForCausalLM
mode, attn = sys.argv[1], sys.argv[2]          # eager|compile, sdpa|eager
B, S = 4, 2048
cfg = LlamaConfig(vocab_size=32000, hidden_size=1024, intermediate_size=4096, num_hidden_layers=24,
                  num_attention_heads=16, num_key_value_heads=16, max_position_embeddings=S,
                  attn_implementation=attn)
model = LlamaForCausalLM(cfg).to("cuda", dtype=torch.bfloat16)
opt = torch.optim.AdamW(model.parameters(), lr=1e-4, fused=True)
fwd = torch.compile(model) if mode == "compile" else model
x = torch.randint(0, 32000, (B, S), device="cuda")
def step():
    loss = fwd(input_ids=x, labels=x).loss
    loss.backward(); opt.step(); opt.zero_grad(set_to_none=True)
t0 = time.time(); step(); torch.cuda.synchronize(); first = time.time() - t0
for _ in range(4): step()
torch.cuda.synchronize(); torch.cuda.reset_peak_memory_stats()
t0 = time.time()
for _ in range(20): step()
torch.cuda.synchronize(); dt = (time.time() - t0) / 20
print(json.dumps({"mode": mode, "attn": attn, "first_step_s": first, "step_ms": dt * 1000,
                  "tokens_per_s": B * S / dt, "peak_gb": torch.cuda.max_memory_allocated() / 1e9}))

Results

GPU	$/hour	Setup	Step (ms)	Tokens/s	Peak memory	First step (s)	M tokens per $
RTX 5090	1.15	eager + sdpa attention	196	41,870	17.3 GB	0.6	131
RTX 5090	1.15	compile + sdpa attention	215	38,040	13.3 GB	21.6	119
RTX 5090	1.15	eager + eager attention	OOM
RTX 5090	1.15	compile + eager attention	OOM
L40S	0.54	eager + sdpa attention	244	33,581	17.3 GB	0.6	224
L40S	0.54	compile + sdpa attention	236	34,691	13.3 GB	34.2	231
L40S	0.54	eager + eager attention	OOM
L40S	0.54	compile + eager attention	391	20,973	39.1 GB	38.9	140
RTX A6000	0.65	eager + sdpa attention	457	17,942	17.3 GB	0.7	99
RTX A6000	0.65	compile + sdpa attention	455	18,005	13.3 GB	39.2	100
RTX A6000	0.65	eager + eager attention	OOM
RTX A6000	0.65	compile + eager attention	546	15,017	39.1 GB	47.1	83
H100	3.86	eager + sdpa attention	152	54,022	17.4 GB	0.4	50
H100	3.86	compile + sdpa attention	160	51,055	13.4 GB	36.4	48
H100	3.86	eager + eager attention	435	18,813	56.0 GB	0.7	18
H100	3.86	compile + eager attention	169	48,469	39.1 GB	43.9	45

"M tokens per $" is training tokens per dollar of GPU time, in millions, at each machine's hourly price. OOM means the run did not fit in the GPU's memory.

What the numbers say

With SDPA attention, compile did not speed up training. Step time compiled against eager: RTX 5090 215 vs 196 ms (9% slower), H100 160 vs 152 ms (5% slower), RTX A6000 455 vs 457 ms (same), L40S 236 vs 244 ms (3% faster). SDPA already runs a fused attention kernel, and at this model size the matmuls dominate, so there was little left for the compiler to fuse.
Compile did save memory: 4 GB, 23%, on every GPU. Peak allocation fell from 17.3 to 13.3 GB. Fused elementwise ops keep fewer intermediate activations for the backward pass, so compile can buy a larger batch even when it does not buy speed.
Compilation took 22 to 47 s. (22 to 39 s with SDPA.) For a short experiment that is a real share of the run, and on the RTX 5090 and the H100, where compiled steps were also slower, it never paid back.
Naive attention is where compile pays. On the H100, eager attention ran at 435 ms per step and 56 GB; compiled, 169 ms (2.6x faster) and 39 GB. On the L40S and the RTX A6000 eager attention ran out of memory without compile and fit with it. On the 32 GB RTX 5090 it did not fit either way.
But the right attention kernel beat the compiler. Eager mode with SDPA was faster than compiled naive attention on every GPU (H100 152 vs 169 ms, L40S 244 vs 391 ms) and used less than half the memory.
Per dollar, the L40S trained the most tokens: 231M tokens per dollar compiled with SDPA, against 131M on the RTX 5090, 100M on the RTX A6000 and 50M on the H100. The H100 was the fastest card (54,022 tokens per second), 1.3x the RTX 5090, at 3.4 times its price.

What to take from it

Use SDPA (or FlashAttention) first. That is the large win and costs nothing.
Then try torch.compile for memory rather than speed at this size: 23% less activation memory can mean a larger batch or a longer context on the same card.
Measure on your GPU. The same compiled model was 9% slower on one card and 3% faster on another.

Caveats

One model size (468M), one batch shape, one run per setup, on one day. Larger models, longer sequences or mode="max-autotune" can change the result; I tested the default mode only.
Pure bf16 weights and optimizer state, no mixed precision with fp32 master weights.
The machines differ in CPU and host; with this batch size the step is GPU-bound, but small differences may come from the host.
Prices are on-demand at launch time and change daily. Billed cost for the four machines, which also ran an embedding benchmark, was $0.14 (RTX 5090), $0.11 (L40S, which also ran a model startup test), $0.08 (RTX A6000) and $0.43 (H100).

Reproduce it

Create an API key and add an SSH key (POST /api/v1/ssh-keys).
Launch a machine with POST /api/v1/instances ({"name": "...", "gpu_type": "L40S", "gpu_count": 1, "ssh_key_id": "...", "max_runtime_minutes": 40}), install torch and transformers==4.57.1 with uv.
Run python train.py compile sdpa and the other three combinations, then delete the instance.

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.

Text embedding throughput per dollar: BGE small, base and large on RTX 5090, L40S, RTX A6000 and H100

Runcrate — Thu, 29 Oct 2026 03:24:15 +0000

What does it cost to embed a million passages with an open model, and does a faster GPU earn its price? I rented four GPUs on Runcrate today (2026-10-29), an RTX 5090, an L40S, an RTX A6000 and an H100 PCIe, measured end-to-end encode() throughput for BGE small, base and large at 128 and 512 tokens, and deleted every machine when the numbers were saved.

Short version: the L40S, at $0.54 an hour, embedded a million passages for the least money in all six cases: $0.032 per million 128-token passages with bge-small, $0.49 per million 512-token passages with bge-large. The H100 was the fastest card only for bge-base and bge-large, and cost 3.6 to 6.8 times more per passage than the L40S. With bge-small, the H100 was slower than the RTX 5090.

Setup

GPUs (one per machine), on-demand price at launch time: RTX 5090 32 GB (Austin, US, $1.15 per hour, 16 vCPUs), L40S 48 GB (Des Moines, US, $0.54, 12 vCPUs), RTX A6000 48 GB (Montreal, CA, $0.65, 28 vCPUs), H100 PCIe 80 GB (US Midwest, $3.86, 20 vCPUs).
Models: BAAI/bge-small-en-v1.5 (33M parameters), BAAI/bge-base-en-v1.5 (109M), BAAI/bge-large-en-v1.5 (335M), fp16.
Software: sentence-transformers 5.7.0, transformers 4.57.1, torch 2.8.0+cu128, Python 3.12.
Input: 4,096 passages per run made of random words from the model's vocabulary, long enough that every passage fills the window, then truncated to exactly 128 or 512 tokens. The content does not change the compute for an encoder; the fixed length makes runs comparable.
Measured: passages per second over at least 20 s of repeated model.encode(texts, batch_size=128) after a warm-up, tokenization included, plus peak GPU memory from torch.cuda.max_memory_allocated. Cost per million passages = hourly price / (passages per second x 3600) x 1,000,000.

The script

import json, sys, time, random, torch
from sentence_transformers import SentenceTransformer
model_id, seqlen = sys.argv[1], int(sys.argv[2])
m = SentenceTransformer(model_id, device="cuda", model_kwargs={"torch_dtype": torch.float16})
m.max_seq_length = seqlen
random.seed(0)
words = [w for w in m.tokenizer.get_vocab() if w.isalpha() and len(w) > 2][:20000]
texts = [" ".join(random.choice(words) for _ in range(seqlen)) for _ in range(4096)]
m.encode(texts[:512], batch_size=128)  # warm-up
torch.cuda.synchronize()
done, t0 = 0, time.time()
while time.time() - t0 < 20:
    m.encode(texts, batch_size=128); done += len(texts)
torch.cuda.synchronize()
print(json.dumps({"model": model_id, "seqlen": seqlen, "passages_per_s": done / (time.time() - t0)}))

Results

128-token passages, fp16, batch 128

Model	GPU	$/hour	Passages/s	Tokens/s	Peak memory	$ per 1M passages
bge-small-en-v1.5	RTX 5090	1.15	6,415	821,094	0.2 GB	$0.050
bge-small-en-v1.5	L40S	0.54	4,714	603,443	0.2 GB	$0.032
bge-small-en-v1.5	RTX A6000	0.65	3,681	471,130	0.2 GB	$0.049
bge-small-en-v1.5	H100	3.86	4,952	633,882	0.2 GB	$0.217
bge-base-en-v1.5	RTX 5090	1.15	4,077	521,843	0.5 GB	$0.078
bge-base-en-v1.5	L40S	0.54	3,054	390,925	0.5 GB	$0.049
bge-base-en-v1.5	RTX A6000	0.65	2,273	290,931	0.5 GB	$0.079
bge-base-en-v1.5	H100	3.86	4,123	527,706	0.6 GB	$0.260
bge-large-en-v1.5	RTX 5090	1.15	1,748	223,731	1.1 GB	$0.183
bge-large-en-v1.5	L40S	0.54	1,374	175,936	1.1 GB	$0.109
bge-large-en-v1.5	RTX A6000	0.65	875	111,987	1.1 GB	$0.206
bge-large-en-v1.5	H100	3.86	2,391	306,061	1.1 GB	$0.448

512-token passages, fp16, batch 128

Model	GPU	$/hour	Passages/s	Tokens/s	Peak memory	$ per 1M passages
bge-small-en-v1.5	RTX 5090	1.15	1,713	877,005	0.7 GB	$0.186
bge-small-en-v1.5	L40S	0.54	1,193	610,867	0.7 GB	$0.126
bge-small-en-v1.5	RTX A6000	0.65	1,068	546,867	0.7 GB	$0.169
bge-small-en-v1.5	H100	3.86	1,428	731,085	0.7 GB	$0.751
bge-base-en-v1.5	RTX 5090	1.15	1,033	528,998	1.4 GB	$0.309
bge-base-en-v1.5	L40S	0.54	763	390,605	1.4 GB	$0.197
bge-base-en-v1.5	RTX A6000	0.65	603	308,838	1.4 GB	$0.299
bge-base-en-v1.5	H100	3.86	1,156	591,974	1.5 GB	$0.927
bge-large-en-v1.5	RTX 5090	1.15	411	210,330	2.3 GB	$0.778
bge-large-en-v1.5	L40S	0.54	305	156,109	2.3 GB	$0.492
bge-large-en-v1.5	RTX A6000	0.65	234	120,064	2.3 GB	$0.770
bge-large-en-v1.5	H100	3.86	598	306,330	2.3 GB	$1.792

What the numbers say

The L40S gave the most passages per dollar for every model and length. For 10 million 512-token chunks with bge-base that is about $2.0 on the L40S, $3.0 on the RTX A6000, $3.1 on the RTX 5090 and $9.3 on the H100 PCIe.
Small models don't fill a big GPU. bge-small ran at 4,952 passages per second on the H100 and 6,415 on the RTX 5090, and on every card its tokens per second changed by 16% or less between 128 and 512 tokens. The GPU is waiting on tokenization and Python between batches, so the per-batch CPU work and the CPU itself decide the speed, not the GPU.
The H100 pulled ahead only when the model was large enough. bge-large at 512 tokens: 598 passages per second on the H100, 411 on the RTX 5090, 305 on the L40S, 234 on the RTX A6000. Still, the L40S did the same work for 27% of the H100's cost.
Memory is not the constraint. Peak allocation was 2.3 GB or less at batch 128, even for bge-large at 512 tokens. Any of these cards could serve several embedding models at once.
The RTX 5090 and the RTX A6000 cost about the same per passage (within 11% on every row), the 5090 doing it 1.6 to 2.0x faster.

What to take from it

For bulk embedding with BGE-class models, a mid-range data-center card at a low hourly price was the efficient choice here. Paying for an H100 bought speed only for the large model, at 3.6 to 4.1 times the L40S's cost per passage.
If you use a small embedding model, throughput depends on the CPU side: tokenize ahead of time, use a fast tokenizer, or run several workers per GPU before you reach for a bigger card.
Real corpora have mixed lengths, and sentence-transformers sorts each call's inputs by length, so short real-world text will run faster than these fixed-length 512-token numbers.

Caveats

One run per setup and machine, on one day. Treat differences under about 10% as noise.
The machines differ in CPU count (12 to 28 vCPUs) and CPU model, which matters most for the small models (point 2).
No ONNX, TensorRT or text-embeddings-inference server: plain sentence-transformers in fp16, the way most pipelines start. A dedicated server would raise the small-model numbers.
Prices are on-demand at launch time and change daily. Billed cost for the four machines, which also ran a training benchmark, was $0.14 (RTX 5090), $0.11 (L40S, which also ran a model startup test), $0.08 (RTX A6000) and $0.43 (H100).

Reproduce it

Create an API key and add an SSH key (POST /api/v1/ssh-keys).
Launch a machine with POST /api/v1/instances ({"name": "...", "gpu_type": "L40S", "gpu_count": 1, "ssh_key_id": "...", "max_runtime_minutes": 40}), install torch and sentence-transformers with uv.
Run python embed.py BAAI/bge-base-en-v1.5 512 (the script above), then delete the instance.

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.

Fresh Hugging Face download vs local cache: time to first token for a 7B model on a Runcrate L40S

Runcrate — Thu, 29 Oct 2026 03:24:13 +0000

How long does a GPU machine sit idle before a 7B model answers its first request, and how much of that is the download? I rented one L40S on Runcrate today (2026-10-29), timed every phase of starting Qwen2.5-7B-Instruct from a fresh Hugging Face download and from a copy already on the machine's disk, and deleted the machine when the numbers were saved.

Short version: from a fresh download, the first token came 48 to 55 s after the Python process started. With the weights already on local disk it came after 18 to 19 s. The download was 30 s of that (15.2 GB at about 515 MB/s), and the part a cache cannot remove was the 14 s it took to read 15 GB from disk into GPU memory.

Setup

GPU: one Runcrate L40S 48 GB (Des Moines, US, $0.54 per hour on demand at launch time), 12 vCPUs, 70 GB RAM.
Model: Qwen/Qwen2.5-7B-Instruct, bf16 safetensors, 15.24 GB on disk (4 shards).
Software: torch 2.8.0+cu128, transformers 4.57.1, huggingface-hub 0.36.2 with hf-xet 1.7.1 (the default download backend), Python 3.12.
One fresh process per measurement, timing four phases: Python imports, snapshot_download (a real download, or a cache check when the files are present), from_pretrained(..., dtype=bfloat16, device_map="cuda"), and generating one token for a short chat prompt.
Six starts: two fresh downloads with the defaults, one fresh download with HF_XET_HIGH_PERFORMANCE=1, one from the local cache after evicting the files from the page cache, and two from the local cache right after a previous run.

The script

import json, os, sys, time
T0 = time.time()
import torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer
r = {"import_s": round(time.time() - T0, 1)}
MID = "Qwen/Qwen2.5-7B-Instruct"
t = time.time(); path = snapshot_download(MID, allow_patterns=["*.json", "*.safetensors", "*.txt"])
r["download_s"] = round(time.time() - t, 1)
t = time.time(); tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map="cuda")
torch.cuda.synchronize(); r["load_to_gpu_s"] = round(time.time() - t, 1)
t = time.time()
ids = tok.apply_chat_template([{"role": "user", "content": "Say hello in five words."}],
                              add_generation_prompt=True, return_tensors="pt").to("cuda")
model.generate(ids, max_new_tokens=1, do_sample=False)
torch.cuda.synchronize(); r["first_token_s"] = round(time.time() - t, 2)
r["total_s"] = round(time.time() - T0, 1)
print(json.dumps(r))

A fresh start is rm -rf $HF_HOME/hub/models--Qwen--Qwen2.5-7B-Instruct before the run. To evict a cached copy from RAM without root (/proc/sys/vm/drop_caches was not writable in this container) I called os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED) on every blob file.

Results

Start	Python imports (s)	Download (s)	Load to GPU (s)	First token (s)	Total (s)	GPU cost of the wait
Fresh download, defaults (run 1)	10.2	29.6	14.0	0.57	54.5	$0.0082
Fresh download, defaults (run 2)	3.7	29.6	13.7	0.54	47.6	$0.0071
Fresh download, HF_XET_HIGH_PERFORMANCE=1	4.0	32.5	14.1	0.50	51.1	$0.0077
Local disk cache, page cache evicted	3.7	0.1	13.9	0.54	18.3	$0.0027
Local disk cache, in page cache (run 1)	3.8	0.1	14.7	0.56	19.1	$0.0029
Local disk cache, in page cache (run 2)	3.8	0.1	14.9	0.53	19.4	$0.0029

GPU cost of the wait is the total time at the L40S's hourly price.

What the numbers say

The download was 60% of a fresh start. 29.6 s in both default runs for 15.24 GB, about 515 MB/s or 4.1 Gbit/s. With the weights on disk the same phase took 0.1 s (a cache check).
HF_XET_HIGH_PERFORMANCE=1 did not help here (32.5 s against 29.6 s). At 4 Gbit/s on 12 vCPUs the default settings already used what this machine's network gave.
Loading to the GPU took 14 s every time, cached or not. That is 15.24 GB at about 1.1 GB/s, and dd read the largest shard at 1.0 GB/s from this disk, both right after eviction and right after a previous read. So on this machine the load is bound by disk reads, and keeping the file in RAM did not change it. I can't prove the eviction took effect, but the load time did not move either way.
The first token itself took about half a second (0.50 to 0.57 s), including tokenizing the prompt and the first CUDA kernels. It is not where startup time goes.
The first Python import of the session took 10 s; later ones 4 s. Bytecode compilation and cold library files cost 6 s once per machine.

What to take from it

If your machines start often, keep the weights on the machine (or on a disk you reattach) instead of pulling them from the Hub each time: here it cut time to first token from about 50 s to about 19 s.
After that, the floor is reading the weights. 15 GB at 1 GB/s is 14 s, so a 70B model in bf16 (about 140 GB) would need roughly two minutes just to load at the same rate, and the download about 4.5 minutes more at 515 MB/s. Both are extrapolations, not measured.
At $0.54 an hour, a fresh start's wait cost under a cent. The money matters less than the latency: a scale-to-zero endpoint that downloads on every cold start makes its first user wait close to a minute.

Caveats

One machine, one region, one day, one model. Network speed to the Hub varies by region and time of day; disk speed varies by machine.
My first script deleted the wrong cache directory (this image sets HF_HOME=/workspace/hf), so its second "fresh" run was served from cache. I reran both fresh downloads and the evicted start on the same machine before it was deleted; those are the numbers above, and the discarded runs are kept with the raw logs.
transformers from_pretrained only. vLLM and other servers add their own startup work (CUDA graphs, KV cache allocation) on top of the load.

Reproduce it

Create an API key and add an SSH key (POST /api/v1/ssh-keys).
Launch an L40S with POST /api/v1/instances ({"name": "...", "gpu_type": "L40S", "gpu_count": 1, "ssh_key_id": "...", "max_runtime_minutes": 40}), install torch and transformers==4.57.1 accelerate hf_xet with uv.
Run the script above once after deleting the model's cache directory and once with it in place, then delete the instance.

The same machine also ran an embedding throughput test and a torch.compile training test; the whole run, about 12 minutes of billed L40S time, was billed $0.11.

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.

Whisper large-v3 hours of audio per dollar on four GPUs: RTX 5090, L40S, RTX A6000 and H100, five setups measured

Runcrate — Wed, 28 Oct 2026 15:17:27 +0000

How many hours of audio can Whisper large-v3 transcribe for a dollar? It depends on the GPU, but it depends even more on how you run it. I rented four GPUs on Runcrate today (2026-10-28), an RTX 5090, an L40S, an RTX A6000 and an H100 PCIe, ran the same 2.5 hours of English talks through five Whisper setups on each, measured speed and word error rate, and deleted every machine when the results were on disk.

Short version: batched faster-whisper with the turbo model transcribed 315 hours of audio per hour on the RTX 5090, about 267 hours of audio per dollar, at a 3.1% word error rate. Plain sequential large-v3 with beam search, the way most tutorials start, was 6.6x slower than the same weights batched on that GPU and had a higher error rate (5.7% against 3.2%).

Setup

GPUs (one per machine), on-demand price at launch time: RTX 5090 32 GB (Austin, US, $1.18 per hour), L40S 48 GB (Kansas City, US, $1.14), RTX A6000 48 GB (US Midwest, $0.66), H100 PCIe 80 GB (US Midwest, $3.86).
Audio: the long-form TED talk test set from the distil-whisper project (distil-whisper/tedlium-long-form): 11 full TED talks, 8,905 s (2 h 28 min) of 16 kHz English, with reference transcripts.
Five setups, each in its own process:
- faster-whisper 1.2.1 (CTranslate2 4.8.4), BatchedInferencePipeline, batch 16, with large-v3 in fp16, large-v3 in int8 (int8_float16), and large-v3-turbo in fp16;
- transformers 4.57.1 pipeline("automatic-speech-recognition"), large-v3 in fp16, 30 s chunks, batch 16, SDPA attention;
- faster-whisper large-v3 in fp16, plain model.transcribe (sequential, beam 5), the setup most tutorials start with.
Measured: wall time to transcribe all 11 talks after a one-minute warm-up (model load excluded), speed as a multiple of real time, word error rate over all talks with Whisper's English text normalizer, and peak GPU memory from nvidia-smi sampled every 0.5 s.
torch 2.8.0+cu128, Python 3.12, on all four machines.

The commands

Launch (the instance_type_id comes from GET /api/v1/instances/types):

curl -s https://www.runcrate.ai/api/v1/instances \
  -H "Authorization: Bearer $RUNCRATE_API_KEY" -H "Content-Type: application/json" \
  -d '{"name":"whisper-l40s","instance_type_id":"<id from /instances/types>",
       "ssh_key_id":"<your key id>","max_runtime_minutes":80}'

On the machine:

curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH=$HOME/.local/bin:$PATH
uv venv --python 3.12 /opt/env && . /opt/env/bin/activate
uv pip install torch faster-whisper 'transformers==4.57.1' accelerate soundfile pandas pyarrow jiwer --torch-backend=cu128
# CTranslate2 needs the cuBLAS and cuDNN that came with torch on its library path
export LD_LIBRARY_PATH=$(python -c "import os, nvidia.cublas, nvidia.cudnn; print(os.path.join(nvidia.cublas.__path__[0],'lib')+':'+os.path.join(nvidia.cudnn.__path__[0],'lib'))"):$LD_LIBRARY_PATH
for CFG in fw_large_v3_fp16_b16 fw_large_v3_int8_b16 fw_turbo_fp16_b16 hf_large_v3_fp16_b16 fw_large_v3_fp16_seq; do
  python whisper_bench.py $CFG > results/$CFG.json
done

The two faster-whisper modes in whisper_bench.py:

from faster_whisper import WhisperModel, BatchedInferencePipeline
model = WhisperModel("large-v3", device="cuda", compute_type="float16")  # or "int8_float16"
# batched
segs, _ = BatchedInferencePipeline(model=model).transcribe(audio, language="en", batch_size=16)
# sequential
segs, _ = model.transcribe(audio, language="en", beam_size=5)
text = " ".join(s.text for s in segs)

Then DELETE /api/v1/instances/<id> and a check that the instance list is empty.

Results

faster-whisper large-v3-turbo, fp16, batch 16

GPU	$/hour	x realtime	WER	Peak GPU memory	Audio hours per $	$ per 1,000 audio hours
RTX 5090	1.18	315x	3.1%	4.6 GiB	267	$3.75
L40S	1.14	192x	3.1%	4.5 GiB	168	$5.94
RTX A6000	0.66	121x	3.1%	4.4 GiB	183	$5.45
H100 PCIe	3.86	196x	3.1%	4.6 GiB	51	$19.65

faster-whisper large-v3, fp16, batch 16

GPU	$/hour	x realtime	WER	Peak GPU memory	Audio hours per $	$ per 1,000 audio hours
RTX 5090	1.18	171x	3.2%	10.2 GiB	145	$6.89
L40S	1.14	109x	3.2%	10.3 GiB	96	$10.43
RTX A6000	0.66	72x	3.2%	9.8 GiB	109	$9.15
H100 PCIe	3.86	118x	3.2%	10.0 GiB	30	$32.80

faster-whisper large-v3, int8, batch 16

GPU	$/hour	x realtime	WER	Peak GPU memory	Audio hours per $	$ per 1,000 audio hours
RTX 5090	1.18	172x	3.2%	8.4 GiB	146	$6.86
L40S	1.14	109x	3.2%	8.3 GiB	95	$10.49
RTX A6000	0.66	73x	3.2%	8.1 GiB	110	$9.08
H100 PCIe	3.86	119x	3.1%	8.3 GiB	31	$32.36

transformers large-v3, fp16, batch 16

GPU	$/hour	x realtime	WER	Peak GPU memory	Audio hours per $	$ per 1,000 audio hours
RTX 5090	1.18	47x	2.9%	27.9 GiB	40	$25.05
L40S	1.14	23x	2.9%	27.6 GiB	20	$48.93
RTX A6000	0.66	19x	2.9%	27.5 GiB	29	$34.92
H100 PCIe	3.86	36x	2.9%	27.9 GiB	9	$106.93

faster-whisper large-v3, fp16, sequential, beam 5

GPU	$/hour	x realtime	WER	Peak GPU memory	Audio hours per $	$ per 1,000 audio hours
RTX 5090	1.18	26x	5.7%	5.6 GiB	22	$45.74
L40S	1.14	19x	5.5%	5.5 GiB	17	$60.00
RTX A6000	0.66	10x	6.0%	5.4 GiB	15	$65.35
H100 PCIe	3.86	19x	5.5%	5.6 GiB	5	$207.53

"Audio hours per $" is the speed divided by the hourly price, so it assumes the GPU is kept busy. The four machines were billed $2.23 in total for the whole run, including boot, installs and downloads: $0.30 (RTX 5090), $0.39 (L40S), $0.35 (RTX A6000) and $1.19 (H100 PCIe).

What surprised me

The RTX 5090 was the fastest GPU here, ahead of the H100 PCIe, in every setup: 171x against 118x for batched large-v3, 315x against 196x for turbo. At about a third of the hourly price it gave 4.3 to 5.2x the audio per dollar. The batched pipeline also does audio decoding, voice activity detection and feature extraction on the CPU, so the GPU is not the only limit; I did not profile where the time went on each machine.
int8 gave no speedup on any of the four GPUs. It saved about 1.8 GiB of memory and changed the error rate by 0.1 points at most. If memory is not your problem, fp16 is as good.
Turbo was 1.7 to 1.8x faster than large-v3 at the same error rate (3.1% against 3.2%) on this English set. Turbo keeps large-v3's encoder and cuts the decoder from 32 layers to 4.
Sequential beam-5 transcription was the worst on both speed and accuracy. 5.5 to 6.0% WER against 3.2% for the same weights batched, and 5.7 to 7.1x slower. On the L40S the first 250 characters of the first talk were identical in all three large-v3 setups, so the gap comes from later in the talks; long-form sequential decoding conditions each window on the text before it, which is a known source of repeated or dropped passages. I did not diff the transcripts line by line.
transformers had the best error rate (2.9%) and the highest cost. 3 to 5x slower than batched faster-whisper with the same weights, and 28 GiB peak memory against 10 GiB.

What to take from it

For bulk English transcription, batched faster-whisper with turbo is the setup to start from, and a consumer-class 32 GB card did it at about $3.75 per 1,000 hours of audio here.
Don't pay for int8 conversion or an H100 to make Whisper faster. Batching is the large win, and the model choice (turbo) the second.
If the last 0.3 points of WER matter more than cost, the transformers pipeline was the most accurate in this test.

Caveats

One run per setup, per machine, on one day. Treat differences under about 10% as noise.
One English test set of TED talks: clean, single-speaker audio. Noisy calls, accents or other languages will give different error rates, and turbo is known to lose more accuracy than large-v3 on some languages.
Peak memory comes from nvidia-smi, which includes what the frameworks reserve, not only what they use.
The CPU, disk and network differ between the four machines (6 to 20 vCPUs), and the batched pipeline uses the CPU, so part of the GPU-to-GPU difference may come from the rest of the machine.
Prices are on-demand at launch time and move daily.

Reproduce it

Create an API key and add an SSH key (POST /api/v1/ssh-keys).
Pick a GPU from GET /api/v1/instances/types and launch with the curl above, max_runtime_minutes set as a safety net.
Run the block above, copy results/ back, delete the instance.

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.

Full fine-tune vs LoRA vs QLoRA on one L40S: Qwen2.5-1.5B, memory, speed and held-out loss, measured

Runcrate — Wed, 28 Oct 2026 15:15:31 +0000

"Use LoRA, it needs far less memory" and "use QLoRA if LoRA doesn't fit" are the standard advice for fine-tuning on one GPU. I wanted to see what each one costs on a small model, where the weights are not the problem, so I rented an L40S (48 GB) on Runcrate today (2026-10-28) and fine-tuned Qwen2.5-1.5B-Instruct three ways on the same data for the same number of steps. Two machines, both deleted when the results were on disk.

Short version: full fine-tuning needed 36.9 GiB at micro-batch 4 and ran out of memory at micro-batch 8. LoRA needed 14.5 GiB, trained 1.4x faster and reached a slightly lower held-out loss. QLoRA was slower than both and used more memory than LoRA, not less.

Setup

GPU: one L40S (48 GB, 44.4 GiB usable), Kansas City, US, $1.14 per hour on demand at launch time.
Model: Qwen/Qwen2.5-1.5B-Instruct (1.54 billion parameters).
Data: yahma/alpaca-cleaned, shuffled with a fixed seed: 4,000 examples for training (753,486 tokens), the next 300 held out. Qwen chat template, loss on the answer tokens only, at most 512 tokens per example.
Training: one epoch, 250 optimizer steps of 16 examples each, cosine schedule with 10 warm-up steps, gradient clipping at 1.0, AdamW, no gradient checkpointing. A plain PyTorch loop, the same for all three:
- full: fp32 master weights, bf16 autocast, learning rate 1e-5;
- LoRA: bf16 base, rank 16, alpha 32, dropout 0.05 on all attention and MLP projections (18.5 M trainable parameters, 1.2% of the model), learning rate 2e-4;
- QLoRA: the same LoRA on a 4-bit NF4 base (double quantization, bf16 compute, bitsandbytes 0.50.2), prepared with peft's prepare_model_for_kbit_training.
Two runs: micro-batch 8 with 2 accumulation steps first. Full fine-tuning ran out of memory there, so a second machine ran all three at micro-batch 4 with 4 accumulation steps (still 16 per step).
Measured: training wall time (excluding model load and evaluation), tokens per second, peak memory from torch.cuda.max_memory_allocated, loss on the 300 held-out examples before and after.
Software: torch 2.8.0+cu128, transformers 4.57.1, peft 0.17.1, accelerate 1.15.0, Python 3.12.

The commands

Launch (the instance_type_id comes from GET /api/v1/instances/types):

curl -s https://www.runcrate.ai/api/v1/instances \
  -H "Authorization: Bearer $RUNCRATE_API_KEY" -H "Content-Type: application/json" \
  -d '{"name":"ft-l40s","instance_type_id":"<id from /instances/types>",
       "ssh_key_id":"<your key id>","max_runtime_minutes":60}'

On the machine:

curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH=$HOME/.local/bin:$PATH
uv venv --python 3.12 /opt/env && . /opt/env/bin/activate
uv pip install torch 'transformers==4.57.1' 'peft==0.17.1' bitsandbytes accelerate --torch-backend=cu128
for CFG in full lora qlora; do MB=4 ACC=4 python ft_bench.py $CFG > results/${CFG}_mb4.json; done

The parts of ft_bench.py that differ between the three:

if CFG == "full":
    model = AutoModelForCausalLM.from_pretrained(M, torch_dtype=torch.float32).cuda()
    params, lr = list(model.parameters()), 1e-5
else:
    if CFG == "qlora":
        q = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                               bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
        model = AutoModelForCausalLM.from_pretrained(M, quantization_config=q, device_map={"": 0})
        model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=False)
    else:
        model = AutoModelForCausalLM.from_pretrained(M, torch_dtype=torch.bfloat16).cuda()
    model = get_peft_model(model, LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]))
    params, lr = [p for p in model.parameters() if p.requires_grad], 2e-4
opt = torch.optim.AdamW(params, lr=lr, weight_decay=0.0)
# training step: with torch.autocast("cuda", dtype=torch.bfloat16): loss = model(...).loss

Then DELETE /api/v1/instances/<id> and a check that the instance list is empty.

Results

Method	Micro-batch x accumulation	Trainable	Train time	Tokens/s	Peak memory (PyTorch)	Held-out loss	GPU cost of training
full	4 x 4	1,543.7 M	210 s	3,595	36.9 GiB	1.103 to 1.065	$0.066
LoRA	4 x 4	18.5 M	152 s	4,970	14.5 GiB	1.102 to 1.058	$0.048
QLoRA	4 x 4	18.5 M	215 s	3,507	16.5 GiB	1.144 to 1.090	$0.068
full	8 x 2	1,543.7 M	out of memory
LoRA	8 x 2	18.5 M	161 s	4,680	25.8 GiB	1.102 to 1.056	$0.051
QLoRA	8 x 2	18.5 M	237 s	3,184	30.7 GiB	1.144 to 1.089	$0.075

The cost column is training time only, at $1.14 per hour. The two machines were billed $0.18 and $0.24 in total, including boot, installs, downloads, evaluation and the run that failed.

All three models gave nearly the same answers to three fixed prompts afterwards. For "Rewrite this sentence in the passive voice: The cat chased the mouse." all three answered "The mouse was chased by the cat."

What surprised me

Full fine-tuning of a 1.5B model does not fit in 48 GB at micro-batch 8, without gradient checkpointing. fp32 weights, fp32 gradients and the two AdamW states are 16 bytes per parameter, about 23 GiB before a single activation. At micro-batch 4 it peaked at 36.9 GiB. The OOM came in the backward pass with 39.6 GiB already allocated.
QLoRA saved nothing here. On a 1.5B model the bf16 weights are only about 3 GiB, so quantizing them to 4 bits can save at most about 2 GiB, and the activations at these batch sizes are much larger than that. QLoRA's peak was 2 GiB higher than LoRA's at micro-batch 4 and 4.9 GiB higher at micro-batch 8. peft's prepare_model_for_kbit_training casts every layer that is not quantized to fp32, including Qwen's 151,936-token embedding, and the 4-bit layers are dequantized on every forward and backward pass; my guess is that these two account for the extra memory and for QLoRA being the slowest of the three, but I did not profile it.
Quantizing the base model cost almost as much loss as the fine-tune removed. QLoRA's held-out loss before training was 1.144, against 1.102 for the bf16 model, a 0.042 penalty. Training removed 0.054, about as much as LoRA removed (0.044), so QLoRA finished at 1.090, worse than where LoRA and full fine-tuning finished.
LoRA matched full fine-tuning on held-out loss (1.058 against 1.065) with 1.2% of the parameters trainable, while training 1.4x faster and using 40% of the memory. On this data and at these learning rates, full fine-tuning bought nothing.
Halving the micro-batch made LoRA faster, not slower (152 s against 161 s) and cut its peak memory from 25.8 to 14.5 GiB. The examples vary in length and each micro-batch is padded to its longest one, so smaller micro-batches probably waste less compute on padding.

What to take from it

For a model this size on a 48 GB card, plain LoRA in bf16 is the default to beat: least memory, fastest, and no worse on held-out loss than full fine-tuning here.
QLoRA is a tool for models whose bf16 weights don't fit. Below a few billion parameters, the activations dominate, and the 4-bit base costs speed and starting quality for no memory back.
If you need full fine-tuning, micro-batch and gradient checkpointing decide whether it fits; check the peak memory on a few steps before you rent a bigger GPU.

Caveats

One run per setting, one seed, one GPU type, one day. Treat differences under about 10% as noise, and the loss differences between full and LoRA as a tie.
One epoch on 4,000 Alpaca examples, starting from an instruct model that already follows instructions, so the loss moves only a little. A harder task or more data could favour full fine-tuning.
The learning rates (1e-5 for full, 2e-4 for LoRA) are common defaults, not tuned. Full fine-tuning at a higher rate might do better.
No gradient checkpointing, no fused or 8-bit optimizers, no FlashAttention 2 (PyTorch SDPA). Each would change the memory numbers.
Price is the on-demand price at launch time and moves daily.

Reproduce it

Create an API key and add an SSH key (POST /api/v1/ssh-keys).
Pick an L40S from GET /api/v1/instances/types and launch with the curl above, max_runtime_minutes set as a safety net.
Run the block above, copy results/ back, delete the instance.

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.

Ollama Q4 vs vLLM bf16 on a 24 GB A10: 3x the single-user speed, 1.2 points of GSM8K, and where vLLM still wins

Runcrate — Wed, 28 Oct 2026 15:14:18 +0000

If you have one 24 GB GPU and a 7B model to serve, the usual advice is "use vLLM for throughput, use Ollama (llama.cpp) for convenience". I wanted numbers for that on the same card, the same model and the same requests, so I rented one NVIDIA A10 (24 GB) on Runcrate today (2026-10-28), served Qwen2.5-7B-Instruct three ways, and deleted the machine when the results were on disk.

Short version: on this card, Ollama with the Q4_K_M build was faster than vLLM in bf16 at every load I tried, from 1 to 16 users at once, and cost less per token. Q4 scored 90.0% on 250 GSM8K questions against 91.2% for bf16. vLLM kept the better time to first token once more than one user was waiting.

Setup

GPU: one NVIDIA A10, 24 GB (23,028 MiB usable, 150 W limit), Dulles, US, $1.68 per hour on demand at launch time.
Model: Qwen2.5-7B-Instruct, three ways:
- Ollama 0.40.12, qwen2.5:7b-instruct-q4_K_M (GGUF, 4-bit, llama.cpp runner),
- Ollama 0.40.12, qwen2.5:7b-instruct-q8_0 (GGUF, 8-bit),
- vLLM 0.11.0, Qwen/Qwen2.5-7B-Instruct in bf16 (torch 2.8.0+cu128, transformers 4.57.6), --max-model-len 4096 --gpu-memory-utilization 0.92.
Ollama settings: OLLAMA_NUM_PARALLEL=16 OLLAMA_CONTEXT_LENGTH=4096, so it can run 16 requests at once with a 4,096-token context each. All 29 layers were on the GPU.
Client: one Python script for all three servers, through each server's OpenAI-compatible /v1/chat/completions, streamed. 32 fixed "explain X to a new engineer" prompts, temperature 0, max_tokens 256. Every request in every run produced the full 256 tokens, so all three servers did the same amount of work. 1, 4 and 16 users at once (12, 32 and 96 requests), one warm-up request first. The client ran on the same machine.
Quality check: the first 250 questions of the GSM8K test set, zero-shot, greedy, up to 512 tokens, 16 at a time, answer read from the last number.

The commands

Launch (the instance_type_id comes from GET /api/v1/instances/types):

curl -s https://www.runcrate.ai/api/v1/instances \
  -H "Authorization: Bearer $RUNCRATE_API_KEY" -H "Content-Type: application/json" \
  -d '{"name":"ollama-a10","instance_type_id":"<id from /instances/types>",
       "ssh_key_id":"<your key id>","max_runtime_minutes":90}'

On the machine, Ollama:

curl -fsSL https://ollama.com/install.sh | sh
systemctl stop ollama   # run our own server with our settings
OLLAMA_NUM_PARALLEL=16 OLLAMA_CONTEXT_LENGTH=4096 OLLAMA_KEEP_ALIVE=-1 ollama serve &
ollama pull qwen2.5:7b-instruct-q4_K_M
python client.py http://localhost:11434/v1 qwen2.5:7b-instruct-q4_K_M speed 16 96
python client.py http://localhost:11434/v1 qwen2.5:7b-instruct-q4_K_M gsm8k

vLLM:

uv venv --python 3.12 /opt/env && . /opt/env/bin/activate
uv pip install vllm==0.11.0 'transformers<4.58' openai pandas pyarrow --torch-backend=cu128
vllm serve Qwen/Qwen2.5-7B-Instruct --dtype bfloat16 --max-model-len 4096 \
  --gpu-memory-utilization 0.92 --port 8000 &
python client.py http://localhost:8000/v1 Qwen/Qwen2.5-7B-Instruct speed 16 96
python client.py http://localhost:8000/v1 Qwen/Qwen2.5-7B-Instruct gsm8k

The heart of client.py, per request:

stream = await client.chat.completions.create(
    model=MODEL, messages=[{"role": "user", "content": prompt(i)}],
    temperature=0, max_tokens=256, stream=True,
    stream_options={"include_usage": True}, seed=0)
async for c in stream:
    ...  # time to first content chunk, then usage.completion_tokens at the end

Both servers reported usage in the stream, so token counts come from the servers, not from counting chunks.

Results

Server	Users	Output tok/s	Per-user tok/s (median)	Median TTFT	p95 TTFT	$ per 1M output tokens
Ollama Q4_K_M	1	89	90	35 ms	40 ms	$5.25
Ollama Q4_K_M	4	239	61	110 ms	136 ms	$1.95
Ollama Q4_K_M	16	618	41	405 ms	483 ms	$0.75
Ollama Q8_0	1	58	58	42 ms	44 ms	$8.09
Ollama Q8_0	4	197	50	128 ms	152 ms	$2.37
Ollama Q8_0	16	484	32	456 ms	511 ms	$0.96
vLLM bf16	1	30	30	47 ms	48 ms	$15.40
vLLM bf16	4	119	30	72 ms	100 ms	$3.93
vLLM bf16	16	449	28	106 ms	173 ms	$1.04

The dollar column is $1.68 per hour divided by the output tokens per hour at that load, so it assumes the GPU stays that busy.

Server	GPU memory in use	GSM8K (250, greedy)	GSM8K wall time
Ollama Q4_K_M	8.4 GiB	90.0% (225/250)	104 s
Ollama Q8_0	11.3 GiB	90.4% (226/250)	140 s
vLLM bf16	21.3 GiB	91.2% (228/250)	125 s

vLLM's memory figure is what it reserves up front (--gpu-memory-utilization 0.92), not what it needed: 4.61 GiB of that was KV cache, room for 86,352 tokens. Ollama allocated 3.5 GiB of KV cache for 16 slots of 4,096 tokens.

Other timings from the same run: ollama pull took 22 s for Q4_K_M (4.7 GB) and 34 s for Q8_0, and the first request after a pull loaded the model in under 3 s. vLLM took 61 s from vllm serve to answering, with the weights already downloaded.

What surprised me

A single user got 3x the speed from Q4. 89 tokens per second against 30. Generating one token means reading every weight once, so a model that is 4.7 GB instead of 15 GB is read about three times faster on the same memory bandwidth. The A10's bandwidth (about 600 GB/s) is the limit here, not its compute.
vLLM's per-user speed did not drop at all from 1 to 16 users (30, 30, 28 tokens per second), while Ollama Q4's went from 90 to 41. vLLM batches the 16 streams into one pass over the weights, so 16 users cost almost the same as one. Ollama also batches, but each extra user cost it more. At 16 users Ollama Q4 still led on total throughput (618 against 449 tokens per second), but the gap had closed from 3x to 1.4x. I did not test past 16 users.
vLLM answered first under load. At 16 users the median time to first token was 106 ms for vLLM and 405 ms for Ollama Q4. If your users notice the wait before the first word more than the speed after it, that matters more than the throughput column.
Q8 was not a free middle ground. It was slower than Q4 at every load and scored one more GSM8K question (226 against 225). Against bf16, Q4 and bf16 disagreed on 13 of 250 questions: bf16 got 8 of them right and Q4 got 5. A 1.2-point gap on 250 questions is within what reruns could move.

What to take from it

On a 24 GB card with a 7B model and up to 16 concurrent users, a 4-bit GGUF under Ollama gave more tokens per dollar than bf16 under vLLM, at a small cost in GSM8K accuracy.
This is not Ollama against vLLM in general. vLLM can also serve 4-bit and 8-bit models (AWQ, GPTQ, FP8 on newer cards), and that comparison would be fairer to it. Here vLLM ran the full bf16 weights because that is what most people start with.
If you serve many users at once and care about time to first token, vLLM's behaviour under load (flat per-user speed, low TTFT) is the reason to choose it. Measure at your own concurrency.

Caveats

One run per setting, on one machine, on one day. Treat differences under about 10% as noise.
256 output tokens and short prompts (about 30 tokens). Long prompts would shift the picture towards prefill speed, where the two engines differ in other ways.
GSM8K is one narrow test of quality. Quantization can hurt other tasks more or less.
Price is the on-demand price at launch time and moves daily.

Reproduce it

Create an API key and add an SSH key (POST /api/v1/ssh-keys).
Pick an A10 (or any 24 GB card) from GET /api/v1/instances/types and launch with the curl above, max_runtime_minutes set as a safety net.
Run the blocks above, copy the results back, delete the instance.

The whole run, including both installs, the model downloads, the speed tests and three GSM8K passes, was billed $0.54 (about 19 minutes at $1.68 per hour; billing starts when the machine reports running, 5 minutes after the create call).

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.

SDXL images per dollar on four GPUs: RTX A6000, L40S, RTX 5090 and H100, eager vs torch.compile

Runcrate — Tue, 27 Oct 2026 16:28:10 +0000

How many SDXL images does a dollar buy on a rented GPU? It depends on the card, the batch size and whether you pay for torch.compile, so I measured it. Today (2026-10-27) I rented four single-GPU machines on Runcrate, ran the same SDXL script on each, and terminated them as soon as the numbers were on disk.

Short version: an RTX A6000 made 1,000 images for $0.91, an RTX 5090 for $0.93, an L40S for $1.01 and an H100 for $1.80. The H100 was the fastest by far, 1.67 s per image against 4.94 s on the A6000, and the most expensive per image.

Setup

Model: stabilityai/stable-diffusion-xl-base-1.0, fp16 weights, with the madebyollin/sdxl-vae-fp16-fix VAE. Base model only, no refiner.
Image: 1024 x 1024, 30 steps, the pipeline's default scheduler and guidance, one fixed prompt ("a photo of an astronaut riding a horse on a beach at sunset, detailed, 35mm").
Software: diffusers 0.42.1, torch 2.8.0+cu128, Python 3.12.
Four configurations per GPU: batch 1 and batch 4 (num_images_per_prompt), each in eager mode and with torch.compile(pipe.unet, fullgraph=True) plus channels-last memory format. One warm-up generation before each timed run; 8 images timed at batch 1, 16 at batch 4.
Machines (on-demand price at launch time):

Runcrate GPU	VRAM	Region	$/hour
RTX A6000	48 GB	US Midwest	0.66
L40S	48 GB	Des Moines, US	1.14
RTX 5090	32 GB	Gdansk, PL	1.33
H100 PCIe	80 GB	Des Moines, US	3.88

Cost per 1,000 images is hourly price / images per hour x 1,000, with the GPU kept busy. It does not include setup time; that is below.

The commands

Launch (one call per GPU; the instance_type_id comes from GET /api/v1/instances/types):

curl -s https://www.runcrate.ai/api/v1/instances \
  -H "Authorization: Bearer $RUNCRATE_API_KEY" -H "Content-Type: application/json" \
  -d '{"name":"sdxl-a6000","instance_type_id":"<id from /instances/types>",
       "ssh_key_id":"<your key id>","max_runtime_minutes":50}'

On the machine:

curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH=$HOME/.local/bin:$PATH
uv venv --python 3.12 /opt/env && . /opt/env/bin/activate
uv pip install torch diffusers transformers accelerate safetensors --torch-backend=cu128
python sdxl_bench.py

The core of sdxl_bench.py:

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe = StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", vae=vae,
        torch_dtype=torch.float16, variant="fp16", use_safetensors=True).to("cuda")

def run(batch, n_batches):
    pipe(PROMPT, num_inference_steps=30, height=1024, width=1024, num_images_per_prompt=batch)  # warm-up
    torch.cuda.synchronize(); t = time.time()
    for _ in range(n_batches):
        pipe(PROMPT, num_inference_steps=30, height=1024, width=1024, num_images_per_prompt=batch)
    torch.cuda.synchronize()
    return (time.time() - t) / (batch * n_batches)   # seconds per image

run(1, 8); run(4, 4)                                  # eager
pipe.unet.to(memory_format=torch.channels_last)
pipe.unet = torch.compile(pipe.unet, fullgraph=True)
run(1, 8); run(4, 4)                                  # compiled

Then DELETE /api/v1/instances/<id> and a check that the instance list is empty.

Results

GPU	Config	Seconds per image	Images per hour	$ per 1,000 images	Peak VRAM
RTX A6000	eager, batch 1	5.79	622	1.06	9.0 GB
RTX A6000	eager, batch 4	5.65	637	1.04	14.6 GB
RTX A6000	compiled, batch 1	4.94	728	0.91	9.0 GB
RTX A6000	compiled, batch 4	5.13	701	0.94	14.6 GB
L40S	eager, batch 1	3.48	1,034	1.10	9.0 GB
L40S	eager, batch 4	3.39	1,064	1.07	14.6 GB
L40S	compiled, batch 1	3.21	1,121	1.02	9.0 GB
L40S	compiled, batch 4	3.19	1,130	1.01	14.6 GB
RTX 5090	eager, batch 1	2.77	1,301	1.02	9.0 GB
RTX 5090	eager, batch 4	2.66	1,355	0.98	14.6 GB
RTX 5090	compiled, batch 1	2.53	1,424	0.93	9.0 GB
RTX 5090	compiled, batch 4	2.53	1,421	0.94	14.6 GB
H100 PCIe	eager, batch 1	2.25	1,603	2.42	9.0 GB
H100 PCIe	eager, batch 4	1.94	1,853	2.09	14.6 GB
H100 PCIe	compiled, batch 1	1.80	2,003	1.94	9.0 GB
H100 PCIe	compiled, batch 4	1.67	2,156	1.80	14.6 GB

What torch.compile costs up front (the first compiled call, measured on the machine) and how many images it takes to earn that back at batch 1:

GPU	Compile, batch 1	Compile, batch 4	Saved per image (batch 1)	Break-even
RTX A6000	188 s	255 s	0.85 s	about 220 images
L40S	82 s	121 s	0.27 s	about 300 images
RTX 5090	96 s	133 s	0.24 s	about 400 images
H100 PCIe	94 s	130 s	0.45 s	about 210 images

Batch 4 recompiles (new input shape), so a server that mixes batch sizes pays both.

Setup on each machine, before the first image:

GPU	Launch to SSH	diffusers install	Model download (about 7 GB)	Pipeline load
RTX A6000	288 s	29 s	73 s	3.9 s
L40S	190 s	under 1 s	24 s	2.0 s
RTX 5090	122 s	15 s	25 s	2.8 s
H100 PCIe	307 s	7 s	27 s	2.5 s

What surprised me

Three of the four cards were within 11% of each other per image, and the H100 cost about twice as much per image. The A6000, RTX 5090 and L40S landed between $0.91 and $1.01 per 1,000 images. The H100 was 3x faster than the A6000 and cost 5.9x more per hour.
Batch 4 barely helped, except on the H100. SDXL at 1024 x 1024 already fills these GPUs with one image: batch 4 changed throughput by -4% to +4% on the A6000, L40S and 5090. Only the H100 gained (16% eager, 8% compiled). Batching mostly adds latency for the person waiting for the first image.
torch.compile cut the time per image by 8% to 20%, if you generate a few hundred images per process. The biggest gain was on the A6000 (5.79 to 4.94 s), which also had the slowest compile (188 s). For a one-off batch of 50 images, eager mode is the better deal.
Peak memory was 9.0 GB at batch 1 and 14.6 GB at batch 4, so SDXL base fits on any of these cards with room to spare. VRAM size did not matter for this workload; compute did.
The model download varied 3x between machines (24 s to 73 s for the same 7 GB). For short jobs, keep the weights on a volume.

Caveats

One run per GPU, 8 to 16 timed images per configuration. Treat differences under about 5% as noise.
One prompt, one resolution, one scheduler, 30 steps. Fewer steps, a distilled model (SDXL Turbo, Lightning) or a different scheduler change the absolute numbers; rerun before relying on them.
No refiner, no ControlNet, no LoRA, no xFormers or other attention kernels beyond PyTorch's built-in SDPA, no TensorRT.
Prices are the on-demand prices at launch time on 2026-10-27 and move daily.

Reproduce it

Create an API key and add an SSH key (POST /api/v1/ssh-keys).
Pick a GPU from GET /api/v1/instances/types and launch with the curl above, max_runtime_minutes set as a safety net.
Run the block above, copy results/ back, delete the instance.

The four machines, including setup time and compile warm-ups, were billed $1.02 together ($0.17 for the A6000, $0.15 for the L40S, $0.19 for the RTX 5090, $0.50 for the H100).

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.

vLLM FP8 vs bf16 on an H100: 1.4x the throughput, and what it did to GSM8K accuracy

Runcrate — Tue, 27 Oct 2026 16:27:34 +0000

vLLM can quantize a bf16 checkpoint to FP8 while it loads it: add --quantization fp8 and nothing else changes. On an H100 that should mean more tokens per second for the same money, but I wanted the actual numbers, and I wanted to know what it does to answer quality. So I rented one H100 on Runcrate today (2026-10-27), served the same 7B model in bf16 and then in FP8, ran the same load and the same 300 math questions against both, and terminated the machine as soon as the results were on disk.

Short version: FP8 gave 1.40x the output throughput at 64 concurrent requests (6,147 against 4,401 tokens/s) and 1.55x the speed for a single user. On GSM8K it answered 270 of 300 questions right against 272 for bf16.

Setup

GPU: one H100 PCIe (80 GB), US Midwest, $3.86 per hour on demand at launch time.
Model: Qwen/Qwen2.5-7B-Instruct, the regular bf16 checkpoint for both modes. FP8 here is vLLM's online quantization: weights are converted to FP8 at load time with a per-tensor scale, activations are scaled dynamically. The KV cache stays bf16 in both modes.
Server: vLLM 0.11.0 (torch 2.8.0+cu128, transformers 4.57.6), --dtype bfloat16 --max-model-len 4096 --gpu-memory-utilization 0.90, plus --quantization fp8 for the FP8 run. Fresh server per mode.
Throughput: vllm bench serve, random prompts, 512 tokens in and exactly 256 out (--ignore-eos), concurrency 1, 8, 32 and 64 (16, 64, 192 and 320 prompts), seed 42. Client on the same machine.
Quality: the first 300 questions of the GSM8K test set, zero-shot, greedy (temperature=0), up to 768 output tokens, final number parsed from a #### <answer> line, 64 requests in flight.

The commands

Launch (the instance_type_id comes from GET /api/v1/instances/types):

curl -s https://www.runcrate.ai/api/v1/instances \
  -H "Authorization: Bearer $RUNCRATE_API_KEY" -H "Content-Type: application/json" \
  -d '{"name":"fp8-h100","instance_type_id":"<id from /instances/types>",
       "ssh_key_id":"<your key id>","max_runtime_minutes":60}'

On the machine:

curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH=$HOME/.local/bin:$PATH
uv venv --python 3.12 /opt/env && . /opt/env/bin/activate
uv pip install vllm==0.11.0 'transformers<4.58' pandas pyarrow --torch-backend=cu128

M=Qwen/Qwen2.5-7B-Instruct
for MODE in bf16 fp8; do
  EXTRA=""; [ $MODE = fp8 ] && EXTRA="--quantization fp8"
  vllm serve $M --dtype bfloat16 $EXTRA --max-model-len 4096 \
    --gpu-memory-utilization 0.90 --port 8000 &
  until curl -sf localhost:8000/v1/models >/dev/null; do sleep 1; done
  for C in 1 8 32 64; do
    case $C in 1) N=16;; 8) N=64;; 32) N=192;; 64) N=320;; esac
    vllm bench serve --backend vllm --model $M --dataset-name random \
      --random-input-len 512 --random-output-len 256 --ignore-eos \
      --num-prompts $N --max-concurrency $C --seed 42 \
      --save-result --result-dir results --result-filename bench_${MODE}_c$C.json
  done
  python gsm8k.py $M > results/gsm8k_$MODE.json   # 300 questions, greedy, OpenAI client
  kill %1; wait
done

gsm8k.py is 40 lines: it loads openai/gsm8k from the Hugging Face Hub, sends each question with the instruction "Solve this math word problem step by step. On the last line write only the final answer as '#### '.", and compares the last number with the reference.

Results

Memory, as reported by vLLM at startup:

Mode	Weights in GPU memory	KV cache	KV cache size	Server ready after
bf16	14.25 GiB	51.27 GiB	959,936 tokens	66.5 s
FP8	8.14 GiB	57.38 GiB	1,074,320 tokens	67.5 s

Throughput and latency:

Concurrency	Mode	Output tok/s	Per-user tok/s	Median TTFT	p99 TTFT	$ per 1M output tokens
1	bf16	104	105	26 ms	28 ms	10.33
1	FP8	160	162	21 ms	25 ms	6.70
8	bf16	766	99	119 ms	141 ms	1.40
8	FP8	1,173	153	88 ms	109 ms	0.91
32	bf16	2,490	84	163 ms	502 ms	0.43
32	FP8	3,709	130	221 ms	375 ms	0.29
64	bf16	4,401	77	106 ms	939 ms	0.24
64	FP8	6,147	108	98 ms	755 ms	0.17

"Per-user tok/s" is 1000 / median time per output token. Cost per 1M output tokens is hourly price / (output tokens per second x 3600) x 1,000,000, counting generated tokens only.

GSM8K, first 300 test questions:

Mode	Correct	Accuracy	Wall time (64 in flight)
bf16	272 / 300	90.7%	15.3 s
FP8	270 / 300	90.0%	12.1 s

The two modes gave the same final answer on 272 of 300 questions. bf16 alone was right on 10 questions, FP8 alone on 8.

What surprised me

The biggest gain was for one user, not for the full batch. At concurrency 1, FP8 decoded at 162 tokens/s against 105 (1.55x). Single-stream decoding reads every weight once per token, so halving the bytes of the weights goes almost straight into speed. At 64 users the gain was 1.40x, still large, but the batch is more compute bound there.
Startup cost nothing extra. The online conversion added about a second to a 67 s server start. There is no separate FP8 checkpoint to find or trust.
The quality cost was within noise on this test. Two questions out of 300 is less than the disagreement between the modes in both directions (10 against 8). The final answer still differed on 28 questions, so FP8 does change outputs; it just did not make them worse on average here.
The freed memory mostly went to KV cache, and this test did not need it. About 6 GiB more cache (+12% tokens) matters with long contexts and many users; with 768-token requests and 64 users the cache was never full in either mode. The throughput gain above comes from faster matmuls and fewer bytes read, not from bigger batches.
The p99 TTFT at concurrency 32 was better with FP8 but the median was worse (221 ms against 163 ms). One run per setting; I would not read anything into that.

Caveats

One model (7B), one GPU, one run per setting. Larger models, where weights take most of the memory, may gain more from FP8; smaller ones less.
GSM8K with 300 questions has a standard error of about 1.7 points at 90% accuracy. It shows FP8 did not break math reasoning on this model; it cannot show a difference under about 3 points.
This is online FP8 (per-tensor). Pre-quantized FP8 checkpoints with calibrated or per-channel scales, and FP8 KV cache (--kv-cache-dtype fp8), are different settings and were not tested.
Ada and Hopper GPUs (L40S, RTX 6000 Ada, H100, H200) have FP8 tensor cores. On older cards vLLM falls back to weight-only FP8 kernels, and the speed math changes.
Price is the on-demand price at launch time and moves daily.

Reproduce it

Create an API key and add an SSH key (POST /api/v1/ssh-keys).
Pick an H100 from GET /api/v1/instances/types and launch with the curl above, max_runtime_minutes set as a safety net.
Run the block above, copy results/ back, delete the instance.

The whole run, including install, model download, both server starts and both benchmarks, was billed $0.58.

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.

vLLM prefix caching on vs off: 7.5x the throughput with a 4,000-token system prompt on one L40S

Runcrate — Tue, 27 Oct 2026 16:26:58 +0000

Most chat traffic sends the same system prompt in front of every request. vLLM can keep the KV cache for that shared prefix and skip recomputing it (automatic prefix caching, on by default since the V1 engine). I wanted to know what that is worth in numbers, so I rented one L40S on Runcrate today (2026-10-27), ran the same chat-shaped load with prefix caching on and off, and terminated the machine as soon as the results were on disk.

Short version: with a 4,000-token shared system prompt and 64 users at once, turning prefix caching off cut output throughput from 1,902 to 254 tokens per second and pushed p99 time to first token from 0.38 s to 23.9 s.

Setup

GPU: one L40S (48 GB), Kansas City, US, $1.14 per hour on demand at launch time.
Model: Qwen/Qwen2.5-7B-Instruct, bf16, no quantization.
Server: vLLM 0.11.0 (torch 2.8.0+cu128, transformers 4.57.6), --max-model-len 8192 --gpu-memory-utilization 0.90. vLLM reported 14.25 GiB of weights and 24.26 GiB of KV cache (454,272 tokens).
Two modes: default (prefix caching on), and --no-enable-prefix-caching (off). Same server flags otherwise, fresh server per mode.
Load: vllm bench serve with the random dataset. Every request is one shared prefix (1,000 or 4,000 tokens, the "system prompt") followed by 200 unique tokens, and exactly 128 output tokens (--ignore-eos). Concurrency 1, 16 and 64 (24, 160 and 384 requests). The client ran on the same machine, so no network is in the numbers.
Cache hit rate comes from vLLM's own /metrics counters (vllm:prefix_cache_hits / vllm:prefix_cache_queries), read before and after every run.

The commands

Launch (the instance_type_id comes from GET /api/v1/instances/types):

curl -s https://www.runcrate.ai/api/v1/instances \
  -H "Authorization: Bearer $RUNCRATE_API_KEY" -H "Content-Type: application/json" \
  -d '{"name":"prefix-l40s","instance_type_id":"<id from /instances/types>",
       "ssh_key_id":"<your key id>","max_runtime_minutes":60}'

On the machine:

curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH=$HOME/.local/bin:$PATH
uv venv --python 3.12 /opt/env && . /opt/env/bin/activate
uv pip install vllm==0.11.0 'transformers<4.58' --torch-backend=cu128

M=Qwen/Qwen2.5-7B-Instruct
for MODE in on off; do
  EXTRA=""; [ $MODE = off ] && EXTRA="--no-enable-prefix-caching"
  vllm serve $M --dtype bfloat16 $EXTRA --max-model-len 8192 \
    --gpu-memory-utilization 0.90 --port 8000 &
  until curl -sf localhost:8000/v1/models >/dev/null; do sleep 1; done
  for P in 1000 4000; do
    for C in 1 16 64; do
      case $C in 1) N=24;; 16) N=160;; 64) N=384;; esac
      vllm bench serve --backend vllm --model $M --dataset-name random \
        --random-prefix-len $P --random-input-len 200 --random-output-len 128 --ignore-eos \
        --num-prompts $N --max-concurrency $C --seed $((P + C)) \
        --save-result --result-dir results --result-filename bench_${MODE}_p${P}_c$C.json
      curl -s localhost:8000/metrics | grep -E '^vllm:prefix_cache_(queries|hits)'
    done
  done
  kill %1; wait
done

Then DELETE /api/v1/instances/<id> and a check that the instance list is empty.

Results

Prefix	Users	Cache	Requests/s	Output tok/s	Median TTFT	p99 TTFT	Median time per token	Hit rate
1,000	1	on	0.37	48	40 ms	44 ms	20.8 ms	85%
1,000	1	off	0.36	47	103 ms	106 ms	20.8 ms
1,000	16	on	5.27	675	163 ms	208 ms	22.5 ms	88%
1,000	16	off	3.72	476	314 ms	1,416 ms	31.1 ms
1,000	64	on	14.61	1,870	303 ms	773 ms	31.8 ms	88%
1,000	64	off	6.63	849	507 ms	5,940 ms	71.5 ms
4,000	1	on	0.37	47	57 ms	63 ms	21.1 ms	96%
4,000	1	off	0.33	42	377 ms	385 ms	21.1 ms
4,000	16	on	5.21	667	98 ms	147 ms	23.4 ms	99%
4,000	16	off	1.90	243	850 ms	4,987 ms	59.1 ms
4,000	64	on	14.86	1,902	172 ms	384 ms	32.1 ms	99.6%
4,000	64	off	1.99	254	1,936 ms	23,935 ms	240 ms

What that means for the bill, at $1.14 per hour, with the GPU kept busy at that concurrency:

Prefix	Users	$ per 1,000 requests, cache on	cache off	Ratio
1,000	16	0.060	0.085	1.4x
1,000	64	0.022	0.048	2.2x
4,000	16	0.061	0.167	2.7x
4,000	64	0.021	0.160	7.5x

What surprised me

With the cache on, the length of the system prompt almost stopped mattering. At 64 users, the 4,000-token prefix ran at 14.86 requests/s and the 1,000-token prefix at 14.61. The median time to first token was even lower with the longer prefix (172 ms against 303 ms), because a larger share of each prompt was already cached (99.6% hit rate against 88%). Without the cache, going from 1,000 to 4,000 tokens of prefix cut throughput by 3.3x.
The cost of "off" shows up in decode speed too, not only in TTFT. Median time per output token at 64 users went from 32 ms to 240 ms with a 4,000-token prefix. Every new request has to prefill 4,200 tokens, and those prefills share the GPU with everyone else's decoding, so every user in the batch slows down, not only the one who just arrived.
The tail is where it hurts. p99 time to first token at 64 users and a 4,000-token prefix was 23.9 s with the cache off. The median was 1.9 s. Some users wait for a long queue of prefills ahead of them.
For a single user, the cache saves latency but not money. At concurrency 1 the GPU is bound by decoding one stream, so requests per second barely moved (0.37 against 0.33). The 4,000-token prefix still cost 377 ms of prefill per request without the cache and 57 ms with it.
The first request pays. The 85% hit rate at 1,000 tokens and concurrency 1 is close to the share of each prompt that is shared (1,000 of 1,200 tokens). The cache never helps the unique part of a prompt.

What to take from it

Prefix caching is already on in current vLLM. These "off" numbers are also what you get with the cache on if your prompts do not actually share a prefix: a timestamp, user name or request id at the top of the system prompt changes the first tokens and every request misses. Put the parts that never change first and the parts that change last.
If your traffic is many users behind one long system prompt, size the GPU from the cache-on numbers, but check the hit rate in /metrics in production to see whether you really get them.

Caveats

One run per setting, on one machine, on one day. Treat differences under about 10% as noise.
Synthetic prompts from vLLM's random dataset: one prefix shared by every request. Real traffic mixes several system prompts, multi-turn history and different lengths; hit rates will be lower.
The cache could hold the whole prefix easily here (454k tokens of KV cache). With many distinct long prefixes and a full cache, evictions would lower the hit rate.
Price is the on-demand price at launch time and moves daily.

Reproduce it

Create an API key and add an SSH key (POST /api/v1/ssh-keys).
Pick an L40S from GET /api/v1/instances/types and launch with the curl above, max_runtime_minutes set as a safety net.
Run the block above, copy results/ back, delete the instance.

The whole run, including install, model download and both server starts, was billed $0.38.

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.

4 GPUs, 2x speed: why PyTorch DDP stalled on a PCIe-only 4x RTX A6000, and two flags that took it to 3.6x

Runcrate — Mon, 26 Oct 2026 09:22:37 +0000

Four GPUs should train about four times faster than one. On a rented 4x RTX A6000 machine this morning (2026-10-26), plain PyTorch DDP gave me 2x. This is the notebook of finding out why, and the two flags that took it to 3.6x.

Setup

Machine: one Runcrate 4x RTX A6000 (48 GB each) VM, US Midwest, $2.64/hour for the whole machine at launch time. Two separate machines of the same type over the morning: the first for the baseline, the second for the fixes.
Topology (nvidia-smi topo -m): every GPU pair is PHB, i.e. GPU to GPU traffic crosses the CPU's PCIe host bridge. No NVLink. PCIe gen 3 x16 per card.
Model: a GPT-2 style decoder written in plain PyTorch, two sizes: 124M parameters (12 layers, 768 wide) and 355M (24 layers, 1024 wide). Random tokens, sequence 1024, micro-batch 8 per GPU, bf16 autocast, fused AdamW.
Software: torch 2.8.0+cu128 (first machine) and 2.11.0+cu128 (second), NCCL as shipped in the wheel.
Metric: tokens/s from the median step time over 40 steps after 10 warm-up steps. Weak scaling: each GPU always gets its own micro-batch of 8.

The commands

# baseline: 1, 2 and 4 processes on the same machine
for N in 1 2 4; do torchrun --standalone --nproc_per_node $N train_ddp.py --size small; done

# all-reduce bandwidth, nccl-tests style (bus bandwidth = algbw x 2(n-1)/n)
torchrun --standalone --nproc_per_node 4 allreduce.py

# the fixes
torchrun --standalone --nproc_per_node 4 train_ddp.py --size small --hook bf16            # bf16 gradient compression
torchrun --standalone --nproc_per_node 4 train_ddp.py --size small --accum 4              # all-reduce every 4 micro-batches
torchrun --standalone --nproc_per_node 4 train_ddp.py --size small --accum 4 --hook bf16  # both

The two fixes are a few lines each in the training loop:

from torch.distributed.algorithms.ddp_comm_hooks import default_hooks
ddp = DDP(model, device_ids=[local_rank], gradient_as_bucket_view=True)
ddp.register_comm_hook(None, default_hooks.bf16_compress_hook)   # send gradients as bf16

for k in range(accum):                                            # gradient accumulation
    ctx = ddp.no_sync() if k < accum - 1 else contextlib.nullcontext()
    with ctx, torch.autocast("cuda", dtype=torch.bfloat16):
        loss = model_loss(ddp, next_batch()) / accum
        loss.backward()
opt.step(); opt.zero_grad(set_to_none=True)

Baseline: 2x from 4 GPUs

Model	GPUs	Tokens/s	Speed-up vs 1 GPU
124M	1	61,907	1.00x
124M	2	77,287	1.25x
124M	4	127,682	2.06x
355M	1	25,762	1.00x
355M	2	30,431	1.18x
355M	4	48,611	1.89x

The all-reduce explains it. NCCL bus bandwidth between the four cards was 4.3 GB/s (2 GPUs: 3.9 GB/s), flat from 8 MB to 1 GB messages. DDP all-reduces fp32 gradients: about 500 MB per step for the 124M model. At 2.9 GB/s of algorithm bandwidth that is roughly 170 ms of communication against a 130 ms compute step, and 4 GPUs measured about 255 ms per step. The GPUs spend more time talking than computing.

The fixes, on 4 GPUs

Model	Variant	Tokens/s	Speed-up vs 1 GPU (same variant)	$ per 1B training tokens
124M	1 GPU	64,912	1.00x	2.82 (one GPU's share, $0.66/h)
124M	4 GPUs, plain DDP	129,032	1.99x	5.68
124M	4 GPUs, bf16 gradients	172,144	2.65x	4.26
124M	4 GPUs, accumulate 4	211,834	3.20x	3.46
124M	4 GPUs, both	234,175	3.54x	3.13
355M	1 GPU	27,109	1.00x	6.76
355M	4 GPUs, plain DDP	51,711	1.91x	14.18
355M	4 GPUs, bf16 gradients	73,653	2.72x	9.96
355M	4 GPUs, accumulate 4	86,507	3.10x	8.48
355M	4 GPUs, both	99,152	3.66x	7.40

(1-GPU tokens/s with accumulation 4: 66,189 and 27,893, so the "both" rows are 88% and 89% scaling efficiency against the same recipe on one GPU.)

What surprised me

The interconnect, not the GPU, set the speed. 4.3 GB/s of all-reduce bandwidth is a fraction of what PCIe gen 3 x16 can carry. With every pair going through the host bridge, plain DDP on four cards only doubled throughput.
Forcing peer-to-peer made it worse. NCCL_P2P_LEVEL=SYS (use P2P even across the host bridge) halved the all-reduce bandwidth to 2.3 GB/s and dropped 124M training from 129k to 80k tokens/s. NCCL's default choice was right; don't override it on a PHB machine.
Two cheap changes recovered most of it. bf16 gradient compression halves the bytes; accumulating 4 micro-batches per all-reduce cuts the number of all-reduces by 4. Together: 3.5x to 3.7x on four GPUs. Accumulation changes your effective batch size (32 per GPU here), so retune the learning rate if you adopt it.
Per token, one GPU still won. Even with both fixes, a token cost about 10% more on four GPUs than on one, because four GPUs at 88% efficiency is still four GPUs. Rent the 4x machine when wall-clock time matters (a 3.5x shorter run), not to save money.
The bigger model scaled slightly better, as expected: more compute per byte of gradient.

Caveats

Random tokens and a synthetic model: this measures throughput, not convergence. bf16 gradient compression slightly changes numerics.
Two machines of the same type, one run per configuration. Topology varies between machines; run nvidia-smi topo -m and a 30-second all-reduce test before a long job.
Machines with NVLink behave completely differently; this is a PCIe-only result.
Prices are on-demand at launch time on 2026-10-26.

Reproduce it

Launch a 4x GPU instance through POST /api/v1/instances with max_runtime_minutes set.
uv venv /opt/env && . /opt/env/bin/activate && uv pip install torch numpy --torch-backend=cu128.
Copy train_ddp.py and allreduce.py (about 100 lines together; the core is the snippet above) and run the torchrun commands.
Delete the instance.

Both 4x machines together, including setup and every configuration above, were billed $0.92.

If you want to try it on your own model: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the runs described above; raw logs are kept with the post.

How long until a rented GPU runs CUDA? Launch to first matmul on six GPU types, timed

Runcrate — Mon, 26 Oct 2026 09:19:54 +0000

"How long until my GPU is ready?" is the question I care about most when I rent one for a short job, and it is the number nobody publishes. So this morning (2026-10-26) I launched single-GPU machines of six types on Runcrate through the API, one after another in one batch, and timed every step until a real CUDA matmul finished. Then I deleted each machine the second it was done.

What I measured

For each machine, seconds since the create call returned:

API running: the instance API reports running and hands out the IP.
First SSH: my script's first successful ssh root@ip echo ok (it retries every 2 s once it has the IP).
First CUDA op: after SSH, install uv, create a venv, uv pip install torch --torch-backend=cu128, then multiply two 4096x4096 matrices on the GPU and synchronize.

I also read two clocks on the machine itself, because they tell you where the time goes: kernel boot (uptime -s) and when ssh.service came up (systemctl show ssh -p ActiveEnterTimestamp).

The commands

# 1. launch (instance_type_id from GET /api/v1/instances/types)
curl -s https://www.runcrate.ai/api/v1/instances \
  -H "Authorization: Bearer $RUNCRATE_API_KEY" -H "Content-Type: application/json" \
  -d '{"name":"cold-l40s","instance_type_id":"<id>","ssh_key_id":"<key id>","max_runtime_minutes":30}'

# 2. poll every 10 s until status is running and ip is set
curl -s https://www.runcrate.ai/api/v1/instances/<id> -H "Authorization: Bearer $RUNCRATE_API_KEY"

# 3. on the machine
curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH=$HOME/.local/bin:$PATH
uv venv -q --python 3.12 /opt/env && . /opt/env/bin/activate
uv pip install -q torch --torch-backend=cu128
python -c "import torch,time;t=time.time();x=torch.randn(4096,4096,device='cuda');y=x@x;torch.cuda.synchronize();print(time.time()-t)"

# 4. delete
curl -s -X DELETE https://www.runcrate.ai/api/v1/instances/<id> -H "Authorization: Bearer $RUNCRATE_API_KEY"

Results

All VMs, Ubuntu image, one GPU each, launched within 4 seconds of each other at 09:02 UTC.

Runcrate GPU	Region	$/hour	API running	First SSH	First CUDA op	Billed for the whole run
L40S	Des Moines, US	1.14	151 s	164 s	186 s	$0.010
A100 80GB	Montreal, CA	1.73	303 s	319 s	337 s	$0.016
RTX A4000	New York, US	1.04	452 s	463 s	482 s	$0.008
V100 16GB	New York, US	0.36	611 s	625 s	673 s (see below)	$0.006
RTX 6000 Ada	US Midwest	1.08	620 s	635 s	670 s	$0.014
RTX A6000	US Midwest	0.66	629 s	647 s	707 s	$0.014

And from the machines' own clocks, same runs:

Runcrate GPU	Kernel booted	ssh.service up
L40S	65 s	149 s
A100 80GB	116 s	293 s
RTX A4000	143 s	455 s
V100 16GB	30 s	61 s
RTX 6000 Ada	76 s	97 s
RTX A6000	95 s	121 s

The torch step on its own (venv + wheel install + first matmul): 10 to 48 s for the install, 0.2 to 1.0 s for the first matmul, which is mostly CUDA context creation.

What surprised me

The spread is 4x, and it is not about the GPU. The L40S was doing math in about 3 minutes, the A6000 in almost 12. Same API, same image, same script. Cold start depends on where the machine comes from far more than on which card is in it, so if you launch often, measure the specific region you use.
On three types, the machine was ready long before the API said so. The V100, RTX 6000 Ada and A6000 booted and had sshd listening in 1 to 2 minutes, but the API reported them running about 8 minutes later. Nothing in my script could have used them earlier, because the IP arrives with running. The upside: billing starts at running, which is why the bills in the table are a cent or two.
The newest PyTorch wheels no longer run on a V100. uv pip install torch --torch-backend=cu128 resolved to torch 2.11.0, which warns that it was built for compute capability 7.5 and up, and the matmul then failed with cudaErrorNoKernelImageForDevice. The V100 is 7.0. On an earlier V100 run the same command had resolved to torch 2.8.0, which worked. If you rent Volta cards, pin torch (torch==2.8.* worked here) instead of taking the latest.
Not every create call works. Over the day I made 18 create calls. Two (an RTX 4090 and an RTX 5090 VM in the same region) came back within 7 seconds with 500 provider_unavailable, and one type was gone between listing it and launching it (400 ... no longer available). Both failures are fast and cost nothing, so the right client behaviour is: read /instances/types right before launching, and fall back to the next option on a 4xx/5xx.
A container is a different animal. In an earlier batch I launched an RTX 5090 as a container instead of a VM: its clock said it started 4 seconds after the create call. If startup time matters more than a full VM, check the deployment_type column.
My own polling made my first batch useless. I first polled nine machines every 2 seconds, hit the API's rate limit (HTTP 429) for several minutes, and the "running" times from that batch were wrong by minutes. This table is from a second batch polling every 10 seconds. Poll slowly.

Caveats

One launch per GPU type in this table, at one time of day. Cold start depends on what is in stock at that moment; treat it as a sample, not a promise.
"First CUDA op" includes downloading a 900 MB torch wheel. A pre-built image or a volume with a cached venv removes most of the 10 to 48 s.
The prices are the on-demand prices at launch time and move daily.

Reproduce it

Create an API key, add an SSH key (POST /api/v1/ssh-keys).
For each type you care about: launch with the curl above, poll every 10 s or slower, SSH in as soon as there is an IP, run the torch block, delete.
Read uptime -s and systemctl show ssh -p ActiveEnterTimestamp on the machine if you want to know where the time went.

The six machines in the table were billed $0.07 in total.

If you want to time it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the runs described above; raw logs are kept with the post.

vLLM on three GPUs: Qwen2.5-7B tokens per second and cost per 1M tokens on RTX A6000, L40S and H100

Runcrate — Mon, 26 Oct 2026 09:05:51 +0000

I wanted one number before picking a GPU for a small self-hosted model: what does a million generated tokens cost on each card, once the server is actually busy? Spec sheets don't answer that, so I rented three single-GPU machines on Runcrate this morning (2026-10-26), ran the same vLLM benchmark on each, and terminated them as soon as the numbers were on disk.

Setup

Model: Qwen/Qwen2.5-7B-Instruct, bf16, no quantization.
Server: vLLM 0.11.0 (torch 2.8.0+cu128, transformers 4.57.6), --max-model-len 4096 --gpu-memory-utilization 0.90, everything else default.
Load: vllm bench serve with random prompts, 512 tokens in and exactly 256 tokens out (--ignore-eos), at concurrency 1, 8, 32 and 64 (16, 64, 192 and 320 prompts), seed 42. The client ran on the same machine as the server, so there is no network in these numbers.
Machines (on-demand price at launch time):

Runcrate GPU	VRAM	Region	$/hour
RTX A6000	48 GB	Des Moines, US	0.74
L40S	48 GB	Kansas City, US	1.14
H100 PCIe	80 GB	Montreal, CA	3.55

Cost per 1M output tokens below is simply hourly price / (output tokens per second x 3600) x 1,000,000. It counts only generated tokens; the 512 prompt tokens per request are free riders in this math.

The commands

Launch (one call per GPU; the instance_type_id comes from GET /api/v1/instances/types):

curl -s https://www.runcrate.ai/api/v1/instances \
  -H "Authorization: Bearer $RUNCRATE_API_KEY" -H "Content-Type: application/json" \
  -d '{"name":"vllm-a6000","instance_type_id":"<id from /instances/types>",
       "ssh_key_id":"<your key id>","max_runtime_minutes":45}'

max_runtime_minutes is the safety net: the machine stops itself even if my script dies. Then, on the machine:

curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH=$HOME/.local/bin:$PATH
uv venv --python 3.12 /opt/env && . /opt/env/bin/activate
uv pip install vllm==0.11.0 'transformers<4.58' --torch-backend=cu128

vllm serve Qwen/Qwen2.5-7B-Instruct --dtype bfloat16 --max-model-len 4096 \
  --gpu-memory-utilization 0.90 --port 8000 &

for C in 1 8 32 64; do
  case $C in 1) N=16;; 8) N=64;; 32) N=192;; 64) N=320;; esac
  vllm bench serve --backend vllm --model Qwen/Qwen2.5-7B-Instruct \
    --dataset-name random --random-input-len 512 --random-output-len 256 --ignore-eos \
    --num-prompts $N --max-concurrency $C --seed 42 \
    --save-result --result-dir results --result-filename bench_c$C.json
done

Then DELETE /api/v1/instances/<id> and a check that the list is empty.

Results

GPU	Concurrency	Output tok/s	Per-user tok/s	Median TTFT	p99 TTFT	$ per 1M output tokens
RTX A6000	1	42	43	82 ms	85 ms	4.86
RTX A6000	8	307	40	289 ms	529 ms	0.67
RTX A6000	32	1,012	32	479 ms	1,940 ms	0.20
RTX A6000	64	1,727	32	259 ms	3,347 ms	0.12
L40S	1	48	49	48 ms	49 ms	6.56
L40S	8	354	45	161 ms	299 ms	0.89
L40S	32	1,082	34	279 ms	1,050 ms	0.29
L40S	64	1,887	33	146 ms	2,169 ms	0.17
H100 PCIe	1	104	105	25 ms	26 ms	9.47
H100 PCIe	8	776	100	62 ms	125 ms	1.27
H100 PCIe	32	2,566	87	233 ms	410 ms	0.38
H100 PCIe	64	4,515	79	101 ms	960 ms	0.22

"Per-user tok/s" is 1000 / median time per output token, i.e. how fast one stream feels to the person reading it.

How long each machine took before it could serve anything (measured on the machine):

GPU	vLLM install	Model download (15 GB)	`vllm serve` until ready	First chat reply
RTX A6000	29 s	103 s	114 s	1.22 s
L40S	7 s	78 s	67 s	0.85 s
H100 PCIe	4 s	13 s	54 s	0.56 s

What surprised me

The L40S is barely faster than the A6000 at decoding. It has roughly twice the bf16 compute on paper, but single-stream decode on a 7B model is limited by memory bandwidth, and the two cards are close there. 48 vs 42 tokens/s at concurrency 1, 1,887 vs 1,727 at 64. Per dollar, the A6000 wins at every load level I tried.
The H100 is not the per-token deal, it is the latency deal. It produced 2.6x the A6000's throughput at concurrency 64 for 4.8x the hourly price. But every user still saw 79 tokens/s at that load, against 32 on the other two cards, and the p99 time to first token stayed under a second. If your product is a chat UI with real people waiting, that is what you are paying for.
At concurrency 1, every GPU is expensive. $4.86 to $9.47 per million tokens. Batching is where the money is: going from 1 to 64 concurrent requests cut the cost per token by about 40x on all three cards. If your traffic can't keep the batch full, a smaller, older card idles more cheaply.
p99 TTFT at concurrency 64 on the A6000 was 3.3 s, while the median was 259 ms. Some requests wait for a whole batch slot. Look at tail latency before you take the cost-per-token number at face value.
Setup time varied more than benchmark time. The 15 GB model download took 13 s on one machine and 103 s on another. For short experiments, that download is a real part of the bill; put the weights on a volume if you will start the server more than once.

Caveats

One run per GPU, on one morning. Treat differences under about 10% as noise.
Random prompts, fixed 512/256 shape. Real chat traffic has prefix reuse and varied lengths; prefix caching would help all three cards.
These are PCIe cards, one per machine. No tensor parallelism, no FP8, no speculative decoding. An FP8 checkpoint would likely change the H100 math more than the others.
Prices are the on-demand prices at launch time on 2026-10-26 and move daily.

Reproduce it

Create an API key and add an SSH key (POST /api/v1/ssh-keys).
Pick an instance type from GET /api/v1/instances/types and launch with the curl above, max_runtime_minutes set.
Run the setup and benchmark block, copy results/ back, delete the instance.

The three machines, including setup time, were billed $1.07 together ($0.15 for the A6000, $0.29 for the L40S, $0.64 for the H100).

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.