DEV Community: Qaiser Mehdi

RTX 3090 vs A100 for a 7B model on vLLM: the cheaper GPU depends on your answer time

Qaiser Mehdi — Sun, 01 Nov 2026 07:23:30 +0000

The setup

Same model (Qwen2.5-7B-Instruct), same vLLM (0.27.1, default flags), same load: prompts of about 512 tokens, 256-token answers, 1 to 256 requests at once, 60 seconds per level. Two rented cards: an RTX 3090 at $0.259/hour and an A100 80GB PCIe at $1.017/hour.

Same load: the cheap card wins

1 at once: 3090 $1.81, A100 $3.16 per 1M output tokens
4 at once: 3090 $0.49, A100 $0.80
16 at once: 3090 $0.16, A100 $0.24
64 at once: 3090 $0.08, A100 $0.11

At every load we ran on both, the 3090 cost less per token. The A100 does 2.2x to 3x more tokens per second at the same load, but it costs 3.9x more per hour.

Same answer time: the A100 wins

Answers don't take the same time on both cards. At 64 at once the A100 answered in 6.0 s (p50), the 3090 in 17.9 s. So compare at the answer time your users accept:

About 3 s: only the A100 can do it. The 3090 already takes 6.3 s with one request at a time.
About 6 to 7 s: 3090 at 4 at once (p50 6.8 s, p95 7.2 s) costs $0.490 per 1M. A100 at 64 at once (p50 6.0 s, p95 7.1 s) costs $0.106. The A100 is about 4.6x cheaper.
About 18 s: 3090 at 64 at once and A100 at 256 at once both cost $0.082.

What it means

"Which GPU is cheapest" has no answer until you say how fast answers must be and how many requests you can keep in flight. Price per hour alone can point the wrong way when answers must be fast.

How it was counted

Cost = GPU price x the measured window, which runs until the last in-flight request finishes (that time is paid for too). Requests and tokens came from vLLM's own /metrics before and after each level, and a separate client-side meter counted the same things. They matched exactly at every level on both cards. 0 errors.

Limits

These costs assume the GPU is busy the whole time; idle hours in a real day raise them, and only real traffic shows by how much. One run per level, two hosts on different days, synthetic prompts, two vast.ai prices. Other prices change the answer.

Both reports, raw /metrics and server logs: https://github.com/qaisermehdi3-coder/qvunex/tree/main/benchmarks/data/planb-a100-2026-10-29

If you want this for your own model and settings: send the model, your vLLM flags and typical prompt and answer length, and I'll run a short mini-report on a rented GPU for free. The full report is paid work: qvunexaudit@gmail.com

Same RTX 3090, 22x cheaper per token: what concurrency does to vLLM cost

Qaiser Mehdi — Wed, 28 Oct 2026 05:42:59 +0000

I rented one RTX 3090 at $0.259/hour, served Qwen2.5-7B-Instruct with vLLM 0.27.1 and default flags, and kept it busy for 60 seconds at 1, 4, 16 and 64 requests at once. Each request had a prompt of about 512 tokens and a 256-token answer.

Results

1 at once: 40 output tokens/s, answer in 6.3 s, $1.81 per 1M output tokens
4 at once: 147 tokens/s, 6.8 s, $0.49 per 1M
16 at once: 442 tokens/s, 9.1 s, $0.16 per 1M
64 at once: 878 tokens/s, 17.9 s, $0.08 per 1M

0 errors at every level.

What it means

The card costs the same per hour whatever you do with it. At 64 requests in flight it produced 22 times more tokens than at 1, so each token cost 22 times less. The price is waiting: each answer took about 2.9 times longer.

So "how much does my GPU cost per token" has no single answer. It depends on how many requests you keep in flight, and that is a choice you make with your latency target.

How the cost was counted

Cost = GPU price x the measured window. The window runs until the last in-flight request finishes (62.8 to 71.7 s here), because that time is paid for too. Requests and tokens came from vLLM's own /metrics counters before and after each level, and a client-side meter counted the same things separately. They matched exactly at every level.

What this does not tell you

These costs assume the GPU is busy the whole time. A real server is not. If it serves requests 30% of the day, each request really costs about 3.3x these numbers, and only your own traffic shows that share. Also: one card, one run per level, synthetic prompts. The 1-at-once window includes a one-time kernel compile.

Start-up was 122.9 s from launch to /health with the model already on disk. If you scale to zero, every wake pays that before the first answer: about $0.009 per wake at this price.

Report, raw /metrics and server log: https://github.com/qaisermehdi3-coder/qvunex/tree/main/benchmarks/data/planb-3090-2026-10-27

If you want this for your own model and settings and don't want to run anything on your servers, I rent the same GPU and run it for you, as paid work: qvunexaudit@gmail.com

Two vLLM servers on one GPU: the startup check that bites, and the fix

Qaiser Mehdi — Sat, 24 Oct 2026 17:27:42 +0000

Someone asked on the vLLM tracker how to run a small embedding model next to a chat model on one GPU, so the card isn't half idle. I tested it on a rented RTX 3090 (24 GB) with vLLM 0.27.1: BAAI/bge-small-en-v1.5 for embeddings and Qwen2.5-1.5B-Instruct for chat, each in its own vllm serve.

It works

Embedding server at --gpu-memory-utilization 0.10, chat server at 0.70. Both came up and served with 0 errors. Together they used 18,317 of 24,576 MiB.

The startup check

Start the chat server first at 0.9, then the embedding server at 0.10, and the second one refuses to start:

ValueError: Free memory on device cuda:0 (0.95/23.8 GiB) on startup is less than desired GPU memory utilization (0.1, 2.38 GiB).

At startup vLLM checks that free memory is at least your fraction of the whole card. The chat server had taken almost everything, so 0.10 (2.38 GiB) didn't fit, even though the embedding model never uses that much.

The fix: give the small model a small number

The embedding model has no KV cache to size. vLLM runs it as a pooling model, its log has no "Available KV cache memory" line, and it stayed at about 450 to 580 MiB whether I gave it 0.10, 0.05 or 0.03.

At 0.03 (0.71 GiB) it started fine next to the 0.9 chat server: 23,110 MiB + 552 MiB on one card. With --gpu-memory-utilization 0.03 --max-model-len 512 --max-num-seqs 32 it used 454 MiB at the same throughput as at 0.10.

What sharing costs

Each server busy alone, then both busy at the same time (30 seconds each, 0 errors):

embeddings: 1,761 to 1,609 texts/s (-8.6%)
chat: 2,607 to 2,298 output tokens/s (-11.8%)

There's no isolation between them, so that's the price.

Leave headroom

The chat server used more than its fraction: 17,710 MiB at 0.70, where 0.70 of the card is 17,203 MiB. Its own log says so too: 12.53 GiB of KV cache in use, 11.9 GiB to fit the request.

Limits

One card, one vLLM version, two small models, short runs, one try each. Starting both servers at the same moment worked once here, but one run isn't proof, so start them one at a time.

Script and raw output: https://github.com/qaisermehdi3-coder/qvunex/tree/main/benchmarks/data/coloc-3090-2026-10-23

The thread: https://github.com/vllm-project/vllm/issues/60599

If you run vLLM on your own GPUs and want to see how much of the paid time serves nothing, there's a 2-minute guide here: https://github.com/qaisermehdi3-coder/qvunex/blob/main/TRY-IT.md

If you want it measured on your setup, I take it on as paid work: qvunexaudit@gmail.com

A 3090 vs an L40S: 1.09x at batch 1, 0.46x at batch 128

Qaiser Mehdi — Sun, 18 Oct 2026 13:01:09 +0000

People ask "is a 3090 as fast as an L40S?" and want one number. I measured both on my own rented machines, same script, and there is no one number. It depends on how many requests you run at once.
Setup: vLLM 0.27.1, Qwen2.5 1.5B and 7B (plain and AWQ), 128 output tokens per request, CUDA graphs on, batch 1, 8, 32 and 128, 3 repeats each, median used. RTX 3090 on vast.ai (Netherlands, 24 of 48 vCPUs). L40S on vast.ai (Romania, 8 of 64 vCPUs). The three repeats of each setting landed within 0.5% of each other on the 3090 and within 2.6% on the L40S (worst case), so these are not noise.

Speed: 3090 as a multiple of the L40S

7B fp16:
batch 1 - 1.03x
batch 8 - 1.04x
batch 32 - 1.11x
batch 128 - 0.72x
7B AWQ:
batch 1 - 1.09x
batch 8 - 1.01x
batch 32 - 0.71x
batch 128 - 0.46x
1.5B fp16:
batch 1 - 1.09x
batch 128 - 0.70x
At small batches the two cards are about even. That is roughly what memory bandwidth says (936 vs 864 GB/s). At batch 128 the L40S pulls far ahead, because the work stops being about memory and starts being about compute, and the L40S has much more of it.
So "the 3090 is 1.08x an L40S" is true at batch 1 and wrong at batch 128.

Cost per million output tokens

At the prices I actually paid ($0.414/h for the 3090, $0.826/h for the L40S), counting only time the GPU was busy:
7B fp16:
batch 1 - 3090 $2.22, L40S $4.59
batch 128 - 3090 $0.041, L40S $0.059
7B AWQ:
batch 1 - 3090 $0.77, L40S $1.67
batch 128 - 3090 $0.040, L40S $0.036
At low batch the 3090 is about half the cost per token. At batch 128 the gap closes, and for 7B AWQ the L40S is already cheaper. Which card is cheaper depends on your traffic, not on the card.

Two things this does not tell you

One card is one card. A different 3090 I measured earlier was clearly slower on every setting, with the same script and driver. If you are choosing hardware, measure two units, not one.
And every cost above assumes the GPU is busy. A real server is not. If it serves requests 30% of the time, the real cost per token is about 3.3x these numbers. That idle share can be the biggest number in the bill, and many dashboards don't show it.

The data

All 192 runs, the exact script, and the machine details are public:
https://github.com/qaisermehdi3-coder/qvunex/tree/main/benchmarks/data/3090-own
https://github.com/qaisermehdi3-coder/qvunex/tree/main/benchmarks/data/l40s-own
If you run vLLM and want to see your own idle share, there is a 2-minute guide here (one file, reads only /metrics):
https://github.com/qaisermehdi3-coder/qvunex/blob/main/TRY-IT.md
If you want this measured on your own setup, I take it on as paid work: qvunexaudit@gmail.com

The same H100 hour cost $1.26 or $0.22 per million tokens

Qaiser Mehdi — Tue, 13 Oct 2026 00:46:48 +0000

A GPU hour has one price. What it delivers does not. Here is one measurement that shows how big the gap can be.

The setup. One rented H100 SXM 80GB on vast.ai at $2.689 per hour, 20 vCPUs. vLLM 0.27.1 (official image). Qwen2.5-0.5B-Instruct in fp16, unique prompts so nothing was cached. 28 September 2026. Every number below comes from vLLM's own /metrics counters, and they matched what my client recorded exactly.

Two loads, three rounds each.

One request at a time: each request decoded at 648-649 tokens/s. The whole server produced 589-593 tokens/s. That is about $1.26 per 1M output tokens.
8 requests at once: each request got a bit slower, 549-573 tokens/s. But the whole server produced 3,194-3,348 tokens/s. That is about $0.22-0.23 per 1M output tokens.

Same GPU, same hour, same price. About 5.5x apart.

The common mistake. Many cost estimates take one request's speed and divide the hourly price by it. At 8 at once, one request ran at about 549-573 tokens/s, which would price the hour at about $1.30-1.36 per 1M tokens. The real cost was $0.22-0.23. In each round the whole server made 5.65-5.84x what one request suggests.

So pricing from one request's speed can overstate cost by about 5.7x at this load. At other loads the gap is different. That is the point.

What decides your real cost.

How many requests your server actually runs at once, not what a benchmark ran.
The idle time inside the hour. A GPU that waits still bills, so cost per token is the hourly price over everything the server produced in that hour, idle included.
For scale-to-zero, the boot. On this same H100, one cold wake was about 75 of 77.3 billed seconds before the server could answer.

Same ratio on a smaller card. On a T4 with the same model and test, at 8 at once each request decoded at 122-124 tokens/s while the whole server made 701-708 tokens/s. Again about 5.7x.

What I can and cannot say. This is a small model, one GPU, three rounds per load. A 7B or 70B model, another server, or longer prompts will give different numbers. I am not saying your cost is $0.22. I am saying the number depends on your load, and you can only know it by measuring your own server.

The method, the raw numbers and the open-source tool are here, if useful: github.com/qaisermehdi3-coder/qvunex

If you have measured cost per token on your own servers at different loads, I would like to compare.

Ready is not warm: one SGLang cold start on a T4

Qaiser Mehdi — Mon, 12 Oct 2026 22:51:26 +0000

I measure what GPU time actually buys. One number surprised me this month, so here it is with the raw timeline.

The setup. One T4 on Colab. SGLang 0.5.21. Qwen2.5-0.5B-Instruct in fp16. I started a clock just before launching the server, the way a scale-to-zero platform starts billing when a worker is created. Then 8 short chat requests, then 60 seconds of health checks, then stop. One run, 5 October 2026.

The timeline.

0 s: server launched (billing starts)
~371 s: prefill CUDA graph capture finished (it took 287.7 s)
386.7 s: /health returns 200, log says "ready to roll"
~563 s: first chat request finishes (it took about 176 s)
~579 s: second request finishes (about 16 s), the other six within the same second
639.6 s: billing stops

After that, 8 more requests on the same server took 0.08 to 0.22 s each.

What this means for the bill. If you split the bill at the health check, boot looks like 60% of it (386.7 of 639.6 s). But the first user got an answer about 563 s in. That is 88% of the bill gone before anyone was served.

So "ready" was not the end of the wait. It was about two thirds of it.

What I can and cannot say.

The server's own counters matched my client exactly: 8 requests, 264 prompt tokens, 80 output tokens.
The log does not say what the first 176 s were spent on, so I am not claiming a cause.
Each health check in the tail shows up in the SGLang log as a one-token prefill on the GPU, but the request counters do not move. So "the GPU did work" and "a request was served" are different things.
For comparison, on vLLM 0.27.1 on the same card, the 8 requests after ready took 5.8 s in total. Boot there was about 165 of 171 billed seconds.
This is one run on one small card. A bigger GPU or another model can look different. That is the point: measure your own.

If you run scale-to-zero, record three times, not one:

billing start to health check OK
billing start to the first finished request
the idle tail after the last request

Then price the whole billed time, not just the part where requests were running.

The method and the open-source tool I used are here, if useful: github.com/qaisermehdi3-coder/qvunex

If you have measured this on other cards or servers, I would like to see your numbers.