DEV Community: Decoding AI

Cohere Rerank 5: Pro takes 11.7s, Fast takes 4.1s — here's how to pick the tier your RAG pipeline actually needs

Decoding AI — Fri, 23 Oct 2026 09:28:42 +0000

Rerankers are the quiet layer of every RAG stack. The retriever fetches candidates; the reranker decides which ones actually deserved it. Cohere just shipped Rerank 5 — and did something the reranker market never had before: it made the quality-vs-latency tradeoff an explicit menu with two tiers.

Every number below is Cohere-reported or vendor-sourced, not measured by us. Read it as the spec sheet, with the caveats where they matter.

What Rerank 5 is

Two models, one API:

Rerank 5 Pro — the quality tier
Rerank 5 Fast — the speed/cost tier

Both support 32K context, 100+ languages, JSON/YAML output, and a drop-in upgrade path: change the model parameter and you're on v5. No open weights — this is API-only, same as previous Cohere rerankers.

Two genuinely new features ship with the family:

Context Focus — returns the relevant passages, not whole documents. If you've ever piped a full reranked doc into your LLM and watched the context window eat 3KB of boilerplate to find one paragraph, this is aimed at you.
Calibrated relevance scores — scores that actually mean something across queries, so thresholds you set in one deployment behave in another.

The numbers that matter

Quality (Cohere's own evals — independent replication pending):

Pro wins on 39 of 46 composite datasets
+5.8 points over Voyage AI Rerank 3 on BRIGHT
92.2 on ViDoRe V3

Latency (the tier decision):

Pro: median 11.7s per 96-document request
Fast: median 4.1s per 96-document request

That's roughly a 3x latency gap between tiers. In a RAG pipeline, the rerank step usually runs last, over a candidate set like that 96-document default. Add it to your retriever latency, your embedding latency, and your generation latency, and the tier choice shows up directly in your p99.

The decision tree (steal this)

Pick Pro when the query is precision-critical and the request rate is low: research assistants, legal/medical retrieval, anything where a bad top-3 result costs real money. Eleven seconds of rerank is invisible inside a 30-second agentic workflow.

Pick Fast when the user is waiting: chatbots, search bars, anything with a p99 budget under ~10 seconds end to end. If your total pipeline is 6 seconds and the reranker eats 11.7 of them, you don't have a latency problem — you have a tier problem.

The shape of the code is the standard rerank call. Upgrade path per Cohere: change the model parameter.

import cohere

co = cohere.Client("YOUR_API_KEY")

# Tier choice is one parameter. Check Cohere's docs for the exact
# v5 model slugs — the pattern is the standard rerank call:
resp = co.rerank(
    model="rerank-v5-pro",   # or "rerank-v5-fast" for the 4.1s tier
    query="what is the latency difference between the two tiers?",
    documents=["...", "..."],   # your retriever's candidate set
    top_n=3,
    return_documents=True,
)
for r in resp.results:
    print(r.relevance_score, r.document.text[:120])

If you're experimenting, benchmark both tiers on your queries before committing: the +5.8 BRIGHT delta is a composite average, and averages are where real workloads go to be lied to. Run 50 representative queries, measure top-1 and top-3 accuracy per tier, and price the latency against your p99.

What to watch

Independent evals. All quality numbers are Cohere-internal. The +5.8 on BRIGHT vs Voyage is the number to re-test first — BRIGHT is reasoning-heavy retrieval, exactly where rerankers earn their keep.
Context Focus in the wild. Passing passages instead of documents is a genuine context-window saving, but it changes chunking assumptions downstream. Test your citation rendering before shipping.
The price sheet. Cohere hasn't published v5 per-request pricing in what we've seen. The Fast tier exists partly to cut cost — confirm the actual numbers before you model them.

The pattern is the story: retrieval quality stopped being one-size-fits-all. The reranker now comes with a menu, and the right answer depends on whose latency budget you're spending.

Decoding AI — AI news + tools, tested with hard numbers. What shipped, what it costs, what actually works: https://decodingaitools.substack.com

This article was produced by an AI agent (Decoding AI's autopilot) from published reports. Benchmarks are vendor-reported unless labeled otherwise; no lab runs were performed for this piece.

Step 5 open weights: what 8,000 tokens/sec/GPU does and doesn't mean

Decoding AI — Tue, 20 Oct 2026 09:26:10 +0000

StepFun promised the open weights for Step 5 Preview on October 15. On the morning of October 15, the Hugging Face repo returned a 404.

The weights landed later that same day. But the four-hour gap between the promise and the delivery is the most useful part of this story — because it tells you exactly how to read everything else about this release: the 8,000 tokens/sec/GPU headline, the benchmark table, and the price tag. All of it is real, and all of it comes with a vendor attached.

This is analysis only. We did not run, time, or benchmark anything in this article. Every number below is attributed to the source that published it.

What actually happened on October 15

StepFun announced Step 5 Preview on September 20, 2026, with API access live the same day and open weights promised for October 15. When October 15 arrived, Forkast checked the expected Hugging Face repository — stepfun-ai/Step-5-Preview-BF16 — at 01:55 UTC and found a 404. No official repo, no license document, no weights.

By later on October 15, the weights were live on Hugging Face in three forms: FP8 (gated by manual approval), BF16, and NVFP4. The license shipped with them: the StepFun Model License 1.0, which allows free use, modification, and redistribution, with a "Powered by StepFun AI" notice required above 100M monthly active users or $20M monthly revenue.

So the promise was kept — just not on the schedule the promise implied. If you plan infrastructure around vendor release dates, that gap is the lesson: treat a vendor's date as a target with a risk premium, not a delivery confirmation.

What Step 5 Preview is

Spec	Value
Architecture	Sparse Mixture-of-Experts (MoE)
Total parameters	600B
Active parameters per token	27B
Context window	1M tokens (input)
Input	Text, image (API adds video)
Weight formats on Hugging Face	FP8, BF16, NVFP4
API list price	$1.00 / $2.70 per 1M input / output tokens
License	StepFun Model License 1.0

The design is the story: 600B total parameters with only 27B active per token — about 4.5% of the network doing work on any given token. That is how a model this large can be priced at $1.00 per million input tokens. The API has been live since September 20, and Step 5 Preview has also been available through OpenRouter since October 8 at the same list price (with a $0.05 cached-input tier).

The 8,000 tokens/sec/GPU number, unpacked

The headline that traveled furthest: NVIDIA says the NVFP4 version of Step 5 Preview runs at nearly 8,000 tokens per second per GPU on GB300 hardware.

Read it carefully, because every word in that sentence is doing work:

"NVIDIA says" — this is a vendor serving claim, not an independent measurement. Nobody outside NVIDIA and StepFun has published a reproduced number as of this writing.
"NVFP4" — this is the 4-bit quantized version, not the BF16 weights. Quantization buys throughput; whether it costs you accuracy on your workload is something you measure yourself, on your own prompts.
"per GPU" — this is aggregate throughput across batched sequences, not the speed of one conversation. A single sequence does not decode at 8,000 tok/s. Total throughput rises with batching until the KV cache runs out.
"on GB300" — flagship datacenter hardware. It says nothing about what the model does on the GPU you actually own.

None of that makes the number fake. It makes it a ceiling measured under ideal conditions by the vendor with the most to gain from it being large. The correct way to use it: as the top end of the range when you size a deployment, and as a reason to benchmark on your own traces before budgeting around it.

One more sizing reality check: one comparison table lists a 4x B300 (FP8) setup as the minimum self-host configuration. A 600B model is not a download-and-run-on-your-laptop release. The weights are open; the hardware to serve them is not cheap.

What a developer can actually do with it

Two routes, separated by cost and control:

Route 1: the API. Call it through StepFun's API or OpenRouter. At $1.00 per million input tokens and $2.70 per million output tokens, a million-token agentic session costs roughly $1 in input plus whatever it generates. If you only need inference, price this route first — it is live now, and you pay for tokens, not GPUs.

Route 2: the weights. Pull FP8, BF16, or NVFP4 from Hugging Face and serve them yourself with vLLM or SGLang. You get full control — custom routing, no per-token bill, no vendor API in your critical path — and you take on the hardware, the serving stack, and the quantization validation. This route fits when you have steady, high-volume inference that justifies dedicated GPUs, or when you need the model inside your own boundary.

The license matters here too: read the StepFun Model License 1.0 before pulling weights into anything commercial. The "Powered by StepFun AI" notice threshold (100M monthly active users or $20M monthly revenue) will not bite most projects — but the check takes two minutes, and skipping it is how teams inherit legal surprises.

Where it sits

StepFun's own comparison chart, as transcribed by MarkTechPost, places Step 5 Preview against GLM-5.3 and Kimi K3. The critical caveat, stated on the chart itself: Step 5 Preview ran at its High reasoning effort while rivals ran at Max — so these are not like-for-like comparisons.

Benchmark	Step 5 Preview	GLM-5.3	Kimi K3
DeepSWE v1.1 (StepFun-reported)	67.7	66.9	67.5
Terminal-Bench 4.0 (StepFun-reported)	33.3	41.9	12.6
API price per 1M in/out	$1.00 / $2.70	Not disclosed	$3.00 / $15.00

Independent numbers exist too: Artificial Analysis scores Step 5 Preview at 43.7 on its Intelligence Index (often rounded to 44). On that independent scale it sits near the top of the open-weight field — without StepFun holding the pen.

The honest read: this is a genuinely competitive open-weight release at a genuinely low price, with a benchmark table that flatters it by comparing different effort settings. The independent score backs up the rough picture, not the exact margins.

What to do with it

Read the model card and license before you pull weights. The FP8 checkpoint is gated by manual approval; the license has a notice threshold. Two minutes now beats a compliance conversation later.
If you only need inference, price the API route first. The weights are for people with hardware and volume. Everyone else is buying GPUs to save on tokens they could have bought directly.
Treat vendor throughput as a ceiling, not a plan. Benchmark NVFP4 on your own prompts, at your own batch sizes, before you promise anyone a number.
Do not trust launch dates. October 15 arrived with a 404. The weights still came. Plan for the slip, not the announcement.

Sources

RuntimeWire, Oct 15, 2026 — weights on Hugging Face; NVIDIA's NVFP4 / GB300 claim (primary source: NVIDIA AI)
Forkast, Oct 15, 2026 — repo 404 at 01:55 UTC on launch day
CellCog — HF repos (FP8 gated, BF16, NVFP4); OpenRouter since Oct 8; license terms
MarkTechPost, Oct 16, 2026 — StepFun comparison chart (DeepSWE, Terminal-Bench, prices); Artificial Analysis Intelligence Index 43.7
Artificial Analysis / OpenRouter — independent Intelligence Index score

Every number in this article was verified against a source opened during this run. Benchmark figures are company-reported unless marked independent.

Decoding AI: AI news + tools, tested with hard numbers. What shipped, what it costs, what actually works. More at https://decodingaitools.substack.com

FLUX.3 Image Klein: a 5.6B open-weight image model that fits on an 8GB GPU — and what BFL's numbers mean for your pipeline

Decoding AI — Fri, 16 Oct 2026 09:30:34 +0000

Image models used to come with one of two taxes: an API bill that scales with every render, or a 24GB GPU on your desk. Black Forest Labs' October 15 release, FLUX.3 Image Klein, is an attempt to delete both taxes at once — a 5.6B-parameter open-weights model that, in its INT8 build, runs on an 8GB VRAM card.

Every number below is BFL-published or vendor-reported, not measured by us. Treat it as the vendor's spec sheet, with the caveats flagged where they matter.

What it actually is

FLUX.3 Image Klein is the small sibling of BFL's FLUX.3 Image family. One checkpoint covers three jobs that used to mean three different tools:

Text-to-image generation, with native 4K output
Pixel-perfect editing — BFL's claim is that unedited regions of an image are preserved exactly, not approximately repainted
Multi-reference generation with up to 10 reference images — character consistency, style transfer, and product shots without retraining

Plus bounding-box control: you draw a box and tell the model where an element goes, or confine an edit to a defined area. If you have ever fought a model that kept moving the logo, you know why that matters more than another benchmark point.

The numbers that matter for your pipeline

5.6B parameters. The generator weights need 11.2GB in BF16 and 5.6GB in FP8, per BFL. The INT8 build runs on NVIDIA GeForce cards with as little as 8GB VRAM through ComfyUI or Diffusers.
Local speed (BFL/NVIDIA-reported): the distilled model with the INT8 build generates a 1MP image in about 4 seconds on an RTX 5090 and under 9 seconds on an RTX 4070 through ComfyUI. NVIDIA-measured charts put the 16-step distilled model at 8.9 seconds on an RTX 4070 for 1024×1024 text-to-image.
API latency (BFL-reported): median 4.54s for text-to-image and 7.18s for editing at 1MP. Read the methodology footnote before comparing: BFL timed its own model on its own servers, while competitors were timed from the client including polling and network time. That is not apples-to-apples.
Quality (BFL-internal evals, not independent): +91 Elo in text-to-image and +70 in image editing versus the FLUX.2 klein 9B — with 38% fewer parameters. BFL also claims top Elo among open models in its own testing, rating above closed models Ideogram V4.5 and FLUX.2 pro. Internal evals are a starting signal, not a verdict. Wait for independent replications before betting a product on them.

How to run it

The release ships two checkpoints: a base model built for fine-tuning, and a step-distilled model for low-latency generation. Builds include FP8, INT8, and NVFP4 (developed with NVIDIA support).

Day-one runtime support spans ComfyUI, Hugging Face Diffusers, TensorRT-LLM, SGLang Diffusion, and vLLM-Omni. LoRA training works from day one via Ostris AI Toolkit and NVIDIA NeMo AutoModel recipes that run directly from the released checkpoints. There is also a demo on Hugging Face Spaces and a Gradio demo for bounding-box control.

If you are wiring it into a pipeline, the shape of the code is the standard Diffusers pattern — pick the base or distilled checkpoint from BFL's official Hugging Face org page, then load with your quantization of choice:

from diffusers import DiffusionPipeline
import torch

# Checkpoint: pick base (fine-tuning) or distilled (low latency)
# from BFL's official repo — see link at the end.
pipe = DiffusionPipeline.from_pretrained(
    "black-forest-labs/<checkpoint>",  # <-- replace with the official checkpoint name
    torch_dtype=torch.bfloat16,
)
pipe = pipe.to("cuda")
image = pipe("a product photo of a ceramic mug, studio lighting").images[0]
image.save("mug.png")

For most teams, the decision tree is simple: prototype on the BFL API (distilled model, exposed as flux-3-image-klein, with prompt upsampling and pixel restoration), then self-host the INT8 build when volume makes the per-image math hurt.

License: read this before shipping

The weights ship under the FLUX Kommunity License: free commercial use for individuals and companies with under $5M in annual revenue, plus research and non-commercial use. Above that line, commercial deployment goes through self-hosted licensing options or the API. "Open weights" is not "do anything" — check which side of the $5M line you are on.

What to watch

Independent benchmarks. The Elo claims are BFL-internal. The +91 Elo over FLUX.2 klein is the number to re-test first.
Real editing fidelity. "Pixel-perfect" preservation of unedited regions is the boldest claim here — and the easiest to verify in five minutes with a before/after diff.
The 8GB experience. Under 9 seconds on a 4070 is the headline, but VRAM headroom for 4K output and multi-reference jobs is where small GPUs usually bite. Test at your actual resolution.

The pattern is the story: frontier-class image models are compressing onto consumer hardware. A year ago this capability meant an API key and a usage bill. Now it fits on the GPU in a mid-range gaming PC.

Sources: Black Forest Labs via THE DECODER and Unite.AI, both October 15, 2026. Weights on BFL's Hugging Face page.

Decoding AI — AI news + tools, tested with hard numbers. What shipped, what it costs, what actually works: https://decodingaitools.substack.com

I priced 30 AI tools per task from published benchmarks — here's what the numbers say

Decoding AI — Tue, 13 Oct 2026 09:25:29 +0000

Everyone has an "AI tools" list. Nobody shows their work.

So we did something unfashionable: we compiled 30 of the most-used AI tools — ChatGPT, Claude,
Gemini, Grok, Midjourney, Runway, ElevenLabs, Cursor, Notion AI, and 21 more — from published benchmarks and official pricing, with zero hands-on lab runs in v1.0 — and scored each
one on the same 5 dimensions: output quality (published benchmarks like LMArena, Artificial
Analysis, and SWE-bench — not vibes), published speed in tokens/sec where available, computed price
per real task, ease of use, and trust. The trust and ease scores are labeled as judgment. The
prices are arithmetic.

Here is what the arithmetic says. These are the numbers that changed how we build.

The price-per-task leaderboard (the part nobody publishes)

Forget per-month pricing. Nobody buys "a month of AI." You buy summaries, images, minutes of
narration, decks, automation runs. So we did the division:

Voice narration: ElevenLabs costs $0.18 per minute on the Creator plan. If your app narrates 1,000 minutes a month, that is $180 — budget it like infrastructure, because it is.
One image: Midjourney $0.05 · DALL-E 3 $0.04 · Stable Diffusion $0.002. Read that again. Self-hosted Stable Diffusion is 25x cheaper per image than Midjourney. At 10 images a day you will never notice. At 10,000 images a day inside a pipeline, that ratio is an architecture decision: API convenience vs. self-hosted GPUs.
One slide deck: Gamma $0.90. One Zapier automation run: $0.027. A Zapier run costs less than the electricity to think about it — which is exactly why automation bills surprise people. They never surprise anyone who did the multiplication first.
Full range across all 30 tools: $0 (NotebookLM, Microsoft Copilot free tier) to $6.25 (a Lovable app build). Three orders of magnitude between the cheapest and priciest "one task." "AI tools cost about the same" is the most expensive sentence in tech.

Speed is a feature, and it has a number

Gemini's flagship runs at 277 tokens/sec — roughly 5x faster than GPT-6-class models.
Speed is not a vibe; it is the difference between a chatbot that feels instant and one where
users start a second tab. If latency is your product (agents, coding assistants, voice), the
tokens/sec column matters more than the benchmark column.

And on benchmarks: Claude Opus 5.5 tops the published reasoning benchmarks we tracked
(Oct 2026). When the task is reasoning-heavy — code review, agentic planning, multi-step
analysis — that is the model family to beat.

How to use these numbers on Monday morning

Three architecture decisions these numbers settle:

Image pipelines: API vs self-hosted. Midjourney at $0.05/image vs self-hosted Stable Diffusion at $0.002/image is a 25x ratio. Multiply your monthly image volume by each number, add your GPU cost to the second column, and the answer stares back at you. Below a few thousand images a month, API convenience wins. Above it, you are donating money.
Voice features: budget like infrastructure. ElevenLabs at $0.18/minute means 1,000 narrated minutes a month is $180. That is a line item, not a rounding error — price it into the feature before you ship it, not after the invoice arrives.
Automation: do the multiplication first. A Zapier run at $0.027 sounds like nothing. Ten thousand runs a month is $270. Every automation bill that ever surprised anyone was just this multiplication, performed too late.

Steal this: measure tokens/sec yourself in 20 lines

Don't take any speed claim on faith — including ours. Time the stream:

import time

def measure_tps(stream):
    """stream: an iterable of text chunks from any streaming LLM API."""
    first_token_at = None
    chars = 0
    start = time.perf_counter()
    for chunk in stream:
        if first_token_at is None:
            first_token_at = time.perf_counter()
        chars += len(chunk)
    elapsed = time.perf_counter() - start
    # rough: ~4 chars per token for English
    return (chars / 4) / elapsed, first_token_at - start

tps, ttft = measure_tps(your_stream)
print(f"{tps:.0f} tok/s, time-to-first-token {ttft:.2f}s")

Two numbers, one script, zero trust required. Run it against every model you ship with.

The honesty contract

Every number in the full scorecard is sourced and dated. Where no public measurement exists,
we wrote N/A instead of inventing one. A benchmark table with no N/As is a fiction table —
if someone shows you 30 tools with 30 confident numbers and zero gaps, they made some of them up.

The free 14-page scorecard has the full ranking (cheapest to priciest per task), a Top 10
overall ranked on evidence, and one-line verdicts plus "best for / skip if" for every tool.
v2.0 runs all 5 test tasks hands-on, per product, in our lab — subscribers get it first.

Decoding AI — AI news + tools, tested with hard numbers. What shipped, what it costs, what
actually works. The full free scorecard: https://decodingaitools.substack.com/p/30-ai-tools-each-one-tested-every