DEV Community: QuantID

Hugging Face Official Benchmarks: Daily Update 2026-11-02

QuantID — Sun, 01 Nov 2026 15:00:40 +0000

This is the daily update of the Hugging Face Official Benchmarks map, which reads every Hugging Face official benchmark leaderboard straight from the public API.

As of 2026-11-02 (KST): 57 official benchmarks, 1,183 leaderboard entries.

What changed since the last snapshot

VDEBench: new #1 black-forest-labs/FLUX.2-klein-4B (score 0.767), previously Qwen/Qwen-Image-Edit.
safeagents-judge-bench: new #1 vllm-sr/d3-ultra (score 0.994), previously no entries.
14 new leaderboard entries across all boards.

Organizations by number of #1 spots

Rank	Organization	#1 spots
1	FINAL-Bench	14
2	deepseek-ai	5
3	zai-org	5
4	inclusionAI	3
5	moonshotai	3
6	MiniMaxAI	2
7	Qwen	2
8	XiaomiMiMo	2
9	ornith-ai	2
10	Edge0	1

Current #1 on every official benchmark

Benchmark	#1 model	Score	Entries
MMLU-Pro	FINAL-Bench/Darwin-180B-RSI	88.12	144
gpqa	FINAL-Bench/Darwin-180B-RSI	94.44	114
hle	deepseek-ai/DeepSeek-V4.1-Flash	63.9	96
SWE-bench_Verified	ornith-ai/Ornith-1.5-397B	86	68
open-asr-leaderboard	Edge0/ARK-ASR-3B	4.76	65
WildClawBench	Qwen/Qwen3.8-2.4T-A95B	708	51
SWE-bench_Pro	PCL-RSI-lab/Seshat-agent	94.94	47
terminal-bench-2.0	mindlab-research/Macaron-V1-Venti	87.6	38
terminal-bench-2.1	deepseek-ai/DeepSeek-V4.1-Flash	90.6	37
ParseBench	ProKope-AI/ProKope-4B-Eagle-Eye	86.01	35
SWE-bench_Multilingual	inclusionAI/Ling-3.1-flash	88.5	28
deep-swe	deepseek-ai/DeepSeek-V4.1-Flash-1017	77	27
aime_2026	FINAL-Bench/Darwin-180B-RSI	100	26
ScreenSpot-Pro	meta-models/Muse-Glimmer-30B	75.4	24
olmOCR-bench	SoMarkAI/OXR-1.0	87.84	23
Claw-Eval	ornith-ai/Ornith-1.5-397B	81.4	22
gsm8k	XiaomiMiMo/MiMo-V2.5-Pro	99.6	21
hmmt_feb_2026	FINAL-Bench/Darwin-180B-RSI	100	19
Toolathlon	XiaomiMiMo/MiMo-V2.6-Pro-RL	76.9	18
Real5-OmniDocBench	PaddlePaddle/PaddleOCR-VL-1.6	93.19	18
OmniCompass	moonshotai/Kimi-K3	39.6	16
MDPBench	FINAL-Bench/Darwin-180B-RSI	83.65	16
terminal-bench	inclusionAI/Ling-3.1-flash	46.03	15
RedlineBench	crosbylegal/gemini-4-argon	59.5	15
apex-agents	moonshotai/Kimi-K3	41	13
skillsbench	inclusionAI/Ling-3.1-flash	68.7	12
MMMU_Pro	FINAL-Bench/Darwin-180B-RSI	79.48	12
LEXam-hard	FINAL-Bench/Darwin-180B-RSI	45.72	12
Vaani-Benchmark-V1.0	gnani/gnani	13	11
ExtractBench	FINAL-Bench/Darwin-180B-RSI-R3	90.29	11
VDEBench	black-forest-labs/FLUX.2-klein-4B	0.767	10
LEXam	FINAL-Bench/Darwin-180B-RSI	68.94	10
yc-bench	MiniMaxAI/MiniMax-M3	2100000	10
typed-decisions	FINAL-Bench/Darwin-27B-ZTC-v5	0.812	9
agents-last-exam	deepseek-ai/DeepSeek-V4.1-Flash-1017	35.4	9
Long-Horizon-Terminal-Bench	moonshotai/Kimi-K3	6	9
safeagents-judge-bench	vllm-sr/d3-ultra	0.994	8
terminal-bench-science	deepseek-ai/DeepSeek-V4.1-Flash-1017	12.9	8
WBench	robbyant/lingbot-world-fast	78.8	8
physical-engineerings-last-exam	zai-org/GLM-5.2	26	7
ResearchClawBench	zai-org/GLM-5.2	20.709230769230768	7
Video-MME-v2	MiniMaxAI/MiniMax-M3	85.4	7
ifstruct-v1.0	FINAL-Bench/Darwin-180B-RSI	98.95	6
EvasionBench	zai-org/GLM-4.7	82.91	5
arguana	FINAL-Bench/Darwin-180B-RSI-R3	83.83	4
chi-bench	zai-org/GLM-5.1	18.7	3
mmi	Qwen/Qwen3-Omni-30B-A3B-Instruct	13.54	3
BRIGHT	FINAL-Bench/Darwin-BRIGHT-Retriever	0.307	2
APEX-v1-extended	zai-org/GLM-4.7	51.7	2
decision-index-leaderboard	FINAL-Bench/Darwin-27B-ZTC	63.03	1
vlabench_primitive_ft_lerobot_video	nvidia/GR00T-N1-2B	39.7	1

How to read this

Most leaderboard entries are self-reported by model authors, and evaluation settings differ between submissions (sampling, voting, thinking budget). An entry is marked verified only where Hugging Face marks it so. Ties at rank 1 count for each tied organization.

Disclosure: QuantID is a technology partner of VIDRAFT, whose FINAL-Bench organization appears in these rankings. This post reports the public leaderboard data as published on Hugging Face; it is not an independent evaluation.

Hugging Face Official Benchmarks: Daily Update 2026-11-01

QuantID — Sat, 31 Oct 2026 15:00:48 +0000

This is the daily update of the Hugging Face Official Benchmarks map, which reads every Hugging Face official benchmark leaderboard straight from the public API.

As of 2026-11-01 (KST): 57 official benchmarks, 1,169 leaderboard entries.

What changed since the last snapshot

ParseBench: new #1 ProKope-AI/ProKope-4B-Eagle-Eye (score 86.01), previously KDLAI/KDL-Frontier-Parser-nano.
SWE-bench_Pro: new #1 PCL-RSI-lab/Seshat-agent (score 94.94), previously inclusionAI/Ling-3.1-flash.
BRIGHT: new #1 FINAL-Bench/Darwin-BRIGHT-Retriever (score 0.307), previously mteb/baseline-bm25s.
decision-index-leaderboard: new #1 FINAL-Bench/Darwin-27B-ZTC (score 63.03), previously no entries.
arguana: new #1 FINAL-Bench/Darwin-180B-RSI-R3 (score 83.83), previously google/embeddinggemma-300m.
terminal-bench-science: new #1 deepseek-ai/DeepSeek-V4.1-Flash-1017 (score 12.9), previously no entries.
New official benchmarks: VDEBench, physical-engineerings-last-exam, safeagents-judge-bench, asimov-locomotion-bench, mmi, OmniCompass.
29 new leaderboard entries across all boards.

Organizations by number of #1 spots

Rank	Organization	#1 spots
1	FINAL-Bench	14
2	deepseek-ai	5
3	zai-org	5
4	Qwen	3
5	inclusionAI	3
6	moonshotai	3
7	MiniMaxAI	2
8	XiaomiMiMo	2
9	ornith-ai	2
10	Edge0	1

Current #1 on every official benchmark

Benchmark	#1 model	Score	Entries
MMLU-Pro	FINAL-Bench/Darwin-180B-RSI	88.12	144
gpqa	FINAL-Bench/Darwin-180B-RSI	94.44	114
hle	deepseek-ai/DeepSeek-V4.1-Flash	63.9	96
SWE-bench_Verified	ornith-ai/Ornith-1.5-397B	86	68
open-asr-leaderboard	Edge0/ARK-ASR-3B	4.76	65
WildClawBench	Qwen/Qwen3.8-2.4T-A95B	708	51
SWE-bench_Pro	PCL-RSI-lab/Seshat-agent	94.94	47
terminal-bench-2.0	mindlab-research/Macaron-V1-Venti	87.6	38
terminal-bench-2.1	deepseek-ai/DeepSeek-V4.1-Flash	90.6	37
ParseBench	ProKope-AI/ProKope-4B-Eagle-Eye	86.01	35
SWE-bench_Multilingual	inclusionAI/Ling-3.1-flash	88.5	28
deep-swe	deepseek-ai/DeepSeek-V4.1-Flash-1017	77	27
aime_2026	FINAL-Bench/Darwin-180B-RSI	100	26
ScreenSpot-Pro	meta-models/Muse-Glimmer-30B	75.4	24
olmOCR-bench	SoMarkAI/OXR-1.0	87.84	23
Claw-Eval	ornith-ai/Ornith-1.5-397B	81.4	22
gsm8k	XiaomiMiMo/MiMo-V2.5-Pro	99.6	21
hmmt_feb_2026	FINAL-Bench/Darwin-180B-RSI	100	19
Toolathlon	XiaomiMiMo/MiMo-V2.6-Pro-RL	76.9	18
Real5-OmniDocBench	PaddlePaddle/PaddleOCR-VL-1.6	93.19	18
OmniCompass	moonshotai/Kimi-K3	39.6	16
MDPBench	FINAL-Bench/Darwin-180B-RSI	83.65	16
terminal-bench	inclusionAI/Ling-3.1-flash	46.03	15
RedlineBench	crosbylegal/gemini-4-argon	59.5	15
apex-agents	moonshotai/Kimi-K3	41	13
skillsbench	inclusionAI/Ling-3.1-flash	68.7	12
MMMU_Pro	FINAL-Bench/Darwin-180B-RSI	79.48	12
LEXam-hard	FINAL-Bench/Darwin-180B-RSI	45.72	12
Vaani-Benchmark-V1.0	gnani/gnani	13	11
ExtractBench	FINAL-Bench/Darwin-180B-RSI-R3	90.29	11
LEXam	FINAL-Bench/Darwin-180B-RSI	68.94	10
yc-bench	MiniMaxAI/MiniMax-M3	2100000	10
agents-last-exam	deepseek-ai/DeepSeek-V4.1-Flash-1017	35.4	9
Long-Horizon-Terminal-Bench	moonshotai/Kimi-K3	6	9
typed-decisions	FINAL-Bench/Darwin-27B-ZTC-v5	0.812	8
terminal-bench-science	deepseek-ai/DeepSeek-V4.1-Flash-1017	12.9	8
WBench	robbyant/lingbot-world-fast	78.8	8
physical-engineerings-last-exam	zai-org/GLM-5.2	26	7
ResearchClawBench	zai-org/GLM-5.2	20.709230769230768	7
Video-MME-v2	MiniMaxAI/MiniMax-M3	85.4	7
ifstruct-v1.0	FINAL-Bench/Darwin-180B-RSI	98.95	6
VDEBench	Qwen/Qwen-Image-Edit	0.757	5
EvasionBench	zai-org/GLM-4.7	82.91	5
arguana	FINAL-Bench/Darwin-180B-RSI-R3	83.83	4
chi-bench	zai-org/GLM-5.1	18.7	3
mmi	Qwen/Qwen3-Omni-30B-A3B-Instruct	13.54	3
BRIGHT	FINAL-Bench/Darwin-BRIGHT-Retriever	0.307	2
APEX-v1-extended	zai-org/GLM-4.7	51.7	2
decision-index-leaderboard	FINAL-Bench/Darwin-27B-ZTC	63.03	1
vlabench_primitive_ft_lerobot_video	nvidia/GR00T-N1-2B	39.7	1

How to read this

Korea Holds 31% of Hugging Face Official Benchmark #1 Spots With 4% of the Entries

QuantID — Thu, 29 Oct 2026 08:02:58 +0000

TL;DR

On 29 October 2026 (updated 23:30 KST) the Hugging Face Official Benchmarks Map counted 53 official benchmarks, 48 with a leaderboard, 1,135 entries, 449 models and 108 organizations. Grouping each leaderboard's #1 by the publisher's country:

Country	#1 spots	Share	Entries
China	24	50%	668
Korea	15	31%	48
USA	2	4%	233
India	1	2%	1
Unknown	6	13%	126

Korea submits 4% of all entries and holds 31% of the #1 spots. All 15 come from one organization.

Efficiency: few entries, many wins

The ratio of #1 spots to entries is the clearest difference. China's organizations hold 24 #1 spots on 668 entries (one per 28 entries). US organizations hold 2 on 233 (one per 117). Korea holds 15 on 48 (one per 3). By organization, FINAL-Bench (VIDRAFT, Korea) entered 16 leaderboards and is #1 on 15.

Organization	#1 spots
FINAL-Bench (Korea)	15
deepseek-ai (China)	4
zai-org (China)	4
inclusionAI (China)	3
Qwen (China)	3

Depth: one organization carries Korea

China's 24 #1 spots are spread across DeepSeek, Zhipu, Qwen, Moonshot, MiniMax, Xiaomi and others. Korea's 15 all come from FINAL-Bench. Korea's position on this map therefore reflects one organization's results, not a broad national base. Breadth would make the position more stable.

How to read a #1

Board size differs. MMLU-Pro has 144 entries, GPQA 114, HLE 96, while many boards have under 20, and some new boards have only one or two entries.
Margins can be small. Several boards have a #1 within a point of #2.
Scores are self-reported by publishers, as Hugging Face lists them. The map does not re-run evaluations.
The map is live. Numbers in this post are from 29 October 2026, 23:30 KST.

FAQ

How many Hugging Face official benchmarks are there? 53 on 29 October 2026; 48 have a leaderboard.

Which country holds the most #1 spots? China, with 24. Korea is second with 15.

Are these results verified? No. They are publisher-reported scores as published on Hugging Face.

Originally published on QuantID Measurement Lab.

Hugging Face Official Benchmarks Map: 52 Boards, 1,131 Entries, and a New Retrieval Board

QuantID — Wed, 28 Oct 2026 08:38:49 +0000

TL;DR

The Hugging Face Official Benchmarks Map reads every dataset tagged as an official benchmark on Hugging Face, and every leaderboard attached to it, straight from the public API each time the page opens. On 28 October 2026 (17:36 KST) it showed:

Measure	Value
Official benchmarks	52
Benchmarks with a leaderboard	45
Leaderboard entries	1,131
Distinct models	438
Organizations	104

Since our 24 October update (51 benchmarks, 1,118 entries), one benchmark and 13 entries were added. The newest field on the map is Retrieval, with BRIGHT, a reasoning-intensive retrieval benchmark.

What fields do the leaderboards cover?

Field	Leaderboards
Agents & terminal	17
Science & knowledge	6
Coding	5
Documents & OCR	5
Law & finance	4
Vision & video	4
Math	3
Other	3
Retrieval	2
Speech	2
Robotics	1

Agent and terminal benchmarks are now the largest group by far: 17 of 52 leaderboards. Coding, documents, science and math together account for 19.

Which boards draw the most entries?

Benchmark	Field	Entries
TIGER-Lab/MMLU-Pro	Science & knowledge	144
Idavidrein/gpqa	Science & knowledge	114
cais/hle	Science & knowledge	96
SWE-bench/SWE-bench_Verified	Coding	68
hf-audio/open-asr-leaderboard	Speech	63
internlm/WildClawBench	Agents & terminal	51
ScaleAI/SWE-bench_Pro	Coding	47

Participation is concentrated on a few classic knowledge benchmarks. The three science boards above hold 354 of 1,131 entries (31.3%). Many agent boards have fewer than 20 entries.

Who holds the #1 spots?

The map counts each leaderboard once, for the organization at rank 1.

Organization	#1 spots
FINAL-Bench	12
deepseek-ai	4
zai-org	4
inclusionAI	3
moonshotai	3

Twenty organizations hold at least one #1. All scores on these leaderboards are self-reported by the publishers, as Hugging Face shows them.

What is BRIGHT, the new retrieval board?

In most retrieval benchmarks, the right document shares words with the query. In BRIGHT it mostly does not, so a system has to work out what the query needs before it can find the document. Scores are nDCG@10 and stay low: the current leader reports 0.307 and BM25, the classic keyword baseline, reports 0.11. A new board with two entries is a good example of why the map also shows entry counts: a #1 on a board with 144 entries and a #1 on a board with 2 entries are different measurements.

How to read the map

Entry count first. A rank means more when many models compete.
Gap to #2. Several boards have leaders within one point of the runner-up (for example MDPBench 83.65 against 83.6).
Self-reported. Hugging Face lists the scores publishers submit. The map does not re-run any evaluation.
Live. Numbers change as new results are indexed. This post records the state at 17:36 KST on 28 October 2026.

FAQ

How many official benchmarks does Hugging Face have?
52 on 28 October 2026, of which 45 have at least one leaderboard entry.

How many models appear on these leaderboards?
438 distinct models from 104 organizations, in 1,131 entries.

Which field has the most leaderboards?
Agents & terminal, with 17.

Are these scores verified?
No. They are self-reported by publishers. The map reports them as published.

Where can I see the live data?
At huggingface.co/spaces/quantid/huggingface-official-benchmark-leaderboards.

Originally published on QuantID Measurement Lab.

How Do You Show Two Training Runs Are Equal? A Paired Test on Unseen Text, Using an NPU vs GPU Case

QuantID — Mon, 26 Oct 2026 05:54:22 +0000

Disclosure: the case study below is VIDRAFT's own result. VIDRAFT is a technology partner of QuantID. This post reviews the measurement design; it is not an independent verification.

TL;DR

To claim two training runs reached the same quality, you need two things: an evaluation set the models truly never saw, and a paired comparison on identical inputs. The case here is VIDRAFT's VKORE-325M, pretrained on a Rebellions ATOM-Max NPU and, as a reference, on a B200 GPU. On 1,024 unseen windows the losses are 0.7993 and 0.7994; the paired difference is -0.0001 with a standard error of 0.0007.

What makes a fair comparison between two runs?

Hold everything fixed except the variable you test. In this case the model, initialization, data, batch and learning-rate schedule were identical, and only the hardware differed.

Matching training curves are encouraging but not proof. Training loss is measured on data the model is fitting.

Is "held out" really held out?

This case contains a useful lesson. The first evaluation used 256 fixed windows from the end of the training file and was called "held out". But training sampled random windows from the whole file, about 0.72 passes over it, so those windows were partly seen during training. The comparison between the two runs was still fair, because both saw the same file, but the label was wrong.

The fix was to evaluate on a shard of the source dataset that was never part of the training corpus. The training file came from FineWeb-Edu sample/10BT shards 000 and 001; the new evaluation used shard 013.

Check: before calling a set held out, read the training sampler, not only the data split. Random-window samplers cover the whole file.

Why compare windows in pairs instead of averages?

Loss varies far more between text windows than between two good models. If you compare two averages, that window-to-window variation dominates the error bar.

Evaluating both models on identical windows and taking the per-window difference removes it. Here the per-model standard error is about 0.0049, while the paired standard error is 0.0007, about seven times tighter.

Quantity	Value
NPU loss, unseen (1,024 windows)	0.7993
GPU loss, unseen	0.7994
Paired mean difference (NPU minus GPU)	-0.0001
Paired standard error	0.0007
Windows where NPU is lower	50.1%

The honest reading is "no detectable difference at a resolution of about 0.0014 (two standard errors)". That is a stronger statement than "the averages look close", and a weaker one than "identical".

What else should a reader check?

Reproducibility. Both weights and the evaluation script are public, so the paired test can be rerun.
Scope. The result concerns a 325M base model on 5B tokens. It says nothing about larger scales.
What is not measured. Speed and cost per run are separate questions. The case also notes that the NPU alone does not enable training; it depends on the developer's own training stack.

FAQ

Why bits per byte? With a byte-level vocabulary, loss per byte divided by ln 2 gives bits per byte (1.1531 vs 1.1533 here), which is comparable across tokenizers.

Is 1,024 windows enough? For a paired test with this variance, yes: the standard error of the difference is 0.0007.

Could a single seed hide differences? Possibly. A stronger design repeats each run with several seeds; this case reports one run per hardware.

Who measured this? VIDRAFT measured its own models. This post reviews the design.

Hugging Face Official Benchmarks: Daily Update 2026-10-24

QuantID — Fri, 23 Oct 2026 15:01:23 +0000

This is the daily update of the Hugging Face Official Benchmarks map, which reads every Hugging Face official benchmark leaderboard straight from the public API.

As of 2026-10-24 (KST): 51 official benchmarks, 1,118 leaderboard entries.

What changed since the last snapshot

New official benchmarks: decision-index-leaderboard, decision-index-vision-leaderboard.
5 new leaderboard entries across all boards.

Organizations by number of #1 spots

Rank	Organization	#1 spots
1	FINAL-Bench	11
2	deepseek-ai	4
3	inclusionAI	4
4	zai-org	4
5	MiniMaxAI	2
6	XiaomiMiMo	2
7	moonshotai	2
8	ornith-ai	2
9	Edge0	1
10	KDLAI	1

Current #1 on every official benchmark

Benchmark	#1 model	Score	Entries
MMLU-Pro	FINAL-Bench/Darwin-180B-RSI	88.12	142
gpqa	FINAL-Bench/Darwin-180B-RSI	94.44	115
hle	deepseek-ai/DeepSeek-V4.1-Flash	63.9	98
SWE-bench_Verified	ornith-ai/Ornith-1.5-397B	86	68
open-asr-leaderboard	Edge0/ARK-ASR-3B	4.76	63
WildClawBench	Qwen/Qwen3.8-2.4T-A95B	708	51
SWE-bench_Pro	inclusionAI/Ling-3.1-flash	68.74	47
terminal-bench-2.0	mindlab-research/Macaron-V1-Venti	87.6	40
terminal-bench-2.1	deepseek-ai/DeepSeek-V4.1-Flash	90.6	37
ParseBench	KDLAI/KDL-Frontier-Parser-nano	76.36	34
SWE-bench_Multilingual	inclusionAI/Ling-3.1-flash	88.5	28
deep-swe	deepseek-ai/DeepSeek-V4.1-Flash-1017	77	27
aime_2026	FINAL-Bench/Darwin-180B-RSI	100	27
ScreenSpot-Pro	meta-models/Muse-Glimmer-30B	75.4	24
Claw-Eval	ornith-ai/Ornith-1.5-397B	81.4	23
olmOCR-bench	SoMarkAI/OXR-1.0	87.84	22
hmmt_feb_2026	FINAL-Bench/Darwin-180B-RSI	100	20
gsm8k	XiaomiMiMo/MiMo-V2.5-Pro	99.6	19
Toolathlon	XiaomiMiMo/MiMo-V2.6-Pro-RL	76.9	18
Real5-OmniDocBench	PaddlePaddle/PaddleOCR-VL-1.6	93.19	17
MDPBench	FINAL-Bench/Darwin-180B-RSI	83.65	16
RedlineBench	crosbylegal/gemini-4-argon	59.5	14
apex-agents	moonshotai/Kimi-K3	41	13
skillsbench	inclusionAI/Ling-3.1-flash	68.7	12
LEXam-hard	FINAL-Bench/Darwin-180B-RSI	45.72	12
ExtractBench	FINAL-Bench/Darwin-180B-RSI-R3	90.29	11
MMMU_Pro	FINAL-Bench/Darwin-180B-RSI	79.48	11
Vaani-Benchmark-V1.0	gnani/gnani	13	11
agents-last-exam	deepseek-ai/DeepSeek-V4.1-Flash-1017	35.4	10
yc-bench	MiniMaxAI/MiniMax-M3	2100000	10
LEXam	FINAL-Bench/Darwin-180B-RSI	68.94	10
Long-Horizon-Terminal-Bench	moonshotai/Kimi-K3	6	9
typed-decisions	FINAL-Bench/Darwin-27B-ZTC-v5	0.812	8
terminal-bench	inclusionAI/Ling-3.1-flash	46.03	8
WBench	robbyant/lingbot-world-fast	78.8	8
Video-MME-v2	MiniMaxAI/MiniMax-M3	85.4	7
ResearchClawBench	zai-org/GLM-5.2	20.709230769230768	7
ifstruct-v1.0	FINAL-Bench/Darwin-180B-RSI	98.95	6
EvasionBench	zai-org/GLM-4.7	82.91	5
arguana	google/embeddinggemma-300m	71.535	3
chi-bench	zai-org/GLM-5.1	18.7	3
APEX-v1-extended	zai-org/GLM-4.7	51.7	2
vlabench_primitive_ft_lerobot_video	nvidia/GR00T-N1-2B	39.7	1
BRIGHT	mteb/baseline-bm25s	0.11	1

How to read this

Hugging Face Official Benchmarks: Daily Update 2026-10-23

QuantID — Thu, 22 Oct 2026 15:00:37 +0000

This is the daily update of the Hugging Face Official Benchmarks map, which reads every Hugging Face official benchmark leaderboard straight from the public API.

As of 2026-10-23 (KST): 49 official benchmarks, 1,113 leaderboard entries.

What changed since the last snapshot

agents-last-exam: new #1 deepseek-ai/DeepSeek-V4.1-Flash-1017 (score 35.4), previously XiaomiMiMo/MiMo-V2.6-Pro-RL.
deep-swe: new #1 deepseek-ai/DeepSeek-V4.1-Flash-1017 (score 77), previously deepseek-ai/DeepSeek-V4.1-Flash.
5 new leaderboard entries across all boards.

Organizations by number of #1 spots

Rank	Organization	#1 spots
1	FINAL-Bench	11
2	deepseek-ai	4
3	inclusionAI	4
4	zai-org	4
5	MiniMaxAI	2
6	XiaomiMiMo	2
7	moonshotai	2
8	ornith-ai	2
9	Edge0	1
10	KDLAI	1

Current #1 on every official benchmark

Benchmark	#1 model	Score	Entries
MMLU-Pro	FINAL-Bench/Darwin-180B-RSI	88.12	142
gpqa	FINAL-Bench/Darwin-180B-RSI	94.44	115
hle	deepseek-ai/DeepSeek-V4.1-Flash	63.9	97
SWE-bench_Verified	ornith-ai/Ornith-1.5-397B	86	68
open-asr-leaderboard	Edge0/ARK-ASR-3B	4.76	63
WildClawBench	Qwen/Qwen3.8-2.4T-A95B	708	51
SWE-bench_Pro	inclusionAI/Ling-3.1-flash	68.74	47
terminal-bench-2.0	mindlab-research/Macaron-V1-Venti	87.6	40
terminal-bench-2.1	deepseek-ai/DeepSeek-V4.1-Flash	90.6	37
ParseBench	KDLAI/KDL-Frontier-Parser-nano	76.36	34
SWE-bench_Multilingual	inclusionAI/Ling-3.1-flash	88.5	28
aime_2026	FINAL-Bench/Darwin-180B-RSI	100	27
deep-swe	deepseek-ai/DeepSeek-V4.1-Flash-1017	77	26
ScreenSpot-Pro	meta-models/Muse-Glimmer-30B	75.4	24
Claw-Eval	ornith-ai/Ornith-1.5-397B	81.4	23
olmOCR-bench	SoMarkAI/OXR-1.0	87.84	22
hmmt_feb_2026	FINAL-Bench/Darwin-180B-RSI	100	20
gsm8k	XiaomiMiMo/MiMo-V2.5-Pro	99.6	19
Toolathlon	XiaomiMiMo/MiMo-V2.6-Pro-RL	76.9	18
Real5-OmniDocBench	PaddlePaddle/PaddleOCR-VL-1.6	93.19	17
MDPBench	FINAL-Bench/Darwin-180B-RSI	83.65	16
RedlineBench	crosbylegal/gemini-4-argon	59.5	14
apex-agents	moonshotai/Kimi-K3	41	13
skillsbench	inclusionAI/Ling-3.1-flash	68.7	12
LEXam-hard	FINAL-Bench/Darwin-180B-RSI	45.72	12
ExtractBench	FINAL-Bench/Darwin-180B-RSI-R3	90.29	11
MMMU_Pro	FINAL-Bench/Darwin-180B-RSI	79.48	11
LEXam	FINAL-Bench/Darwin-180B-RSI	68.94	10
yc-bench	MiniMaxAI/MiniMax-M3	2100000	10
Vaani-Benchmark-V1.0	gnani/gnani	13	10
agents-last-exam	deepseek-ai/DeepSeek-V4.1-Flash-1017	35.4	9
Long-Horizon-Terminal-Bench	moonshotai/Kimi-K3	6	9
typed-decisions	FINAL-Bench/Darwin-27B-ZTC-v5	0.812	8
WBench	robbyant/lingbot-world-fast	78.8	8
terminal-bench	inclusionAI/Ling-3.1-flash	46.03	7
Video-MME-v2	MiniMaxAI/MiniMax-M3	85.4	7
ResearchClawBench	zai-org/GLM-5.2	20.709230769230768	7
ifstruct-v1.0	FINAL-Bench/Darwin-180B-RSI	98.95	6
EvasionBench	zai-org/GLM-4.7	82.91	5
chi-bench	zai-org/GLM-5.1	18.7	3
arguana	google/embeddinggemma-300m	71.535	3
APEX-v1-extended	zai-org/GLM-4.7	51.7	2
vlabench_primitive_ft_lerobot_video	nvidia/GR00T-N1-2B	39.7	1
BRIGHT	mteb/baseline-bm25s	0.11	1

How to read this

Hugging Face Official Benchmarks: Daily Update 2026-10-20

QuantID — Mon, 19 Oct 2026 15:00:54 +0000

This is the daily update of the Hugging Face Official Benchmarks map, which reads every Hugging Face official benchmark leaderboard straight from the public API.

As of 2026-10-20 (KST): 49 official benchmarks, 1,107 leaderboard entries.

What changed since the last snapshot

olmOCR-bench: new #1 SoMarkAI/OXR-1.0 (score 87.84), previously infly/Infinity-Parser2-Pro.
5 new leaderboard entries across all boards.

Organizations by number of #1 spots

Rank	Organization	#1 spots
1	FINAL-Bench	11
2	inclusionAI	4
3	zai-org	4
4	XiaomiMiMo	3
5	deepseek-ai	3
6	MiniMaxAI	2
7	moonshotai	2
8	ornith-ai	2
9	Edge0	1
10	KDLAI	1

Current #1 on every official benchmark

Benchmark	#1 model	Score	Entries
MMLU-Pro	FINAL-Bench/Darwin-180B-RSI	88.12	142
gpqa	FINAL-Bench/Darwin-180B-RSI	94.44	115
hle	deepseek-ai/DeepSeek-V4.1-Flash	63.9	95
SWE-bench_Verified	ornith-ai/Ornith-1.5-397B	86	68
open-asr-leaderboard	Edge0/ARK-ASR-3B	4.76	63
WildClawBench	Qwen/Qwen3.8-2.4T-A95B	708	51
SWE-bench_Pro	inclusionAI/Ling-3.1-flash	68.74	47
terminal-bench-2.0	mindlab-research/Macaron-V1-Venti	87.6	40
terminal-bench-2.1	deepseek-ai/DeepSeek-V4.1-Flash	90.6	37
ParseBench	KDLAI/KDL-Frontier-Parser-nano	76.36	34
SWE-bench_Multilingual	inclusionAI/Ling-3.1-flash	88.5	28
aime_2026	FINAL-Bench/Darwin-180B-RSI	100	27
deep-swe	deepseek-ai/DeepSeek-V4.1-Flash	74.2	25
ScreenSpot-Pro	meta-models/Muse-Glimmer-30B	75.4	24
Claw-Eval	ornith-ai/Ornith-1.5-397B	81.4	23
olmOCR-bench	SoMarkAI/OXR-1.0	87.84	22
hmmt_feb_2026	FINAL-Bench/Darwin-180B-RSI	100	20
gsm8k	XiaomiMiMo/MiMo-V2.5-Pro	99.6	19
Toolathlon	XiaomiMiMo/MiMo-V2.6-Pro-RL	76.9	18
Real5-OmniDocBench	PaddlePaddle/PaddleOCR-VL-1.6	93.19	17
MDPBench	FINAL-Bench/Darwin-180B-RSI	83.65	15
RedlineBench	crosbylegal/gemini-4-argon	59.5	14
apex-agents	moonshotai/Kimi-K3	41	13
skillsbench	inclusionAI/Ling-3.1-flash	68.7	12
LEXam-hard	FINAL-Bench/Darwin-180B-RSI	45.72	12
ExtractBench	FINAL-Bench/Darwin-180B-RSI-R3	90.29	11
MMMU_Pro	FINAL-Bench/Darwin-180B-RSI	79.48	11
LEXam	FINAL-Bench/Darwin-180B-RSI	68.94	10
yc-bench	MiniMaxAI/MiniMax-M3	2100000	10
Vaani-Benchmark-V1.0	gnani/gnani	13	10
Long-Horizon-Terminal-Bench	moonshotai/Kimi-K3	6	9
typed-decisions	FINAL-Bench/Darwin-27B-ZTC-v5	0.812	8
agents-last-exam	XiaomiMiMo/MiMo-V2.6-Pro-RL	31.6	8
WBench	robbyant/lingbot-world-fast	78.8	8
ResearchClawBench	zai-org/GLM-5.2	20.709230769230768	7
Video-MME-v2	MiniMaxAI/MiniMax-M3	85.4	7
ifstruct-v1.0	FINAL-Bench/Darwin-180B-RSI	98.95	6
terminal-bench	inclusionAI/Ling-3.1-flash	46.03	6
EvasionBench	zai-org/GLM-4.7	82.91	5
chi-bench	zai-org/GLM-5.1	18.7	3
arguana	google/embeddinggemma-300m	71.535	3
APEX-v1-extended	zai-org/GLM-4.7	51.7	2
vlabench_primitive_ft_lerobot_video	nvidia/GR00T-N1-2B	39.7	1
BRIGHT	mteb/baseline-bm25s	0.11	1

How to read this

Hugging Face Official Benchmarks: Daily Update 2026-10-18

QuantID — Sun, 18 Oct 2026 03:18:26 +0000

This is the daily update of the Hugging Face Official Benchmarks map, which reads every Hugging Face official benchmark leaderboard straight from the public API.

As of 2026-10-18 (KST): 49 official benchmarks, 1,102 leaderboard entries.

This is the first snapshot in the series, so it records the baseline: who is ranked first on each board today. Later posts will report only what changed.

Organizations by number of #1 spots

Rank	Organization	#1 spots
1	FINAL-Bench	11
2	inclusionAI	4
3	zai-org	4
4	XiaomiMiMo	3
5	deepseek-ai	3
6	MiniMaxAI	2
7	moonshotai	2
8	ornith-ai	2
9	Edge0	1
10	KDLAI	1

Current #1 on every official benchmark

Benchmark	#1 model	Score	Entries
MMLU-Pro	FINAL-Bench/Darwin-180B-RSI	88.12	142
gpqa	FINAL-Bench/Darwin-180B-RSI	94.44	115
hle	deepseek-ai/DeepSeek-V4.1-Flash	63.9	95
SWE-bench_Verified	ornith-ai/Ornith-1.5-397B	86	68
open-asr-leaderboard	Edge0/ARK-ASR-3B	4.76	63
WildClawBench	Qwen/Qwen3.8-2.4T-A95B	708	51
SWE-bench_Pro	inclusionAI/Ling-3.1-flash	68.74	47
terminal-bench-2.0	mindlab-research/Macaron-V1-Venti	87.6	40
terminal-bench-2.1	deepseek-ai/DeepSeek-V4.1-Flash	90.6	37
ParseBench	KDLAI/KDL-Frontier-Parser-nano	76.36	34
SWE-bench_Multilingual	inclusionAI/Ling-3.1-flash	88.5	28
aime_2026	FINAL-Bench/Darwin-180B-RSI	100	27
deep-swe	deepseek-ai/DeepSeek-V4.1-Flash	74.2	25
ScreenSpot-Pro	meta-models/Muse-Glimmer-30B	75.4	23
Claw-Eval	ornith-ai/Ornith-1.5-397B	81.4	23
olmOCR-bench	infly/Infinity-Parser2-Pro	87.6	21
hmmt_feb_2026	FINAL-Bench/Darwin-180B-RSI	100	20
gsm8k	XiaomiMiMo/MiMo-V2.5-Pro	99.6	18
Toolathlon	XiaomiMiMo/MiMo-V2.6-Pro-RL	76.9	18
Real5-OmniDocBench	PaddlePaddle/PaddleOCR-VL-1.6	93.19	17
MDPBench	FINAL-Bench/Darwin-180B-RSI	83.65	15
RedlineBench	crosbylegal/gemini-4-argon	59.5	14
apex-agents	moonshotai/Kimi-K3	41	13
skillsbench	inclusionAI/Ling-3.1-flash	68.7	12
LEXam-hard	FINAL-Bench/Darwin-180B-RSI	45.72	12
ExtractBench	FINAL-Bench/Darwin-180B-RSI-R3	90.29	11
MMMU_Pro	FINAL-Bench/Darwin-180B-RSI	79.48	10
LEXam	FINAL-Bench/Darwin-180B-RSI	68.94	10
yc-bench	MiniMaxAI/MiniMax-M3	2100000	10
Vaani-Benchmark-V1.0	gnani/gnani	13	10
Long-Horizon-Terminal-Bench	moonshotai/Kimi-K3	6	9
agents-last-exam	XiaomiMiMo/MiMo-V2.6-Pro-RL	31.6	8
WBench	robbyant/lingbot-world-fast	78.8	8
typed-decisions	FINAL-Bench/Darwin-27B-ZTC-v5	0.812	7
ResearchClawBench	zai-org/GLM-5.2	20.709230769230768	7
Video-MME-v2	MiniMaxAI/MiniMax-M3	85.4	7
ifstruct-v1.0	FINAL-Bench/Darwin-180B-RSI	98.95	6
terminal-bench	inclusionAI/Ling-3.1-flash	46.03	6
EvasionBench	zai-org/GLM-4.7	82.91	5
chi-bench	zai-org/GLM-5.1	18.7	3
arguana	google/embeddinggemma-300m	71.535	3
APEX-v1-extended	zai-org/GLM-4.7	51.7	2
vlabench_primitive_ft_lerobot_video	nvidia/GR00T-N1-2B	39.7	1
BRIGHT	mteb/baseline-bm25s	0.11	1

How to read this

Measuring the Real Cost of an LLM Call: Tokens, Precision, and Honest Numbers

QuantID — Mon, 12 Oct 2026 10:34:35 +0000

TL;DR

The price per million tokens is the sticker, not the cost. The real cost of an LLM call depends on how many tokens you actually spend, the precision the provider serves at, how much the request waits, and whether the endpoint is even up. This post is a measurement checklist for putting a defensible number on an LLM call, written from the view of a team whose product is measurement.

The sticker is not the cost

Two providers can quote the same price per million tokens and cost you very different amounts. The gap hides in four places.

Token spend. Reasoning models and verbose system prompts change how many tokens a task actually burns. The unit price is fixed, the unit count is not.
Precision. A model served at a lower numeric precision is cheaper to run, and the provider may or may not tell you which precision you are getting. Precision is a cost and quality lever at the same time.
Latency. First-token latency and tokens per second decide whether a call fits a user-facing budget. Public price pages rarely publish these.
Availability. A cheap endpoint that returns a 503 under load has an effective cost of infinity for that request.

Precision deserves its own line

When you compare two deployments of the same model, the flags on the command line are not the whole story. The served weights can differ in precision even when the launch looks identical, and that changes both speed and output. If you are benchmarking, pin the precision you are actually measuring and state it. A fair comparison names the precision; an unfair one hides it.

A simple way to keep this honest is to treat precision as an explicit weight in your cost model rather than folding it silently into a single price number. State the assumption, then let readers adjust it.

Latency is a cost, not a vibe

First-token latency and tokens per second are measurable, and they belong in the cost story. The catch is that public APIs often do not expose them, so you have to call the endpoint yourself with the same prompt and record the numbers. One call is a sample, not a law. Report how many calls, and from where, because the answer moves with both.

A measurement checklist

Before you publish a cost number for an LLM call, confirm:

You counted actual tokens for a representative task, not just the unit price.
You named the serving precision you measured.
You measured first-token latency and throughput yourself, with the call count stated.
You checked availability under the load you care about, not at idle.
You separated numbers you measured from numbers you assumed, and labeled the assumptions.

If a comparison is missing any of these, it is a quote, not a measurement.

Why this matters now

As more products route across many models, the question is no longer which model is best in the abstract. It is which deployment gives the quality you need at a cost you can defend. That is a measurement problem, and measurement problems reward discipline over marketing.

Note on relationships: QuantID works with technology partners on model and serving research; where a partner is relevant we disclose it as a partner rather than an endorsement.

FAQ

Is price per million tokens useless? No. It is one input. The cost is price times actual token count, adjusted for precision, latency, and availability.

How do I measure precision if the provider does not say? Measure behavior and speed on fixed prompts, and state that the served precision is unverified when it is.

Why measure latency myself? Because most public APIs do not expose first-token latency, and it is decisive for user-facing budgets.

One call or many? Many. One call is a sample. State the count and the origin.

Eleven Number-One Records: Measuring a Model That Swept Math, Science, Law and Decisions

QuantID — Sun, 11 Oct 2026 02:50:52 +0000

TL;DR

A single self-improving model family holds eleven public number-one benchmark records at once, across mathematics, science, law, structured output, and decisions. This is a measurement note: the eleven, their scores, and why the spread matters more than any single win.

The eleven records

#	Benchmark	Field	Result
1	AIME 2026	Math	100% (perfect)
2	HMMT 2026	Math	100% (perfect)
3	GPQA Diamond	Science	94.44%
4	MMLU-Pro	Knowledge	88.12%
5	MMMU-Pro	Multimodal	79.48%
6	LEXam	Law	68.94%
7	LEXam-hard	Law	45.72%
8	ExtractBench	Extraction	90.29%
9	IFStruct	Structured output	98.95%
10	MDPBench	Decision process	83.65%
11	S1MB	Decision engine	Borda 89.58, Task Avg 66.46

Why the spread is the real signal

A single top score can come from tuning to one test. Eleven number-one records across unrelated fields cannot. Math, science, law, and typed decisions stress different capabilities, so holding the top of all of them at once is evidence of a general method rather than a lucky fit.

Two of the records are perfect scores, on AIME 2026 and HMMT 2026. When a benchmark saturates, the honest reading is that it no longer discriminates at the top, which is itself a measurement result and a reason to move evaluation toward harder, decision-oriented tests such as S1MB.

How these were produced

The family is self-improving: it runs a recursive loop tied to external verification rather than to its own opinion. For decisions, where the output is typed, scoring uses a zero-token method, which keeps the measurement deterministic and reproducible.

Check it yourself

S1MB number one model: https://github.com/final-bench/s1mb
ZTC decision method: https://github.com/final-bench/ztc
Models: https://huggingface.co/FINAL-Bench

FAQ

How many number-one records does the family hold?
Eleven public number-one benchmark records at once, across math, science, law, structured output, and decisions.

Which are perfect scores?
AIME 2026 and HMMT 2026, both at 100%.

Why does holding many unrelated benchmarks matter?
A single win can reflect tuning to one test. Leading many unrelated fields at once is evidence of a general method.

Are the results reproducible?
Decision results use a deterministic zero-token method, and the leaderboards and models are public.

Reading the S1MB result: why Borda aggregation makes a #1 harder to fake

QuantID — Fri, 09 Oct 2026 08:58:56 +0000

The System One Mosaic Benchmark (S1MB, by hotchpotch) compares 102 models across 137 specialized benchmarks in three task families: Noul (assess a condition), Choice (select an option), and Score (rate on a scale). This is a measurement note on how it scores and why its top position is informative.

Borda, not average

S1MB ranks by a Borda score. Instead of averaging raw task numbers, which lets one easy benchmark dominate the mean, Borda converts each of the 137 benchmarks into a rank and sums rank-based points. That rewards broad, consistent strength over a spike on a few favorable tasks. It is a deliberately conservative aggregation, and it makes a top position harder to manufacture: you cannot buy first place by overfitting a handful of suites.

The current top

Darwin-27B-ZTC-v2 leads with Borda 89.58, ahead of OpenJev-27B (87.50) and AutoJev-27B (87.07). The margin is widest on the Score family (60.21 vs 54.83), which is the ordinal-judgment task where most judges weaken. Leading where the task is hardest, rather than on an easy split, is the signal worth reading.

Why a deterministic judge is clean to measure

A zero-token judge emits a probability distribution in a single forward pass. No sampling means no output variance, so the reported numbers are exact values rather than sample estimates. There is no standard-deviation term to carry and no decoding hyperparameters (temperature, top-p, length) to confound cross-model comparison. The same input reproduces the same score.

On a related board, typed-decisions, the same approach is also #1 at accuracy 0.743 (zero-shot), with calibration KL 0.204 and Brier 0.097. Brier is a strictly proper scoring rule, so confidence cannot be gamed by inflating or deflating it.

DEV Community: QuantID

Hugging Face Official Benchmarks: Daily Update 2026-11-02

What changed since the last snapshot

Organizations by number of #1 spots

Current #1 on every official benchmark

How to read this

Hugging Face Official Benchmarks: Daily Update 2026-11-01

What changed since the last snapshot

Organizations by number of #1 spots

Current #1 on every official benchmark

How to read this

Korea Holds 31% of Hugging Face Official Benchmark #1 Spots With 4% of the Entries

TL;DR

Efficiency: few entries, many wins

Depth: one organization carries Korea

How to read a #1

FAQ

Hugging Face Official Benchmarks Map: 52 Boards, 1,131 Entries, and a New Retrieval Board

TL;DR

What fields do the leaderboards cover?

Which boards draw the most entries?

Who holds the #1 spots?

What is BRIGHT, the new retrieval board?

How to read the map

FAQ

How Do You Show Two Training Runs Are Equal? A Paired Test on Unseen Text, Using an NPU vs GPU Case

TL;DR

What makes a fair comparison between two runs?

Is "held out" really held out?

Why compare windows in pairs instead of averages?

What else should a reader check?

FAQ

Further reading

Hugging Face Official Benchmarks: Daily Update 2026-10-24

What changed since the last snapshot

Organizations by number of #1 spots

Current #1 on every official benchmark

How to read this

Hugging Face Official Benchmarks: Daily Update 2026-10-23

What changed since the last snapshot

Organizations by number of #1 spots

Current #1 on every official benchmark

How to read this

Hugging Face Official Benchmarks: Daily Update 2026-10-20

What changed since the last snapshot

Organizations by number of #1 spots

Current #1 on every official benchmark

How to read this

Hugging Face Official Benchmarks: Daily Update 2026-10-18

Organizations by number of #1 spots

Current #1 on every official benchmark

How to read this

Measuring the Real Cost of an LLM Call: Tokens, Precision, and Honest Numbers

TL;DR

The sticker is not the cost

Precision deserves its own line

Latency is a cost, not a vibe

A measurement checklist

Why this matters now

FAQ

Further reading

Eleven Number-One Records: Measuring a Model That Swept Math, Science, Law and Decisions

TL;DR

The eleven records

Why the spread is the real signal

How these were produced

Check it yourself

FAQ

Reading the S1MB result: why Borda aggregation makes a #1 harder to fake

Borda, not average

The current top

Why a deterministic judge is clean to measure

Links