What these numbers mean
A focused local regression panel. Useful for finding strengths and failure modes, with important limits on cross-model comparisons.
54 selected cases, four families
FAST54 is the reduced selection from the 95-case Contested Quality Suite. It tests 50 contested cases plus four frontier cases. These are selected diagnostic questions, not the full parent benchmarks.
Counts are not a leaderboard
Models used different physical contexts, hardware, concurrency and—in some cases—different scorers or eligible questions. A result shown as 9/15 does not cover the same 17 cases as 15/17. No missing or capped answer is quietly converted into a wrong answer.
Publisher-setting runs use each model’s own scenario recipe. Historical nonthinking runs appear separately and are not recommended-setting comparisons.
Compare the evidence.
Each cell is passes / eligible cases, not the model’s score on the whole parent suite. Hover or tap a cell for its run-specific qualification.
Publisher-setting observations
Each model’s own task-specific recipe. These are still different serving conditions.
Contested cases by component
First samples · pass / eligible| Run | ARC | IFEval | HumanEval+ | Tool-Eval | Environment |
|---|---|---|---|---|---|
| SenseNova Loop 9B · Q4 | 15/17 | 17/24 | 5/5 | 3/4 | Dunamis · RTX 5090 · llama.cpp fork / Vulkan · shared 256K · up to C16 |
| Nanbeige 4.3 33B · AutoRound · pass 1 | 15/17 | 22/24 | 4/5 | 1/3 | Dunamis · RTX 5090 · vLLM · 32K |
| Nanbeige 4.3 33B · AutoRound · pass 2 | 15/17 | 22/24 | 4/5 | 1/3 | Dunamis · RTX 5090 · vLLM · 32K |
| Ciru Flash Next 5.5 · IU4 | 9/15 | 16/24 | 4/5 | 3/4 | Ciru · AMD Strix Halo · IU4 / MTP3 · 32K · C1 |
SenseNova: TC-82 passed the native scorer even though the model supplied a file path rather than an attachment. Nanbeige: TC-80 was excluded due to the old scorer; TC-75 also has a clarification-scoring caveat. Ciru Flash: two ARC responses hit a 256-token cap and are excluded.
Historical nonthinking runs
Different sampling and earlier grading protocols. Displayed separately to preserve the original evidence.
Retained historical observations
First samples · pass / eligible| Run | ARC | IFEval | HumanEval+ | Tool-Eval | Environment |
|---|---|---|---|---|---|
| Nanbeige 4.3 33B · Q6_K | 12/17 | 17/23 | 3/5 | 2/3 | Dunamis · RTX 5090 · llama.cpp / CUDA · 32K |
| Nanbeige 4.3 33B · EXL3 5 bpw | 12/17 | 19/24 | 1/5 | 2/3 | Dunamis · RTX 5090 · ExLlamaV3 / TabbyAPI · 32K |
| Nanbeige 4.3 33B · AutoRound historical | 12/17 | 17/22 | 4/5 | 1/3 | Dunamis · RTX 5090 · vLLM · 32K |
| Nanbeige 4.3 33B · NVFP4 | 11/17 | 19/23 | 4/5 | 1/3 | Dunamis · RTX 5090 · vLLM · 32K |
| Ornith 1.5 · Ciru Ultra | — | 15/24 | 5/5 | 1/3 | Sozo · AMD Strix Halo · 256K · C8 |
| Raon · Ciru Ultra Q8 | 10/17 | 15/24 | 2/5 | 1/3 | Sozo · AMD Strix Halo · 256K · C8 |
Historical Nanbeige runs used nonthinking/default settings and pinned seeds in some components. Ornith’s ARC family was withheld. Raon’s IFEval counts were audited from individual strict grades; one coding non-pass was a one-second execution timeout. TC-80 was excluded from these tool scores.
Keep the frontier separate.
Four selected frontier cases are reported apart from the 50 contested cases. These tiny counts are especially sensitive to one outcome.
SenseNova Loop 9B
0 / 4All four selected frontier cases accounted forIFEval case 1883 repeated the same reasoning block. It was stopped at 58,356 decoded tokens and counted as a failure; no final answer was returned.
Nanbeige 33B AutoRound
3 / 4Each publisher-setting passARC 1/1 and IFEval 2/3. Both passes had identical component counts but differed on individual responses.
Ciru Flash IU4
2 / 3IFEval frontier onlyThe native ARC family was withheld after output cutoffs. The IFEval result is a partial frontier observation, not a complete 4-case total.
What did it cost to run?
SenseNova’s quality run and long-context retrieval test are distinct workloads. Neither is a matched speed comparison with the other models.
ARC phase at up to 16 active requests
14,130 output tokens over the 57.209-second phase. Output includes reasoning, and wall time includes prefill and scheduling. It is not per-user decode speed.
The panel time includes a user-directed loop stop, a 39.2-second gap and a pre-request tool-runner fix. Coding and tool work overlapped. FAST54 per-request TTFT and a single aggregate PP/TG were not established.
Long-context synthetic retrieval
A 244,629-token input prompt followed by 272 generated tokens. This was a single C1 retrieval probe, not the C16 FAST54 workload.
One synthetic retrieval example does not establish quality on arbitrary long documents. Both tests used the Q4 community model with restored FP32 routers, Vulkan and an RTX 5090.
Inspect each run.
Open a run for its recorded environment, settings, frontier coverage and component-specific caveats. All outcomes are first samples unless identified otherwise.
SenseNova Loop 9B · Q4
Environment Dunamis · RTX 5090 · llama.cpp fork / Vulkan · shared 256K · up to C16
Settings Thinking on; temperature 1.0, top-p 0.95, top-k 50; no primary request cap or fixed request seed.
Frontier 0/4; one case manually stopped in a confirmed loop
Source record quality-store-shared-c16-fast54-20261031
- All 17 contested cases completed.
- All 24 contested cases completed; one frontier case was stopped in a loop.
- Chat-adapted HumanEval+ with native base and plus checks.
- TC-80 passed. TC-82 passed natively but supplied a path, not a file attachment.
Nanbeige 4.3 33B · AutoRound · pass 1
Environment Dunamis · RTX 5090 · vLLM · 32K
Settings Thinking on; publisher task-specific sampling; no fixed request seed.
Frontier 3/4
Source record 20261026T223144Z-nanbeige43-autoround-recommended54-pass1-think-32k
- All 17 contested cases eligible.
- All 24 contested cases eligible.
- Chat-adapted HumanEval+.
- TC-80 excluded due to the old negation-scoring defect. The preserved TC-75 failure has a separate clarification-scoring caveat.
Nanbeige 4.3 33B · AutoRound · pass 2
Environment Dunamis · RTX 5090 · vLLM · 32K
Settings Independent second sampled pass; same publisher task-specific settings.
Frontier 3/4
Source record 20261026T223144Z-nanbeige43-autoround-recommended54-pass2-think-32k
- Same family count as pass 1; individual outcomes differ.
- Same family count as pass 1; individual outcomes differ.
- Chat-adapted HumanEval+.
- TC-80 excluded; TC-75 retains the native caveat.
Ciru Flash Next 5.5 · IU4
Environment Ciru · AMD Strix Halo · IU4 / MTP3 · 32K · C1
Settings Publisher nonthinking recipe: temperature 0.7, top-p 0.8, top-k 20, presence penalty 1.5.
Frontier IFEval 2/3; ARC frontier withheld from native collection
Source record ciru-iu4-v55-fast54-recommended-20261026
- Two contested ARC responses hit the 256-token cap; this is an audited 15-case subset, not the same denominator as 17/17 coverage.
- All 24 contested cases completed.
- Chat-adapted HumanEval+.
- TC-80 passed under the corrected scorer.
Nanbeige 4.3 33B · Q6_K
Environment Dunamis · RTX 5090 · llama.cpp / CUDA · 32K
Settings Historical nonthinking/default, with pinned seeds in some components.
Frontier ARC 0/1; IFEval 1/3
Source record 20261026T162354Z-nanbeige43-q6k-contested-fast54-nothink
- 17 eligible contested cases.
- One contested response hit an output cutoff; 23 eligible.
- Five selected coding cases.
- TC-80 excluded due to old grader defect.
Nanbeige 4.3 33B · EXL3 5 bpw
Environment Dunamis · RTX 5090 · ExLlamaV3 / TabbyAPI · 32K
Settings Historical nonthinking/default, with pinned seeds in some components.
Frontier ARC 0/1; IFEval 1/3
Source record 20261026T162400Z-nanbeige43-exl3-5bpw-contested-fast54-nothink
- 17 eligible contested cases.
- 24 eligible contested cases.
- Five selected coding cases.
- TC-80 excluded due to old grader defect.
Nanbeige 4.3 33B · AutoRound historical
Environment Dunamis · RTX 5090 · vLLM · 32K
Settings Historical nonthinking/default; separate from the two publisher-setting passes.
Frontier ARC 0/1; IFEval 0/2 eligible
Source record 20261026T202803Z-nanbeige43-autoround-attempt3-contested-fast54-nothink
- 17 eligible contested cases.
- Two contested responses hit output cutoffs; 22 eligible.
- Five selected coding cases.
- TC-80 excluded due to old grader defect.
Nanbeige 4.3 33B · NVFP4
Environment Dunamis · RTX 5090 · vLLM · 32K
Settings Historical nonthinking/default, with pinned seeds in some components.
Frontier ARC 0/1; IFEval 0/2 eligible
Source record 20261026T213510Z-nanbeige43-nvfp4-attempt3-contested-fast54-nothink
- 17 eligible contested cases.
- One contested response hit a cutoff; 23 eligible.
- Five selected coding cases.
- TC-80 excluded due to old grader defect.
Ornith 1.5 · Ciru Ultra
Environment Sozo · AMD Strix Halo · 256K · C8
Settings Historical nonthinking controlled regression; capped generation.
Frontier IFEval 0/3; ARC family withheld
Source record 20261026T211008Z-sozo-ornith-ultra-fast54
- ARC family was withheld; no interpretable component score.
- 24 contested strict IFEval cases.
- Five selected coding cases.
- TC-80 excluded due to old grader defect.
Raon · Ciru Ultra Q8
Environment Sozo · AMD Strix Halo · 256K · C8
Settings Historical nonthinking controlled regression; capped generation.
Frontier ARC 0/1; IFEval 0/2 eligible, one cutoff
Source record 20261026T212525Z-sozo-raon-ultra-fast54
- 17 contested ARC cases completed.
- Audited one-to-one against the raw strict grades; the collector withheld the family after a frontier cutoff.
- One of the three non-passes was a 1-second execution timeout; a 10-second diagnostic returned the expected result.
- TC-80 excluded; TC-75 has a clarification-matching defect.
The Python generator uses pyecharts==2.0.9 and reads the data and template from its own directory. Model details and run IDs are included in the data file. Full raw traces and retained benchmark stores remain with the original runs. The SenseNova artifact is Q4_K_M with publisher FP32 routers restored, not the original BF16 checkpoint.
Run FAST54 yourself.
The public Contested Quality Suite includes the selected inputs, dependencies and grader sources. Follow its README for current requirements and protocol limits.
Start an OpenAI-compatible model server
cd contested-quality-suite
./benchmark check
./benchmark --endpoint http://127.0.0.1:8080/v1 --model main --label my-model
Use the model ID advertised by your server. ./benchmark --all selects the full 95-case panel and requires the additional dependencies documented in the repository.
Inspect actual model requests
The public default suite does not automatically apply a publisher’s recommended sampler across components. A custom sampling overlay is a different protocol and must retain its settings and grader revision. The public ARC driver’s 256-token cap does not record finish reasons, so inspect server-side telemetry before interpreting a miss.