What these numbers mean
A focused local regression panel. Useful for finding strengths and failure modes, with important limits on cross-model comparisons.
54 selected cases, four families
FAST54 is the reduced selection from the 95-case Contested Quality Suite. It tests 50 contested cases plus four frontier cases. These are selected diagnostic questions, not the full parent benchmarks.
Counts are not a leaderboard
Models used different physical contexts, hardware, concurrency and—in some cases—different scorers or eligible questions. A result shown as 9/15 does not cover the same 17 cases as 15/17. No missing or capped answer is quietly converted into a wrong answer.
Publisher-setting runs use each model’s own scenario recipe. Historical nonthinking runs appear separately and are not recommended-setting comparisons.
Compare the evidence.
Each cell is passes / eligible cases, not the model’s score on the whole parent suite. Hover or tap a cell for its run-specific qualification.
Publisher-setting observations
Each model’s own task-specific recipe. These are still different serving conditions.
Contested cases by component
First samples · pass / eligible| Run | ARC | IFEval | HumanEval+ | Tool-Eval | Environment |
|---|
SenseNova: TC-82 passed the native scorer even though the model supplied a file path rather than an attachment. Nanbeige: TC-80 was excluded due to the old scorer; TC-75 also has a clarification-scoring caveat. Ciru Flash: two ARC responses hit a 256-token cap and are excluded.
Historical nonthinking runs
Different sampling and earlier grading protocols. Displayed separately to preserve the original evidence.
Retained historical observations
First samples · pass / eligible| Run | ARC | IFEval | HumanEval+ | Tool-Eval | Environment |
|---|
Historical Nanbeige runs used nonthinking/default settings and pinned seeds in some components. Ornith’s ARC family was withheld. Raon’s IFEval counts were audited from individual strict grades; one coding non-pass was a one-second execution timeout. TC-80 was excluded from these tool scores.
Keep the frontier separate.
Four selected frontier cases are reported apart from the 50 contested cases. These tiny counts are especially sensitive to one outcome.
SenseNova Loop 9B
0 / 4All four selected frontier cases accounted forIFEval case 1883 repeated the same reasoning block. It was stopped at 58,356 decoded tokens and counted as a failure; no final answer was returned.
Nanbeige 33B AutoRound
3 / 4Each publisher-setting passARC 1/1 and IFEval 2/3. Both passes had identical component counts but differed on individual responses.
Ciru Flash IU4
2 / 3IFEval frontier onlyThe native ARC family was withheld after output cutoffs. The IFEval result is a partial frontier observation, not a complete 4-case total.
What did it cost to run?
SenseNova’s quality run and long-context retrieval test are distinct workloads. Neither is a matched speed comparison with the other models.
ARC phase at up to 16 active requests
14,130 output tokens over the 57.209-second phase. Output includes reasoning, and wall time includes prefill and scheduling. It is not per-user decode speed.
The panel time includes a user-directed loop stop, a 39.2-second gap and a pre-request tool-runner fix. Coding and tool work overlapped. FAST54 per-request TTFT and a single aggregate PP/TG were not established.
Long-context synthetic retrieval
A 244,629-token input prompt followed by 272 generated tokens. This was a single C1 retrieval probe, not the C16 FAST54 workload.
One synthetic retrieval example does not establish quality on arbitrary long documents. Both tests used the Q4 community model with restored FP32 routers, Vulkan and an RTX 5090.
Inspect each run.
Open a run for its recorded environment, settings, frontier coverage and component-specific caveats. All outcomes are first samples unless identified otherwise.
The Python generator uses pyecharts==2.0.9 and reads the data and template from its own directory. Model details and run IDs are included in the data file. Full raw traces and retained benchmark stores remain with the original runs. The SenseNova artifact is Q4_K_M with publisher FP32 routers restored, not the original BF16 checkpoint.
Run FAST54 yourself.
The public Contested Quality Suite includes the selected inputs, dependencies and grader sources. Follow its README for current requirements and protocol limits.
Start an OpenAI-compatible model server
cd contested-quality-suite
./benchmark check
./benchmark --endpoint http://127.0.0.1:8080/v1 --model main --label my-model
Use the model ID advertised by your server. ./benchmark --all selects the full 95-case panel and requires the additional dependencies documented in the repository.
Inspect actual model requests
The public default suite does not automatically apply a publisher’s recommended sampler across components. A custom sampling overlay is a different protocol and must retain its settings and grader revision. The public ARC driver’s 256-token cap does not record finish reasons, so inspect server-side telemetry before interpreting a miss.