llmprobe report card

/Volumes/edata2/models/afm/Qwen3.8-Flash-Next-AFM-MLX-4bit

http://127.0.0.1:10302/v1

Overview

Three independent scores — never averaged

Surface coverage

100%

Core 9/9 core complete

core 100%

extended 86.7%

frontier 88.9%

How much of the standard API surface exists. Missing features are listed on purpose.

Engine conformance

100%

195/195 MUST no MUST fails

Of the surfaces that exist, how correct are the MUST behaviors. Unsupported ≠ fail.

Model capability

94.4%

strong 8 categories

Practical floor for tools, JSON, instructions — graded below floor / capable / strong.

Performance

69.2

tok/s decode 175 ms first token

Informational — hardware-dependent and never scored. Same-machine comparisons only.

Agentic

3/3

Harder multi-step bar. Never blended into capability.

Engine fidelity

97.78%

Same-model only — holds the model constant so the number is the engine.

Outcomes honesty

Pass92

Fail0

Unsupported5

Inconclusive2

Skipped15

Unsupported and inconclusive are not zeros and not fails.

Surface coverage engine

Core 100%

Per tier, never averaged. Click Core / Extended / Frontier to expand every feature under that tier.

Click a tier row to expand · missing features sort first

Feature	Kind	Status	Detail
/v1/models	surface	supported	present
chat/completions	surface	supported	present
error codes	feature	supported	present
finish reasons	feature	supported	present
JSON mode	feature	supported	present
stop + max_tokens	feature	supported	present
streaming + SSE framing	feature	supported	present
tool calling	feature	supported	present
usage tokens	feature	supported	present

✗ reasoning items✗ reasoning history round-trip

Feature	Kind	Status	Detail
reasoning history round-trip	feature	missing	prompt_tokens identical with and without history reasoning — the field is dropped before the chat template, or this model's template strips history reasoning by design
reasoning items	feature	missing	no separate reasoning channel, even with the standard opt-in
completions (legacy)	surface	supported	present
embeddings	surface	supported	present
logprobs	feature	supported	present
max_tokens legacy alias	feature	supported	present
messages	surface	supported	present
messages/count_tokens	surface	supported	present
parallel tool calls	feature	supported	present
responses	surface	supported	present
seed determinism	feature	supported	present
streamed usage	feature	supported	present
structured outputs	feature	supported	present
top_p sampling	feature	supported	present
vision	feature	supported	present

✗ MCP tools

not probed: prompt caching

Feature	Kind	Status	Detail
MCP tools	feature	missing	not probed — llmprobe does not yet host an MCP server to call back into
prompt caching	feature	not probed	not probed at this depth
audio/speech	surface	supported	present
audio/transcriptions	surface	supported	present
background responses	feature	supported	present
images/edits	surface	supported	present
images/generations	surface	supported	present
n>1 choices	feature	supported	present
previous_response_id chaining	feature	supported	present
rate limiting	feature	supported	present

○ chat prefill: none — a trailing assistant message starts a new turn — detected, not scored

○ sampling extensions: top_k honored; min_p, repetition_penalty accepted (acceptance only) — detected, not scored

Engine conformance engine

100%

MUST assertions on implemented surfaces only. Click a surface tile to filter the table. Default view: failures only.

Unsupported and inconclusive are not failures — they do not enter the conformance denominator. Use the filters below to inspect every check.

Test	Surface	Assertion	Evidence	Status

Model capability model

94.4% · strong

Floor check — not an intelligence rank. Category floor is 50%. Click a category to expand its evals (failures first).

Click a category row to expand evals

Eval	Samples	Status	Failure detail
distinguishes the time tool from the weather tool	3/3	pass	—
picks the weather tool from a menu of decoys	3/3	pass	—

Eval	Samples	Status	Failure detail
answers from knowledge instead of calling an irrelevant tool	3/3	pass	—
does not reach for a tool during small talk	3/3	pass	—

Eval	Samples	Status	Failure detail
fills typed arguments with the right types	3/3	pass	—
respects an enum constraint in the tool schema	3/3	pass	—

Eval	Samples	Status	Failure detail
keeps honoring the system prompt after several turns	1/1	pass	—
recalls a fact from an earlier turn	1/1	pass	—
uses a tool result instead of inventing an answer	1/1	pass	—

Eval	Samples	Status	Failure detail
honors a negative constraint	1/1	pass	—
obeys a one-word length constraint	1/1	pass	—
reproduces an exact requested string	1/1	pass	—

Eval	Samples	Status	Failure detail
produces a nested JSON structure on request	1/3	fail	user.pets was [], expected two strings; user.pets was [], expected two strings
emits raw JSON without markdown fences when told to	3/3	pass	—

Eval	Samples	Status	Failure detail
counts letters in a word	1/1	pass	—
follows a transitive chain	1/1	pass	—
multiplies two two-digit numbers	1/1	pass	—

Eval	Samples	Status	Failure detail
attributes a well-known novel	1/1	pass	—
knows a chemical symbol	1/1	pass	—
knows a non-obvious capital city	1/1	pass	—

Agentic model

3/3 tasks

Multi-step tool use in a simulated workspace — harder than the capability floor, never blended into it.

✓

reads the config instead of answering from priors

2 steps

✓

finds where the port really lives and edits only that

5 steps

✓

follows the pointer in build.cfg instead of guessing

4 steps

Engine fidelity engine

97.78%

Same-model comparisons only. Click a slice to see what was measured. Unmeasured slices are named — never zeroed.

Check	Result	Detail
Measured	pass	included in fidelity headline
Score	94.44%	34/36 items answered correctly
Weight	40%	blend weight among measured slices
Battery items	36	34/36 items answered correctly

Check	Result	Detail
Measured	pass	included in fidelity headline
Score	100%	mean top-1 prob 0.951 · margin 0.921 · 12 harder items
Weight	25%	blend weight among measured slices
Requires	logprobs from the engine — absent → slice dropped, not scored zero

Check	Result	Detail
Measured	pass	included in fidelity headline
Score	100%	3/3 greedy prompts reproduced byte-for-byte
Weight	25%	blend weight among measured slices
First divergence	pass	no temperature-0 divergence recorded

Check	Result	Detail
Measured	pass	included in fidelity headline
Score	100%	36/36 items: emitted token = argmax, valid distribution
Weight	10%	blend weight among measured slices
Requires	logprobs from the engine — absent → slice dropped, not scored zero

Performance engine

69.2 tok/s

Informational — never scored and never part of the exit code. Hardware-dependent: only comparable against runs on the same machine.

machine: Apple M3 Ultra · 512 GB · darwin arm64

Decode throughput

69.2

tok/s · median of 3

Time to first token

175

ms · median of 3

Prefill throughput

1328.1

tok/s · median of 3

Measurement	Result	How it was taken
Decode throughput	69.2 tok/s 68.9–69.2 over 3	steady-state generation while writing code
Time to first token	175 ms 175–177 over 3	latency before the first generated token
Prefill throughput	1328.1 tok/s 1327.3–1330.7 over 3	measured on a 2044-token prompt

Signal	Verdict	Detail
Prefix cache	fail not detected	1× — 1.3s cold → 1.2s warm; usage reports 0 of 1543 prompt tokens cached
Concurrency (4 streams)	partial serialized	0.24 efficiency · 18.9 tok/s aggregate vs 19.4 alone · slowest first token 15.7s
Speculative decode	pass 1.99× effective	predictable 39.5 tok/s vs novel 19.9 tok/s · 3.53 tokens per decode step
Sustained load	pass steady	69.2 → 69 tok/s over 176s (-0.3%)

Context scaling — decode, first-token latency and speculation as the prompt grows. A failed rung ends the ladder; larger sizes were not attempted.

Prompt	Decode	First token	Prefill	Speculation
~513	36.5 tok/s	570ms	900 tok/s	3.43 tok/step · 1.14× ceiling
~4.2k	32 tok/s	3.3s	1289 tok/s	3.43 tok/step · 1.14× ceiling
~8.2k	25.7 tok/s	6.6s	1258 tok/s	3.37 tok/step · 1.28× ceiling
~16.3k	28.1 tok/s	14.5s	1127 tok/s	3.69 tok/step · 1.07× ceiling