llmprobe report card

DeepSeek-V4-Flash-MXFP4Experts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-mxfp4-0731.gguf

http://127.0.0.1:10307/v1

Overview

Three independent scores — never averaged

Surface coverage

100%

Core 9/9 core complete

core 100%

extended 73.3%

frontier 88.9%

How much of the standard API surface exists. Missing features are listed on purpose.

Engine conformance

97.3%

179/184 MUST 5 violations

Of the surfaces that exist, how correct are the MUST behaviors. Unsupported ≠ fail.

Model capability

100%

strong 8 categories

Practical floor for tools, JSON, instructions — graded below floor / capable / strong.

Performance

39.4

tok/s decode 560 ms first token

Informational — hardware-dependent and never scored. Same-machine comparisons only.

Agentic

3/3

Harder multi-step bar. Never blended into capability.

Engine fidelity

94.87%

Same-model only — holds the model constant so the number is the engine.

Outcomes honesty

Pass84

Fail5

Unsupported8

Inconclusive2

Skipped15

Unsupported and inconclusive are not zeros and not fails.

Surface coverage engine

Core 100%

Per tier, never averaged. Click Core / Extended / Frontier to expand every feature under that tier.

Click a tier row to expand · missing features sort first

Feature	Kind	Status	Detail
/v1/models	surface	supported	present
chat/completions	surface	supported	present
error codes	feature	supported	present
finish reasons	feature	supported	present
JSON mode	feature	supported	present
stop + max_tokens	feature	supported	present
streaming + SSE framing	feature	supported	present
tool calling	feature	supported	present
usage tokens	feature	supported	present

✗ vision✗ logprobs✗ reasoning items✗ reasoning history round-trip

Feature	Kind	Status	Detail
logprobs	feature	missing	accepted `logprobs` but returned none
reasoning history round-trip	feature	missing	prompt_tokens identical with and without history reasoning — the field is dropped before the chat template, or this model's template strips history reasoning by design
reasoning items	feature	missing	no separate reasoning channel, even with the standard opt-in
vision	feature	missing	rejects image content with HTTP 400
completions (legacy)	surface	supported	present
embeddings	surface	supported	present
max_tokens legacy alias	feature	supported	present
messages	surface	supported	present
messages/count_tokens	surface	supported	present
parallel tool calls	feature	supported	present
responses	surface	supported	present
seed determinism	feature	supported	present
streamed usage	feature	supported	present
structured outputs	feature	supported	present
top_p sampling	feature	supported	present

✗ MCP tools

not probed: prompt caching

Feature	Kind	Status	Detail
MCP tools	feature	missing	not probed — llmprobe does not yet host an MCP server to call back into
prompt caching	feature	not probed	not probed at this depth
audio/speech	surface	supported	present
audio/transcriptions	surface	supported	present
background responses	feature	supported	present
images/edits	surface	supported	present
images/generations	surface	supported	present
n>1 choices	feature	supported	present
previous_response_id chaining	feature	supported	present
rate limiting	feature	supported	present

○ chat prefill: none — a trailing assistant message starts a new turn — detected, not scored

○ sampling extensions: top_k honored; min_p, repetition_penalty accepted (acceptance only) — detected, not scored

Engine conformance engine

97.3%

MUST assertions on implemented surfaces only. Click a surface tile to filter the table. Default view: failures only.

Unsupported and inconclusive are not failures — they do not enter the conformance denominator. Use the filters below to inspect every check.

Test	Surface	Assertion	Evidence	Status

Model capability model

100% · strong

Floor check — not an intelligence rank. Category floor is 50%. Click a category to expand its evals (failures first).

Click a category row to expand evals

Eval	Samples	Status	Failure detail
distinguishes the time tool from the weather tool	3/3	pass	—
picks the weather tool from a menu of decoys	3/3	pass	—

Eval	Samples	Status	Failure detail
answers from knowledge instead of calling an irrelevant tool	3/3	pass	—
does not reach for a tool during small talk	3/3	pass	—

Eval	Samples	Status	Failure detail
fills typed arguments with the right types	3/3	pass	—
respects an enum constraint in the tool schema	3/3	pass	—

Eval	Samples	Status	Failure detail
keeps honoring the system prompt after several turns	1/1	pass	—
recalls a fact from an earlier turn	1/1	pass	—
uses a tool result instead of inventing an answer	1/1	pass	—

Eval	Samples	Status	Failure detail
honors a negative constraint	1/1	pass	—
obeys a one-word length constraint	1/1	pass	—
reproduces an exact requested string	1/1	pass	—

Eval	Samples	Status	Failure detail
emits raw JSON without markdown fences when told to	3/3	pass	—
produces a nested JSON structure on request	3/3	pass	—

Eval	Samples	Status	Failure detail
counts letters in a word	1/1	pass	—
follows a transitive chain	1/1	pass	—
multiplies two two-digit numbers	1/1	pass	—

Eval	Samples	Status	Failure detail
attributes a well-known novel	1/1	pass	—
knows a chemical symbol	1/1	pass	—
knows a non-obvious capital city	1/1	pass	—

Agentic model

3/3 tasks

Multi-step tool use in a simulated workspace — harder than the capability floor, never blended into it.

✓

reads the config instead of answering from priors

3 steps

✓

finds where the port really lives and edits only that

4 steps

✓

follows the pointer in build.cfg instead of guessing

5 steps

Engine fidelity engine

94.87%

Same-model comparisons only. Click a slice to see what was measured. Unmeasured slices are named — never zeroed.

Check	Result	Detail
Measured	pass	included in fidelity headline
Score	91.67%	33/36 items answered correctly
Weight	40%	blend weight among measured slices
Battery items	36	33/36 items answered correctly

Check	Result	Detail
Measured	unsupported	excluded from denominator (not zeroed)
Score	—	engine returned no logprobs
Weight	25%	blend weight among measured slices
Why unmeasured	engine returned no logprobs
Requires	logprobs from the engine — absent → slice dropped, not scored zero

Check	Result	Detail
Measured	pass	included in fidelity headline
Score	100%	3/3 greedy prompts reproduced byte-for-byte
Weight	25%	blend weight among measured slices
First divergence	pass	no temperature-0 divergence recorded

Check	Result	Detail
Measured	unsupported	excluded from denominator (not zeroed)
Score	—	engine returned no top-k logprobs
Weight	10%	blend weight among measured slices
Why unmeasured	engine returned no top-k logprobs
Requires	logprobs from the engine — absent → slice dropped, not scored zero

· Confidence, Logprob consistency not measured

Performance engine

39.4 tok/s

Informational — never scored and never part of the exit code. Hardware-dependent: only comparable against runs on the same machine.

machine: Apple M3 Ultra · 512 GB · darwin arm64

Decode throughput

39.4

tok/s · median of 3

Time to first token

560

ms · median of 3

Prefill throughput

593.6

tok/s · median of 3

Measurement	Result	How it was taken
Decode throughput	39.4 tok/s 39.2–39.9 over 3	steady-state generation while writing code
Time to first token	560 ms 557–595 over 3	latency before the first generated token
Prefill throughput	593.6 tok/s 593.6–594.4 over 3	measured on a 1791-token prompt

Signal	Verdict	Detail
Prefix cache	fail not detected	1× — 2.5s cold → 2.4s warm; usage reports 0 of 1412 prompt tokens cached
Concurrency (4 streams)	partial serialized	0.24 efficiency · 31.2 tok/s aggregate vs 32.5 alone · slowest first token 9.8s
Speculative decode	pass 1.25× effective	predictable 48.5 tok/s vs novel 38.7 tok/s · 4.83 tokens per decode step
Sustained load	pass steady	39.4 → 41.7 tok/s over 211s (+5.9%)

Context scaling — decode, first-token latency and speculation as the prompt grows. A failed rung ends the ladder; larger sizes were not attempted.

Prompt	Decode	First token	Prefill	Speculation
~510	40.4 tok/s	1.3s	396 tok/s	2.75 tok/step · 1.27× ceiling
~4.1k	38.1 tok/s	7.2s	573 tok/s	2.74 tok/step · 1.18× ceiling
~8.2k	36.7 tok/s	14.5s	566 tok/s	2.63 tok/step · 1.22× ceiling
~16.3k	35.4 tok/s	31s	526 tok/s	2.16 tok/step · 1.23× ceiling