# AFM / maclocal.ai

> AFM is a local OpenAI-compatible Swift server for Apple Silicon. maclocal.ai is its human and agent-readable companion documentation; it is not a hosted inference API.

## Canonical contract

- [What's new](https://maclocal.ai/whats-new): Visual AFM release highlights and links to the complete GitHub release history.
- [API overview](https://maclocal.ai/docs/api-reference): Compatibility boundaries and route matrix.
- [Chat completions](https://maclocal.ai/docs/api-reference/chat-completions): Request fields, precedence, response shape, and examples.
- [Streaming](https://maclocal.ai/docs/api-reference/streaming): SSE lifecycle, tool deltas, usage, cancellation, and profiling.
- [Operations](https://maclocal.ai/docs/api-reference/operations): Models, token budgeting, batches, health, and metrics.
- [Errors](https://maclocal.ai/docs/api-reference/errors): Error envelope, correlation IDs, status codes, and retry behavior.
- [OpenAPI 3.1](https://maclocal.ai/openapi.json): Machine-readable public snapshot of the local AFM contract.
- [Full agent context](https://maclocal.ai/llms-full.txt): Compact implementation notes and operating constraints.

## Capability guides

- [WebUI](https://maclocal.ai/docs/webui)
- [Tool calling](https://maclocal.ai/docs/tool-calling)
- [Parameter recipes](https://maclocal.ai/docs/configuration-recipes)
- [Environment variables](https://maclocal.ai/docs/environment-variables)
- [Model architectures](https://maclocal.ai/docs/model-architectures)
- [Apple-native APIs](https://maclocal.ai/docs/apple-native)
- [Performance](https://maclocal.ai/docs/performance)
- [DwarfStar + DeepSeek V4](https://maclocal.ai/docs/dwarfstar-deepseek)

## Source

- [maclocal-api](https://github.com/scouzi1966/maclocal-api): Authoritative runtime implementation.

## Agent operating notes

Default local base URL: http://127.0.0.1:9999

1. Discover first: call GET /v1/models before selecting a model. The list includes the active chat model and Apple embedding models.
2. Budget before long prompts: POST /v1/tokenize returns token IDs, count, and max_model_len when an MLX tokenizer is loaded. POST /v1/count_tokens returns input_tokens. Foundation has no public tokenizer and returns 422 tokenize_unsupported.
3. Correlate every request: send X-Request-ID or OpenAI-Request-ID. AFM echoes both headers; otherwise it mints req_<uuid12>.
4. Cancel with the correlation ID: POST /v1/chat/completions/{id}/cancel uses the request-header ID, not the chatcmpl response ID. A known in-flight request returns cancelled:true; unknown or completed IDs return 404 with cancelled:false.
5. Stream incrementally: SSE events contain data: <json>. Content and reasoning_content arrive as deltas; tool arguments may be split across events. A finish_reason event comes before the optional usage event, then AFM emits data: [DONE].
6. Handle pressure: a saturated MLX chat server can wait for a slot and then return 503 server_busy with Retry-After: 2. Retry with bounded exponential backoff and jitter.
7. Treat strict output as conditional: strict JSON Schema or strict tool definitions require the server's grammar-constraint engine. X-Grammar-Constraints: downgraded means best-effort generation was used.
8. Keep the security boundary explicit: ordinary local endpoints do not add authentication. The default base URL is loopback; binding AFM to a network interface requires the operator to supply the surrounding access controls.

## Chat field precedence

- max_tokens wins over max_completion_tokens.
- repetition_penalty wins over repeat_penalty.
- Request sampling fields override corresponding CLI defaults.
- Request response_format overrides the server's --guided-json default.
- reasoning_effort is normalized into chat_template_kwargs; an explicit enable_thinking=false remains authoritative in the model encoder.
- CLI --stop values and request stop arrays are merged and deduplicated.
- frequency_penalty is accepted but currently ignored.

## Route availability

- Always registered: chat completions, cancellation, models, embeddings, Vision OCR, speech, token budgeting, health, metrics, runtime OpenAPI, and runtime docs. Framework or backend availability can still produce domain-specific errors.
- MLX tokenizer required: /v1/tokenize and /v1/count_tokens.
- Concrete MLX batch service required: /v1/batch/completions, /v1/files, and /v1/batches.
- macOS 13+ required for Apple speech; advanced Vision modes have newer OS requirements documented in the Apple-native guide.

## Compatibility warning

AFM deliberately combines OpenAI-compatible surfaces with explicit AFM extensions. Do not assume every accepted field affects every backend. Use the human compatibility tables and response headers to distinguish implemented behavior from accepted compatibility fields.

## WebUI operating contract

- Start with `-w` or `--webui`; AFM serves the browser client and API from the same local process.
- The model picker reads `/v1/models`. `/models/load` and `/models/unload` are compatibility acknowledgements, not hot-swap commands for a single-model MLX or DwarfStar process.
- Image and PDF attachments require an actually multimodal backend. DwarfStar is text-only.
- The AFM metrics drawer polls `/metrics` once per second only while open.
- The WebUI and ordinary API routes add no authentication. Keep the default loopback binding or provide an authenticated network boundary.

## DwarfStar and DeepSeek V4 operating contract

- DeepSeek V4 is the architecture; DwarfStar is the fixed-schedule Metal executor; DSpark is an optional greedy speculative support model.
- `--mlx-runtime auto` selects DwarfStar only for a local, self-contained executor directory. It must declare `model_type=deepseek_v4`, AFM-native Q8_0 dense weights, packed DwarfStar MXFP4 experts, executor layout version 3 or newer, 32-byte-aligned tensor offsets, and a bundled `dwarfstar-template.gguf`.
- DwarfStar supports text chat, streaming, sampling, reasoning policy, stop sequences, cancellation, resident concurrent sessions, and conservative exact-prefix reuse.
- DwarfStar does not support media, tools, structured-output grammar, repetition/presence penalties, KV quantization, MTP, EAGLE3, logprobs, or MLX batch endpoints.
- DSpark requires `--dspark-support`, runs only at temperature <= 0, accepts 1-16 draft tokens and a 0-1 confidence threshold, and exposes acceptance counters in response metadata.

## Environment variables

Scope: variables consumed or populated by the shipped AFM process and its applied MLX/DwarfStar patches. CI, release-script, and test-harness-only variables are intentionally outside the runtime contract.

### Operator configuration

| Variable | Values and default | Effect |
|---|---|---|
| `MACAFM_MLX_MODEL_CACHE` | path; unset | Adds a read-only root for curated or side-loaded checkpoints. AFM checks flat org/model, models/org/model, and Hugging Face cache layouts below it. It does not redirect downloads in the current build. |
| `HUGGINGFACE_HUB_CACHE` | path; unset | Adds an explicit Hugging Face Hub cache root to model discovery. Checked before HF_HUB_CACHE. |
| `HF_HUB_CACHE` | path; unset | Adds the standard Hugging Face Hub cache root to model discovery. |
| `HF_HOME` | path; unset | Uses <path>/hub as a Hugging Face model-cache candidate. |
| `XDG_CACHE_HOME` | path; unset | Uses <path>/huggingface/hub as a Hugging Face model-cache candidate. |
| `MACAFM_MLX_METALLIB` | file path; unset | Overrides the default.metallib selected for MLX. The path must point to a readable Metal library compatible with the build. |
| `MACAFM_CACHE_PROFILE_PATH` | JSONL file path; unset | Writes prefix-cache timing records. An explicit --cache-profile-path wins; this name wins over the legacy AFM_CACHE_PROFILE_PATH alias. |
| `AFM_CACHE_PROFILE_PATH` | JSONL file path; unset | Legacy alias for MACAFM_CACHE_PROFILE_PATH, used only when neither the CLI option nor the preferred variable is present. |
| `AFM_DEBUG` | 1; off | Enables verbose AFM and patched MLX diagnostic logging, including request, scheduler, cache, and selected model-path details. |
| `AFM_CLEAR_GPU_CACHE` | 1; off | Clears MLX's GPU cache after each request. It can reduce retained-memory pressure, but usually costs throughput and should be used as a mitigation, not a default. |
| `AFM_DSPARK` | 1, true, on, or yes; off | Enables DSpark behavior through the environment fallback. Prefer the typed --dspark-support startup option when possible. |
| `AFM_DSPARK_DRAFT` | integer; runtime default | Overrides the DSpark draft-token limit used by speculative generation. Prefer --dspark-draft-tokens for validated user configuration. |
| `AFM_EAGLE3_BLOCK` | integer; 2 | Overrides the EAGLE3 speculative block size. Intended for controlled tuning of an already configured EAGLE3 server. |
| `MLX_MAX_OPS_PER_BUFFER` | positive integer; MLX default | Overrides MLX's maximum operations per Metal command buffer. If neither MLX scheduling variable is set, AFM supplies 200 for DeepSeek V4 on Ultra chips. |
| `MLX_MAX_MB_PER_BUFFER` | positive integer; MLX default | Overrides MLX's command-buffer memory accounting limit in megabytes. AFM supplies 100000 for DeepSeek V4 on Ultra chips only when both scheduling variables are absent. |

### Diagnostics and tracing

| Variable | Values and default | Effect |
|---|---|---|
| `AFM_GPU_TRACE_OUTPUT` | file path; /tmp/afm-metal.trace | Changes the output path for --gpu-trace. A path supplied by the active capture flow takes precedence. |
| `AFM_GPU_TRACE_TEMPLATE` | shader or unset; system trace | Selects the custom Xcode Metal Shader Profile template when set to shader and the template is installed; otherwise AFM uses Metal System Trace. |
| `AFM_GPU_TRACE_FINALIZE_GRACE_SECONDS` | integer >= 5; 120 | Controls how long AFM allows Instruments to finalize a GPU trace before cleanup. |
| `AFM_PREFIX_CACHE_TRACE_BOUNDARY` | 1, true, or yes; off | Logs prefix-cache boundary decisions for scheduler and serial generation diagnosis. |
| `AFM_PREFIX_CACHE_ALLOW_UNSAFE_EXACT_REPLAY` | integer suffix length; off | Enables an experimental exact-replay shortcut for the requested suffix length. It is explicitly unsafe and must not be used for production correctness. |
| `AFM_PERF` | 1; off | Prints per-token MLX generation timing, including model construction and token-conversion work. |
| `AFM_EAGLE3_PROFILE` | 1; off | Prints EAGLE3 verify, draft, and acceptance timing for speculative-generation analysis. |
| `AFM_DEBUG_PREFILL` | 1; off | Prints Gemma 4 batched-prefill and shared-KV diagnostics when a multi-token batch is present. |
| `AFM_DWARFSTAR_METAL_SOURCE_ROOT` | directory path; auto-discovered | Overrides the directory containing DwarfStar Metal source files. AFM normally derives the individual DS4_METAL_* paths from this root. |
| `AFM_DWARFSTAR_TRACE_PROMPT` | 1; off | Prints DwarfStar prompt roles and token details. Prompt content may be sensitive; do not enable it in shared logs. |

### DeepSeek V4 and benchmark controls

| Variable | Values and default | Effect |
|---|---|---|
| `DSV4_KV_MODE` | sliding, full, or tq; sliding | Selects the DeepSeek V4 KV-cache diagnostic mode. sliding is the production hybrid cache; full uses a simple full cache; tq requires the caller's TurboQuant KV mode. |
| `DSV4_POOL_QUANT` | 0/false/off/no disables; enabled | Controls quantization of DeepSeek V4 pooled long-context cache storage. |
| `AFM_MLX_KERNELS` | native or ds4; set by CLI | Stable runtime bridge populated from --mlx-kernels. When AFM is launched through its CLI, the typed option is authoritative. |
| `VMLX_DSV4_KERNELS` | native or ds4; native | Legacy fallback kernel selector used only when AFM_MLX_KERNELS is absent. |
| `VMLX_DSV4_NATIVE_MXFP4` | 0/false disables; enabled | Controls the native specialized DeepSeek V4 MXFP4 path when the kernel engine is native. |
| `VMLX_DSV4_NATIVE_MXFP8` | 0/false disables; enabled | Controls the native symmetric MXFP8 matrix-vector path. |
| `VMLX_DSV4_ACTIVATION_QAT` | 0/false disables; enabled | Controls the official E4M3 activation fake-quantization round trip before eligible MXFP matrix multiplies. |
| `VMLX_DSV4_STAGED_MOE` | 1/true enables; enabled | Controls the staged native MXFP4 mixture-of-experts path. |
| `VMLX_DSV4_STAGED_SELECTOR` | 1/true enables; disabled | Adds the staged route-selection path; it requires staged MoE. |
| `VMLX_DSV4_SHARED_Q8_STAGE` | 1 enables; disabled | Enables the staged shared symmetric-Q8 expert path when its geometry and selector prerequisites match. |
| `VMLX_DSV4_COOPERATIVE_DOWN` | 1/true enables; disabled | Enables the cooperative native MXFP4 down-projection experiment. |
| `VMLX_DSV4_HALF_MULTIPLY` | 0, all, gate_up, or down; 0 | Selects reduced-precision multiply experiments for the routed gate/up path, down path, or both. 1 and true mean all. |
| `VMLX_DSV4_MXFP4_ROWS_PER_SIMD` | 1, 2, or 4; 2 | Sets the native MXFP4 rows handled per SIMD group; invalid values fall back to 2. |
| `VMLX_DSV4_MXFP4_SIMD_GROUPS` | 1, 2, 4, or 8; 2 | Sets SIMD groups per native MXFP4 threadgroup; invalid values fall back to 2. |
| `VMLX_DSV4_ALIGNED_MXFP4` | 1/true enables; disabled | Enables the aligned MXFP4 Metal-kernel experiment. It cannot be combined with interleaved MXFP4. |
| `VMLX_DSV4_INTERLEAVED_MXFP4` | 1/true enables; disabled | Enables the interleaved MXFP4 Metal layout experiment. |
| `VMLX_DSV4_INTERLEAVED_LANES` | 1/true enables; disabled | Uses interleaved SIMD lanes; it has effect only with VMLX_DSV4_INTERLEAVED_MXFP4. |
| `VMLX_DSV4_THREADGROUP_LUT` | 1/true enables; disabled | Enables the threadgroup lookup-table variant in the fused DeepSeek V4 Metal kernel. |
| `VMLX_DSV4_FUSED_ROUTER` | 0/false/off disables; enabled | Controls the fused route-selection implementation. |
| `VMLX_DSV4_FUSED_HC4` | 0 disables; enabled | Controls the four-way fused HC projection path. |
| `VMLX_DSV4_FUSED_HC_NORM` | 1 enables; disabled | Enables the fused HC normalization experiment. |
| `VMLX_DSV4_FUSED_HC_Q8_TAIL` | non-off enables; disabled | Enables the fused HC, routed-MoE, shared-Q8, and expansion decode tail when the exact tensor contract matches. |
| `VMLX_DSV4_DSPARK_HEAD_GEMV` | 0/false/off disables; enabled | Controls the native DSpark head GEMV when the selected kernel engine is native. |
| `VMLX_DSV4_CACHE_WOA` | 0/false disables; enabled | Caches the evaluated dequantized grouped attention output-projection tensor. Disabling trades memory for repeated work. |
| `VMLX_DSV4_CACHE_LM_HEAD` | 0/false disables; enabled | Caches the evaluated float32 language-model head. Disabling reduces retained memory and increases repeated work. |
| `VMLX_DSV4_Q8_LM_HEAD` | non-off enables; disabled | Quantizes the language-model head to the DeepSeek V4 Q8 path. |
| `VMLX_DSV4_QUANTIZED_GROUPED_WOA` | 0/false/off disables; enabled | Controls the quantized grouped attention output projection. |
| `VMLX_DSV4_QUANTIZED_GROUPED_WOA_QAT` | non-off enables; disabled | Applies activation QAT to the quantized grouped output projection experiment. |
| `VMLX_DSV4_COMPILE_ATTN_PRE` | 0/false/off disables; enabled | Controls compiled decode for attention input projection. |
| `VMLX_DSV4_COMPILE_ATTN_POST` | 0/false/off disables; enabled | Controls compiled decode for attention output projection. |
| `VMLX_DSV4_COMPILE_ATTN_HC` | 0/false/off disables; enabled | Controls compiled decode for the attention HC path. |
| `VMLX_DSV4_COMPILE_FFN` | 0/false/off disables; enabled | Controls compiled decode for the layer feed-forward tail. |
| `VMLX_DSV4_COMPILE_MOE` | 1 enables; disabled | Enables compiled decode around MoE. This stricter opt-in is disabled by default. |
| `VMLX_ENABLE_UNSAFE_COMPILE` | 1/true enables; disabled | Enables whole decode compilation despite known model-switch corruption risk. Controlled benchmarks only; never a safe production default. |
| `MLXPRESS_ENABLE_UNSAFE_COMPILE` | 1/true enables; disabled | Legacy alias for VMLX_ENABLE_UNSAFE_COMPILE. |
| `VMLX_SHARED_GATE_UP_ACTIVATION` | 0/false disables; enabled | Controls reuse of the shared gate/up activation in SwitchGLU. |
| `VMLX_FUSED_GATE_UP_CACHE_LIMIT_BYTES` | integer bytes; unset | Sets the fused gate/up cache limit and takes precedence over the megabyte form. A negative value means unlimited. |
| `VMLX_FUSED_GATE_UP_CACHE_LIMIT_MB` | integer MiB; 512 | Limits duplicated fused gate/up weights. A negative value means unlimited. |
| `BENCH_NO_FUSED_GATE_UP` | 1 disables; off | Disables the fused gate/up weight cache for A/B benchmarks. |
| `BENCH_FUSED_GATE_UP_THRESHOLD` | integer route count; 32 | Changes the decode-shaped route-count threshold for using fused gate/up weights. |
| `VMLX_DSV4_STAGE_PROFILE` | 1; off | Prints fine-grained DeepSeek V4 and SwitchGLU stage timings. |
| `VMLX_DSV4_OUTER_PROFILE` | 1; off | Prints aggregate transformer-build/eval and language-head-build/eval timings. |
| `VMLX_DSV4_NUMERIC_TRACE` | 1; off | Prints token IDs plus sample, mean, RMS, and maximum values for selected tensors. It forces evaluation and is very expensive. |
| `VMLX_DSV4_LOAD_TRACE` | 1; off | Prints DeepSeek V4 model-loading trace events. |
| `MLX_PROFILE_COMMAND_BUFFERS` | 1; off | Enables patched MLX command-buffer and operation-count profiling, including a process-end summary. |

### Inherited or AFM-managed

| Variable | Values and default | Effect |
|---|---|---|
| `PWD` | shell working directory | Inherited from the shell and used to resolve relative model and configuration paths after Metal-library discovery may change the process working directory. |
| `MTL_CAPTURE_ENABLED` | 1 when --gpu-capture is active | Set internally before MLX initializes so Metal GPU capture can begin. Prefer --gpu-capture instead of setting it yourself. |

AFM derives these DwarfStar bridge variables from AFM_DWARFSTAR_METAL_SOURCE_ROOT; configure the root instead of individual files: `DS4_METAL_FLASH_ATTN_SOURCE`, `DS4_METAL_DENSE_SOURCE`, `DS4_METAL_MOE_SOURCE`, `DS4_METAL_DSV4_HC_SOURCE`, `DS4_METAL_UNARY_SOURCE`, `DS4_METAL_DSV4_KV_SOURCE`, `DS4_METAL_DSV4_ROPE_SOURCE`, `DS4_METAL_DSV4_MISC_SOURCE`, `DS4_METAL_ARGSORT_SOURCE`, `DS4_METAL_CPY_SOURCE`, `DS4_METAL_CONCAT_SOURCE`, `DS4_METAL_GET_ROWS_SOURCE`, `DS4_METAL_SUM_ROWS_SOURCE`, `DS4_METAL_SOFTMAX_SOURCE`, `DS4_METAL_REPEAT_SOURCE`, `DS4_METAL_GLU_SOURCE`, `DS4_METAL_NORM_SOURCE`, `DS4_METAL_BIN_SOURCE`, `DS4_METAL_SET_ROWS_SOURCE`.