AFM — local AI infrastructure for Apple Silicon

Agents need more than a generation endpoint. AFM exposes the information required to select a model, stay inside its context window, correlate work, cancel it, batch it, and observe pressure.

Model discovery

curl http://127.0.0.1:9999/v1/models

The response contains the conventional object: "list" and data array, plus an AFM models detail array with capability names. Depending on the server mode it can include:

the active MLX chat model, marked owned_by: "mlx" and loaded: true;
foundation for Apple Foundation Models;
models discovered from local gateway backends;
shipped Apple NaturalLanguage embedding model IDs, advertised even before lazy loading.

Token budgeting

POST /v1/tokenize

Accepts text or the vLLM alias prompt. Returns tokens, count, model, and max_model_len when known.

POST /v1/count_tokens

Accepts the same body and returns the smaller Anthropic-shaped input_tokens and model response.

curl http://127.0.0.1:9999/v1/tokenize \
  -H 'Content-Type: application/json' \
  -d '{"model":"active-model","text":"Plan a local deployment."}'

{
  "tokens": [2321, 264, 3819, 11347, 13],
  "count": 5,
  "model": "active-model",
  "max_model_len": 32768
}

If the requested model alias differs from the loaded MLX model, AFM logs the mismatch and uses the loaded tokenizer. If no MLX tokenizer is available, both routes return HTTP 422 with code: "tokenize_unsupported". Message-array chat-template tokenization is not implemented; budget the rendered text or leave a safety margin.

Direct multiplex batch

POST /v1/batch/completions accepts 1–64 uniquely identified requests and promotes the MLX runtime into batch mode as needed.

{
  "requests": [
    {"custom_id":"doc-a","body":{"messages":[{"role":"user","content":"Summarize A"}],"stream":true}},
    {"custom_id":"doc-b","body":{"messages":[{"role":"user","content":"Summarize B"}],"stream":false}}
  ]
}

The response is one SSE stream. Every JSON event is tagged with custom_id, so consumers demultiplex by that field rather than by arrival order. Each item may request streaming or non-streaming semantics. The transport emits one final [DONE] after every item has completed or failed.

OpenAI-style files and batch jobs

These routes are registered only with the concrete MLX batch service:

Method	Route	Contract
POST	`/v1/files`	Upload JSONL with `purpose=batch`; multipart and raw-body fallback are supported.
GET	`/v1/files/{file_id}`	Read file metadata.
GET	`/v1/files/{file_id}/content`	Download `application/jsonl` content, including completed results.
DELETE	`/v1/files/{file_id}`	Delete a stored file.
POST	`/v1/batches`	Create a job. Only `endpoint: "/v1/chat/completions"` is accepted.
GET	`/v1/batches`	List jobs.
GET	`/v1/batches/{batch_id}`	Read status, counts, and output file ID.
POST	`/v1/batches/{batch_id}/cancel`	Cancel a job only while it is `in_progress`.

Each JSONL input line needs custom_id, method, url, and a chat-completion body. Per-request failures are recorded in output JSONL without failing the entire batch.

Health and metrics

Route	Use	Important limitation
`GET /health`	Liveness probe with status, timestamp, and version.	Healthy means the process responds; it is not a model-quality or available-capacity guarantee.
`GET /metrics`	Prometheus text for active connections, scheduler pressure, tokens, throughput, and timings.	Metric families and labels are AFM/vLLM-inspired operational surfaces, not OpenAI API fields.
`GET /openapi.json`	Compact machine-readable contract bundled into the binary.	Hand-maintained; pair it with model discovery and response headers.
`GET /docs`	Scalar-rendered interactive API reference.	The UI loads Scalar from a CDN; the JSON contract itself is local.

A resilient agent loop

Check /health, then read /v1/models.
If using MLX, call /v1/tokenize before sending a prompt near the context limit.
Mint a stable application correlation ID and send it in X-Request-ID.
Stream long generations and parse usage separately from content.
On operator interruption, cancel with the correlation ID.
On 503 server_busy, respect Retry-After and retry with bounded exponential backoff plus jitter.
Use /metrics to tune concurrency rather than increasing client parallelism blindly.