Headroom

The loop every user runs

Each simulated person is a closed loop: ask, wait, read the answer, think, ask again. A request crosses the network, waits in the server's queue, has its prompt read (prefill), then generates its answer one token at a time (decode), crosses the network back, and is drawn on the device. Because users wait for answers before asking again, a slow box slows its own crowd down, which is how real households and offices behave. So the useful question is how many people a box serves before the answers get bad.

One rule for speed

The server works in iterations. Every iteration reads the model weights it needs once, reads the conversation memory (KV cache) of every sequence in the batch, and does two operations per active parameter per token, plus attention. Its length is the larger of two times, plus a fixed overhead:

iteration time = max( bytes read / memory bandwidth , FLOPs / tensor throughput ) + overhead

• One user generates at roughly bandwidth divided by model size. That is why a 70B model at 4-bit crawls at about 5 tokens a second on a 273 GB/s box however many TFLOPS it has.
• More users share the same weight read, so total throughput climbs with batch size until the iteration turns compute-bound or the KV reads take over. Each user still slows down a little.
• Long prompts are compute-bound. Reading a 30,000-token coding context is where tensor throughput, and prefix caching, earn their keep.
• Mixture-of-experts models read only the experts a token routes to. With one user that is a small slice; with many users routing differently, more experts wake up every iteration and the advantage shrinks. Measured on a DGX Spark, a dense 7B model gains about 20 times total throughput going from 1 to 32 users; gpt-oss-120b gains about 6 times. Routing is skewed toward popular experts, so Headroom treats n users as n^0.65 independent routers, fitted to those measurements.
• Mixture-of-experts prompts split into many small matrix multiplies, one per expert, which run far below peak. That is why a Strix Halo box reads a gpt-oss-120b prompt at about a quarter of a DGX Spark's speed while generating almost as fast.
• Many small experts cost time to launch. llama.cpp and MLX start one kernel per expert per layer, and each start has a fixed cost that no bandwidth figure shows. A model that wakes ten of 512 experts in 48 layers, like Qwen3-Next, generates at about half the speed its size suggests; vLLM-style fused kernels mostly avoid it.

What comes before the first word

The first generated token is not always the first thing a person sees or hears. Headroom counts three things that come before it:

• Hidden reasoning. gpt-oss always thinks before it answers, about 60 tokens even at its lowest effort, and DeepSeek R1 far more. A chat app or an IDE shows that thinking as it streams, so the reader sees progress; a speaker or an e-ink page cannot, and waits for all of it.
• The speech buffer. A speech engine needs about 60 characters, roughly 16 tokens, before it starts talking.
• Tool calls. A voice command to a smart home is usually a tool call the model writes before it replies.

That is why the voice missions reward a model that answers straight away over a bigger one that reasons first.

Memory decides concurrency

After the weights, the rest of usable memory holds KV cache: every token of every live conversation costs a fixed number of bytes that depends on the model's layers and attention heads. Two server designs spend it differently.

• Slots (llama.cpp, Ollama, MLX): a fixed number of parallel slots, each with its own preallocated context. Extra users queue for a slot. A request longer than a slot's context fails, which is the classic "Ollama cannot hold my coding agent's system prompt" surprise.
• Paged (vLLM, SGLang, TensorRT-LLM): one shared pool handed out in blocks. Concurrency is limited by memory, not by a slot count, and a shared system prompt is stored once.

Two boxes can run one model in two ways. vLLM, SGLang, TensorRT-LLM and MLX split every layer across both (tensor parallel), so both read weights at once and generation gets faster, less two exchanges per layer over the link. llama.cpp over RPC gives each box whole layers: a token passes through one box and then the other, so the pair holds a model twice the size but generates no faster than one box would.

Prompt caches work by content: groups that send the same system prompt share one cached copy, and six different prompts (six languages, say) compete for room. Recent llama.cpp and MLX servers also park evicted prompts in host memory and restore them. Ollama is different in one way that matters: a prompt longer than its window is cut to fit and answered anyway, so the answer arrives on time and without its start.

The network is mostly about packets, not bandwidth

Text is tiny. What hurts is round trips (distance, handshakes, radio scheduling) and, on shared radios, per-packet airtime: a Wi-Fi or Bluetooth frame costs the same preamble and acknowledgement whether it carries one token or a kilobyte. Streaming one OpenAI-style event per token to 50 clients multiplies that cost. On LoRa it is fatal: a single 200-byte packet takes more than a second of airtime, and European rules cap transmit time at 10%. Headroom models each shared radio as one queue that every client's traffic waits in. On internet paths a lost packet also stalls the connection until TCP resends it, about a round trip plus 200 ms, which is why a lossy satellite link feels worse than its latency. Wi-Fi and Bluetooth resend at the radio and only pay the airtime.

Verdicts and the redline

Each persona has its own idea of a good answer: a chat user wants text within 2 seconds at reading speed, a Kindle reader wants the whole answer within 20 seconds, a voice assistant wants its first word within 1.2 seconds. A run is right-sized when at least 95% of answers in every group meet their target and even the slowest 5% arrive well inside it. It is tight when those slowest answers already use 80% of the time allowed, or when every slot is taken on a box more than 85% busy: a few more users will tip it over. Busy time alone does not decide it, because a batching server is busy whenever anyone is generating and a batch worker keeps any box busy on purpose. Under 25% busy with answers well inside their targets is overkill: it works, but the money is idle. The redline search doubles the crowd until answers fail, then bisects to the largest crowd that still passes.

Where the numbers come from

Model shapes come from each model's published configuration. Hardware figures are vendor specifications, or estimates backed out of measured speed where the vendor publishes none. Runtime efficiencies are calibrated against published single-user benchmarks, shown below next to what the model predicts.

What it leaves out

• Speculative decoding, which can lift single-user speed 1.5 to 2.5 times on some models.
• Thermal throttling, power-limit modes and background load on the box.
• Images, audio and video inputs; tool execution time beyond each persona's think time.
• Answer quality, beyond a coarse capability tier per model and quantization.
• Software drift: the same box and model moved 50% or more within months of 2025 and 2026 as llama.cpp, vLLM and drivers improved. Every calibration point below carries its build or date.

Against the published measurements below, the model's typical error is about 11%, with a few older or unusual points off by up to 2.5 times. Treat results as estimates. They are most reliable on what runs out first.