The 4x Local Inference Claim Needs a Harness
A local inference engine can look four times faster without making an agent four times more useful.
That is the tension in a benchmark circulating on X this week. The account @akshay_pachaar claims that an engine called Uzu generated Qwen3.5-9B at 92.1 tokens per second on a base M5 MacBook Pro. The same post reports 22.0 tokens per second with llama.cpp and 25.1 with MLX.
Those are interesting numbers. They are not yet an operational result.
The post says the comparison used one engine at a time on the same machine. It does not, by itself, establish the exact commit, model file, quantization, prompt, sampling settings, thermal state, or whether all three runs produced equivalent output. Treat the comparison as an alleged benchmark, not a reproduced fact.
The more important question is what happens after the first token.
A token benchmark is not an agent benchmark
A chat completion mostly measures decoding throughput. An agent measures a control loop.
The loop reads context, chooses a tool, waits for the tool result, updates its state, and decides again. A coding agent may generate a shell command, inspect a compiler error, edit a file, run tests, and ask for another decision. A local security agent adds isolation, evidence collection, and approval boundaries to the same cycle.
The model is only one stage in that pipeline.
A faster decoder can reduce the time spent generating a response. It does not automatically reduce tool latency, context processing, file inspection, verification, or retries. If the faster runtime produces weaker tool choices, the agent can spend the saved milliseconds taking an extra turn.
The useful unit is therefore not tokens per second. It is completed tasks per minute, with the verifier still satisfied.
What the claimed speedup may be measuring
The reported numbers are compatible with a familiar optimization path: speculative decoding.
In speculative decoding, a smaller draft model proposes several tokens. The larger model checks those proposals in batches and accepts the prefix it considers valid. When the draft is well aligned with the target model, the target can produce more than one committed token per expensive pass.
That mechanism can be especially valuable on memory-bound hardware. Ordinary decoding repeatedly streams the target model’s weights through memory for one next-token decision. Verification of several proposed tokens can improve the amount of useful output obtained from each pass.
But the gain depends on acceptance rate. It varies with the model, prompt, context length, draft model, temperature, and hardware kernels. A short, predictable prompt can make a runtime look exceptional. A long tool trace with unusual identifiers can produce a different result.
The post attributes the result to Uzu’s speculative-decoding system and describes a tree-based verifier that can check several candidate sequences. Those details come from the account’s description, not from an independently reproduced run.
The mechanism is plausible. The headline still needs a harness.
The missing denominator is successful work
An agent operator should record at least this comparison:
| Measure | Why it matters |
|---|---|
| Output tokens per second | Measures decoder throughput |
| Prompt processing time | Captures context ingestion cost |
| Tool calls per task | Shows how much work the agent creates |
| Retries per task | Exposes weak decisions hidden by raw speed |
| Successful tasks per minute | Measures useful throughput |
| Verifier failures | Shows whether speed is buying unreliable output |
The last three measures change the conclusion more often than the first one.
If Uzu generates 92 tokens per second but causes a coding agent to call the shell twice as often, the runtime has not delivered a four times faster workflow. If speculative decoding works on prose but loses acceptance on structured tool calls, a chat benchmark overstates the gain where operators need it most.
This is why an agent benchmark must freeze the harness as carefully as it freezes the model. Use the same task set, tools, context, approval rules, and verifier. Measure wall-clock completion, not only generation time.
Speed changes the deployment boundary
The claim is still worth watching even before it is independently confirmed.
A reliable fourfold decoding improvement on commodity Apple hardware would change which agents can run locally. More local throughput means shorter interactive waits, higher task concurrency, and less pressure to send source code or operational data to a cloud endpoint.
It could also change the security boundary. Local inference keeps model inputs on the device, but it does not make the agent safe. The runtime still needs a restricted filesystem, explicit tool permissions, secret isolation, and an approval gate for irreversible actions.
A faster unsafe loop is still an unsafe loop. It simply reaches its mistakes sooner.
There is a second infrastructure cost. Speculative systems introduce more moving parts: a draft model, a verifier path, model-specific kernels, quantized checkpoints, and acceptance logic. Each can affect reproducibility. Operators need to pin the runtime and artifact together, record the exact model checksum, and retain a baseline implementation for comparison.
What to reproduce before trusting the number
The minimum useful reproduction is not a screenshot of a token counter.
Run the same model file and prompt through llama.cpp, MLX, and Uzu on the same machine. Record the runtime version, quantization, context length, warm-up policy, temperature, output length, and power or thermal mode. Repeat the run after a cold start and after a sustained workload.
Then move the comparison into an agent harness. Use a fixed repository task, a fixed tool budget, and a verifier that rejects invalid edits. Report successful tasks per minute, tool calls, retries, prompt processing time, and output quality alongside tokens per second.
If the advantage survives those controls, it becomes an infrastructure result rather than a viral benchmark.
The runtime is part of the model
The industry still talks about local models as if the checkpoint determines the experience. It does not.
The usable system is the checkpoint, quantization, runtime, kernels, draft model, scheduler, context window, tool bridge, and verifier acting together. A runtime that turns one model pass into several accepted tokens can be as important to deployment economics as a new training run.
But the operator’s unit of truth is the completed, verified task. Until the Uzu claim is reproduced inside that loop, 92.1 tokens per second is a promising measurement, not a production advantage.
The real frontier is not faster decoding. It is faster verified work.