The 35B Model That Runs Like a 3B Model
A 35 billion parameter model is not supposed to feel like a phone-sized model.
MiniCPM-V 4.7 is built around exactly that contradiction. OpenBMB released a 35B-A3B mixture-of-experts model that it says can decode at more than 13 tokens per second on an iPhone 17 Pro Max using about 3 GB of memory. The model has roughly 35 billion total parameters, but only about 3 billion active parameters per token.
That headline is easy to misread.
The important change is not simply that the model activates fewer weights. The important change is that the inference system treats expert movement, storage, prediction, quantization, and compute as one problem.
The active parameter number is not the deployment cost
A 35B-A3B model does not need to multiply every parameter for every token. Its router selects a smaller set of experts at each layer. That is where the A3B label comes from.
But inactive does not mean free.
The device still needs access to the expert weights that the router may select next. If those weights live on slower storage, every expert swap can become a visible pause. If the runtime limits the expert pool to avoid those pauses, it may produce faster but weaker inference by preventing the model from using the experts it actually chose.
This is the part that benchmark tables usually hide. Sparse computation reduces arithmetic. It does not automatically solve memory capacity or bandwidth.
LES turns routing into a scheduling problem
OpenBMB’s answer is Lookahead Expert Scheduling, or LES. The project describes it as a system that predicts which experts are likely to be needed by a later layer and preloads them before the router requests them.
The prediction is not allowed to change the model’s routing decision. It is a prefetch hint.
That distinction matters. A bad prediction should cost time, not silently alter the model’s computation. The actual router still selects the experts, while the runtime prioritizes experts that are already loaded and fetches missed experts when required.
The architecture looks roughly like this:
next-layer routing prediction
|
v
prefetch likely experts
|
+-------------------+
| |
compute loaded experts load misses
| |
+---------+---------+
|
actual token routingThe model is still a mixture of experts. The runtime is now a prediction and scheduling loop wrapped around it.
That is the systems-level story. The model architecture creates a sparse workload. The deployment layer makes the sparse workload usable on hardware that cannot keep every expert resident in fast memory.
Why this matters for agents
Most agent infrastructure discussions still assume that the best local model is the one with the largest practical quality score inside a fixed memory budget.
That is too narrow.
An agent does not only need a model that can answer. It needs a model that can stay responsive while inspecting screenshots, reading documents, watching tool output, and deciding whether to act. Latency spikes become control-loop failures when the model is waiting on an expert load instead of advancing the task.
A model that can run locally with stable latency changes the trust boundary too. Images and documents can stay on the device. A tool call can be reviewed locally. A local verifier does not need to ship every observation to a remote endpoint just to obtain a fast enough multimodal response.
Those benefits are not automatic. They depend on the runtime behaving predictably under the real workload, not only on a single benchmark prompt.
The catch is in the claim
OpenBMB’s repository and model card describe 13-plus tokens per second on an iPhone 17 Pro Max, plus support for iOS, Android, and HarmonyOS. The same material says that the 35B-A3B model achieves a higher overall score in less inference time than Qwen3.6 35B-A3B in the team’s evaluation.
Those are useful engineering targets, not independent measurements.
The public material does not establish that every device, quantization, prompt shape, image resolution, or video workload will see the same result. It also does not turn a 35B model into a 3B model for storage, download time, or every memory path. The full expert pool still exists, and the runtime still has to move the right pieces at the right time.
The honest interpretation is narrower and more interesting: OpenBMB is showing that a large sparse model can be made operationally small enough for a particular device class when the inference engine is designed around expert movement.
The deployment stack is becoming part of the model
MiniCPM-V 4.7 ships with support for multiple runtimes and quantized formats, including llama.cpp, Ollama, vLLM, SGLang, GGUF, AWQ, GPTQ, and FP8 variants. That is not just packaging.
Each backend exposes different tradeoffs in memory, scheduling, kernels, and hardware support. A model release that includes the deployment path is increasingly more useful than a checkpoint that leaves the hard part to the operator.
This is also why active parameter counts should be treated as a routing metric, not a complete cost metric. The actual operator cares about at least four numbers:
| Question | Why it matters |
|---|---|
| How many parameters activate per token? | Estimates arithmetic work. |
| How many weights must be stored? | Sets the lower bound on local storage. |
| How fast can experts move? | Determines whether sparse inference stalls. |
| How stable is latency? | Determines whether an agent control loop feels usable. |
The third and fourth rows are where local agents usually fail. A model can fit on paper and still be too slow to operate.
What builders should measure next
If you are evaluating a sparse model for an agent, do not stop at tokens per second from a warm, text-only prompt.
Measure:
- Cold start time after the process has been idle.
- Expert-load stalls during long multimodal prompts.
- Latency variance across images, documents, and video.
- Memory pressure when the agent and its tools share the device.
- Whether prefetch misses change quality, or only latency.
- Recovery behavior when storage is slow or temporarily unavailable.
The verifier belongs in this loop. If the model is going to make local decisions about files, cameras, or tools, the system should record which model variant ran, which quantization was used, and whether the runtime was under memory pressure.
A fast local model without those traces is still a black box, just a smaller one.
The uncomfortable truth about small local models
The next generation of local AI will not be won by parameter counts alone.
It will be won by the teams that make routing, storage, scheduling, quantization, and observability behave like one product. MiniCPM-V 4.7 is a concrete example of that direction: 35B total parameters, roughly 3B active per token, and a runtime whose main job is to keep the other weights from becoming a latency wall.
The model is not small.
The deployment system is getting smarter about where the size matters.