llama.cpp Adds Multi-Token Prediction for Qwen3.6: A Massive Speed Boost for Local AI
llama.cpp has merged Multi-Token Prediction (MTP) support for the Qwen3.6 model family, with Georgi Gerganov calling it a 'significant milestone for the local AI ecosystem.' The change enables up to 2.5x faster inference on commodity hardware, with no additional model required — just three extra flags at run time.
What Is Multi-Token Prediction?
Standard LLM inference generates one token at a time, sequentially. Multi-Token Prediction (MTP) changes this by training and running additional prediction heads that speculate on multiple future tokens simultaneously. During inference, the model generates candidate tokens in parallel and accepts the ones that match — dramatically increasing effective throughput without changing the model's output quality.
Qwen3.6 was released with built-in MTP heads, but until now, llama.cpp didn't support them. That's changed with PR #22673, merged by Aman Gupta.
The Performance Impact
The numbers are striking. Depending on hardware and quantization level:
- Up to 2.5x faster inference on Qwen3.6 27B in llama.cpp
- 20%+ speedup on RTX 3090 Ti with standard GGUF models
- Works across both Qwen3.6 27B and Qwen3.6 35BA3B (the MoE variant)
- No second model required — MTP heads are built into the base GGUF weights
Georgi Gerganov, creator of llama.cpp, described the change as a "significant milestone for the local AI ecosystem," adding that "the performance jump with these changes is massive and elevates local inference on commodity hardware further."
How to Use It
Enabling MTP in llama.cpp is straightforward. Update to the latest build (post-PR #22673) and add the MTP flags to your existing run command. Community guides confirm it works on:
- RTX 3090 / 3090 Ti (24 GB VRAM)
- RTX PRO 6000
- Windows 10/11 and Linux
- Standard GGUF model files from Hugging Face
The DataCamp tutorial has a detailed step-by-step walkthrough for running Qwen3.6 27B with MTP on consumer hardware.
Why This Matters for Local AI
Local inference has always been constrained by the sequential nature of token generation. MTP breaks that bottleneck at the hardware level without requiring cloud compute, larger models, or specialized inference servers. For developers running models on consumer GPUs, this update effectively doubles or more the usable throughput of one of the most capable open models available.
Get the latest llama.cpp build and start MTP-accelerated local inference today at github.com/ggml-org/llama.cpp.
Enjoyed this?
Get more posts like this delivered to your inbox.
