
- Who
- Tarek Ziade
- What
- introduces a Transformers implementation for streaming routed MoE experts from disk
- Quote
The interesting part is how it got that close: streaming only the routed experts, a bounded cache that grouped_mm runs on as is, and a few targeted Metal kernels.
What happened
The author built an expert-streaming path for Transformers and ran Qwen3.8-Flash-Next on a 32 GB Mac, reaching 13.9 tokens per second on short decode, about 6% behind Strata. The approach streams only routed experts from disk through a bounded cache and uses targeted Metal kernels; the generic implementation is proposed in Transformers PR #50717, and the author says the setup remains too slow for their coding-agent workflow.
Why it matters
On a 32 GB Mac, routed-expert streaming brings Transformers to within about 6% of Strata on short decode, while the implementation remains an experimental PR and still runs too slowly for the author’s coding-agent workflow.
The story so far
Tarek Ziade has introduced an expert-streaming implementation for Transformers, proposed in pull request #50717, to facilitate Mixture-of-Experts (MoE) model inference on memory-constrained hardware. When testing the implementation by running Qwen3.8-Flash-Next on a 32 GB Mac, Ziade achieved 13.9 tokens per second on short decode, which he notes is about 6% behind Strata's performance. The technique streams only the routed experts from disk into a bounded cache, utilizing targeted Metal kernels to manage the process. Although this method closes the performance gap, Ziade states the current configuration is still too slow for his personal coding-agent workflow.
