Skip to content

How Expert Streaming Brings Transformers MoE Inference Near Strata on a 32 GB Mac

Hugging Face Blog·· 2 days agoPicks180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
Original
Image: Hugging Face Blog
Who
Tarek Ziade
What
introduces a Transformers implementation for streaming routed MoE experts from disk
Quote
The interesting part is how it got that close: streaming only the routed experts, a bounded cache that grouped_mm runs on as is, and a few targeted Metal kernels.

What happened

The author built an expert-streaming path for Transformers and ran Qwen3.8-Flash-Next on a 32 GB Mac, reaching 13.9 tokens per second on short decode, about 6% behind Strata. The approach streams only routed experts from disk through a bounded cache and uses targeted Metal kernels; the generic implementation is proposed in Transformers PR #50717, and the author says the setup remains too slow for their coding-agent workflow.

Why it matters

On a 32 GB Mac, routed-expert streaming brings Transformers to within about 6% of Strata on short decode, while the implementation remains an experimental PR and still runs too slowly for the author’s coding-agent workflow.

The story so far

Tarek Ziade has introduced an expert-streaming implementation for Transformers, proposed in pull request #50717, to facilitate Mixture-of-Experts (MoE) model inference on memory-constrained hardware. When testing the implementation by running Qwen3.8-Flash-Next on a 32 GB Mac, Ziade achieved 13.9 tokens per second on short decode, which he notes is about 6% behind Strata's performance. The technique streams only the routed experts from disk into a bounded cache, utilizing targeted Metal kernels to manage the process. Although this method closes the performance gap, Ziade states the current configuration is still too slow for his personal coding-agent workflow.

Read next

See the full storyTarek Ziade Implements Expert-Streaming for Transformers to Improve MoE Inference on Memory-Constrained Hardware
This story is in 11/19 AI DailyRead the whole issue