Tarek Ziade has introduced an expert-streaming implementation for Transformers, proposed in pull request #50717, to facilitate Mixture-of-Experts (MoE) model inference on memory-constrained hardware. When testing the implementation by running Qwen3.8-Flash-Next on a 32 GB Mac, Ziade achieved 13.9 tokens per second on short decode, which he notes is about 6% behind Strata's performance.
The technique streams only the routed experts from disk into a bounded cache, utilizing targeted Metal kernels to manage the process. Although this method closes the performance gap, Ziade states the current configuration is still too slow for his personal coding-agent workflow.
AI 根据报道生成 · 1 天前更新