Skip to content

Tarek Ziade Implements Expert-Streaming for Transformers to Improve MoE Inference on Memory-Constrained Hardware

1 reports 1 reporting sources first reported Updated 2 days ago

Understand the event

AI overview

Tarek Ziade has introduced an expert-streaming implementation for Transformers, proposed in pull request #50717, to facilitate Mixture-of-Experts (MoE) model inference on memory-constrained hardware. When testing the implementation by running Qwen3.8-Flash-Next on a 32 GB Mac, Ziade achieved 13.9 tokens per second on short decode, which he notes is about 6% behind Strata's performance. The technique streams only the routed experts from disk into a bounded cache, utilizing targeted Metal kernels to manage the process. Although this method closes the performance gap, Ziade states the current configuration is still too slow for his personal coding-agent workflow.

Generated by AI from reports · Updated 1 day ago

Read next

More changes in today’s briefingGoogle DeepMind Introduces Gemini 4 Neon for Coding, Analysis, and Science

Models in this story

Report timeline

Follow the reports to explore different sides of the event.

11/18
  1. How Expert Streaming Brings Transformers MoE Inference Near Strata on a 32 GB Mac

    The author built an expert-streaming path for Transformers and ran Qwen3.8-Flash-Next on a 32 GB Mac, reaching 13.9 tokens per second on short decode, about 6% behind Strata. The approach streams only routed experts from disk through a bounded cache and uses targeted Metal kernels; the generic implementation is proposed in Transformers PR #50717, and the author says the setup remains too slow for their coding-agent workflow.

Keep up with AI, one email a day

Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue

Event heat trend

Current within comparable coverage
3
Peak within comparable coverage
611/19 01:00 (UTC+8)
24-hour change
-49%

Trends compare the same continuously observed participants, which may cover fewer sources than current heat. Point or click to inspect hourly heat, or use left and right arrow keys.