
- 谁
- Tarek Ziade
- 引文
The interesting part is how it got that close: streaming only the routed experts, a bounded cache that grouped_mm runs on as is, and a few targeted Metal kernels.
发生了什么
作者为 Transformers 构建了一个专家流式传输路径,并在 32 GB Mac 上运行了 Qwen3.8-Flash-Next,在短解码任务中达到了每秒 13.9 个 token,比 Strata 慢约 6%。该方法仅通过有限缓存从磁盘流式传输路由专家,并使用针对性的 Metal 内核;其通用实现已在 Transformers PR #50717 中提出,作者表示该设置对于他们的编程智能体工作流来说仍然太慢。
推荐理由
在 32 GB Mac 上,路由专家流式传输使 Transformers 在短解码任务中的表现与 Strata 的差距缩小至 6% 以内,但该实现仍处于实验性 PR 阶段,且运行速度对于作者的编程智能体工作流来说依然过慢。
这件事到目前为止
Tarek Ziade has introduced an expert-streaming implementation for Transformers, proposed in pull request #50717, to facilitate Mixture-of-Experts (MoE) model inference on memory-constrained hardware. When testing the implementation by running Qwen3.8-Flash-Next on a 32 GB Mac, Ziade achieved 13.9 tokens per second on short decode, which he notes is about 6% behind Strata's performance. The technique streams only the routed experts from disk into a bounded cache, utilizing targeted Metal kernels to manage the process. Although this method closes the performance gap, Ziade states the current configuration is still too slow for his personal coding-agent workflow.
