跳到正文

如何通过专家流式传输让 Transformers MoE 推理在 32 GB Mac 上接近 Strata 的表现

Hugging Face Blog·· 2 天前精选180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
打开原文
原文
图片来自 Hugging Face Blog
谁
Tarek Ziade
引文
The interesting part is how it got that close: streaming only the routed experts, a bounded cache that grouped_mm runs on as is, and a few targeted Metal kernels.

发生了什么

作者为 Transformers 构建了一个专家流式传输路径,并在 32 GB Mac 上运行了 Qwen3.8-Flash-Next,在短解码任务中达到了每秒 13.9 个 token,比 Strata 慢约 6%。该方法仅通过有限缓存从磁盘流式传输路由专家,并使用针对性的 Metal 内核;其通用实现已在 Transformers PR #50717 中提出,作者表示该设置对于他们的编程智能体工作流来说仍然太慢。

推荐理由

在 32 GB Mac 上,路由专家流式传输使 Transformers 在短解码任务中的表现与 Strata 的差距缩小至 6% 以内,但该实现仍处于实验性 PR 阶段,且运行速度对于作者的编程智能体工作流来说依然过慢。

这件事到目前为止

Tarek Ziade has introduced an expert-streaming implementation for Transformers, proposed in pull request #50717, to facilitate Mixture-of-Experts (MoE) model inference on memory-constrained hardware. When testing the implementation by running Qwen3.8-Flash-Next on a 32 GB Mac, Ziade achieved 13.9 tokens per second on short decode, which he notes is about 6% behind Strata's performance. The technique streams only the routed experts from disk into a bounded cache, utilizing targeted Metal kernels to manage the process. Although this method closes the performance gap, Ziade states the current configuration is still too slow for his personal coding-agent workflow.

接下来读什么

查看这件事的完整进展Tarek Ziade Implements Expert-Streaming for Transformers to Improve MoE Inference on Memory-Constrained Hardware
这条收录于 11月19日 AI 日报看这一期日报的全部内容