跳到正文

AWS 在 SageMaker HyperPod 推理中增加了对 NVIDIA Dynamo 的支持

AWS Machine Learning Blog当事人· Vinay Arora·· 2 天前精选Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
打开原文
原文
图片来自 AWS Machine Learning Blog
谁
AWS
引文
Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo , combining Dynamo’s distributed inference capabilities with the infrastructure management of HyperPod to address these challenges.

发生了什么

Amazon SageMaker HyperPod 推理现已支持 NVIDIA Dynamo,这是一个用于多节点和解耦服务的开源分布式推理框架。AWS 展示了使用 DynamoGraphDeployment 或 InferenceEndpointConfig 的部署路径,并对 Qwen3-235B-A22B-FP8 进行了基准测试,结果显示在既定的延迟和生成速度目标下,解耦服务的每 GPU 输出速度达到每秒 207 个 token,而聚合服务为 101 个。另一项 Qwen3.5-397B 测试在两个实例上保留了单节点 99.1% 的吞吐量。

推荐理由

对于 Qwen3-235B-A22B-FP8,在既定的延迟和生成速度目标下,解耦服务的每 GPU 输出速度达到每秒 207 个 token,而聚合服务为 101 个,这为团队选择部署拓扑提供了衡量依据。

这件事到目前为止

AWS has integrated NVIDIA Dynamo, an open-source distributed inference framework, into Amazon SageMaker HyperPod Inference to enable multi-node and disaggregated serving. Developers can now utilize this managed support through either DynamoGraphDeployment or InferenceEndpointConfig to scale large language model inference across multiple instances. In internal benchmarking using Qwen3-235B-A22B-FP8, AWS reported that disaggregated serving achieved 207 output tokens per second per GPU, significantly outpacing the 101 tokens per second per GPU recorded for aggregated serving under comparable latency and generation-speed targets. Additionally, tests with the Qwen3.5-397B model demonstrated the system's efficiency, retaining 99.1 percent of single-node throughput when distributed across two instances.

接下来读什么

查看这件事的完整进展AWS Adds Managed NVIDIA Dynamo Support to SageMaker HyperPod Inference
这条收录于 11月19日 AI 日报看这一期日报的全部内容