Skip to content

AWS Adds NVIDIA Dynamo Support to SageMaker HyperPod Inference

AWS Machine Learning BlogFrom the source· Vinay Arora·· 2 days agoPicksManaged NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
Original
Image: AWS Machine Learning Blog
Who
AWS
What
adds managed support for NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
Quote
Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo , combining Dynamo’s distributed inference capabilities with the infrastructure management of HyperPod to address these challenges.

What happened

Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, an open source distributed inference framework for multi-node and disaggregated serving. AWS demonstrates deployment paths using DynamoGraphDeployment or InferenceEndpointConfig, and benchmarks Qwen3-235B-A22B-FP8, where disaggregated serving reached 207 output tokens per second per GPU versus 101 for aggregated serving under the stated latency and generation-speed targets. A separate Qwen3.5-397B test retained 99.1 percent of single-node throughput across two instances.

Why it matters

For Qwen3-235B-A22B-FP8, disaggregated serving reached 207 output tokens per second per GPU versus 101 for aggregated serving under the stated latency and generation-speed targets, giving teams a measured basis for choosing a deployment topology.

The story so far

AWS has integrated NVIDIA Dynamo, an open-source distributed inference framework, into Amazon SageMaker HyperPod Inference to enable multi-node and disaggregated serving. Developers can now utilize this managed support through either DynamoGraphDeployment or InferenceEndpointConfig to scale large language model inference across multiple instances. In internal benchmarking using Qwen3-235B-A22B-FP8, AWS reported that disaggregated serving achieved 207 output tokens per second per GPU, significantly outpacing the 101 tokens per second per GPU recorded for aggregated serving under comparable latency and generation-speed targets. Additionally, tests with the Qwen3.5-397B model demonstrated the system's efficiency, retaining 99.1 percent of single-node throughput when distributed across two instances.

Read next

See the full storyAWS Adds Managed NVIDIA Dynamo Support to SageMaker HyperPod Inference
This story is in 11/19 AI DailyRead the whole issue