
- Who
- AWS
- What
- adds managed support for NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
- Quote
Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo , combining Dynamo’s distributed inference capabilities with the infrastructure management of HyperPod to address these challenges.
What happened
Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, an open source distributed inference framework for multi-node and disaggregated serving. AWS demonstrates deployment paths using DynamoGraphDeployment or InferenceEndpointConfig, and benchmarks Qwen3-235B-A22B-FP8, where disaggregated serving reached 207 output tokens per second per GPU versus 101 for aggregated serving under the stated latency and generation-speed targets. A separate Qwen3.5-397B test retained 99.1 percent of single-node throughput across two instances.
Why it matters
For Qwen3-235B-A22B-FP8, disaggregated serving reached 207 output tokens per second per GPU versus 101 for aggregated serving under the stated latency and generation-speed targets, giving teams a measured basis for choosing a deployment topology.
The story so far
AWS has integrated NVIDIA Dynamo, an open-source distributed inference framework, into Amazon SageMaker HyperPod Inference to enable multi-node and disaggregated serving. Developers can now utilize this managed support through either DynamoGraphDeployment or InferenceEndpointConfig to scale large language model inference across multiple instances. In internal benchmarking using Qwen3-235B-A22B-FP8, AWS reported that disaggregated serving achieved 207 output tokens per second per GPU, significantly outpacing the 101 tokens per second per GPU recorded for aggregated serving under comparable latency and generation-speed targets. Additionally, tests with the Qwen3.5-397B model demonstrated the system's efficiency, retaining 99.1 percent of single-node throughput when distributed across two instances.
