Skip to content

AWS Adds Managed NVIDIA Dynamo Support to SageMaker HyperPod Inference

1 reports 1 reporting sources first reported Updated 2 days ago

Understand the event

AI overview

AWS has integrated NVIDIA Dynamo, an open-source distributed inference framework, into Amazon SageMaker HyperPod Inference to enable multi-node and disaggregated serving. Developers can now utilize this managed support through either DynamoGraphDeployment or InferenceEndpointConfig to scale large language model inference across multiple instances. In internal benchmarking using Qwen3-235B-A22B-FP8, AWS reported that disaggregated serving achieved 207 output tokens per second per GPU, significantly outpacing the 101 tokens per second per GPU recorded for aggregated serving under comparable latency and generation-speed targets. Additionally, tests with the Qwen3.5-397B model demonstrated the system's efficiency, retaining 99.1 percent of single-node throughput when distributed across two instances.

Generated by AI from reports · Updated 2 days ago

Read next

More changes in today’s briefingGoogle DeepMind Introduces Gemini 4 Neon for Coding, Analysis, and Science

Report timeline

Follow the reports to explore different sides of the event.

11/18
  1. AWS Adds NVIDIA Dynamo Support to SageMaker HyperPod Inference

    Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, an open source distributed inference framework for multi-node and disaggregated serving. AWS demonstrates deployment paths using DynamoGraphDeployment or InferenceEndpointConfig, and benchmarks Qwen3-235B-A22B-FP8, where disaggregated serving reached 207 output tokens per second per GPU versus 101 for aggregated serving under the stated latency and generation-speed targets. A separate Qwen3.5-397B test retained 99.1 percent of single-node throughput across two instances.

Keep up with AI, one email a day

Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue

Event heat trend

Current within comparable coverage
3
Peak within comparable coverage
1011/18 09:00 (UTC+8)
24-hour change
-51%

Trends compare the same continuously observed participants, which may cover fewer sources than current heat. Point or click to inspect hourly heat, or use left and right arrow keys.