Skip to content

AWS Launches SageMaker HyperPod Global Inference Gateway

AWS Machine Learning BlogFrom the source· Dmitry Soldatkin·· 6 hours agoPicksIntroducing Amazon SageMaker HyperPod Global Inference Gateway: One endpoint across Regions and clusters
Original
Image: AWS Machine Learning Blog
Who
AWS
What
introduces SageMaker HyperPod Global Inference Gateway for routing inference requests across clusters and Regions
Quote
Today, SageMaker HyperPod introduces the Global Inference Gateway.

What happened

AWS introduces the Amazon SageMaker HyperPod Global Inference Gateway, a SageMaker-managed endpoint that routes requests across registered HyperPod clusters in one or multiple Regions. It selects clusters using KV cache headroom, queue depth, Region proximity, and error rate, and automatically shifts traffic from failed clusters. The service is available in 13 AWS Regions and uses SigV4, IAM, and AWS PrivateLink.

Why it matters

The gateway lets teams route inference across HyperPod clusters in multiple Regions through one endpoint, with automatic failover and private connectivity handled by SageMaker. Teams can tune routing weights across capacity, queue depth, proximity, and errors.

Read next

See the full storyAWS Introduces SageMaker HyperPod Global Inference Gateway
This story is in 11/21 AI DailyRead the whole issue