Skip to content
K = 0.731Subscribe
Original
ProductsAWS Machine Learning BlogFrom the source· Shantanu Tripathi·· 2 hours agoPicks

Amazon SageMaker Inference Adds Multi-Instance GPU Partitioning for Endpoints

SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints

Image: AWS Machine Learning Blog
Who
Amazon SageMaker
What
adds support for GPU partitioning inference endpoints using NVIDIA Multi-Instance GPU
Quote
Today, Amazon SageMaker Inference supports GPU partitioning, which uses NVIDIA Multi-Instance GPU (MIG) to divide a physical GPU into independent partitions.

What happened

Amazon SageMaker Inference now supports GPU partitioning with NVIDIA Multi-Instance GPU (MIG) on inference component endpoints, letting teams deploy models in isolated GPU partitions. AWS supports the feature on P4d, P5, G7, and G7e instance families in seven Regions. In AWS’s comparison, one H100 mig-3g.40gb partition delivered higher throughput and lower cost per million output tokens than a full A100-40GB for the tested high-concurrency, long-input workloads; the A100 remained cheaper for short-input or low-concurrency workloads.

Why it matters

The post compares an H100 partition with a full A100 across five traffic shapes, showing where the newer GPU slice was faster and cheaper and where the A100 remained cheaper.

Read next

See the full storyAmazon SageMaker Adds NVIDIA MIG Partitioning for Inference Endpoints

The story so far

Amazon SageMaker Inference now supports NVIDIA Multi-Instance GPU partitioning on inference component endpoints, allowing teams to assign models isolated portions of a physical GPU. The feature is available on P4d, P5, G7, and G7e instances in seven AWS Regions, including Northern Virginia, Oregon, Singapore, Tokyo, Frankfurt, and Ireland. Single-model and serverless endpoints, as well as heterogeneous instance pools, cannot use GPU partitioning. AWS reported that in its tests, one H100 MIG 3g.40gb partition delivered higher throughput and lower cost per million output tokens than a full A100-40GB for high-concurrency workloads with long inputs. For short inputs or low concurrency, AWS found the A100 remained cheaper.

Deployment engineeringProduct updates#Product update#Deployment and engineering

Keep up with AI, one email a day

Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue · 2026-11-06