Skip to content
K = 0.731Subscribe

Amazon SageMaker Adds NVIDIA MIG Partitioning for Inference Endpoints

1 reports 1 reporting sources Updated 2 hours ago

Understand the event

AI overview

Amazon SageMaker Inference now supports NVIDIA Multi-Instance GPU partitioning on inference component endpoints, allowing teams to assign models isolated portions of a physical GPU. The feature is available on P4d, P5, G7, and G7e instances in seven AWS Regions, including Northern Virginia, Oregon, Singapore, Tokyo, Frankfurt, and Ireland. Single-model and serverless endpoints, as well as heterogeneous instance pools, cannot use GPU partitioning. AWS reported that in its tests, one H100 MIG 3g.40gb partition delivered higher throughput and lower cost per million output tokens than a full A100-40GB for high-concurrency workloads with long inputs. For short inputs or low concurrency, AWS found the A100 remained cheaper.

Generated by AI from reports · Updated 2 hours ago

Latest development11/7 02:01 (UTC+8)
Amazon SageMaker Inference Adds Multi-Instance GPU Partitioning for EndpointsAWS Machine Learning Blog

Read next

More changes in today’s briefingGoogle DeepMind Rolls Out Gemini Omni 1.2 Flash for Longer Video Generation

Report timeline

Follow the reports to explore different sides of the event.

11/7
  1. AWS Machine Learning BlogPicks
    Amazon SageMaker Inference Adds Multi-Instance GPU Partitioning for Endpoints

    Amazon SageMaker Inference now supports GPU partitioning with NVIDIA Multi-Instance GPU (MIG) on inference component endpoints, letting teams deploy models in isolated GPU partitions. AWS supports the feature on P4d, P5, G7, and G7e instance families in seven Regions. In AWS’s comparison, one H100 mig-3g.40gb partition delivered higher throughput and lower cost per million output tokens than a full A100-40GB for the tested high-concurrency, long-input workloads; the A100 remained cheaper for short-input or low-concurrency workloads.

Keep up with AI, one email a day

Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue

Event heat trend

There is not enough continuous observation data to plot a trend yet.