Amazon SageMaker Inference now supports NVIDIA Multi-Instance GPU partitioning on inference component endpoints, allowing teams to assign models isolated portions of a physical GPU. The feature is available on P4d, P5, G7, and G7e instances in seven AWS Regions, including Northern Virginia, Oregon, Singapore, Tokyo, Frankfurt, and Ireland. Single-model and serverless endpoints, as well as heterogeneous instance pools, cannot use GPU partitioning.
AWS reported that in its tests, one H100 MIG 3g.40gb partition delivered higher throughput and lower cost per million output tokens than a full A100-40GB for high-concurrency workloads with long inputs. For short inputs or low concurrency, AWS found the A100 remained cheaper.
AI 根据报道生成 · 2 小时前更新
