Skip to content

Amazon SageMaker AI Adds KV Cache-Aware Routing for Real-Time Endpoints

AWS Machine Learning BlogFrom the source· Dmitry Soldatkin·· 10 hours agoPicksAmazon SageMaker AI real-time endpoints now support KV cache aware routing
Original
Image: AWS Machine Learning Blog
Who
Amazon SageMaker AI
What
introduces KV cache-aware routing for real-time endpoints
Quote
Today, Amazon SageMaker AI introduces KV cache-aware routing (KV-aware routing), a new routing strategy for real-time endpoints.

What happened

Amazon SageMaker AI adds KV cache-aware routing for real-time endpoints, directing requests to instances that report holding the longest matching cached prompt while accounting for load. In benchmarks against random routing with Llama 3.1 70B Instruct on seven ml.p5.48xlarge instances, it reduced p50 time to first token by up to 97 percent and increased throughput by up to 234 percent under long prompts and load. The feature requires a supported AWS vLLM container and KV cache events enabled.

Why it matters

For teams serving long conversations or shared prompts under memory pressure, routing can follow cache entries as instances evict them, reducing repeated prefill work on supported SageMaker AI endpoints.

The story so far

Amazon SageMaker AI has introduced KV cache-aware routing for real-time endpoints, a new strategy that directs incoming requests to the specific instances holding the longest matching cached prompt while balancing current system load. In benchmark tests using Llama 3.1 70B Instruct models across seven ml.p5.48xlarge instances, the AWS Machine Learning Blog reported that this routing method reduced the p50 time-to-first-token by up to 97 percent and increased overall throughput by up to 234 percent when handling long prompts under load. The feature is available at no additional charge in all commercial AWS Regions that support SageMaker AI real-time inference. To utilize it, users must deploy a supported AWS vLLM container with KV cache events enabled.

Read next

See the full storyAmazon SageMaker AI Introduces KV Cache-Aware Routing for Real-Time Endpoints
This story is in 11/18 AI DailyRead the whole issue