
- Who
- Amazon SageMaker AI
- What
- introduces KV cache-aware routing for real-time endpoints
- Quote
Today, Amazon SageMaker AI introduces KV cache-aware routing (KV-aware routing), a new routing strategy for real-time endpoints.
What happened
Amazon SageMaker AI adds KV cache-aware routing for real-time endpoints, directing requests to instances that report holding the longest matching cached prompt while accounting for load. In benchmarks against random routing with Llama 3.1 70B Instruct on seven ml.p5.48xlarge instances, it reduced p50 time to first token by up to 97 percent and increased throughput by up to 234 percent under long prompts and load. The feature requires a supported AWS vLLM container and KV cache events enabled.
Why it matters
For teams serving long conversations or shared prompts under memory pressure, routing can follow cache entries as instances evict them, reducing repeated prefill work on supported SageMaker AI endpoints.
The story so far
Amazon SageMaker AI has introduced KV cache-aware routing for real-time endpoints, a new strategy that directs incoming requests to the specific instances holding the longest matching cached prompt while balancing current system load. In benchmark tests using Llama 3.1 70B Instruct models across seven ml.p5.48xlarge instances, the AWS Machine Learning Blog reported that this routing method reduced the p50 time-to-first-token by up to 97 percent and increased overall throughput by up to 234 percent when handling long prompts under load. The feature is available at no additional charge in all commercial AWS Regions that support SageMaker AI real-time inference. To utilize it, users must deploy a supported AWS vLLM container with KV cache events enabled.
