Skip to content

Amazon SageMaker AI Introduces KV Cache-Aware Routing for Real-Time Endpoints

1 reports 1 reporting sources first reported Updated 10 hours ago

Understand the event

AI overview

Amazon SageMaker AI has introduced KV cache-aware routing for real-time endpoints, a new strategy that directs incoming requests to the specific instances holding the longest matching cached prompt while balancing current system load. In benchmark tests using Llama 3.1 70B Instruct models across seven ml.p5.48xlarge instances, the AWS Machine Learning Blog reported that this routing method reduced the p50 time-to-first-token by up to 97 percent and increased overall throughput by up to 234 percent when handling long prompts under load. The feature is available at no additional charge in all commercial AWS Regions that support SageMaker AI real-time inference. To utilize it, users must deploy a supported AWS vLLM container with KV cache events enabled.

Generated by AI from reports · Updated 10 hours ago

Read next

More changes in today’s briefingOpenAI Introduces GPT-6.2 Astra and Sol

Report timeline

Follow the reports to explore different sides of the event.

11/18
  1. Amazon SageMaker AI Adds KV Cache-Aware Routing for Real-Time Endpoints

    Amazon SageMaker AI adds KV cache-aware routing for real-time endpoints, directing requests to instances that report holding the longest matching cached prompt while accounting for load. In benchmarks against random routing with Llama 3.1 70B Instruct on seven ml.p5.48xlarge instances, it reduced p50 time to first token by up to 97 percent and increased throughput by up to 234 percent under long prompts and load. The feature requires a supported AWS vLLM container and KV cache events enabled.

Keep up with AI, one email a day

Model releases, price changes and compute developments, with original sources. Free, confirmed by email, and easy to unsubscribe. Read a past issue

Event heat trend

Current within comparable coverage
8
Peak within comparable coverage
1011/18 06:00 (UTC+8)
24-hour change
–

Trends compare the same continuously observed participants, which may cover fewer sources than current heat. Point or click to inspect hourly heat, or use left and right arrow keys.