
- 谁
- Amazon SageMaker AI
- 引文
Today, Amazon SageMaker AI introduces KV cache-aware routing (KV-aware routing), a new routing strategy for real-time endpoints.
发生了什么
Amazon SageMaker AI 为实时终端节点增加了 KV 缓存感知路由,在考虑负载的同时,将请求定向至报告持有最长匹配缓存提示词的实例。在 Llama 3.1 70B Instruct 模型于七个 ml.p5.48xlarge 实例上进行的与随机路由的基准测试对比中,该功能在长提示词和负载环境下,将 p50 首字延迟降低了高达 97%,吞吐量提升了高达 234%。此功能需要使用受支持的 AWS vLLM 容器并启用 KV 缓存事件。
推荐理由
对于在内存压力下处理长对话或共享提示词的团队,路由可以追踪缓存条目在实例中被逐出的情况,从而在受支持的 SageMaker AI 终端节点上减少重复的预填充工作。
这件事到目前为止
Amazon SageMaker AI has introduced KV cache-aware routing for real-time endpoints, a new strategy that directs incoming requests to the specific instances holding the longest matching cached prompt while balancing current system load. In benchmark tests using Llama 3.1 70B Instruct models across seven ml.p5.48xlarge instances, the AWS Machine Learning Blog reported that this routing method reduced the p50 time-to-first-token by up to 97 percent and increased overall throughput by up to 234 percent when handling long prompts under load. The feature is available at no additional charge in all commercial AWS Regions that support SageMaker AI real-time inference. To utilize it, users must deploy a supported AWS vLLM container with KV cache events enabled.
