AWS Details Bi-Directional KV Transfer in vLLM and Reports Faster Multi-Turn Responses

AWS published guidance for bi-directional KV transfer in vLLM, reporting time-to-first-token (TTFT) improvements of up to 1.6x for Qwen3-32B and 3.3x for zai-org/GLM-5-FP8 in its benchmarks. Those tests used one prefill instance and one decode instance on Amazon EC2 P5en instances with EFA, with prefill prefix caching disabled; the results depend on the workload and deployment configuration. The mechanism lets prefill nodes reuse cached context that decode nodes retain across conversation turns. Decode instances normally free a request’s KV blocks when generation finishes. In bi-directional mode, they retain those blocks for a configurable period—480 seconds by default—and publish their location and expiration so they can be reused. AWS’s setup requires vLLM v0.21.0 or later with NixlConnector, bi-directional transfer enabled on both prefill and decode servers, and a router that accounts for KV cache locations. The guide also covers retention controls, thresholds for recomputing context, and steps to reproduce the benchmarks.

Generated by AI from reports · Updated 2 hours ago

There is not enough continuous observation data to plot a trend yet.