How bi-directional KV transfer in vLLM improves TTFT for multi-turn agentic workloads

What happened

AWS explains how bi-directional KV transfer in vLLM lets prefill nodes reuse KV blocks retained by decode nodes across conversation turns. In its 1P1D benchmarks on Amazon EC2 P5en instances with EFA and prefill prefix caching disabled, TTFT improved by up to 1.6x for Qwen3-32B and 3.3x for zai-org/GLM-5-FP8. The guide covers routing, retention controls, recompute thresholds, and reproduction using vLLM v0.21.0 or later with NixlConnector; results depend on workload and deployment configuration.

The story so far

AWS published guidance for bi-directional KV transfer in vLLM, reporting time-to-first-token (TTFT) improvements of up to 1.6x for Qwen3-32B and 3.3x for zai-org/GLM-5-FP8 in its benchmarks. Those tests used one prefill instance and one decode instance on Amazon EC2 P5en instances with EFA, with prefill prefix caching disabled; the results depend on the workload and deployment configuration. The mechanism lets prefill nodes reuse cached context that decode nodes retain across conversation turns. Decode instances normally free a request’s KV blocks when generation finishes. In bi-directional mode, they retain those blocks for a configurable period—480 seconds by default—and publish their location and expiration so they can be reused. AWS’s setup requires vLLM v0.21.0 or later with NixlConnector, bi-directional transfer enabled on both prefill and decode servers, and a router that accounts for KV cache locations. The guide also covers retention controls, thresholds for recomputing context, and steps to reproduce the benchmarks.

Keep reading