How push-based KV transfer improves TTFT and TPOT in vLLM
Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
- Who
- AWS
- What
- explains a push-based NIXL connector for KV transfer in vLLM disaggregated inference
- Quote
This post introduces a push-based NIXL connector that moves post-prefill transfer coordination and setup off the critical path.
What happened
AWS explains its push-based NIXL connector for vLLM, which overlaps destination registration with prefill and moves transfer preparation to a background thread. In a Qwen3-32B benchmark on a 1P1D deployment using Amazon EC2 P5en instances with EFA and prefix caching disabled, push mode delivered up to 3x better TTFT and up to 30 percent better TPOT than pull mode, with results varying by workload and configuration. The post details the architecture, benchmark reproduction, and routing tradeoff of selecting the decode instance earlier.
Why it matters
The comparison clarifies when overlapping KV transfer setup with prefill can reduce latency, while identifying the earlier decode instance selection that deployments must accept to use push mode.
The story so far
AWS introduced a push-based NIXL connector for vLLM and reported up to 3x better time to first token (TTFT) and up to 30 percent better time per output token (TPOT) than pull mode in a Qwen3-32B benchmark. Those results came from a 1P1D deployment on Amazon EC2 P5en instances with EFA and prefix caching disabled; performance varied by workload and configuration. The connector targets inference deployments that separate prefill from decoding. It overlaps destination registration with prefill and moves transfer preparation to a background thread, taking coordination and setup off the post-prefill critical path. Push mode requires choosing the decode instance earlier. AWS said pull mode remains the better fit for workloads that need bidirectional KV transfer or delay decode-instance selection until after prefill to route requests based on load.
Keep reading
Latest daily · Issue 26VAST open-sources EdenLive world model with over 24FPS generation at 720PTomorrow at 08:00, a briefing like this arrives in your inbox.
The same kind of briefing as the one above: a few stories worth reading. Free. Mail starts after you confirm, and every letter has a one-click unsubscribe. Read a past issue · 2026-10-19