Skip to content
K = 0.731

AWS Details Push-Based KV Transfer in vLLM and Its Performance Tradeoffs

1 reports 1 reporting sources 3 hours agoNew

Understand the event

AI overview

AWS introduced a push-based NIXL connector for vLLM and reported up to 3x better time to first token (TTFT) and up to 30 percent better time per output token (TPOT) than pull mode in a Qwen3-32B benchmark. Those results came from a 1P1D deployment on Amazon EC2 P5en instances with EFA and prefix caching disabled; performance varied by workload and configuration. The connector targets inference deployments that separate prefill from decoding. It overlaps destination registration with prefill and moves transfer preparation to a background thread, taking coordination and setup off the post-prefill critical path. Push mode requires choosing the decode instance earlier. AWS said pull mode remains the better fit for workloads that need bidirectional KV transfer or delay decode-instance selection until after prefill to route requests based on load.

Generated by AI from reports · Updated 2 hours ago

Latest development10/20 00:05 (UTC+8)
How push-based KV transfer improves TTFT and TPOT in vLLMAWS Machine Learning Blog

Report timeline

Follow the reports to explore different sides of the event.

10/20
  1. AWS Machine Learning BlogPicks
    How push-based KV transfer improves TTFT and TPOT in vLLM

    AWS explains its push-based NIXL connector for vLLM, which overlaps destination registration with prefill and moves transfer preparation to a background thread. In a Qwen3-32B benchmark on a 1P1D deployment using Amazon EC2 P5en instances with EFA and prefix caching disabled, push mode delivered up to 3x better TTFT and up to 30 percent better TPOT than pull mode, with results varying by workload and configuration. The post details the architecture, benchmark reproduction, and routing tradeoff of selecting the decode instance earlier.

Tomorrow at 08:00, a briefing like this arrives in your inbox.

The same kind of briefing as the one above: a few stories worth reading. Free. Mail starts after you confirm, and every letter has a one-click unsubscribe. Read a past issue

Event heat trend

There is not enough continuous observation data to plot a trend yet.

Keep reading

Latest daily · Issue 26VAST open-sources EdenLive world model with over 24FPS generation at 720P