Run multi-node inference workloads using Ray LLM Deep Learning Containers
- Who
- AWS
- What
- explains how to deploy a large language model across two GPU nodes on Amazon EKS using multi-node Ray LLM Deep Learning Containers
What happened
AWS shows how to deploy Qwen3.5-9B across two GPU nodes on Amazon EKS using Ray LLM Deep Learning Containers, with separate prefill and decode deployments behind one OpenAI-compatible endpoint. The walkthrough configures the application in YAML and uses NIXL over EFA to transfer the KV cache between nodes. It covers cluster provisioning, model invocation, transport verification, and resource cleanup.
Why it matters
The guide provides a transferable deployment pattern for separating prefill and decode, including how to verify the transport backend and avoid a silent fallback to TCP-speed communication.
The story so far
AWS published a walkthrough for deploying Qwen3.5-9B across two GPU nodes on Amazon EKS using Ray LLM Deep Learning Containers, with separate prefill and decode deployments serving requests through a single OpenAI-compatible endpoint. The example configures the application in YAML and uses NVIDIA Inference Xfer Library (NIXL) over Elastic Fabric Adapter (EFA) to transfer the model’s KV cache between nodes. AWS covers provisioning the cluster, invoking the model, verifying the transport between nodes, and cleaning up the resources.
Keep reading
Latest daily · Issue 24NVIDIA and AI4Bharat Introduce Nemotron-pretraining-Indic-v1 and an Open Data Generation PipelineTrending now
- 1Meta Releases Muse Spark Open With Mac and Cloud Support, Topping US Open-Weight Index
- 2Google Launches Gemini 4 Argon Globally, Expands Paid Access as Benchmark Results Vary
- 3OpenAI Updates GPT-6 Astra Pro in ChatGPT With Parallel Agents and Faster Responses
- 4Meta’s New Business Terms Raise Dispute Over Website Crawling and AI Training Rights