How NVIDIA’s Full-Stack Codesign Addresses AI’s Memory Challenge

What happened

NVIDIA argues that full-stack codesign can increase useful work per byte across an AI factory’s memory hierarchy as larger models and longer agentic contexts put pressure on GPU memory. It describes reducing model-weight and KV-cache footprints with NVFP4, moving inactive context to lower-cost tiers, and keeping RAG indexes outside HBM. On SemiAnalysis AgentX, NVIDIA reports up to 10x higher throughput per dollar for GB300 NVL72 than H200 at 100 tokens per second per user, and up to 30x for Vera Rubin NVL72 than GB300 at 125 tokens per second per user.

Read next

• More changes in today’s briefingOpenAI Releases Images 2.6 With SVG Export, GIFs, and Faster Generation

The story so far

NVIDIA outlined five ways its full-stack approach can ease AI inference memory pressure, from shrinking model weights to reusing context across servers. It says NVFP4 reduces model-weight and KV-cache footprints, inactive context can move to lower-cost memory tiers, and retrieval-augmented generation indexes can stay outside high-bandwidth memory (HBM). For SemiAnalysis AgentX, NVIDIA reported up to 10x higher throughput per dollar for GB300 NVL72 than H200 at 100 tokens per second per user, and up to 30x for Vera Rubin NVL72 than GB300 at 125 tokens per second per user.