Mixture-of-Experts kernel engineering on Amazon SageMaker AI
AWS explains three fused MoE optimizations used in Amazon SageMaker AI, reporting up to 10 percent higher throughput and up to 9 percent faster Time to First Token (TTFT). On H200, swapping matrix operands improved throughput by 2–4 percent, while routing FP8 tensor-quantized models to Triton improved throughput by 10 percent. Eliminating unnecessary Triton JIT recompilation improved TTFT by 9 percent at 8-way tensor parallelism; the post identifies availability in vLLM and the AWS LMI v28 container.
Why it matters
The optimization examples offer a transferable approach to MoE serving performance by examining matrix layout, backend selection, and compiler specialization as separate sources of throughput and latency overhead.

