Profiling TorchNeuron (Part 1): System-Level Profiling on AWS Trainium

What happened

This guide explains how to profile TorchNeuron inference on AWS Trainium using torch.profiler and Neuron Explorer, tracing operations from framework dispatch through runtime scheduling to hardware execution. It compares three RMSNorm implementations with the same inputs and profiling configuration. In these runs, torch.rms_norm and torch.compile each fuse the computation into one kernel dispatch, taking about one-sixth of the naive implementation’s device time.

The story so far

Hugging Face published a tutorial on profiling TorchNeuron inference on AWS Trainium, showing how to trace execution from framework dispatch through runtime scheduling to hardware execution with torch.profiler and Neuron Explorer. The guide compares three RMSNorm implementations using the same inputs and profiling configuration. In the reported runs, torch.rms_norm and torch.compile each fuse the computation into a single kernel dispatch and take about one-sixth of the naive implementation’s device time. That result applies to these RMSNorm comparisons, rather than overall inference performance.

Keep reading

Trending now

1. 1Mistral and NVIDIA Announce Industrial Engineering AI Collaboration and Open Benchmarks
2. 2Mistral Releases Voxtral 3 Speech Models, Voice Agents API and LiveKit-Based SDK
3. 3PHP releases security updates after OpenAI says its models helped researchers find critical flaws
4. 4DeepSeek Open-Sources DeepInfer and API Components, With Model and Hardware Limits