How Deepgram optimized voice AI at scale with AWS Trainium
- Who
- AWS and Deepgram
- What
- explained their inference optimization effort porting and serving Deepgram’s Nova-3 speech-to-text model on AWS Trainium2
What happened
Deepgram and AWS ported the Nova-3 speech-to-text model to AWS Trainium2, processing 11–70 percent more audio per dollar on offline batch workloads than Deepgram’s previous compute options, using instance pricing at evaluation time. The implementation combines optimized NKI kernels with eight model replicas on a single trn2.3xlarge instance. A Rust serving runtime added 7.5% throughput over the multiprocessing Python baseline at the throughput-maximizing batch size.
The story so far
Deepgram and AWS ported Deepgram’s Nova-3 speech-to-text model to AWS Trainium2, and AWS reported that the optimized setup processed 11–70% more audio per dollar than Deepgram’s previous compute options on offline, throughput-oriented batch workloads. The comparison used instance pricing at the time of evaluation. The implementation combines optimized NKI kernels with eight model replicas on a single trn2.3xlarge instance. It also uses a Rust serving runtime that connects Deepgram’s model server to Trainium. AWS said the Rust runtime increased throughput by 7.5% over the multiprocessing Python baseline at the batch size that maximized throughput.
Keep reading
Latest daily · Issue 26VAST open-sources EdenLive world model with over 24FPS generation at 720PTomorrow at 08:00, a briefing like this arrives in your inbox.
The same kind of briefing as the one above: a few stories worth reading. Free. Mail starts after you confirm, and every letter has a one-click unsubscribe. Read a past issue · 2026-10-19