Stop guessing auto scaling targets for TTS on Amazon SageMaker AI

What happened

AWS engineer Joseph Cottingham demonstrates how to derive Amazon SageMaker AI auto scaling settings for Kokoro-82M from measured concurrency limits and scale-out time. The walkthrough measures Q_max=50 against a 3,000 ms p95 time-to-first-audio-byte SLO and plans around 320.2 seconds for scale-out, including bounded policy detection lag. Its tooling derives thresholds and cooldowns, checks deployment consistency, and flags limited surge headroom and scale-in instability with a minimum of 1 instance.

Read next

• More changes in today’s briefingMicrosoft Research and Collaborator Resolve the KLS Conjecture With Human-Directed AI

The story so far

AWS engineer Joseph Cottingham demonstrated how to derive autoscaling settings for Kokoro-82M text-to-speech endpoints on Amazon SageMaker AI from measured concurrency limits and scale-out time, with tooling that calculates thresholds and cooldowns and flags capacity and stability risks. The walkthrough deploys the 82-million-parameter model in a custom container on a real-time endpoint. It measures a concurrency limit of Q_max=50 against a 3,000 ms p95 time-to-first-audio-byte service-level objective and uses 320.2 seconds for scale-out planning, including bounded policy detection lag. Those measurements underpin the scaling settings rather than serving as universal targets for other deployments. The tooling also checks deployment consistency and flags limited surge headroom and scale-in instability when the minimum is set to one instance. Running the scale-out measurement requires an ml.g5.xlarge service quota with at least one spare instance slot above the baseline.