AWS Demonstrates Measurement-Based Autoscaling for Kokoro-82M TTS on SageMaker AI

AWS engineer Joseph Cottingham demonstrated how to derive autoscaling settings for Kokoro-82M text-to-speech endpoints on Amazon SageMaker AI from measured concurrency limits and scale-out time, with tooling that calculates thresholds and cooldowns and flags capacity and stability risks. The walkthrough deploys the 82-million-parameter model in a custom container on a real-time endpoint. It measures a concurrency limit of Q_max=50 against a 3,000 ms p95 time-to-first-audio-byte service-level objective and uses 320.2 seconds for scale-out planning, including bounded policy detection lag. Those measurements underpin the scaling settings rather than serving as universal targets for other deployments. The tooling also checks deployment consistency and flags limited surge headroom and scale-in instability when the minimum is set to one instance. Running the scale-out measurement requires an ml.g5.xlarge service quota with at least one spare instance slot above the baseline.

Generated by AI from reports · Updated 52 minutes ago

There is not enough continuous observation data to plot a trend yet.