Skip to content

AWS Adds Workload Recovery and Infrastructure Checks to SageMaker HyperPod

AWS Machine Learning BlogFrom the source· Roy Allela·· 19 hours agoPicksIntroducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod
Original
Image: AWS Machine Learning Blog

What happened

AWS introduces three SageMaker HyperPod capabilities for catching training failures: MultiNodeTrainingValidation checks whether GPUs train correctly, a Kubernetes monitoring agent reports infrastructure problems and affected jobs, and hung job detection identifies stalled Slurm jobs. Hung job detection can cancel a job so Slurm requeues it. The training check is for supported EKS clusters, while the monitoring agent covers EKS and hung job detection is available for Slurm.

Why it matters

Slurm teams can return stalled jobs to the queue after detection, while EKS users can trace training failures to affected pods and nodes; the training check also catches invalid loss on hardware that passes device diagnostics.

Read next

More changes in today’s briefingGoogle DeepMind Introduces Gemini 4 Neon for Coding, Analysis, and Science
This story is in 11/19 AI DailyRead the whole issue