
What happened
AWS introduces three SageMaker HyperPod capabilities for catching training failures: MultiNodeTrainingValidation checks whether GPUs train correctly, a Kubernetes monitoring agent reports infrastructure problems and affected jobs, and hung job detection identifies stalled Slurm jobs. Hung job detection can cancel a job so Slurm requeues it. The training check is for supported EKS clusters, while the monitoring agent covers EKS and hung job detection is available for Slurm.
Why it matters
Slurm teams can return stalled jobs to the queue after detection, while EKS users can trace training failures to affected pods and nodes; the training check also catches invalid loss on hardware that passes device diagnostics.
