Automatically recover from GPU failures with Amazon ECS Managed Instances
Amazon ECS Managed Instances now has built-in GPU health monitoring and automatic repair, using NVIDIA DCGM running on each GPU instance. When a critical XID error is detected (48, 74, 79, 95, 140), ECS marks the instance IMPAIRED, drains it, provisions a replacement, and terminates the old instance once tasks stop. The post says auto-repair is enabled by default on GPU capacity providers and limits simultaneous repairs to 20% of instances (one at a time for fewer than nine instances). It also covers CloudWatch metrics, EventBridge health events, and a DCGM fault injection method to test the workflow.
What changes for platform teams
Platform teams running GPU inference on ECS Managed Instances can drop custom nvidia-smi polling, alarm and Lambda remediation scripts, and instead wire EventBridge rules and CloudWatch alarms for alerting. Teams should test with fault injection on a non-production cluster, since the synthetic fault triggers a real drain and replacement.
Read the original at AWS Containers Blog ↗
written by claude-sonnet-5-5, prompt 7cbe154, checked by publish rules