platformchangelog.dev
44 tools
feedpost

Automatically recover from GPU failures with Amazon ECS Managed Instances

Amazon ECS Managed Instances now has built-in GPU health monitoring and automatic repair, using NVIDIA DCGM running on each GPU instance. When a critical XID error is detected (48, 74, 79, 95, 140), ECS marks the instance IMPAIRED, drains it, provisions a replacement, and terminates the old instance once tasks stop. The post says auto-repair is enabled by default on GPU capacity providers and limits simultaneous repairs to 20% of instances (one at a time for fewer than nine instances). It also covers CloudWatch metrics, EventBridge health events, and a DCGM fault injection method to test the workflow.

What changes for platform teams

Platform teams running GPU inference on ECS Managed Instances can drop custom nvidia-smi polling, alarm and Lambda remediation scripts, and instead wire EventBridge rules and CloudWatch alarms for alerting. Teams should test with fault injection on a non-production cluster, since the synthetic fault triggers a real drain and replacement.

Read the original at AWS Containers Blog ↗

written by claude-sonnet-5-5, prompt 7cbe154, checked by publish rules

Report a mistake

context

How this was written

The summary was written from the original by the model named above, checked by fixed publishing rules, and read by an editor after publication. See the methodology.

Back to the feed