GPU Incident Response in 60 Seconds: An SRE's Guide to eBPF-Based GPU Observability

TL;DR You get paged at 3am: GPU training pipeline missed its SLA. Datadog shows 95% GPU utilization. nvidia-smi agrees. Everything looks green — but the job is 3x slower than expected. You have zero tools to diagnose this. Ingero gives you causal chains in 60 seconds: the host CPU was fighting with DataLoader workers, starving the GPU. You fix it with taskset and go back to sleep — without waking the ML engineer. The 3am Page Every GPU SRE Dreads Your PagerDuty fires: [CRITICAL] GPU Training Pipeline SLA Breached Cluster: prod-gpu-01 (8x H100) Job: nightly-retraining-v3 Expected completion: 02:00 UTC Current status: 47% complete at 03:12 UTC You open your monitoring stack: Datadog GPU Dashboard: GPU Utilization: 95% ✅ GPU Memory: 78% ✅ GPU Temperature: 72°C ✅ Power Draw: 680W ✅ Grafana (DCGM Exporter): dcgm_gpu_utilization: 0.95 ✅ dcgm_fb_used: 62GB ✅ dcgm_sm_clock: 1980MHz ✅ nvidia-smi: +-------------------------------------------+ | GPU Name | GPU-Util | Memory-Usage | |=============