Metrics Mayhem

When working with complex systems, it’s easy to underestimate the impact of seemingly minor changes. This is an incident that caused the disk I/O to surge by 200x, database latency to increase from a few milliseconds to ~30 seconds, and a perfectly running Grafana dashboard to turn into a digital sloth.