Good and Bad Monitoring

> I started with a system where another team would publish data to us in an eventing architecture and would frequently publish corrupt data. It was my team’s responsibility to address anytime data was not ingested correctly into our system. As a result, we had floods of errors in our system. We tried asking them to stop and they said no.

In this particular instance, I would simply respond to the caller with an appropriate error code and be on my way. The other team should be responsible with dealing with such an issue. The writer implies they had no choice but I don’t buy it, if you design the system in such a way to not allow corrupt data to begin with, it becomes the callers responsibility to handle these issues.

One of the most useful things we did was add a button to every alert we sent that said "Was this alert useful: Yes/No".

We would then send the alert creator reports what percent of recipients said yes. That alone got people to realize that a lot of their alerts were unnecessary and get rid of them. As a bonus, the most useful alerts actually got subscriptions from other people on other teams because it was such a useful indicator.

Monitor the general 5xx error rate so you have high SNR.

Cover mistakes with robust probers that should get 200 and then alert on any non-200 response.

If it's like pulling teeth to motivate your users to use dashboards, you're building useless dashboards.

Dashboards are great for at-a-glance metrics roll-up. Build small, single-page, targeted dash that answers questions your user asks, and they'll be used. I want to see the lay of the land and know I'm heading into the weeds, instead of being kicked in the shin by a monitoring alert when I'm already in the weeds.

> Good: Alerts on business level metrics

I agree partially. I would make it more general though: alert on symptoms of problems. Those can be business metrics, like the one you suggested in your example, or they can be system level, like rate of errors, or a queue that's growing out of control.

> Bad: Alerts on machine level metrics

100% agree. There is no excuse for that. A CPU working overtime with no customer impact (no symptoms of problems) is an efficient system. I'm paying for that CPU, I'd like to use it. If I get an alert every time I use something I pay for that will only drive me to pay more so it shuts up - even though there was no problem to begin with.