21–28 of 28 posts
The most important thing that was missed: Good: Alerts on business level metrics Bad: Alerts on machine level metrics Knowing that checkout volume is sharply down is far more valuable than knowing that CPU on one of the checkout servers is way up. Mainly because that high CPU may have no customer effect, so it's really not all that urgent.
If it's like pulling teeth to motivate your users to use dashboards, you're building useless dashboards. Dashboards are great for at-a-glance metrics roll-up. Build small, single-page, targeted dash that answers questions your user asks, and they'll be used. I want to see the lay of the land and know I'm heading into the weeds, instead of being kicked in the shin by a monitoring alert when I'm already in the weeds.
- how widespread is it?
- who's impacted?
- is the alert the root cause, or just a symptom?
Dashboards should tell a story, not just be a bunch of graphs squeezed onto a page. There should be links to drill-down to more detailed dashboards, logs, and traces, to make it as fast and easy as possible to find the fire when you smell smoke, even for someone who's on their first week.
Most dashboards, unfortunately, are useless. But then those have a place too: hanging on a wall somewhere, to show people how not-useless we are.
One of the most useful things we did was add a button to every alert we sent that said "Was this alert useful: Yes/No". We would then send the alert creator reports what percent of recipients said yes. That alone got people to realize that a lot of their alerts were unnecessary and get rid of them. As a bonus, the most useful alerts actually got subscriptions from other people on other teams because it was such a use…
I am going to implement this if I ever design an alert system.
> Bad: HTTP 400 > On the other hand, HTTP 400 level errors mean the client screwed up. This is bad general advice. HTTP 4xx errors mean the client screwed up, OR you screwed up (a change that e.g. increases 404 rate due to eventual consistency, returns 404 for all content, breaks auth, returns the wrong status code, etc.). Either way the content is inaccessible to the client, the person visiting your website doesn't…
> Bad: HTTP 400 > On the other hand, HTTP 400 level errors mean the client screwed up. This is bad general advice. HTTP 4xx errors mean the client screwed up, OR you screwed up (a change that e.g. increases 404 rate due to eventual consistency, returns 404 for all content, breaks auth, returns the wrong status code, etc.). Either way the content is inaccessible to the client, the person visiting your website doesn't…
In the context of alerting, I agree with TFA. You should not be alerting on bad request errors, because you might have no control over it. That said you might want monitoring on it so you can check if the rate jumped at some important point (eg, after a deployment) but I wouldn't look at it on a regular basis. I had something like that on an internal system. The 400 rate would jump all over the place because our edge…
Its empathetic to keep an eye on surges in 4xxs
Earlier quoted context omitted.
In the context of alerting, I agree with TFA. You should not be alerting on bad request errors, because you might have no control over it. That said you might want monitoring on it so you can check if the rate jumped at some important point (eg, after a deployment) but I wouldn't look at it on a regular basis. I had something like that on an internal system. The 400 rate would jump all over the place because our edge…
The client who's calling wrong might want some help getting it right. Its empathetic to keep an eye on surges in 4xxs
Earlier quoted context omitted.
The client who's calling wrong might want some help getting it right. Its empathetic to keep an eye on surges in 4xxs
Sure, but there’s a big world between keeping an eye on surges and waking someone up if it goes out of 3-4 nines.
One of the most useful things we did was add a button to every alert we sent that said "Was this alert useful: Yes/No". We would then send the alert creator reports what percent of recipients said yes. That alone got people to realize that a lot of their alerts were unnecessary and get rid of them. As a bonus, the most useful alerts actually got subscriptions from other people on other teams because it was such a use…