Good and Bad Monitoring
raynorelyp.medium.com
Good and Bad Monitoring
1–10 of 28 posts
Re: Good and Bad Monitoring
#2>HTTP 400 level errors on their own do not indicate a problem.
...on the back-end, but they can help find issues with the front-end. For example: I have an endpoint my front-end calls in the background to update a preview for the user, I found out that some changes in the front-end made all calls invalid (but only in production), monitoring 400 errors helped me figure out the issue very quickly before a customer even complained about it. Sometimes, "the client screwed up" because of you, if you have 90% of your clients messing up in the exact same way it may actually be your fault.
Re: Good and Bad Monitoring
#3During a RCA, you find a specific error message associated with that incident. You deliver a new alert with some documentation about what it catches and what to do. You even generate automatically a ticket when it is raised.
Time passes.
There's a subtle change if the error message. You have another production incident but your alert hasn't fired.
The complexity comes from this: how do you know that an alert is still valid without creating an incident on purpose ?
Re: Good and Bad Monitoring
#4> Eventually, you will add service F and no one will remember to go into the monitoring service and add it, but they will see the tags on the other lambdas and tag the new one correctly.
This does not match my intuition not my experience, unless there’s automation to check or enforce it.
Re: Good and Bad Monitoring
#5Yes, much prefer a workflow where alerts that actually need to be looked at just come into an alert-specific Slack channel with some pretty decent basic info. We did it this way at my last job with Datadog/Slack hooks. It was easy to setup and worked great. Staring at dashboards or even checking them every hour or whatever makes little sense.
Re: Good and Bad Monitoring
#6> Bad: HTTP 400
> On the other hand, HTTP 400 level errors mean the client screwed up.
This is bad general advice. HTTP 4xx errors mean the client screwed up, OR you screwed up (a change that e.g. increases 404 rate due to eventual consistency, returns 404 for all content, breaks auth, returns the wrong status code, etc.). Either way the content is inaccessible to the client, the person visiting your website doesn't care if they get a 404 or a 502, they care that the content is inaccessible. Once you get high enough traffic, monitoring 4xx rate is pretty critical to making sure people can actually use your service. (Or monitor the inverse, i.e. a floor on 2xx rate instead of a ceiling on {4,5}xx rate.)
Re: Good and Bad Monitoring
#7> Bad: HTTP 400 > On the other hand, HTTP 400 level errors mean the client screwed up. This is bad general advice. HTTP 4xx errors mean the client screwed up, OR you screwed up (a change that e.g. increases 404 rate due to eventual consistency, returns 404 for all content, breaks auth, returns the wrong status code, etc.). Either way the content is inaccessible to the client, the person visiting your website doesn't…
In the context of alerting, I agree with TFA. You should not be alerting on bad request errors, because you might have no control over it. That said you might want monitoring on it so you can check if the rate jumped at some important point (eg, after a deployment) but I wouldn't look at it on a regular basis.
I had something like that on an internal system. The 400 rate would jump all over the place because our edge systems had shitty input validation, and bots would crawl us with broken requests ("can I reserve this item starting last week?" kind of thing) with no rate throttling. After a few years the edge validation (and bot detection) got better, but alerting on that would've been worse than useless.
Re: Good and Bad Monitoring
#8> Bad: HTTP 400 > On the other hand, HTTP 400 level errors mean the client screwed up. This is bad general advice. HTTP 4xx errors mean the client screwed up, OR you screwed up (a change that e.g. increases 404 rate due to eventual consistency, returns 404 for all content, breaks auth, returns the wrong status code, etc.). Either way the content is inaccessible to the client, the person visiting your website doesn't…
This.
In my experience, 4xx and 5xx are only valuable to find the right place to look, but in no way do indicate if client or server failed.
Re: Good and Bad Monitoring
#9> Bad: HTTP 400 > On the other hand, HTTP 400 level errors mean the client screwed up. This is bad general advice. HTTP 4xx errors mean the client screwed up, OR you screwed up (a change that e.g. increases 404 rate due to eventual consistency, returns 404 for all content, breaks auth, returns the wrong status code, etc.). Either way the content is inaccessible to the client, the person visiting your website doesn't…
In the context of alerting, I agree with TFA. You should not be alerting on bad request errors, because you might have no control over it. That said you might want monitoring on it so you can check if the rate jumped at some important point (eg, after a deployment) but I wouldn't look at it on a regular basis. I had something like that on an internal system. The 400 rate would jump all over the place because our edge…
Yeah I agree that false positives are a risk with monitoring 4xx rate. I've never seen a satisfactory "bulletproof" way to monitor 4xx rate, it's inherently difficult and simultaneously important to monitor.
It's easy in retrospect to say "oh that was a waste of time because it was just bots" but you don't know that until you investigate. I ask myself "if I see elevated 4xx's, at what point do I start to care if they're caused by a bug?" and set monitor thresholds somewhere around there.
Re: Good and Bad Monitoring
#10Inflated error rates due to invalid 500 errors is definitely a thing I've seen it at my last 2 jobs. I think it comes from a lack of confidence with developers when it comes to setting the http status code to something different. It starts with a genuine 500 error caused by an unexpected invalid request the app can't handle and is crashing or some sort of bad behaviour, they put in some sort of exception handling so now it can handle it, they now have code that solves the bug that is impacting the server, but they've still got to return something to the clients request which is still not valid. A 400 error would almost always be appropriate, or perhaps another more specific 4xx would be appropriate but 500 is what was already being returned. Anything else is a change which might impact the client in an unexpected way. It takes a lot of confidence to make a potentially breaking change that goes beyond merely fixing a bug even when you are reasonably certain it's the right change.
Once you've got it in your code base a few times to return a 500 after you've caught an exception it starts to set a president that others will follow the example.