Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things We Forgot to Monitor

word.bitly.com

Post reply on HN
Print view View original

41–50 of 66 posts

Re: Things We Forgot to Monitor

#41

Post by berkay »

jsmeaton wrote: post #26

We had a perfect storm of problems only 2 weeks ago. 1. A vendor tomcat application had a memory leak, consumed all the RAM on a box, and crashed with an OOM 2. The warm standby application was slightly misconfigured, and was unable to take over when the primary app crashed 3. Our nagios was configured to email us, but something had gone wrong with ssmtp 2 days prior, and was unable to contact google apps 3a. No one…

You may want to check OpsGenie heartbeat monitoring, or essentially implement the same idea yourself. Our heartbeat monitoring expects to receive messages (via email or API) from monitoring tools periodically and notifies you via push/SMS/phone if we don't receive it over 10 minutes. I think this pattern is very useful to ensure that alert notifications is working.

Re: Things We Forgot to Monitor

#42

Post by aryastark »

marcosdumay wrote: post #33
tantalor wrote: post #8

Some people, when confronted with a problem, think “I know, I'll send an email whenever it happens.” Now they have two problems.

I really don't get where you are going with that. Are you arguing that alerts are useless, and we must fix the issue for once? Because if so, I'd point that some things can not be fixed (because the Earth is finite, we don't know all things, etc) and you are better alerted sooner, rather than later. Now, if you are arguing that email is not the right medium for an alert, well, what medium is better? Really, I can't t…

I think you're being obtuse.

Once you start sending emails for things, you start sending emails for everything. It's easy to fall into the trap of not accurately categorizing what is critical (like real, real, critical, I mean it this time guys!) and what are merely statuses. So what happens is everything starts being ignored, and your systems become obscure black boxes again.

Re: Things We Forgot to Monitor

#43

Post by InclinedPlane »

otterley wrote: post #4

Swap rate (as opposed to space consumed) is probably the #1 metric that monitoring agents fail to report. One thing that drives me nuts is how frequently monitoring agents/dashboards report and graph only free memory on Linux, which gives misleading results. It's fine to report it, but to make sense of it, you have to stack free memory along with cached and buffered memory, if you care about what's actually available…

> One thing that drives me nuts is how frequently monitoring agents/dashboards report and graph only free memory on Linux, which gives misleading results. It's fine to report it, but to make sense of it, you have to stack free memory along with cached and buffered memory, if you care about what's actually available for applications to use.

Even that is misleading. It's actually non-trivial to find out exactly how much "freeable" memory one has on a linux system these days as not all the cached memory bits are truly freeable.

Re: Things We Forgot to Monitor

#45

Post by InclinedPlane »

marcosdumay wrote: post #33
tantalor wrote: post #8

Some people, when confronted with a problem, think “I know, I'll send an email whenever it happens.” Now they have two problems.

I really don't get where you are going with that. Are you arguing that alerts are useless, and we must fix the issue for once? Because if so, I'd point that some things can not be fixed (because the Earth is finite, we don't know all things, etc) and you are better alerted sooner, rather than later. Now, if you are arguing that email is not the right medium for an alert, well, what medium is better? Really, I can't t…

The point being: delegating prioritization and categorization to a human in real-time is lazy and dangerous. As much as possible humans should only receive notifications when something requires action or is too complex to determine that programatically.

Re: Things We Forgot to Monitor

#46

Post by rodgerd »

otterley wrote: post #4

Swap rate (as opposed to space consumed) is probably the #1 metric that monitoring agents fail to report. One thing that drives me nuts is how frequently monitoring agents/dashboards report and graph only free memory on Linux, which gives misleading results. It's fine to report it, but to make sense of it, you have to stack free memory along with cached and buffered memory, if you care about what's actually available…

Even then there's some wrinkles; the anon shared memory used by e.g. the Oracle SGA will show up as cached memory, but evicting it is a no-no.

Re: Things We Forgot to Monitor

#48

Post by jlgaddis »

I have gear in three different facilities and I'm typically visiting any of them unless I'm installing hardware or replacing it. Shortly after starting at $job, I realized there was no monitoring of the RAID arrays in the servers we have. That could have ended badly.

Re: Things We Forgot to Monitor

#49

Post by jlgaddis »

Be sure to monitor your monitoring system as well (preferably from outside your network/datacenters)! If you don't have anything else in place, you can use Pingdom to monitor one website/server for free [0].

I was off work for a few months recently (motorcycle wreck) and removed my e-mail accounts from my phone. Now, I have all my alerts go to a specific e-mail address and those are the only mails I receive on my phone. It has really helped me overcome the problem of ignoring messages.

[0]: https://www.pingdom.com/free/

Re: Things We Forgot to Monitor

#50

Post by hueving »

aryastark wrote: post #42
marcosdumay wrote: post #33

Earlier quoted context omitted.

I really don't get where you are going with that. Are you arguing that alerts are useless, and we must fix the issue for once? Because if so, I'd point that some things can not be fixed (because the Earth is finite, we don't know all things, etc) and you are better alerted sooner, rather than later. Now, if you are arguing that email is not the right medium for an alert, well, what medium is better? Really, I can't t…

I think you're being obtuse. Once you start sending emails for things, you start sending emails for everything . It's easy to fall into the trap of not accurately categorizing what is critical (like real, real, critical, I mean it this time guys!) and what are merely statuses . So what happens is everything starts being ignored, and your systems become obscure black boxes again.

I think you were the one being obtuse. There is no assumption that you will start receiving useless email status updates. In fact, most reasonable monitoring tools only email when a status changes to a problem state.
Post reply on HN