Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things We Forgot to Monitor

word.bitly.com

Post reply on HN
Print view View original

31–40 of 66 posts

Re: Things We Forgot to Monitor

#31

Post by otterley »

marcosdumay wrote: post #30
otterley wrote: post #4

Swap rate (as opposed to space consumed) is probably the #1 metric that monitoring agents fail to report. One thing that drives me nuts is how frequently monitoring agents/dashboards report and graph only free memory on Linux, which gives misleading results. It's fine to report it, but to make sense of it, you have to stack free memory along with cached and buffered memory, if you care about what's actually available…

Swap rate still looks like the wrong metric. It'd be better to have the rate of swap lookups, excluding all writes.

swap-in rate, to be more specific. swap-outs aren't incredibly worrisome.

Re: Things We Forgot to Monitor

#32

Post by acdha »

otterley wrote: post #31
marcosdumay wrote: post #30

Earlier quoted context omitted.

Swap rate still looks like the wrong metric. It'd be better to have the rate of swap lookups, excluding all writes.

swap-in rate, to be more specific. swap-outs aren't incredibly worrisome.

That's backwards: things like mmap() will generate page-in activity during normal operation. page-outs means that the operating system had to evict something to satisfy other memory requests, which is what you really want to know.

Re: Things We Forgot to Monitor

#33

Post by marcosdumay »

tantalor wrote: post #8

Some people, when confronted with a problem, think “I know, I'll send an email whenever it happens.” Now they have two problems.

I really don't get where you are going with that.

Are you arguing that alerts are useless, and we must fix the issue for once? Because if so, I'd point that some things can not be fixed (because the Earth is finite, we don't know all things, etc) and you are better alerted sooner, rather than later.

Now, if you are arguing that email is not the right medium for an alert, well, what medium is better? Really, I can't think of any single candidate. Yeah, email may go down, that's why you complement it with some system external to your network (a VPS is cheap, a couple of them in different providers is almost flawless, and way cheaper than any proprietary dashboard). Yes there is some delay involved, that should be of a few minutes at most, because you create some addresses specifically for the alerts, and make all hell break loose then a message gets there. Some standard IM protocol that federated between all your net (and external point of control), could be reached from anywhere, and had plenty of support on all kinds of computers would be better, but it does not exist.

Re: Things We Forgot to Monitor

#34

Post by marcosdumay »

jsmeaton wrote: post #26

We had a perfect storm of problems only 2 weeks ago. 1. A vendor tomcat application had a memory leak, consumed all the RAM on a box, and crashed with an OOM 2. The warm standby application was slightly misconfigured, and was unable to take over when the primary app crashed 3. Our nagios was configured to email us, but something had gone wrong with ssmtp 2 days prior, and was unable to contact google apps 3a. No one…

> and have set up a basic ssmtp check to SMS us if there is an issue.

And what will happen when the network (or the alert server) is down?

You must put some check outside your network, with independent infrastructure. Adding another protocol on the same net is still subject to Murphy law.

Re: Things We Forgot to Monitor

#35

Post by 0x0 »

mnw21cam wrote: post #13
caw wrote: post #9

Earlier quoted context omitted.

Similar to free inodes, you should also check for maximum number of directories. dir_index option helps, but I've seen it become a problem.

There's a maximum number of directories? On what filesystem is that?

I got bit by this once, i think it was related to a maximum of 32k hardlinks per inode, which effectively sets a limit of 32k subdirs since each subdir has a hardlink to ".."

Re: Things We Forgot to Monitor

#36

Post by otterley »

acdha wrote: post #32
otterley wrote: post #31

Earlier quoted context omitted.

swap-in rate, to be more specific. swap-outs aren't incredibly worrisome.

That's backwards: things like mmap() will generate page-in activity during normal operation. page-outs means that the operating system had to evict something to satisfy other memory requests, which is what you really want to know.

swapouts and pageouts aren't identical in Linux, and are instrumented separately (pswpout and pgpgout, respectively; see /proc/vmstat). mmap() and other page-ins won't be counted under the swap statistics.

A pageout might suggest memory pressure, but not nearly as much as a swapout does. (pgmajfault is a better indicator.) Writing dirty pages is just something the kernel does even when there's no memory pressure at all. Also, unfortunately you can't use pgpgout for anything useful as ordinary file writes are counted there.

Re: Things We Forgot to Monitor

#38

Post by rhizome »

marcosdumay wrote: post #33
tantalor wrote: post #8

Some people, when confronted with a problem, think “I know, I'll send an email whenever it happens.” Now they have two problems.

I really don't get where you are going with that. Are you arguing that alerts are useless, and we must fix the issue for once? Because if so, I'd point that some things can not be fixed (because the Earth is finite, we don't know all things, etc) and you are better alerted sooner, rather than later. Now, if you are arguing that email is not the right medium for an alert, well, what medium is better? Really, I can't t…

Some standard IM protocol that federated between all your net (and external point of control), could be reached from anywhere, and had plenty of support on all kinds of computers would be better, but it does not exist

I would recommend an SMS sent via GSM modem for out-of-band emergency notifications.

Re: Things We Forgot to Monitor

#39

Post by alister »

marcosdumay wrote: post #33
tantalor wrote: post #8

Some people, when confronted with a problem, think “I know, I'll send an email whenever it happens.” Now they have two problems.

I really don't get where you are going with that. Are you arguing that alerts are useless, and we must fix the issue for once? Because if so, I'd point that some things can not be fixed (because the Earth is finite, we don't know all things, etc) and you are better alerted sooner, rather than later. Now, if you are arguing that email is not the right medium for an alert, well, what medium is better? Really, I can't t…

I got the GP's point immediately: He means that system administrators already get an enormous volume of email. Send them another email and it'll get ignored, deleted, or put at the bottom of a gigantic to-do list.

For airline pilots, an excessive number of warnings themselves (bells, alarms, audible warnings) are known to distract the pilots and cause errors.

Re: Things We Forgot to Monitor

#40

Post by berkay »

marcosdumay wrote: post #34
jsmeaton wrote: post #26

We had a perfect storm of problems only 2 weeks ago. 1. A vendor tomcat application had a memory leak, consumed all the RAM on a box, and crashed with an OOM 2. The warm standby application was slightly misconfigured, and was unable to take over when the primary app crashed 3. Our nagios was configured to email us, but something had gone wrong with ssmtp 2 days prior, and was unable to contact google apps 3a. No one…

> and have set up a basic ssmtp check to SMS us if there is an issue. And what will happen when the network (or the alert server) is down? You must put some check outside your network, with independent infrastructure. Adding another protocol on the same net is still subject to Murphy law.

Independent infrastructure is a good idea but not always feasible for everyone. At OpsGenie, to resolve this problem, we came up with a solution we refer as "heartbeat monitoring". This basically allows monitoring tools to send periodic heartbeat messages to us that indicate that the tools is up and can reach us. If we don't receive heartbeat messages from them in 10 minutes, we generate an alert and notify the admins. Not out of band management but does the trick to prevent situations like jsmeaton described.

http://support.opsgenie.com/customer/portal/articles/759603-...

Post reply on HN