Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things We Forgot to Monitor

word.bitly.com

Post reply on HN
Print view View original

51–60 of 66 posts

Re: Things We Forgot to Monitor

#51

Post by DrJ »

AznHisoka wrote: post #2

Also: 1) Maximum # of open file descriptors 2) Whether your slave DB stopped replicating because of some error. 3) Whether something is screwed up in your SOLR/ElasticSearch instance so it doesn't respond to search queries, but respond to simple heartbeat pings. 4) If your Redis db stopped saving to disk because of lack of space, or not enough memory, or you forgot to set overcommit memory. 5) If you're running out o…

Feels like this list (and the original post) are problems caused by:

* lack of proper/default monitoring advocated for your tools (2), (4).

* Choosing poor (default/recommended) settings (1), (4).

* Keeping stateless server/instances when you don't need to (5), (6).

* Not tracking performance as part of monitoring (3), (4)

Albeit, I have made the same mistakes too.

edit: formatting

Re: Things We Forgot to Monitor

#52

Post by dredmorbius »

tantalor wrote: post #8

Some people, when confronted with a problem, think “I know, I'll send an email whenever it happens.” Now they have two problems.

Hospitals have a similar problem -- too many devices with too many alarms. As many as 10,000/day in a busy nursing floor.

NPR covered this a few days back, I've written on it at more length:

http://www.npr.org/blogs/health/2014/01/24/265702152/silenci...

http://www.reddit.com/r/dredmorbius/comments/1x0p1b/npr_sile...

Re: Things We Forgot to Monitor

#54

Post by dredmorbius »

hueving wrote: post #50
aryastark wrote: post #42

Earlier quoted context omitted.

I think you're being obtuse. Once you start sending emails for things, you start sending emails for everything . It's easy to fall into the trap of not accurately categorizing what is critical (like real, real, critical, I mean it this time guys!) and what are merely statuses . So what happens is everything starts being ignored, and your systems become obscure black boxes again.

I think you were the one being obtuse. There is no assumption that you will start receiving useless email status updates. In fact, most reasonable monitoring tools only email when a status changes to a problem state.

most reasonable monitoring tools

20+ years of experience tells me most monitoring tools aren't reasonable.

Re: Things We Forgot to Monitor

#55

Post by dredmorbius »

The corollary of this post is "things we've been monitoring and/or alerting on which we shouldn't have been".

Starting at a new shop, one of the first things I'll do is:

1. Set up a high-level "is the app / service / system responding sanely" check which lets me know, from the top of the stack, whether or not everything else is or isn't functioning properly.

2. Go through the various alerting and alarming systems and generally dialing the alerts way back. If it's broken at the top, or if some vital resource is headed to the red, let me know. But if you're going to alert based on a cascade of prior failures (and DoS my phone, email, pager, whatever), then STFU.

In Nagios, setting relationships between services and systems, for alerting services, setting thresholds appropriately, etc., is key.

For a lot of thresholds you're going to want to find out why they were set to what they were and what historical reason there was for that. It's like the old pot roast recipe where Mom cut off the ends of the roast 'coz that's how Grandma did it. Not realizing it was because Grandma's oven was too small for a full-sized roast....

Sadly, that level of technical annotation is often lacking in shops, especially where there's been significant staff turnover through the years.

I'm also a fan of some simple system tools such as sysstat which log data that can then be graphed for visualization.

Re: Things We Forgot to Monitor

#56

Post by dredmorbius »

rmc wrote: post #27
tantalor wrote: post #8

Some people, when confronted with a problem, think “I know, I'll send an email whenever it happens.” Now they have two problems.

"What if the email goes down? I know I'll send an email"

That's actually a case where sending a regular ping mail to several sentinal systems which report on the LACK of an email can be useful.

Reminds me of a few times the email queues got backed up to hell and beyond. Fuck you, Yahoo.

Re: Things We Forgot to Monitor

#57

Post by teddyh »

AznHisoka wrote: post #2

Also: 1) Maximum # of open file descriptors 2) Whether your slave DB stopped replicating because of some error. 3) Whether something is screwed up in your SOLR/ElasticSearch instance so it doesn't respond to search queries, but respond to simple heartbeat pings. 4) If your Redis db stopped saving to disk because of lack of space, or not enough memory, or you forgot to set overcommit memory. 5) If you're running out o…

X) Number of cgroups. We were getting slow performance, apparently related to slow IO, but nothing stood out as being the culprit. Turns out, since vsftpd was creating cgroups and not removing them, the pseudo-filesystem /sys/fs/cgroup had myriads of subdirectories (each representing a cgroup), and whenever something wanted to create a new cgroup or access the list of cgroups, this counted as listing that pseudo-directory, which counted as IO.

Fixed by using the undocumented option isolate_network=NO in vsftpd.conf.

Re: Things We Forgot to Monitor

#60

Post by otterley »

justincormack wrote: post #25
otterley wrote: post #4

Swap rate (as opposed to space consumed) is probably the #1 metric that monitoring agents fail to report. One thing that drives me nuts is how frequently monitoring agents/dashboards report and graph only free memory on Linux, which gives misleading results. It's fine to report it, but to make sense of it, you have to stack free memory along with cached and buffered memory, if you care about what's actually available…

Yes I can't find the socket backlog anywhere in Linux. FreeBSD exposes it via kqueue http://www.freebsd.org/cgi/man.cgi?query=kqueue through the data item in EVFILT_READ.

With FreeBSD it's even easier; you can use "netstat -L".
Post reply on HN