Also, are your API endpoints working properly.
Things We Forgot to Monitor
21–30 of 66 posts
Re: Things We Forgot to Monitor
#22Also: 1) Maximum # of open file descriptors 2) Whether your slave DB stopped replicating because of some error. 3) Whether something is screwed up in your SOLR/ElasticSearch instance so it doesn't respond to search queries, but respond to simple heartbeat pings. 4) If your Redis db stopped saving to disk because of lack of space, or not enough memory, or you forgot to set overcommit memory. 5) If you're running out o…
> Maximum # of open file descriptors Augh. I ran one of my servers hard into that wall, and now it's something I watch. At least I learned from that mistake.
I learned the hard way that MySQL creates a file descriptor for every database partition you create. Someone had a script that created a new partition every week...
Re: Things We Forgot to Monitor
#23Earlier quoted context omitted.
> Maximum # of open file descriptors Augh. I ran one of my servers hard into that wall, and now it's something I watch. At least I learned from that mistake.
I learned the hard way that MySQL creates a file descriptor for every database partition you create. Someone had a script that created a new partition every week...
So after 5000 years you were running out?
Re: Things We Forgot to Monitor
#24Interestingly, an out-of-the-box Munin configuration on Debian contains nearly all of these. I recommend setting up Munin and having a look at what it monitors by default, even if you don't intend to use it as your monitoring solution.
Installation on Debian/Ubuntu is also as simple as installing the munin package (munin-node for subsequent hosts) and pointing a webserver at the right directory.
Extremely valuable when something is acting up.
Re: Things We Forgot to Monitor
#25Swap rate (as opposed to space consumed) is probably the #1 metric that monitoring agents fail to report. One thing that drives me nuts is how frequently monitoring agents/dashboards report and graph only free memory on Linux, which gives misleading results. It's fine to report it, but to make sense of it, you have to stack free memory along with cached and buffered memory, if you care about what's actually available…
Yes I can't find the socket backlog anywhere in Linux. FreeBSD exposes it via kqueue http://www.freebsd.org/cgi/man.cgi?query=kqueue through the data item in EVFILT_READ.
Re: Things We Forgot to Monitor
#26We had a perfect storm of problems only 2 weeks ago.
1. A vendor tomcat application had a memory leak, consumed all the RAM on a box, and crashed with an OOM
2. The warm standby application was slightly misconfigured, and was unable to take over when the primary app crashed
3. Our nagios was configured to email us, but something had gone wrong with ssmtp 2 days prior, and was unable to contact google apps
3a. No one was paying any attention to our server metric graphs / We didn't have good enough "pay attention to these specific graphs because they are currently outside the norm"
A very embarrassing day for us that one.
We're now working on better graphing, and have set up a basic ssmtp check to SMS us if there is an issue. Monitoring is hard.
Re: Things We Forgot to Monitor
#27Some people, when confronted with a problem, think “I know, I'll send an email whenever it happens.” Now they have two problems.
"What if the email goes down? I know I'll send an email"
Re: Things We Forgot to Monitor
#28Earlier quoted context omitted.
I learned the hard way that MySQL creates a file descriptor for every database partition you create. Someone had a script that created a new partition every week...
So after 5000 years you were running out?
I forget the details, but practically speaking the database keeled over after some 200 or 500 files were open at the same time.
Re: Things We Forgot to Monitor
#29We monitor outgoing smtp and http connections from anything that requires those services.
And the best general advice I have is split your alerts into "stuff that I need to know is broken" and "stuff that just helps me diagnose other problems". You don't want to be disturbing your on-call people for stuff that doesn't directly affect your service (or isn't even something you can fix).
Re: Things We Forgot to Monitor
#30Swap rate (as opposed to space consumed) is probably the #1 metric that monitoring agents fail to report. One thing that drives me nuts is how frequently monitoring agents/dashboards report and graph only free memory on Linux, which gives misleading results. It's fine to report it, but to make sense of it, you have to stack free memory along with cached and buffered memory, if you care about what's actually available…
Swap rate still looks like the wrong metric. It'd be better to have the rate of swap lookups, excluding all writes.