Things We Forgot to Monitor
word.bitly.com
Things We Forgot to Monitor
1–10 of 66 posts
Re: Things We Forgot to Monitor
#2Also:
1) Maximum # of open file descriptors
2) Whether your slave DB stopped replicating because of some error.
3) Whether something is screwed up in your SOLR/ElasticSearch instance so it doesn't respond to search queries, but respond to simple heartbeat pings.
4) If your Redis db stopped saving to disk because of lack of space, or not enough memory, or you forgot to set overcommit memory.
5) If you're running out of space in a specific partition you usually store random stuff like /var/log.
I've had my ass bitten by all of the above :)
Re: Things We Forgot to Monitor
#3Also: 1) Maximum # of open file descriptors 2) Whether your slave DB stopped replicating because of some error. 3) Whether something is screwed up in your SOLR/ElasticSearch instance so it doesn't respond to search queries, but respond to simple heartbeat pings. 4) If your Redis db stopped saving to disk because of lack of space, or not enough memory, or you forgot to set overcommit memory. 5) If you're running out o…
> Maximum # of open file descriptors
Augh. I ran one of my servers hard into that wall, and now it's something I watch. At least I learned from that mistake.
Re: Things We Forgot to Monitor
#4Swap rate (as opposed to space consumed) is probably the #1 metric that monitoring agents fail to report.
One thing that drives me nuts is how frequently monitoring agents/dashboards report and graph only free memory on Linux, which gives misleading results. It's fine to report it, but to make sense of it, you have to stack free memory along with cached and buffered memory, if you care about what's actually available for applications to use.
Another often-overlooked metric that's important for web services in particular is the TCP accept queue depth, per listening port. Once the accept queue is drained, remote clients will get ECONNREFUSED, which is a bad place to be. This value is somewhat difficult to attain, though, because AFAIK Linux doesn't expose it.
Re: Things We Forgot to Monitor
#5Also: 1) Maximum # of open file descriptors 2) Whether your slave DB stopped replicating because of some error. 3) Whether something is screwed up in your SOLR/ElasticSearch instance so it doesn't respond to search queries, but respond to simple heartbeat pings. 4) If your Redis db stopped saving to disk because of lack of space, or not enough memory, or you forgot to set overcommit memory. 5) If you're running out o…
6) Free inodes (as distinct from space) per filesystem.
Re: Things We Forgot to Monitor
#6About reboot monitoring, I suggest to use kdump to dump the oops information and save it for later debugging and understanding of the issue. It may even be an uncorrectable memory or pcie error you are seeing and the info is logged in the oops but is hard to figure otherwise. Also, if you consistently hit a single kernel bug you may want to fix it or workaround it.
Re: Things We Forgot to Monitor
#7You're using icanhazip.com in production? I see from a quick Google search that Puppy Linux seems to use it in some scripts, but how reliable is it?
Re: Things We Forgot to Monitor
#8Some people, when confronted with a problem, think
“I know, I'll send an email whenever it happens.” Now they have two problems.
Re: Things We Forgot to Monitor
#9Also: 1) Maximum # of open file descriptors 2) Whether your slave DB stopped replicating because of some error. 3) Whether something is screwed up in your SOLR/ElasticSearch instance so it doesn't respond to search queries, but respond to simple heartbeat pings. 4) If your Redis db stopped saving to disk because of lack of space, or not enough memory, or you forgot to set overcommit memory. 5) If you're running out o…
6) Free inodes (as distinct from space) per filesystem.
Similar to free inodes, you should also check for maximum number of directories. dir_index option helps, but I've seen it become a problem.
Re: Things We Forgot to Monitor
#10You're using icanhazip.com in production? I see from a quick Google search that Puppy Linux seems to use it in some scripts, but how reliable is it?
$ curl -i -k -L icanhazip.com
HTTP/1.1 200 OK
Date: Mon, 10 Feb 2014 20:13:28 GMT
Server: Apache
Content-Length: 15
Content-Type: text/plain; charset=UTF-8
X-RTFM: Learn about this site at http://bit.ly/14DAh2o and don't abuse the service
X-YOU-SHOULD-APPLY-FOR-A-JOB: If you're reading this, apply here: http://rackertalent.com/
X-ICANHAZNODE: icanhazip2.nugget
Would seem only fair. :D