You're using icanhazip.com in production? I see from a quick Google search that Puppy Linux seems to use it in some scripts, but how reliable is it?
jsonip.com is also usable in production.
Things We Forgot to Monitor
11–20 of 66 posts
Re: Things We Forgot to Monitor
#12Also, are your backups working.
Re: Things We Forgot to Monitor
#13Earlier quoted context omitted.
6) Free inodes (as distinct from space) per filesystem.
Similar to free inodes, you should also check for maximum number of directories. dir_index option helps, but I've seen it become a problem.
There's a maximum number of directories? On what filesystem is that?
Re: Things We Forgot to Monitor
#14Interestingly, an out-of-the-box Munin configuration on Debian contains nearly all of these. I recommend setting up Munin and having a look at what it monitors by default, even if you don't intend to use it as your monitoring solution.
Re: Things We Forgot to Monitor
#15Also: 1) Maximum # of open file descriptors 2) Whether your slave DB stopped replicating because of some error. 3) Whether something is screwed up in your SOLR/ElasticSearch instance so it doesn't respond to search queries, but respond to simple heartbeat pings. 4) If your Redis db stopped saving to disk because of lack of space, or not enough memory, or you forgot to set overcommit memory. 5) If you're running out o…
> Maximum # of open file descriptors Augh. I ran one of my servers hard into that wall, and now it's something I watch. At least I learned from that mistake.
Related to this, if you've ever built/run anything on Solaris, you probably found out the hard way that even in modern times, fdopen() in 32-bit apps only allows up to 255 fds because they oh so badly want to preserve ages old ABI. Funny bug to hit at runtime in production when you aren't aware of this compatibility "feature".
Re: Things We Forgot to Monitor
#16Earlier quoted context omitted.
Similar to free inodes, you should also check for maximum number of directories. dir_index option helps, but I've seen it become a problem.
There's a maximum number of directories? On what filesystem is that?
ext3 without dir_index has a limit of 32K directories in any one directory.
Where I saw it crop up was 32K folders under /tmp on a cluster system. So no it's not a limit on number of directories entirely (that's inodes), but rather how many subdirectories you can have.
http://en.wikipedia.org/wiki/Ext4#Features <-- Fixes 32K limit
Re: Things We Forgot to Monitor
#17Earlier quoted context omitted.
There's a maximum number of directories? On what filesystem is that?
ext3 without dir_index has a limit of 32K directories in any one directory. Where I saw it crop up was 32K folders under /tmp on a cluster system. So no it's not a limit on number of directories entirely (that's inodes), but rather how many subdirectories you can have. http://en.wikipedia.org/wiki/Ext4#Features <-- Fixes 32K limit
ext3/4 has really poor large-directory performance, even with dir_index, especially if you are constantly removing and readding nodes. I would highly recommend XFS for large-directory use cases.
Re: Things We Forgot to Monitor
#18You're using icanhazip.com in production? I see from a quick Google search that Puppy Linux seems to use it in some scripts, but how reliable is it?
[deleted]
Re: Things We Forgot to Monitor
#19Great post except it's 2014 and i would use zabbix over nagios any day of the week!!!
Re: Things We Forgot to Monitor
#20You're using icanhazip.com in production? I see from a quick Google search that Puppy Linux seems to use it in some scripts, but how reliable is it?
[deleted]