Lessons Learned from a Redis Outage at Yipit • Hacker News Board

Re: Lessons Learned from a Redis Outage at Yipit

#11

Post by fduran » Wed, Oct 03, 2012, 7:26 PM UTC

jtreminio wrote: post #2

I'm no server admin, but it seems to be a recurring theme where big issues are narrowed down to disk space running out. Is there not something that can automatically check this and send out alerts?

I have this little script running on cron on all of my servers https://gist.github.com/1870429

Re: Lessons Learned from a Redis Outage at Yipit

#12

Post by tedchs » Thu, Oct 04, 2012, 3:01 AM UTC

I have used Monit for years for basic server monitoring. It's a very tiny daemon with a single, simple config file. Basically I can tell it "when disk space exceeds X, or RAM exceeds Y, or CPU exceeds Z, or process identified by pidfile foo.pid isn't running, or I can't ping something, email me". No monitoring servers, no network polling, no SNMP, no monthly fees. Sounds like five lines of Monit config would have saved these guys. See the config file docs at http://mmonit.com/monit/documentation/monit.html .

Re: Lessons Learned from a Redis Outage at Yipit

#13

Post by bsg75 » Thu, Oct 04, 2012, 3:01 PM UTC

This issue is oddly similar to issues seen at a prior gig, where MSSQL and MySQL transaction logs (replication bin logs for MySQL), consumed excess disk space when large operations did fully replicate (for various reasons), and the log volume filled.

Monitoring helps, but unless your Ops staff knows what to do with a misbehaving database (RDBMS or other), it falls on the DBA or equivalent.