Retrospective from Postmark on outages (MongoDB)

I get the impression that one of the biggest issues was missed. They did not test the standard load against the secondary server, they assigned a machine of lower specs to the task and there's nothing in the future actions that indicates they'll change it... Even if they go for the new and shiny, they can end up in the same situation when their master fails.

I hope they just overlooked that in the blog post, rather than actually not correcting this first.

I know I shouldn't care but I am always weary of doing business with any company using MongoDB.

10gen put out claiming MongoDB was so much faster than SQL solutions but it seemed obvious to me than turning off fsync would make the SQL solutions run at about the same speed. Plus, why would I want to run my database in mode where it is easy to lose data? MongoDB may be a useful product but their marketing is deceptive, which will lead to companies using it in inappropriate situations.

Are they using MongoDB for the wrong purposes? The task they have seems similar to logging and if so aren't there much better software to do that? A logging server perhaps?

MongoDB does have what it calls capped collections - the semantics are the same as a circular log. No idea why they don't use that storage format.

In any event this problem is one of success. That is the kind of problem I prefer having.

That's my takeaway as well.

I have no opinions on MongoDB, but it really seems like this particular problem was because they skimped on disaster recovery, ie. their failover hardware was less powerful than their production hardware. That was the root cause of their downtime, which is inadequate planning.

That's spending money on car insurance, but realizing only after you get into an accident that the car insurance covers almost nothing. It means you've wasted your money paying for the insurance. They paid for the secondary failover hardware, but it was effectively useless since they were down for 2 days. The only thing it mitigated, possibly, was how long they were down for, but the primary objective of the hardware, ie. keep them up in case of a disaster, was a complete failure.

I've worked at a company that was completely down for a day worldwide due to a "disaster", even though we had spent millions on diesel fuel generators, etc. I blame the "checkbox" mentality where people only look to satisfy requirements, but no one actually has ownership over the process and the details. Unfortunately, in my case, no one got fired over this complete misstep, which is another problem... zero accountability.