Retrospective from Postmark on outages (MongoDB) • Hacker News Board

Re: Retrospective from Postmark on outages (MongoDB)

#11

Post by viraptor » Sat, Mar 31, 2012, 4:41 PM UTC

steve8918 wrote: post #10

viraptor wrote: post #2

I get the impression that one of the biggest issues was missed. They did not test the standard load against the secondary server, they assigned a machine of lower specs to the task and there's nothing in the future actions that indicates they'll change it... Even if they go for the new and shiny, they can end up in the same situation when their master fails. I hope they just overlooked that in the blog post, rather t…

That's my takeaway as well. I have no opinions on MongoDB, but it really seems like this particular problem was because they skimped on disaster recovery, ie. their failover hardware was less powerful than their production hardware. That was the root cause of their downtime, which is inadequate planning. That's spending money on car insurance, but realizing only after you get into an accident that the car insurance c…

Seems like Netflix's chaos monkey is not a bad idea actually. I don't mean you have to kill your services randomly while there are users on them... but switching from your master to secondary (why are you even making a distinction anyway?) should be a pretty standard operation.

Even normal upgrades (hardware fails - it's a question of when, not if) could be handled transparently just by making the "secondary" server a first-class citizen.

Re: Retrospective from Postmark on outages (MongoDB)

#12

Post by thibaut_barrere » Sat, Mar 31, 2012, 4:42 PM UTC

rogerbinns wrote: post #7

megaman821 wrote: post #4

I know I shouldn't care but I am always weary of doing business with any company using MongoDB. 10gen put out claiming MongoDB was so much faster than SQL solutions but it seemed obvious to me than turning off fsync would make the SQL solutions run at about the same speed. Plus, why would I want to run my database in mode where it is easy to lose data? MongoDB may be a useful product but their marketing is deceptive,…

Have you noticed that the problem isn't MongoDB? Every case I've seen has been the data traffic exceeding the capacity of the system. There is no database that works well under those circumstances. Another way of putting it is that these businesses have been sufficiently successful that they have outgrown their own planning and deployment. As for your second paragraph, MongoDB has had journalling for quite a while, s…

> Every case I've seen has been the data traffic exceeding the capacity of the system. There is no database that works well under those circumstances.

Is there a benchmark somewhere comparing the memory/disk consumption of MongoDB vs. other datastores?

If there's a significant overhead (and my early tests tend to show that there was - but I didn't make a strict benchmark though), then it would become very related to MongoDB then.

(honest and real question, I'm a MongoDB user btw, as well as Redis, MySQL, Postgresql etc).

Re: Retrospective from Postmark on outages (MongoDB)

#13

Post by ninjastar99 » Sat, Mar 31, 2012, 5:19 PM UTC

Really tried to love Postmark for about 3 months now. Constant, almost daily issues forced us to unfollow them on Twitter (it literally became an annoyance seeing issues daily). Then we switched to Mailgun last week, and we are very happy. +1 to Mailgun

Re: Retrospective from Postmark on outages (MongoDB)

#14

Post by cnagele » Sat, Mar 31, 2012, 6:49 PM UTC

megaman821 wrote: post #4

I know I shouldn't care but I am always weary of doing business with any company using MongoDB. 10gen put out claiming MongoDB was so much faster than SQL solutions but it seemed obvious to me than turning off fsync would make the SQL solutions run at about the same speed. Plus, why would I want to run my database in mode where it is easy to lose data? MongoDB may be a useful product but their marketing is deceptive,…

Hi everyone. Chris here from Postmark.

I would say that MongoDB durability used to be an issue, but now with journaling and replica sets it's not as much of a concern.

There were two reasons why the secondary was less capable than the primary. First, the data had become very fragmented due to our frequent purging. And second, we were in the middle of an upgrade to our servers and that one had not been tackled yet. The primary failure came at a bad time. I could have clarified that better in the post.

Regarding capped collections, yes they are faster. The problem is that they can't be sharded. With our dataset that would not allow us to scale.

Re: Retrospective from Postmark on outages (MongoDB)

#15

Post by rogerbinns » Sat, Mar 31, 2012, 8:08 PM UTC

thibaut_barrere wrote: post #12

rogerbinns wrote: post #7

Earlier quoted context omitted.

Have you noticed that the problem isn't MongoDB? Every case I've seen has been the data traffic exceeding the capacity of the system. There is no database that works well under those circumstances. Another way of putting it is that these businesses have been sufficiently successful that they have outgrown their own planning and deployment. As for your second paragraph, MongoDB has had journalling for quite a while, s…

> Every case I've seen has been the data traffic exceeding the capacity of the system. There is no database that works well under those circumstances. Is there a benchmark somewhere comparing the memory/disk consumption of MongoDB vs. other datastores? If there's a significant overhead (and my early tests tend to show that there was - but I didn't make a strict benchmark though), then it would become very related to…

The main overhead for MongoDB's storage is that the "column name" (keys) is stored in every record rather than just once as with traditional SQL database and some of the other NoSQL solutions. That is why you'll often see developers using very short key names, and one use for an "ORM" to translate between developer friendly names and short stored names.

Of course this can be solved fairly easily by the MongoDB developers by having a table mapping between short tokens/numbers and the long names. This is the ticket:

https://jira.mongodb.org/browse/SERVER-863

This is someone's measurements with different key names:

http://christophermaier.name/blog/2011/05/22/MongoDB-key-nam...

Re: Retrospective from Postmark on outages (MongoDB)

#16

Post by willvarfar » Sat, Mar 31, 2012, 8:57 PM UTC

rogerbinns wrote: post #7

megaman821 wrote: post #4

I know I shouldn't care but I am always weary of doing business with any company using MongoDB. 10gen put out claiming MongoDB was so much faster than SQL solutions but it seemed obvious to me than turning off fsync would make the SQL solutions run at about the same speed. Plus, why would I want to run my database in mode where it is easy to lose data? MongoDB may be a useful product but their marketing is deceptive,…

Have you noticed that the problem isn't MongoDB? Every case I've seen has been the data traffic exceeding the capacity of the system. There is no database that works well under those circumstances. Another way of putting it is that these businesses have been sufficiently successful that they have outgrown their own planning and deployment. As for your second paragraph, MongoDB has had journalling for quite a while, s…

As I remember it:

The journaling is only fsynced every so-often - its not like it magically gives you anything like the D in ACID.

This leaves you relying on replication for durability.

And everyone has problems when they lose one of the cluster.

Luckily it wasn't the lot: http://blog.empathybox.com/post/19574936361/getting-real-abo...

I would love to be corrected; we'd all sleep easier.

Re: Retrospective from Postmark on outages (MongoDB)

#17

Post by bsg75 » Sat, Mar 31, 2012, 10:01 PM UTC

Two is one. One is none.

This includes equality in failover systems. The common issue in all of these cases is not the DB engine, OS, stack, whatever, but the infrastructure.

Re: Retrospective from Postmark on outages (MongoDB)

#18

Post by rogerbinns » Sat, Mar 31, 2012, 10:13 PM UTC

willvarfar wrote: post #16

rogerbinns wrote: post #7

Earlier quoted context omitted.

Have you noticed that the problem isn't MongoDB? Every case I've seen has been the data traffic exceeding the capacity of the system. There is no database that works well under those circumstances. Another way of putting it is that these businesses have been sufficiently successful that they have outgrown their own planning and deployment. As for your second paragraph, MongoDB has had journalling for quite a while, s…

As I remember it: The journaling is only fsynced every so-often - its not like it magically gives you anything like the D in ACID. This leaves you relying on replication for durability. And everyone has problems when they lose one of the cluster. Luckily it wasn't the lot: http://blog.empathybox.com/post/19574936361/getting-real-abo... I would love to be corrected; we'd all sleep easier.

How about the actual facts:

http://www.mongodb.org/display/DOCS/Journaling

So yes it is periodic by default (in the millisecond range). However you can wait on any request until it has become durable.

As for replication, people seem to have some hate for that, but the reality is that a journalled system that has failed (any database/operating system) will take a large amount of time to come back up, replay/recover journals etc. Not that useful.

Re: Retrospective from Postmark on outages (MongoDB)

#19

Post by thibaut_barrere » Sun, Apr 01, 2012, 8:12 AM UTC

rogerbinns wrote: post #15

thibaut_barrere wrote: post #12

Earlier quoted context omitted.

> Every case I've seen has been the data traffic exceeding the capacity of the system. There is no database that works well under those circumstances. Is there a benchmark somewhere comparing the memory/disk consumption of MongoDB vs. other datastores? If there's a significant overhead (and my early tests tend to show that there was - but I didn't make a strict benchmark though), then it would become very related to…

The main overhead for MongoDB's storage is that the "column name" (keys) is stored in every record rather than just once as with traditional SQL database and some of the other NoSQL solutions. That is why you'll often see developers using very short key names, and one use for an "ORM" to translate between developer friendly names and short stored names. Of course this can be solved fairly easily by the MongoDB develo…

Thanks for the links.

My question goes further though, as someone who has worked with, and implemented too, column-based stores: I'm curious to compare the respective space/ram consumption for the data part, too.

I think I'll write such a benchmark one day :)