Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MongoDB's Write Lock

blog.pythonisito.com

Post reply on HN
Print view View original

31–40 of 76 posts

Re: MongoDB's Write Lock

#31

Post by ilaksh »

The thing is that you are still supposed to keep the whole working set in memory and use sharding if its larger than that. Which means that none of this is really relevant.

Except that now with that new graph showing such good performance on reads during paging people are going to get confused.

Anyway you can get 32GB of RAM for $232 or 48GB for $636. Which means that for 90% of applications, you actually don't need to shard. And if the journaling works (people with a vested interest in relational systems really have to hope that it doesn't), then I don't have to worry about my data disappearing, even if I only have one database server, which is also not a recommended design with MongoDB (or any database really).

I have years of experience with SQL Server, Oracle and MySQL. However, MongoDB is the most attractive database now because it makes the object-relational impedance mismatch go away. http://en.wikipedia.org/wiki/Object-relational_impedance_mis...

If I can write code (in CoffeeScript using a library like Mongolian Deadbeef) like this

    posts.insert
      pageId: "hallo"
      title: "Hallo"
      body: "Welcome to my new blog!"
      created: new Date

    posts.findOne
      pageId: "hallo"
    , (err, post) ->

    posts.find().limit(5).sort(created: 1).toArray (err, array) ->
then whey would I want to deal with separate steps of setting up the relational database tables, creating stored procedures, creating a software layer to map my objects to my tables etc., or hiring a DBA?

I believe that most of the hate for MongoDB is fueled by a survival instinct. The popularity of databases like MongoDB threaten to make years of experience obsolete and threaten the existence of the DBA profession. Relational databases are great, but they were an optimization designed to solve certain problems that most people today just don't have, and now they have become an unfortunate institutionalized dogma.

Re: MongoDB's Write Lock

#32

Post by timr »

milkshakes wrote: post #28
DenisM wrote: post #16

Earlier quoted context omitted.

Sure it might be useful, I just don't think it has longevity in it. In other words it will be surpassed by another system, aiming to solve the same set of problems, but designed with concurrency in mind from the start. MongoDB developers will end up spending all their efforts to retrofit concurrency, and the community of users will simply move on.

In other words it will be surpassed by another system, aiming to solve the same set of problems, but designed with concurrency in mind from the start. MongoDB developers will end up spending all their efforts to retrofit concurrency, and the community of users will simply move on. Fine, I'll bite. Your assertion that MongoDB was not designed "with concurrency in mind" is simply wrong. They have in fact put a lot of t…

"Did you even read the article? Did you look at the benchmarks? The entire point of it was it's not as bad as you would think, and it's getting better."

It's still quite bad -- the 2.0 benchmark shows a dropoff of 1000 read qps with as few as 60 faults per second. The 1.8 metrics drop to damn near zero reads under the same scenario. That's obviously worse, but a 33% drop in read capacity for 60 faults/sec is bad, no matter how you look at it.

Another way of thinking about it: if you're throwing 3,000 qps at a mongo instance, and expecting only 60 faults/s, then you need to ensure that 2940/3000 (98%) of your working set fits in RAM. Maybe there are toy problems where that's true as the DB grows to terabytes of persistent storage, but there aren't very many.

Re: MongoDB's Write Lock

#33

Post by JanezStupar »

ilaksh wrote: post #31

The thing is that you are still supposed to keep the whole working set in memory and use sharding if its larger than that. Which means that none of this is really relevant. Except that now with that new graph showing such good performance on reads during paging people are going to get confused. Anyway you can get 32GB of RAM for $232 or 48GB for $636. Which means that for 90% of applications, you actually don't need…

From multiple years of NoSQL experience. The Object/Relational impedance mismatch stays right there where you leave it.

Using K/V stores will only help you write data. But the whole impedance nightmare is right there, waiting for you to try and make some sense of the data. Especially if you want to do relations.

And you are dead wrong about relational DB's. It is either due to habit or because of being a better fit that users demand representations of data that are best served from a relational source. So you better plan your data models wisely, because you WILL pump this data into a relational source, sooner or later. It would be wise do keep a schema around all the time.

In the end it is merely a question of data normalization and the use case at hand.

Re: MongoDB's Write Lock

#34

Post by viraptor »

DenisM wrote: post #21
cperciva wrote: post #9

Earlier quoted context omitted.

I'm not convinced. I'm not saying that MongoDB is well designed -- I don't think it is -- but it seems to me that a process-wide write lock would be perfectly fine for a data store which is designed to cluster at a one-process-per-CPU-core level.

So one core is processing one request at a time, right? If it spends any time at all being queued up on disk IO or network traffic or anything else, that core is burning up XX watts for no good reason. A more efficient system, designed for concurrency, will have higher HW utilization and lower cost. For a sign of things to come, I invite you to take a look at how relational database vendors are fighting to squeeze si…

I'm sorry, but "being queued up on disk IO" and "burning up XX watts for no good reason" are not related at all. For decades we have systems which do not do busy loops on external store access. They go into low power state if they really have nothing else to do in that time. They also use crazy "new" things like interrupts, notifications and wait queues. This article actually explains how mongodb does not wait for IO, but handles other requests in the meantime. (within the current known limitations of writer/writer locks, etc. of course - i'm not saying it's all perfect)

Did you really read the post before arguing this?

Re: MongoDB's Write Lock

#35

Post by amalag »

I think Mongo is great for some use cases. There are some use cases where the flexible json data just makes sense. Regarding his benchmarks, he turned off journaling. Would love to see them with journaling turned on, see how much is relevant.

Re: MongoDB's Write Lock

#36

Post by DenisM »

milkshakes wrote: post #28
DenisM wrote: post #16

Earlier quoted context omitted.

Sure it might be useful, I just don't think it has longevity in it. In other words it will be surpassed by another system, aiming to solve the same set of problems, but designed with concurrency in mind from the start. MongoDB developers will end up spending all their efforts to retrofit concurrency, and the community of users will simply move on.

In other words it will be surpassed by another system, aiming to solve the same set of problems, but designed with concurrency in mind from the start. MongoDB developers will end up spending all their efforts to retrofit concurrency, and the community of users will simply move on. Fine, I'll bite. Your assertion that MongoDB was not designed "with concurrency in mind" is simply wrong. They have in fact put a lot of t…

>that is hard and takes time.

And I'm arguing that's going to be a hell of ride, which is more likely to break their backs than yield success.

I read up on the link you posted elsewhere (thanks for that, by the way) and well, it's just as bad as I thought.

Imagine, in pre-2.0 world, there's a greedy reader, and a subsequent writer queued up on the lock. All subsequent readers are blocked until writer quits, which can't quit before greedy reader does. This is a nightmare. That was before 2.0, now the greedy reader will yield, writer will finish, and all the pending readers will be unblocked. This is only an improvement if you consider the nightmarish previous situation. There are still two problems: 1) single writer blocks all readers on a shard while in progress 2) as soon as a new writer is queued up behind the reader lock, all subsequent readers are queued up again.

Does this look like optimal resource use to you? It does not to me.

Let's contrast this with "legacy" engines:

Sybase: readers/writer lock has granularity of a 8kb page. If you're not touching the same page someone else is writing you're fine. (*) they might have moved on since the 1990-s, I haven't looked.

Microsoft: reader/writer lock has granularity of a row. If you're not reading a row someone is writing, you're fine. That was in the 1990s, they have since moved on to snapshots, but have not yet made them default option, I think.

PostgreSQL or Oracle: 1) readers are reading a snapshot and never block writers 2) writers block each other, and granularity of locking is single row. If you're not writing the same row someone else is writing you're fine.

SQL Lite - readers do not block writers, there is a database-wide writer/writer lock. Note that this is a very lightweight desktop-oriented database, not a cloud solution.

MongoDB - reader/writer lock granularity is a shard, the part of the database apportioned to a single CPU core. If you happen to read data on the same shard someone is writing, or is planning to write you're not fine at all. Their plans are "collection level locking".

So I get it, you're saying they planned to add serious concurrency later. I agree on that - they planned. Where you and I disagree is that they will likely fail, because retrofitting concurrency is exceptionally hard. I just can't believe that anyone who knows what he's getting into would actually agree to get into this.

I understand you need to compromise something when you start out, but I think concurrency is the worst possible choice.

Re: MongoDB's Write Lock

#37

Post by DenisM »

timr wrote: post #32
milkshakes wrote: post #28

Earlier quoted context omitted.

In other words it will be surpassed by another system, aiming to solve the same set of problems, but designed with concurrency in mind from the start. MongoDB developers will end up spending all their efforts to retrofit concurrency, and the community of users will simply move on. Fine, I'll bite. Your assertion that MongoDB was not designed "with concurrency in mind" is simply wrong. They have in fact put a lot of t…

"Did you even read the article? Did you look at the benchmarks? The entire point of it was it's not as bad as you would think, and it's getting better." It's still quite bad -- the 2.0 benchmark shows a dropoff of 1000 read qps with as few as 60 faults per second. The 1.8 metrics drop to damn near zero reads under the same scenario. That's obviously worse, but a 33% drop in read capacity for 60 faults/sec is bad, no…

To be fair, there is a class of problems that fits in RAM, e.g. financial systems. (For more detailed examples, consider the YC startup MemSQL). However these scenarios are write-intensive and demand very high concurrency.

Re: MongoDB's Write Lock

#38

Post by lwat »

ilaksh wrote: post #31

The thing is that you are still supposed to keep the whole working set in memory and use sharding if its larger than that. Which means that none of this is really relevant. Except that now with that new graph showing such good performance on reads during paging people are going to get confused. Anyway you can get 32GB of RAM for $232 or 48GB for $636. Which means that for 90% of applications, you actually don't need…

"whey would I want to deal with separate steps of setting up the relational database tables"

Really? What if you accidentally use 'createDate' instead of 'created' in one spot of your system. You'll get crap data and you won't even know! Or what if some other part of the system relies on a 'post' having a 'pageId' but there's no way to enforce it in your DB? It's called DATA CONSISTENCY and if you don't see the value in stuff like that then you've never worked on a system of significant size.

You NoSQL fans don't seem to realize that these 'separate steps' you don't want to 'deal with' arised from many decades of data storage experience in the real world. Ignore them at your peril.

Re: MongoDB's Write Lock

#39

Post by xxqs »

ketralnis wrote: post #12
xxqs wrote: post #4

Why is everyone paying so much attention to MongoDB? It has been criticized a lot for its design and implementation problems, but still for some reason it's so popular. To name a few, * word-unaligned memory structures, which leads to incompatibility with virtually any non-x86 CPU architecture * explicitly little-endian processing in the server, so there is no way to run the original code on any big-endian CPU archit…

Yeah, how dare people like what you don't like? > It has been criticized a lot [...] but still for some reason it's so popular Since you provide no data or sources for "criticized a lot" it's no surprise that you don't provide the same for "so popular". I assume you mean "I've seen some headlines on Hacker News about it". > incompatibility with virtually any non-x86 CPU architecture [...] no way to run the original c…

ok, details on the criticism:

1. Look at their BSON format specification: http://bsonspec.org/#/specification They name it a new "standard", and at the same time they have things like "\x11" -> Timestamp, a special internal MongoDB data for replication. It's like the IP protocol specification would have vendor-specific parts.

2. The BSON format is not word-aligned, which makes it quite inefficient to process in memory. Besides, the current Mongo server accesses it with word-unaligned pointers, which is only possible on x86 architecture.

to me, it just tells that the database engine design was given to the wrong people.

Re: MongoDB's Write Lock

#40

Post by DenisM »

JanezStupar wrote: post #33
ilaksh wrote: post #31

The thing is that you are still supposed to keep the whole working set in memory and use sharding if its larger than that. Which means that none of this is really relevant. Except that now with that new graph showing such good performance on reads during paging people are going to get confused. Anyway you can get 32GB of RAM for $232 or 48GB for $636. Which means that for 90% of applications, you actually don't need…

From multiple years of NoSQL experience. The Object/Relational impedance mismatch stays right there where you leave it. Using K/V stores will only help you write data. But the whole impedance nightmare is right there, waiting for you to try and make some sense of the data. Especially if you want to do relations. And you are dead wrong about relational DB's. It is either due to habit or because of being a better fit t…

>In the end it is merely a question of data normalization

Can you elaborate?

I understand any JSON can be shredded into a third normal form (minus the ordering problem, but let's leave that aside for now), is this what you refer to?

Post reply on HN