MongoDB's Write Lock • Hacker News Board

Re: MongoDB's Write Lock

#21

Post by DenisM » Sat, Dec 31, 2011, 11:52 PM UTC

cperciva wrote: post #9

DenisM wrote: post #7

> MongoDB, as some of you may know, has a process-wide write lock. I've never taken time to see what MogoDB db is, but thanks to this opening sentence, now I know everything I ever wanted to know about this system. Having worked for 13 years on database system design I am pretty confident that a system not designed with concurrency in mind cannot be retrofitted with any decent concurrency later. Thank you Rick for sa…

I'm not convinced. I'm not saying that MongoDB is well designed -- I don't think it is -- but it seems to me that a process-wide write lock would be perfectly fine for a data store which is designed to cluster at a one-process-per-CPU-core level.

So one core is processing one request at a time, right? If it spends any time at all being queued up on disk IO or network traffic or anything else, that core is burning up XX watts for no good reason. A more efficient system, designed for concurrency, will have higher HW utilization and lower cost.

For a sign of things to come, I invite you to take a look at how relational database vendors are fighting to squeeze single-digit percentages from their engines to beat benchmarks such as TPC-C or TPC-E. NoSQL will get there too - fight for efficiency.

Re: MongoDB's Write Lock

#22

Post by nknight » Sat, Dec 31, 2011, 11:53 PM UTC

xxqs wrote: post #17

nknight wrote: post #15

Earlier quoted context omitted.

The world runs on x86. You might have some legacy systems running SPARC or POWER, but those systems are unlikely to reside in MongoDB's target market anyway. Arguing everyone else should ditch code that happens to not work on your pet architecture is a pretty self-centered worldview.

actually the number of ARM processors is growing, and not only in the mobile sector. There have been some efforts to bring ARM architecture into the server market. Also China is building its own MIPS-based supercomputer. Also the SPARC architecture is actually developing, although it's a pity to see it swallowed by Oracle. IBM is still shipping PowerPC servers. besides, there are huge SPARC-only datacenters still run…

> There have been some efforts to bring ARM architecture into the server market.

None of which have really gone anywhere.

> China is building its own MIPS-based supercomputer.

You think a lot of supercomputers are going to be running Web-targeted NoSQL datastores?

> Also the SPARC architecture is actually developing

So its partisans have insisted for the past decade. The real world doesn't much care. Sun/Oracle was less than 2% total market share for servers in 2010, by the way -- and that includes their x86 sales.

> IBM is still shipping PowerPC servers.

How many? To whom? They had 13% market share in 2010, and they sell a lot more x86 servers than POWER.

> besides, there are huge SPARC-only datacenters still running.

There won't be much longer, and the ones that are mostly run legacy infrastructure.

There's not even a convincing case for 10% of new servers running not-x86. Greenfield systems don't care about not-x86. It's just irrelevant to pretty much everybody. There's no reason for the MongoDB guys to make it any sort of priority.

Re: MongoDB's Write Lock

#23

Post by xxqs » Sat, Dec 31, 2011, 11:55 PM UTC

DenisM wrote: post #16

old-gregg wrote: post #10

Earlier quoted context omitted.

You sound like a "database system design" is just one specific thing that has a well-defined definition which is known to and agreed on by everyone. Ughh... fine, not arguing. But in that case MongoDB is not a "database system" then. How about I call it a tool that holds your data for you and gives it back later. This tool has certain properties that make it work (or not work) for certain applications. It is just tha…

Sure it might be useful, I just don't think it has longevity in it. In other words it will be surpassed by another system, aiming to solve the same set of problems, but designed with concurrency in mind from the start. MongoDB developers will end up spending all their efforts to retrofit concurrency, and the community of users will simply move on.

I guess some other team will eventually spin off an alternative server engine, still keeping the API compatibility, but improving or re-designing the poor parts. Let's wait another year or two. Pity is that as an application developer, I cannot rely on Mongo as a long-term backend solution.

Re: MongoDB's Write Lock

#24

Post by nknight » Sat, Dec 31, 2011, 11:56 PM UTC

DenisM wrote: post #18

nknight wrote: post #11

Earlier quoted context omitted.

Why is it all the "experienced" database people I encounter remind me of flat earthers? Are you really so hung up on yourselves that you'll watch the entire industry pass you by rather than admit the traditional models might not always be the best models for every situation?

If you would kindly refrain from personal attacks, we might have a productive conversation, and hacker news might continue to be a place to have such conversations.

If there were an argument to attack, I'd gladly attack it. You didn't offer one, just snide remarks and an appeal to authority. Kindly refrain from content-free posts.

Re: MongoDB's Write Lock

#25

Post by nknight » Sat, Dec 31, 2011, 11:59 PM UTC

DenisM wrote: post #21

cperciva wrote: post #9

Earlier quoted context omitted.

I'm not convinced. I'm not saying that MongoDB is well designed -- I don't think it is -- but it seems to me that a process-wide write lock would be perfectly fine for a data store which is designed to cluster at a one-process-per-CPU-core level.

So one core is processing one request at a time, right? If it spends any time at all being queued up on disk IO or network traffic or anything else, that core is burning up XX watts for no good reason. A more efficient system, designed for concurrency, will have higher HW utilization and lower cost. For a sign of things to come, I invite you to take a look at how relational database vendors are fighting to squeeze si…

> queued up on disk IO

And there's where your legacy understanding fails. You're really not supposed to be doing a lot of disk IO in modern datastores in the first place. Your working set should be in RAM.

> beat benchmarks

If my vendor is investing time in beating benchmarks instead of solving real problems, I'm finding a new vendor.

Re: MongoDB's Write Lock

#26

Post by xxqs » Sun, Jan 01, 2012, 12:13 AM UTC

nknight wrote: post #22

xxqs wrote: post #17

Earlier quoted context omitted.

actually the number of ARM processors is growing, and not only in the mobile sector. There have been some efforts to bring ARM architecture into the server market. Also China is building its own MIPS-based supercomputer. Also the SPARC architecture is actually developing, although it's a pity to see it swallowed by Oracle. IBM is still shipping PowerPC servers. besides, there are huge SPARC-only datacenters still run…

> There have been some efforts to bring ARM architecture into the server market. None of which have really gone anywhere. > China is building its own MIPS-based supercomputer. You think a lot of supercomputers are going to be running Web-targeted NoSQL datastores? > Also the SPARC architecture is actually developing So its partisans have insisted for the past decade. The real world doesn't much care. Sun/Oracle was l…

a thoughtful software designer would build an endian-neutral and memory-aligned architecture in the first place. In case of Mongo, the designers don't seem to really understand how the computer hardware operates on the low level.

for the rest of your points, neither you nor me can tell what would be the most selling server architecture in 3 years. Do you intend to design software with unpredictable lifespan? I don't.

Re: MongoDB's Write Lock

#27

Post by DenisM » Sun, Jan 01, 2012, 12:15 AM UTC

rogerbinns wrote: post #20

DenisM wrote: post #7

> MongoDB, as some of you may know, has a process-wide write lock. I've never taken time to see what MogoDB db is, but thanks to this opening sentence, now I know everything I ever wanted to know about this system. Having worked for 13 years on database system design I am pretty confident that a system not designed with concurrency in mind cannot be retrofitted with any decent concurrency later. Thank you Rick for sa…

There was a poxy operating system a few years ago. It only ran on one 32 bit architecture, didn't have multi-processing, had limited device support and a rather bizarre set of dev tools. Then they tidied things up a bit. But when they added multi-processing these morons just used a single big kernel lock. What a bunch of idiots. Obviously anyone using that operating system was blind and stupid and there were far bett…

I would argue Linux ended up with decent concurrency because both Sun and IBM poured huge heaps of dollars into making it so with their contributions, plus huge efforts from other contributors. If someone pours same amount of effort into MongoDB they might succeed with a retrofit/rewrite as well. It's relatively rare though.

Re: MongoDB's Write Lock

#28

Post by milkshakes » Sun, Jan 01, 2012, 12:15 AM UTC

DenisM wrote: post #16

old-gregg wrote: post #10

Earlier quoted context omitted.

You sound like a "database system design" is just one specific thing that has a well-defined definition which is known to and agreed on by everyone. Ughh... fine, not arguing. But in that case MongoDB is not a "database system" then. How about I call it a tool that holds your data for you and gives it back later. This tool has certain properties that make it work (or not work) for certain applications. It is just tha…

Sure it might be useful, I just don't think it has longevity in it. In other words it will be surpassed by another system, aiming to solve the same set of problems, but designed with concurrency in mind from the start. MongoDB developers will end up spending all their efforts to retrofit concurrency, and the community of users will simply move on.

In other words it will be surpassed by another system, aiming to solve the same set of problems, but designed with concurrency in mind from the start. MongoDB developers will end up spending all their efforts to retrofit concurrency, and the community of users will simply move on.

Fine, I'll bite.

Your assertion that MongoDB was not designed "with concurrency in mind" is simply wrong. They have in fact put a lot of thought into it, they simply made decisions that you either do not understand, or happen to disagree with.

Nobody at 10gen is arguing that a global write lock is the best idea in the world. They are acutely aware of the drawbacks. They initially implemented it that way because it significantly reduces the complexity of offering concurrency in the first place. There's a reason that a NoSQL solution with more granular locking doesn't exist yet. It's much more difficult to construct. There is an ongoing, documented, transparent effort to migrate toward more granular locking, but that is hard and takes time. In the meantime, they have been working on mitigating the drawbacks of their decision by aggressively yielding whenever possible during long running operations.

Did you even read the article? Did you look at the benchmarks? The entire point of it was it's not as bad as you would think, and it's getting better.

Re: MongoDB's Write Lock

#29

Post by milkshakes » Sun, Jan 01, 2012, 12:33 AM UTC

DenisM wrote: post #21

cperciva wrote: post #9

Earlier quoted context omitted.

I'm not convinced. I'm not saying that MongoDB is well designed -- I don't think it is -- but it seems to me that a process-wide write lock would be perfectly fine for a data store which is designed to cluster at a one-process-per-CPU-core level.

So one core is processing one request at a time, right? If it spends any time at all being queued up on disk IO or network traffic or anything else, that core is burning up XX watts for no good reason. A more efficient system, designed for concurrency, will have higher HW utilization and lower cost. For a sign of things to come, I invite you to take a look at how relational database vendors are fighting to squeeze si…

since you refuse to actually learn about what's going on before arguing about it, i'll just leave this here, lifted directly from 10gen's official page on concurrency[1] (nevermind the fact that this is directly addressed in TFA)

mongod threads yield their lock (read or write) in two classes of situations:

* yield-on-page-fault – v2.0 implements a yield-on-page-fault feature which results in much more concurrency than one would achieve with a pure reader/writer lock. For common operational cases, file system page faults are detected in advanced and handled outside of any lock, then the lock is resumed. Not all fault situations yield, but many do. This results in v2.0 having much better concurrency in practice than v1.8.

* yield-on-long-operation – mongod also yields periodically on common operations that are extremely long running. The goal here is to allow interleaving so that other operations which are quick-running can execute soon.

http://www.mongodb.org/display/DOCS/How+does+concurrency+wor...

Re: MongoDB's Write Lock

#30

Post by latch » Sun, Jan 01, 2012, 12:45 AM UTC

nknight wrote: post #5

> In MongoDB version 2.0 and higher, this is addressed by detecting the likelihood of a page fault and releasing the lock before faulting. I'm assuming MongoDB tries to detect this with OS-specific syscalls. Has there been any attempt to determine whether it would be even faster and/or more portable to just unconditionally "read" the pages before acquiring the lock?

I forget who, but a fairly popular implementation of MongoDB once posted about their experience, and they mentioned that they always did a find before doing an update.

Every now and again you'll see this approach get suggested in the groups.