Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MongoDB's Write Lock

blog.pythonisito.com

Post reply on HN
Print view View original

51–60 of 76 posts

Re: MongoDB's Write Lock

#51

Post by orthecreedence »

> If you are able to do this, it turns out that the global write lock really doesn't affect you. Blocking reads for a few nanoseconds while a write completes turns out to be a non-issue. (I have not measured this, but I suspect that the acquisition of the global write lock takes significantly longer than the actual write.)

Actually, it does affect you. I have worked with mongodb in production in a high-write scenario with about 1000 clients and it slowed to a crawl. All data was in memory. The server was not breaking much of a sweat. mongostat showed upwards of 60 queued reads/writes at any given time.

The only solution was to shard, but I feel like an enormous server like the one we were using should be able to handle 1000 writing clients.

Keep in mind this was version 1.8. I no longer work at the company where this happened and cannot testify to the performance of 2.0, but 1.8 has abysmal write performance.

Re: MongoDB's Write Lock

#52

Post by marshray »

DenisM wrote: post #7

> MongoDB, as some of you may know, has a process-wide write lock. I've never taken time to see what MogoDB db is, but thanks to this opening sentence, now I know everything I ever wanted to know about this system. Having worked for 13 years on database system design I am pretty confident that a system not designed with concurrency in mind cannot be retrofitted with any decent concurrency later. Thank you Rick for sa…

thanks to this opening sentence, now I know everything I ever wanted to know about this system

In other words, your analytical faculties have been intentionally shut down.

I am pretty confident that a system not designed with concurrency in mind cannot be retrofitted with any decent concurrency later.

I think you're probably right from a software engineering perspective. But you haven't actually shown that Mongo was "not designed with concurrency in mind".

I've implemented in-memory persistent storage systems with a global RW lock. Depending on the characteristics of the load it can be ridiculously fast. E.g., on the order of tens of microseconds for typical transactions.

Complex schemes like MVCC with multimaster replication are an amazing accomplishment and are great for general purpose SQL-driven databases. But it's not the only reasonable design choice in the world and it's certainly not the optimal one for every problem.

Re: MongoDB's Write Lock

#53

Post by dolinsky »

latch wrote: post #30
nknight wrote: post #5

> In MongoDB version 2.0 and higher, this is addressed by detecting the likelihood of a page fault and releasing the lock before faulting. I'm assuming MongoDB tries to detect this with OS-specific syscalls. Has there been any attempt to determine whether it would be even faster and/or more portable to just unconditionally "read" the pages before acquiring the lock?

I forget who, but a fairly popular implementation of MongoDB once posted about their experience, and they mentioned that they always did a find before doing an update. Every now and again you'll see this approach get suggested in the groups.

Sounds like a great extension of the benchmarks provided in the article.

Re: MongoDB's Write Lock

#54

Post by ehthere »

jhancock wrote: post #50
DenisM wrote: post #36

Earlier quoted context omitted.

>that is hard and takes time. And I'm arguing that's going to be a hell of ride, which is more likely to break their backs than yield success. I read up on the link you posted elsewhere (thanks for that, by the way) and well, it's just as bad as I thought. Imagine, in pre-2.0 world, there's a greedy reader, and a subsequent writer queued up on the lock. All subsequent readers are blocked until writer quits, which can…

My position on mongo is thus: Its goal is humongous data sets, hence the name. Until well proven, I'm not the type to use it for huge data, but will keep an eye out for case studies. I have used Mongo on two projects with reasonably small data sets. My largest collection at the moment is 5 million, and that's basically a log. Other collections are less than 100,000. I've been running mongo 1.6 for a year on these two…

'I don't need to think much about retrofitting the data for all instances of that model. I just add an attribute where its needed for the new use case, ensure I have basic checking in my ruby model object and my system keeps incrementally improving.'

That's exactly the same as adding a new column to your DB with NULL as the default value.

Re: MongoDB's Write Lock

#55

Post by ilaksh »

lwat wrote: post #38
ilaksh wrote: post #31

The thing is that you are still supposed to keep the whole working set in memory and use sharding if its larger than that. Which means that none of this is really relevant. Except that now with that new graph showing such good performance on reads during paging people are going to get confused. Anyway you can get 32GB of RAM for $232 or 48GB for $636. Which means that for 90% of applications, you actually don't need…

"whey would I want to deal with separate steps of setting up the relational database tables" Really? What if you accidentally use 'createDate' instead of 'created' in one spot of your system. You'll get crap data and you won't even know! Or what if some other part of the system relies on a 'post' having a 'pageId' but there's no way to enforce it in your DB? It's called DATA CONSISTENCY and if you don't see the value…

I think that back in the day, people got along fine in many circumstances without relational tables, using traditional hierarchical stores.

A good solution for dealing with changing/varied schemas is to record multiple schema versions. This is the type of problem that will occur on small systems just as well as large ones, any time the requirements change, and of course how you handle those changes is a key aspect of your system design. And a relational setup has the same type of problem in keeping the requirements, schema, application and deployment in sync. I don't think I implied that there is no value in facilities for data consistency, and I did not mean to imply that a relational database cannot have some advantages, but overall for most cases my current belief is that there are more advantages, in most circumstances, especially if you are using something like CoffeeScript/Node.js, with MongoDB.

Re: MongoDB's Write Lock

#56

Post by ilaksh »

JanezStupar wrote: post #33
ilaksh wrote: post #31

The thing is that you are still supposed to keep the whole working set in memory and use sharding if its larger than that. Which means that none of this is really relevant. Except that now with that new graph showing such good performance on reads during paging people are going to get confused. Anyway you can get 32GB of RAM for $232 or 48GB for $636. Which means that for 90% of applications, you actually don't need…

From multiple years of NoSQL experience. The Object/Relational impedance mismatch stays right there where you leave it. Using K/V stores will only help you write data. But the whole impedance nightmare is right there, waiting for you to try and make some sense of the data. Especially if you want to do relations. And you are dead wrong about relational DB's. It is either due to habit or because of being a better fit t…

I don't think that it is accurate to describe MongoDB as a key/value store.

My designs for use of MongoDB have always included my own versioned schema representation to be used, for starters, in generating data entry screens.

Which NoSQL databases did you use?

Re: MongoDB's Write Lock

#57

Post by mnutt »

orthecreedence wrote: post #51

> If you are able to do this, it turns out that the global write lock really doesn't affect you. Blocking reads for a few nanoseconds while a write completes turns out to be a non-issue. (I have not measured this, but I suspect that the acquisition of the global write lock takes significantly longer than the actual write.) Actually, it does affect you. I have worked with mongodb in production in a high-write scenario…

Were your writes changing the size of the documents so that mongo had to move them? I've had this happen and it'll cause mongo to grind to a halt.

Re: MongoDB's Write Lock

#58

Post by yason »

DenisM wrote: post #7

> MongoDB, as some of you may know, has a process-wide write lock. I've never taken time to see what MogoDB db is, but thanks to this opening sentence, now I know everything I ever wanted to know about this system. Having worked for 13 years on database system design I am pretty confident that a system not designed with concurrency in mind cannot be retrofitted with any decent concurrency later. Thank you Rick for sa…

Perhaps you should read the whole article, then? As it happens, the article itself was exactly about how they actually made MongoDB work well even with the dreary global lock because global locking for in-memory data is nearly nothing. They had no reason to get rid of the global lock, it works well enough. The author said he suspects that obtaining the lock actually takes more time than generally doing the actual database update; I don't know but it sounds about right.

Re: MongoDB's Write Lock

#59

Post by dextorious »

jhancock wrote: post #50
DenisM wrote: post #36

Earlier quoted context omitted.

>that is hard and takes time. And I'm arguing that's going to be a hell of ride, which is more likely to break their backs than yield success. I read up on the link you posted elsewhere (thanks for that, by the way) and well, it's just as bad as I thought. Imagine, in pre-2.0 world, there's a greedy reader, and a subsequent writer queued up on the lock. All subsequent readers are blocked until writer quits, which can…

My position on mongo is thus: Its goal is humongous data sets, hence the name. Until well proven, I'm not the type to use it for huge data, but will keep an eye out for case studies. I have used Mongo on two projects with reasonably small data sets. My largest collection at the moment is 5 million, and that's basically a log. Other collections are less than 100,000. I've been running mongo 1.6 for a year on these two…

"""My position on mongo is thus: Its goal is humongous data sets, hence the name. Until well proven, I'm not the type to use it for huge data, but will keep an eye out for case studies."""

Actually Mongo is bad for really humongous data sets.

It works well if the working data set (the data you commonly need) can fit in memory.

Of course this doesn't scale very well with say several terabytes of data, while there are Oracle databases that handle a lot more...

In the case you Mongo you go to sharding etc and things get complicated in your app handling.

Re: MongoDB's Write Lock

#60

Post by JanezStupar »

ilaksh wrote: post #56
JanezStupar wrote: post #33

Earlier quoted context omitted.

From multiple years of NoSQL experience. The Object/Relational impedance mismatch stays right there where you leave it. Using K/V stores will only help you write data. But the whole impedance nightmare is right there, waiting for you to try and make some sense of the data. Especially if you want to do relations. And you are dead wrong about relational DB's. It is either due to habit or because of being a better fit t…

I don't think that it is accurate to describe MongoDB as a key/value store. My designs for use of MongoDB have always included my own versioned schema representation to be used, for starters, in generating data entry screens. Which NoSQL databases did you use?

I have extensive experience with Lotus Domino and CouchDB.

I know enough about MongoDB, that I know its not all that much different as far as design and usage patterns go. This link [1] tells me that basically mongo's data structure is indeed a Key/Value.

The schema you are talking about is not the schema I mean. What I was talking about that for any nontrivial Key/Value based database system it would be prudent to keep a recipe of how to normalize the data to 3rd Normal Form. Keeping this 3rd Normal Form schema around would greatly ease many troubles that arise from using NoSQL databases.

So what were you thinking about when you say "schema" - is it "relational" (normalized) schema or is it just the recipe that tells you what particular fields are for?

[1]: http://www.mongodb.org/display/DOCS/Schema+Design

Post reply on HN