Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MongoDB's Write Lock

blog.pythonisito.com

Post reply on HN
Print view View original

61–70 of 76 posts

Re: MongoDB's Write Lock

#61

Post by JanezStupar »

prodigal_erik wrote: post #47
latch wrote: post #43

Earlier quoted context omitted.

> The Object/Relational impedance mismatch stays right there where you leave it. Somewhat umbrella...but I do think that document-based storage does make a difference. It certainly doesn't erase it and, as you say, it varies based on the use-case at hand, but I think most people would consider the development experience to have less friction (and that's certainly been the overwhelming anecdotal evidence I've heard (a…

I have to wonder whether that's because many devs started using schemaless databases for the first time over the last couple of years, and haven't yet really experienced the nightmare of data which was scribbled on by various forgotten buggy versions of the apps and never rigorously migrated (because they're self-selected to regard that as unimportant). I once worked on a Lotus Notes-based system with documents event…

Exactly, I have done most of my work on legacy (think 10-15 years worth of data, millions of documents that were spawned in countless application versions, without any schema tracking whatsoever - everything is implicit in the document itself) Lotus Domino applications.

Since there is no explicit database schema in these types of databases, what you didn't do at write, you have to do at read. And usually you want to use the latest view or representation of data, what do you do with data that wasn't there ten versions ago. What do you do about data fields of wrong format? What do you do with data that is "orphaned" and cannot be referenced to other data. Yet it still is data and still is important.

Don't get me wrong, I Love NoSQL and I like to use it. I just have enough experience with it to know that it is definitely not a silver bullet.

By the way: If someone is looking to hire a guy who is not afraid of tackling this kind of issue, contact me. I have plenty of experience with coercion of non relational data into a form suitable for analysis.

Re: MongoDB's Write Lock

#62

Post by yxhuvud »

DenisM wrote: post #16
old-gregg wrote: post #10

Earlier quoted context omitted.

You sound like a "database system design" is just one specific thing that has a well-defined definition which is known to and agreed on by everyone. Ughh... fine, not arguing. But in that case MongoDB is not a "database system" then. How about I call it a tool that holds your data for you and gives it back later. This tool has certain properties that make it work (or not work) for certain applications. It is just tha…

Sure it might be useful, I just don't think it has longevity in it. In other words it will be surpassed by another system, aiming to solve the same set of problems, but designed with concurrency in mind from the start. MongoDB developers will end up spending all their efforts to retrofit concurrency, and the community of users will simply move on.

Sqlite is also not designed with concurrency in mind. That doesn't mean it doesn't fill a purpose or that it won't be with us for a long time.

Re: MongoDB's Write Lock

#63

Post by philjackson »

ehthere wrote: post #54
jhancock wrote: post #50

Earlier quoted context omitted.

My position on mongo is thus: Its goal is humongous data sets, hence the name. Until well proven, I'm not the type to use it for huge data, but will keep an eye out for case studies. I have used Mongo on two projects with reasonably small data sets. My largest collection at the moment is 5 million, and that's basically a log. Other collections are less than 100,000. I've been running mongo 1.6 for a year on these two…

'I don't need to think much about retrofitting the data for all instances of that model. I just add an attribute where its needed for the new use case, ensure I have basic checking in my ruby model object and my system keeps incrementally improving.' That's exactly the same as adding a new column to your DB with NULL as the default value.

Not really.

Mongo has the notion of undefined and Null. You can just start putting the new field on new records without having to backfill. Also, you don't have to do the migration thing, which can get messy in big teams (from my experience).

Moving to a doc store from an RDMS really does bring with it an odd sense of freedom when it comes to the schema.

Re: MongoDB's Write Lock

#64

Post by JanezStupar »

DenisM wrote: post #40
JanezStupar wrote: post #33

Earlier quoted context omitted.

From multiple years of NoSQL experience. The Object/Relational impedance mismatch stays right there where you leave it. Using K/V stores will only help you write data. But the whole impedance nightmare is right there, waiting for you to try and make some sense of the data. Especially if you want to do relations. And you are dead wrong about relational DB's. It is either due to habit or because of being a better fit t…

>In the end it is merely a question of data normalization Can you elaborate? I understand any JSON can be shredded into a third normal form (minus the ordering problem, but let's leave that aside for now), is this what you refer to?

Yes this is exactly what I mean. Lets start from[1].

The kosher way of designing document oriented database is to simply embed everything within a document. This is great for operations such as full text search, retrieving whole objects, etc. We call this denormalized form.

On the other hand, sometimes we want to analyze our data, e.g. we want to extract subsets of data and view them in isolation. With completely denormalized form this is expensive, since we either need to manually touch each and every document and extract data or we need to maintain indices that help us out. Both are extremely resource intensive.

The third option is that we chop our objects into smaller objects and then link them together. But this will mean that retrieving a whole document will take longer (multiple database requests) it also adds an overhead of eliminating duplicates (two objects may appear identical, but really are not), etc...

Denormalization gives you horizontal scalability, but takes away ad-hoc querying. It also wastes storage (document size is minor issue, Indices however will kill you [2]).

Normalized data will take away horizontal scalability, give you ad-hoc querying, and save storage space.

In the end for any kind of nontrivial system, you will eventually reach a point, where you will need to maintain two storages - a normalized and non-normalized form. The only difference is what your primary problem is and this sets whether you start out from Normalized or Denormalized storage, this will be your primary storage and source. The other kind will be an offline slave that will offer secondary functionality.

E.g: If you start out from relational data and you want to build a FTS, you WILL have to denormalize data. On the other hand, if you start from Object/Document store and want to offer ad-hoc analytics, you WILL have to normalize your data. Its good to keep it in the back of your head.

[1]: http://www.mongodb.org/display/DOCS/Schema+Design [2]: An application I worked on had 2GB (100 million documents) worth of data, however completely indexed database would take 25GB of storage and Index rebuild would take ~8 hours.

Re: MongoDB's Write Lock

#65

Post by philjackson »

dextorious wrote: post #59
jhancock wrote: post #50

Earlier quoted context omitted.

My position on mongo is thus: Its goal is humongous data sets, hence the name. Until well proven, I'm not the type to use it for huge data, but will keep an eye out for case studies. I have used Mongo on two projects with reasonably small data sets. My largest collection at the moment is 5 million, and that's basically a log. Other collections are less than 100,000. I've been running mongo 1.6 for a year on these two…

"""My position on mongo is thus: Its goal is humongous data sets, hence the name. Until well proven, I'm not the type to use it for huge data, but will keep an eye out for case studies.""" Actually Mongo is bad for really humongous data sets. It works well if the working data set (the data you commonly need) can fit in memory. Of course this doesn't scale very well with say several terabytes of data, while there are…

"In the case you Mongo you go to sharding etc and things get complicated in your app handling."

Can you elaborate on this?

Re: MongoDB's Write Lock

#66

Post by megaman821 »

steve8918 wrote: post #46

Does anyone know what the performance differences would be between MongoDB and SQL Server/Oracle if they all had enough RAM to hold the entire dataset in memory? I'm only guessing but it seems to me that any database with their entire dataset in memory would be very fast, no?

There were some slides showing PostgreSQL with fsync turned off performed about the same as MongoDB. There is no MongoDB secret sauce that makes it any faster than well established relational dbs with a few configuration tweaks to make the comparison even.

Re: MongoDB's Write Lock

#67

Post by [deleted] »

rogerbinns wrote: post #20
DenisM wrote: post #7

> MongoDB, as some of you may know, has a process-wide write lock. I've never taken time to see what MogoDB db is, but thanks to this opening sentence, now I know everything I ever wanted to know about this system. Having worked for 13 years on database system design I am pretty confident that a system not designed with concurrency in mind cannot be retrofitted with any decent concurrency later. Thank you Rick for sa…

There was a poxy operating system a few years ago. It only ran on one 32 bit architecture, didn't have multi-processing, had limited device support and a rather bizarre set of dev tools. Then they tidied things up a bit. But when they added multi-processing these morons just used a single big kernel lock. What a bunch of idiots. Obviously anyone using that operating system was blind and stupid and there were far bett…

[deleted]

Re: MongoDB's Write Lock

#68

Post by ilaksh »

JanezStupar wrote: post #60
ilaksh wrote: post #56

Earlier quoted context omitted.

I don't think that it is accurate to describe MongoDB as a key/value store. My designs for use of MongoDB have always included my own versioned schema representation to be used, for starters, in generating data entry screens. Which NoSQL databases did you use?

I have extensive experience with Lotus Domino and CouchDB. I know enough about MongoDB, that I know its not all that much different as far as design and usage patterns go. This link [1] tells me that basically mongo's data structure is indeed a Key/Value. The schema you are talking about is not the schema I mean. What I was talking about that for any nontrivial Key/Value based database system it would be prudent to k…

"Document-based" is more accurate. One thing that's nice about MongoDB is that you get a little bit more than Map/Reduce as far as querying.

The idea is that when a type is embedded or linked (using the terms from the Schema Design link you gave) there is a record of that.

And the data types for the fields also need to be recorded. It would be nice if attributes for which there exists a type (collection) were correctly specified as attributes of that type rather some other more general type (otherwise you may need to correctly associate those fields later for analysis).

I plan to record my schema something like this (although probably will want to include some other information like field descriptions):

    [HTML]
    version: 1
    default: text    

    [Hash]
    version: 1
    default: text

    [Post]
    version: 1
    (id_: ObjectID)
    postid: text
    title: text
    author: {linked} User 
    authorname: {embedded} User.screenname
    comments: {embedded} Comments
    
    [User]
    version: 1
    (id_: ObjectID)
    screenname: text
    shash: hash    
    
    
    [Comments]
    (id_: ObjectID)
    version: 1
    text: text
    created: date
    name: text

    [Comments]
    (id_: ObjectID)
    version: 2
    text: html
    created: date
    name: text
    approved: boolean
Also, each record (here I am referring to the actual data collections, not the schema collection) will contain the id and version number of the type (referring to the schema collection). This is for generating screens. As fields are added, changed, or removed, new type versions are recorded.

I am still hopeful that I can avoid having to dump the data into regular relational tables for most ad hoc reporting purposes by generating stuff like this http://www.mongodb.org/display/DOCS/Querying and http://www.mongodb.org/display/DOCS/Aggregation+Framework from a GUI tool to support filters, arrangement of fields, and grouping/aggregation.

Re: MongoDB's Write Lock

#69

Post by MichaelGG »

petrohi wrote: post #45
DenisM wrote: post #36

Earlier quoted context omitted.

>that is hard and takes time. And I'm arguing that's going to be a hell of ride, which is more likely to break their backs than yield success. I read up on the link you posted elsewhere (thanks for that, by the way) and well, it's just as bad as I thought. Imagine, in pre-2.0 world, there's a greedy reader, and a subsequent writer queued up on the lock. All subsequent readers are blocked until writer quits, which can…

In Microsoft SQL Server row/extent/table locks are of transactional semantics and often turned off with nolock option. What really matters for concurrency is page latch, which is per 8K page. In SQLite readers actually do block writers by default. Writing transactions are committed with lock escalation steps. First shared lock is acquired, then reserved, then pending and finally exclusive. Pending blocks new shared l…

I recently found out that turning on nolock is a horrible idea. It doesn't just take you out of transactions. That is, you won't just get uncommitted data, but apparently can get completely inconsistent data as internal structures are updated. That is, even rows not part of a current transaction might not be seen, if you use nolock.

Edit: Turning ON nolock, that is.

Re: MongoDB's Write Lock

#70

Post by cpleppert »

milkshakes wrote: post #28
DenisM wrote: post #16

Earlier quoted context omitted.

Sure it might be useful, I just don't think it has longevity in it. In other words it will be surpassed by another system, aiming to solve the same set of problems, but designed with concurrency in mind from the start. MongoDB developers will end up spending all their efforts to retrofit concurrency, and the community of users will simply move on.

In other words it will be surpassed by another system, aiming to solve the same set of problems, but designed with concurrency in mind from the start. MongoDB developers will end up spending all their efforts to retrofit concurrency, and the community of users will simply move on. Fine, I'll bite. Your assertion that MongoDB was not designed "with concurrency in mind" is simply wrong. They have in fact put a lot of t…

I question whether the tradeoffs that they made are the result of careful planning. Using memory mapped files for a database isn't something a lot of people do because you give up all control over concurrency. There is no way to bandaid concurrent properties back on once you have made such an architectual choice. A thread that may block and be unable to give up a fine grained lock can be avoided in a traditional database system but not in one like mongodb.

At one point the mongodb guys were talking about moving to a record backend that looked a lot like any traditional relational database. Why would they do that if they felt that they had made the right architectual choice in the first place? More importantly, does anyone really believe that the tradeoffs in their current design are necessary?

Post reply on HN