Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Year with MongoDB

blog.engineering.kiip.me

Post reply on HN
Print view View original

71–80 of 153 posts

Re: A Year with MongoDB

#71

Post by bradgessler »

raverbashing wrote: post #68
judofyr wrote: post #40

Earlier quoted context omitted.

So it's actually 100% the same as you would do with on-the-fly migrations in SQL: (1) Add column and add code moves the old data when you access it. Deploy. (2) Let it run for a while. Run a background job that migrates the rest (this might be done days or months later). (3) Remove the column and the custom code. The more I hear about "schemaless" the more I realize that it doesn't make any difference at all.

Absolutely not!! It's more like this You have old user table with for example: login and user name In MongoDB this is a JSON object {login:'user', name:'User Name'} You want to add 'shoe size'. So you add 1 - the shoe size to the signup/user editing form 2 - next user created is like this: {login:'user', name:'User Name', 'shoe_size': 7} 3 - Old users DON'T get a shoe_size added automatically to their document, but n…

That's still a migration, its just incremental.

Re: A Year with MongoDB

#72

Post by FuzzyDunlop »

sirn wrote: post #13

> We changed the structure of our heaviest used models a couple times in the past year, and instead of going back and updating millions of old documents, we simply added a “version” field to the document and the application handled the logic of reading both the old and new version. This flexibility was useful for both application developers and operations engineers. Ugh, this sounds like a maintenance nightmare. How…

> I'll admit I'm a non-believer, but every time I see "Schemaless" in MongoDB, I think "oh, so you're implementing schema in your application?"

I saw what may have well been 'schemaless' in an RBDMS recently, and the application code for it was far from pretty. I couldn't migrate at all; the results were far too inconsistent to pull it off reliably (you know something is wrong when boolean values are replaced with 'Y' and 'N', which of course both evaluate to 'true').

That being said, I tried to implement something else with Node.js and MongoDB, and I found it quite manageable. As long as the application implements a schema well, you should still be able to infer it when looking at the database direct.

To that extent, I'd take that over using an RBDMS as a key/value store for serialised data, because that's typically useless without the application that parses it.

Re: A Year with MongoDB

#74

Post by tolitius »

great post. direct and to the point, although there are many more flaws that I am sure you could have shared.

we tried MongoDB to consume and analyze market feeds, and it failed miserably. I can add a couple of things to your list:

* if there is a pending write due to an fsync lock, all reads are blocked: https://jira.mongodb.org/browse/SERVER-4243

* data loss + 10gen's white lies: https://jira.mongodb.org/browse/SERVER-3367?focusedCommentId...

* _re_ sharding is hard. shard key is should be chosen once and for all => that alone kills the schemaless advantage

* moving chunks between shards [manually or auto] can take hours / days depending on the dataset (but we talking big data, right?)

* aggregate (if any complex: e.g. not SUM, COUNT, MIN, MAX) over several gigs of data takes hours (many minutes at best). Not everything can be incremental..

Those are just several. MongoDB has an excellent marketing => Meghan Gill is great at what she does. But besides that, the tech is not quite there (yet?).

Nice going with Riak + PostgreSQL. I would also give Redis a try for things that you keep in memory => set theory ftw! :)

Re: A Year with MongoDB

#75

Post by paulsutter »

madworld wrote: post #16
paulsutter wrote: post #4

This article is an excellent articulation of the strengths and (fixable) issues with mongoDB. I like MongoDB a lot, and the improvements suggested would really strengthen the product and could make me more comfortable to use it in more serious applications.

How is the global write lock "fixable" without a major rewrite of the codebase? Like the article suggested, it would be one thing if they did it for transaction support. In reality, from looking at the code, it seems like the global write lock came from not wanting to solve the hard problems other people are solving.

A major rewrite of the core engine is exactly what's needed. Sounds like a fun project. If they don't do it, someone else will.

Adoption is hard to replace. Modifying API's is really hard. Rewriting a core engine? Reasonable and in this case probably necessary given the issues.

There are lots of people here who could replace that core engine and someone should if the MongoDB guys can't.

Re: A Year with MongoDB

#76

Post by [deleted] »

madworld wrote: post #16
paulsutter wrote: post #4

This article is an excellent articulation of the strengths and (fixable) issues with mongoDB. I like MongoDB a lot, and the improvements suggested would really strengthen the product and could make me more comfortable to use it in more serious applications.

How is the global write lock "fixable" without a major rewrite of the codebase? Like the article suggested, it would be one thing if they did it for transaction support. In reality, from looking at the code, it seems like the global write lock came from not wanting to solve the hard problems other people are solving.

[deleted]

Re: A Year with MongoDB

#77

Post by [deleted] »

madworld wrote: post #16
paulsutter wrote: post #4

This article is an excellent articulation of the strengths and (fixable) issues with mongoDB. I like MongoDB a lot, and the improvements suggested would really strengthen the product and could make me more comfortable to use it in more serious applications.

How is the global write lock "fixable" without a major rewrite of the codebase? Like the article suggested, it would be one thing if they did it for transaction support. In reality, from looking at the code, it seems like the global write lock came from not wanting to solve the hard problems other people are solving.

[deleted]

Re: A Year with MongoDB

#78

Post by pbiggar »

bradgessler wrote: post #71
raverbashing wrote: post #68

Earlier quoted context omitted.

Absolutely not!! It's more like this You have old user table with for example: login and user name In MongoDB this is a JSON object {login:'user', name:'User Name'} You want to add 'shoe size'. So you add 1 - the shoe size to the signup/user editing form 2 - next user created is like this: {login:'user', name:'User Name', 'shoe_size': 7} 3 - Old users DON'T get a shoe_size added automatically to their document, but n…

That's still a migration, its just incremental.

It's the same concept, but it's very different in practice. For example, adding a field with a default value in postgres will lock the table during the migration, which may be killer. If you use postgres for big data sets, what you'll end up implementing for your migrations looks a lot like what schemaless gives you for free.

Re: A Year with MongoDB

#79

Post by pbiggar »

judofyr wrote: post #40
mitchellh wrote: post #15

Earlier quoted context omitted.

> Ugh, this sounds like a maintenance nightmare. How do you deal with adding extra field to the document? Do you ever feel the need of running on-the-fly migration of old versions? (But when you do, shouldn't running a migration for all documents a better idea?) Yes, we did on-the-fly migration as we loaded old data in. Doing full data migration was not really an option because querying from MongoDB on un-indexed dat…

So it's actually 100% the same as you would do with on-the-fly migrations in SQL: (1) Add column and add code moves the old data when you access it. Deploy. (2) Let it run for a while. Run a background job that migrates the rest (this might be done days or months later). (3) Remove the column and the custom code. The more I hear about "schemaless" the more I realize that it doesn't make any difference at all.

That process sounds more like "implementing schemaless in SQL" than "implementing schema in NoSQL".

Re: A Year with MongoDB

#80

Post by armon »

tolitius wrote: post #74

great post. direct and to the point, although there are many more flaws that I am sure you could have shared. we tried MongoDB to consume and analyze market feeds, and it failed miserably. I can add a couple of things to your list: * if there is a pending write due to an fsync lock, all reads are blocked: https://jira.mongodb.org/browse/SERVER-4243 * data loss + 10gen's white lies: https://jira.mongodb.org/browse/SER…

I work at Kiip, and I can confirm that our "non-durable purely in-memory solution" is Redis.
Post reply on HN