Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Year with MongoDB

blog.engineering.kiip.me

Post reply on HN
Print view View original

81–90 of 153 posts

Re: A Year with MongoDB

#81

Post by ismarc »

jasonmccay wrote: post #49
mitchellh wrote: post #47

Earlier quoted context omitted.

> Part of the lesson here is that if you're doing MongoDB on EC2, you should have more than enough RAM for your working set. We had more than enough RAM for our working set. Unfortunately, due to MongoDB's poor memory managed and non-counting B-trees, even our hot data would sometimes be purged out of memory for cold, unused data, causing serious performance degradation.

I understand your point, but the performance issues still stem off of poor IO performance on Amazon EBS. As we continue to use it, we continue to find it to be the source of most people's woes. If you have solid (even reasonable) IO, then moving things in and out of working memory is not painful. We have some customers on non-EBS spindles that have very large working sets (as compared to memory) ... faulting 400-500…

We are using Mongo in ec2 and raid10 with 6 ebs drives out performs ephemeral disks when the dataset won't fit in RAM in a raw upsert scenario (our actual data, loading in historical data). The use if mmap and relying on the OS to page in/out the appropriate portions is painful, particularly because we end up with a lot of moves (padding factor varies between 1.8 and 1.9 and because of our dataset, using a large field on insert and clearing in update was less performant than upserts and moves).

There's really two knobs to turn on Mongo, RAM and disk speed. Our particular cluster doesn't have enough RAM for the dataset to fit in memory, but could double its performance (or more) if each key range was mmapped individually rather than the entire datastore the shard is responsible for just because of how the OS manages pages. We haven't broken down to implement it yet, but with the performance vs. cost tradeoffs, we may have to pretty soon.

Re: A Year with MongoDB

#82

Post by Jare »

bradgessler wrote: post #71
raverbashing wrote: post #68

Earlier quoted context omitted.

Absolutely not!! It's more like this You have old user table with for example: login and user name In MongoDB this is a JSON object {login:'user', name:'User Name'} You want to add 'shoe size'. So you add 1 - the shoe size to the signup/user editing form 2 - next user created is like this: {login:'user', name:'User Name', 'shoe_size': 7} 3 - Old users DON'T get a shoe_size added automatically to their document, but n…

That's still a migration, its just incremental.

Incremental can make all the difference between zero- and hours of downtime. Do not underestimate the importance of this extra agility in the modern world of almost daily updates to web apps and backends.

Re: A Year with MongoDB

#83

Post by Jebus »

Uncompressed field names - If you store 1,000 documents with the key “foo”, then “foo” is stored 1,000 times in your data set

Oh my god. I didn't know about this. And I hate short, meaningless and anti-intuitive field names. Please fix it mongodb devs!

Re: A Year with MongoDB

#84

Post by AdrianRossouw »

mtrn wrote: post #35
luca_garulli wrote: post #27

Hey, by reading all the bad things seems that OrientDB would fit better than MongoDB for them: - Non-counting B-Trees: OrientDB uses MVRB-Tree that has the counter. size() requires 0ns - Poor Memory Management: OrientDB uses MMAP too but with many settings to optimize it usage - Uncompressed field names: the same as OrientDB - Global write lock: this kills your concurrency! OrientDB handles read/write locks at segmen…

Thanks for writing OrientDB! - I tried it, but I was pressed for time, so I needed something that more or less worked instantly for my requirements - which in the end was elasticsearch. TL; I researched MongoDB and OrientDB for a side-project with a bit heavy data structure (10M+ docs, 800+ fields on two to three levels). MongoDB was blazingly fast, but it segfaulted somewhere in the process (also index creation need…

i love ES, but i don't really feel comfortable with it as a primary datastore. We tend to use couchdb to write to, and ES to query against. It all happens automagically with a single shell command.

I won't use ES on it's own, because I have experienced situations in the past where the dynamic type mapping functionality gets confused, ie: the first time it sees a field, it indexes it as an integer, but then one of the later records has 'n/a' instead of a number. The entire record became unquery-able after that, even if it might have stored the original data.

You could fix this by creating the mapping by hand, BEFORE any data has been imported, as it can't be modified later. But what you have then is a situation where you have to maintain a schema to not get it to 'randomly' ignore data.

You also can't just tell ES to rebuild an index when you need to mess with the mappings, you have to actually create a new index, change the mappings and then reimport the data into the new index (possibly from the existing index).

It actually also feels right to me to split storing the data versus querying the data between separate applications, because they have different enough concerns, that being able to scale them out differently is a boon sometimes.

Re: A Year with MongoDB

#85

Post by justinsb »

corford wrote: post #56
justinsb wrote: post #26

Earlier quoted context omitted.

Care to explain? I believe for Redis, "appendfsync everysec" is the default. The poster's point was that MySQL and Postgres both ship with something like "appendfsync always", and you have to opt-in to the the less safe mode if you want to get more performance. Redis ships with the less safe mode pre-selected, and so has higher performance out-of-the-box.

You're right. The Postgres equivalent to "appendfsync always" is "synchronous_commit = on". Which AFAIK is the default. However, one of the nice things about redis is that even if you run "appendfsync everysec" you never run the risk of corruption. You're only risk is losing a maximum of 2 seconds worth of data. If you missed it, there's a wonderful blog post by antirez covering all of this (and a lot more) here: htt…

All modern relational databases use this approach (it's called ARIES), so all offer the same guarantees as Redis.

Re: A Year with MongoDB

#86

Post by trimbo »

radicalbyte wrote: post #25
sirn wrote: post #13

> We changed the structure of our heaviest used models a couple times in the past year, and instead of going back and updating millions of old documents, we simply added a “version” field to the document and the application handled the logic of reading both the old and new version. This flexibility was useful for both application developers and operations engineers. Ugh, this sounds like a maintenance nightmare. How…

> oh, so you're implementing schema in your application? Isn't that where the schema belongs? Each document represents a conceptual whole. It doesn't contain fields which have to be NULL simply because they weren't in previous versions of the schema. I've been an rdbms guy (datawarehousing/ETL) for a long time now, I've seen a lot of large databases which have been in production for considerable time. They get messy.…

> isn't that where the schema belongs? [In the application]

Well, unless you have, you know, multiple applications accessing said data. Then it's kind of important to keep it in sync, which is why RDBMS exist and operate the way they do.

In my experience, on a long enough timeline, the probability of needing multi-application access for your data goes to 1.

Re: A Year with MongoDB

#87

Post by chx »

rdtsc wrote: post #10

> Safe off by default I think that is fixed now. But this the single most appalling design decision they could have made while also claiming their product was a "database". (And this has been discussed here before so just do a search if you will). This wasn't a bug, it was a deliberate design decision. Ok, that that would have been alright if they put a bright red warning on their front page. "We disabled durability…

Quote the manual" the option is --journal, and is on by default in version 1.9.2+ on 64-bit platforms"

Re: A Year with MongoDB

#88

Post by taligent »

tolitius wrote: post #74

great post. direct and to the point, although there are many more flaws that I am sure you could have shared. we tried MongoDB to consume and analyze market feeds, and it failed miserably. I can add a couple of things to your list: * if there is a pending write due to an fsync lock, all reads are blocked: https://jira.mongodb.org/browse/SERVER-4243 * data loss + 10gen's white lies: https://jira.mongodb.org/browse/SER…

MongoDB is successful because of more than just marketing.

It has great tool support, decent documentation, books and is accessible. Plus the whole transition from MySQL concept makes it easy to grab onto.

Re: A Year with MongoDB

#89

Post by moe »

justinsb wrote: post #85
corford wrote: post #56

Earlier quoted context omitted.

You're right. The Postgres equivalent to "appendfsync always" is "synchronous_commit = on". Which AFAIK is the default. However, one of the nice things about redis is that even if you run "appendfsync everysec" you never run the risk of corruption. You're only risk is losing a maximum of 2 seconds worth of data. If you missed it, there's a wonderful blog post by antirez covering all of this (and a lot more) here: htt…

All modern relational databases use this approach (it's called ARIES), so all offer the same guarantees as Redis.

And to praise redis some more; it also comes with the sanest (async) replication support of any database. By far.

It's literally a one-liner in the config file. No bootstrap needed, maintenance free, absolutely no strings attached. 10 seconds and you're done.

Which means you'll actually use it from day 1 and never worry about it. Compare that to any other database.

Re: A Year with MongoDB

#90

Post by latch »

jbellis wrote: post #14
rdtsc wrote: post #10

> Safe off by default I think that is fixed now. But this the single most appalling design decision they could have made while also claiming their product was a "database". (And this has been discussed here before so just do a search if you will). This wasn't a bug, it was a deliberate design decision. Ok, that that would have been alright if they put a bright red warning on their front page. "We disabled durability…

According to http://www.mongodb.org/display/DOCS/getLastError+Command it is still unsafe by default.

No, the journal file is turned on server-side and you'll get a journal append every 100ms (by default).
Post reply on HN