Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Year with MongoDB

blog.engineering.kiip.me

Post reply on HN
Print view View original

31–40 of 153 posts

Re: A Year with MongoDB

#31

Post by siavosh »

rdtsc wrote: post #10

> Safe off by default I think that is fixed now. But this the single most appalling design decision they could have made while also claiming their product was a "database". (And this has been discussed here before so just do a search if you will). This wasn't a bug, it was a deliberate design decision. Ok, that that would have been alright if they put a bright red warning on their front page. "We disabled durability…

I would bet that there are many apps out there that don't do any reconciliation, and the data will forever be lost unknown. Sadly only 1% of the customers will notice something weird, 0.001% of those will call support saying something is off, and then 99% of those calls will be ignored as customer incompetence. Scary indeed.

Re: A Year with MongoDB

#32

Post by gregbair »

If you're going to have a (management|engineering|whatever) blog for your company/project, have a link to your company/project home page prominently somewhere on the blog.

Re: A Year with MongoDB

#33

Post by Philadelphia »

The impression I got after hearing some of the 10gen developers speak at a conference is that MongoDB has the same essential problem as PHP. It was written by people without a lot of formal knowledge who, for whatever reason, aren't interested in researching what's been tried before, what works, and what doesn't. Because of that, they're always trying to reinvent the wheel, and make flawed design decisions that keep causing problems.

Re: A Year with MongoDB

#35

Post by mtrn »

luca_garulli wrote: post #27

Hey, by reading all the bad things seems that OrientDB would fit better than MongoDB for them: - Non-counting B-Trees: OrientDB uses MVRB-Tree that has the counter. size() requires 0ns - Poor Memory Management: OrientDB uses MMAP too but with many settings to optimize it usage - Uncompressed field names: the same as OrientDB - Global write lock: this kills your concurrency! OrientDB handles read/write locks at segmen…

Thanks for writing OrientDB! - I tried it, but I was pressed for time, so I needed something that more or less worked instantly for my requirements - which in the end was elasticsearch.

TL;

I researched MongoDB and OrientDB for a side-project with a bit heavy data structure (10M+ docs, 800+ fields on two to three levels). MongoDB was blazingly fast, but it segfaulted somewhere in the process (also index creation needs extra time and isn't really ad-hoc). OrientDB wasn't as fast and a little harder to do the initial setup but the inserting speed was ok - for a while (500k docs or so) and then it degraded. I also looked at CouchDB, but I somehow missed the ad-hoc query infrastructure.

My current solution, which works nice for the moment is elasticsearch; it's fast - and it's possible to get a prototype from 0 to 10M docs in about 50 minutes - or less, if you load balance the bulk inserts on a cluster - which is so easy to setup it's scary - and then let a full copy of the data settle on each machine in the background.

Disclaimer - since this is a side project, I did only minimal research on each of the technologies (call it 5 minute test) and ES clearly won the first round over both MongoDB and OrientDB.

Re: A Year with MongoDB

#36

Post by alexchamberlain »

radicalbyte wrote: post #25
sirn wrote: post #13

> We changed the structure of our heaviest used models a couple times in the past year, and instead of going back and updating millions of old documents, we simply added a “version” field to the document and the application handled the logic of reading both the old and new version. This flexibility was useful for both application developers and operations engineers. Ugh, this sounds like a maintenance nightmare. How…

> oh, so you're implementing schema in your application? Isn't that where the schema belongs? Each document represents a conceptual whole. It doesn't contain fields which have to be NULL simply because they weren't in previous versions of the schema. I've been an rdbms guy (datawarehousing/ETL) for a long time now, I've seen a lot of large databases which have been in production for considerable time. They get messy.…

I am a schemaless believer.

BUT (big BUT) have you ever tried to program to a schema that has a XML blob? It's really difficult and painfully slow on the app side. The ops guys like it, because it's easy for them to maintain.

Re: A Year with MongoDB

#37

Post by mitchellh »

fusiongyro wrote: post #30
mitchellh wrote: post #7

Earlier quoted context omitted.

[Note: I wrote the blog post] I'm not at all against horizontally scaling. However, I don't believe that horizontally scaling should be necessary doing a mere 200 updates to per second to a data store that isn't even fsyncing writes to disk. Think of it in terms of cost per ops. Let's just say 200 update ops per second is the point at which you need to shard (not scientific, but let's just use that as a benchmark sin…

I like your post, and I agree with your conclusion, but I have to say I'm puzzled by your decision to back MongoDB with EBS. Were you running MongoDB atop EC2 instances as well? Can you elaborate on this a little?

We were running running MongoDB atop EC2 instances. We chose to back MongoDB with EBS because that was the only reasonable way to get base backups (via snapshots) of the database. Although 10gen recommends using replica sets for backup, we also wanted a way to durably backup our database since there was so much important data in it (user accounts, billing, and so on).

On the other hand, we run PostgreSQL straight on top of a RAID of ephemeral drives, which has had good throughput compared to EBS so far. The reason we're able to do this is because PostgreSQL provides a mechanism for doing base backups safely without having to snapshot the disk[1]. Therefore, we just do an S3 base backup of our entire data (which uploads at 65 MB/s from within EC2) fairly infrequently, while doing WAL shipping very frequently.

[1]: http://www.postgresql.org/docs/8.1/static/backup-online.html

Re: A Year with MongoDB

#38

Post by aschobel »

We love MongoDB at Catch, it's been our primary backing store for all user data for over 20 months now.

  > Catch.com
  > Data Size: 50GB
  > Total Documents 27,000,000
  > Operations per second: 450 (Create, reads, updates, etc.)
  > Lock % average 0%
  > CPU load average 0%
Global Lock isn't ideal, but Mongo is so fast it hasn't been an issue for us. You need to keep on slow queries and design your schema and indexes correctly.

We don't want page faults on indexes, we design them to keep them in memory.

I don't get the safety issue, 20 months and we haven't lost any user data. shrug

Re: A Year with MongoDB

#39

Post by bretthoerner »

aschobel wrote: post #38

We love MongoDB at Catch, it's been our primary backing store for all user data for over 20 months now. > Catch.com > Data Size: 50GB > Total Documents 27,000,000 > Operations per second: 450 (Create, reads, updates, etc.) > Lock % average 0% > CPU load average 0% Global Lock isn't ideal, but Mongo is so fast it hasn't been an issue for us. You need to keep on slow queries and design your schema and indexes correctly…

> I don't get the safety issue, 20 months and we haven't lost any user data. shrug

Nobody loses any user data until they do.

Re: A Year with MongoDB

#40

Post by judofyr »

mitchellh wrote: post #15
sirn wrote: post #13

> We changed the structure of our heaviest used models a couple times in the past year, and instead of going back and updating millions of old documents, we simply added a “version” field to the document and the application handled the logic of reading both the old and new version. This flexibility was useful for both application developers and operations engineers. Ugh, this sounds like a maintenance nightmare. How…

> Ugh, this sounds like a maintenance nightmare. How do you deal with adding extra field to the document? Do you ever feel the need of running on-the-fly migration of old versions? (But when you do, shouldn't running a migration for all documents a better idea?) Yes, we did on-the-fly migration as we loaded old data in. Doing full data migration was not really an option because querying from MongoDB on un-indexed dat…

So it's actually 100% the same as you would do with on-the-fly migrations in SQL:

(1) Add column and add code moves the old data when you access it. Deploy.

(2) Let it run for a while. Run a background job that migrates the rest (this might be done days or months later).

(3) Remove the column and the custom code.

The more I hear about "schemaless" the more I realize that it doesn't make any difference at all.

Post reply on HN