Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Year with MongoDB

blog.engineering.kiip.me

Post reply on HN
Print view View original

61–70 of 153 posts

Re: A Year with MongoDB

#61

Post by aaronblohowiak »

siavosh wrote: post #31
rdtsc wrote: post #10

> Safe off by default I think that is fixed now. But this the single most appalling design decision they could have made while also claiming their product was a "database". (And this has been discussed here before so just do a search if you will). This wasn't a bug, it was a deliberate design decision. Ok, that that would have been alright if they put a bright red warning on their front page. "We disabled durability…

I would bet that there are many apps out there that don't do any reconciliation, and the data will forever be lost unknown. Sadly only 1% of the customers will notice something weird, 0.001% of those will call support saying something is off, and then 99% of those calls will be ignored as customer incompetence. Scary indeed.

so one in ten million data loss events will be acknowledged by 10gen?

Re: A Year with MongoDB

#62

Post by radicalbyte »

alexchamberlain wrote: post #36
radicalbyte wrote: post #25

Earlier quoted context omitted.

> oh, so you're implementing schema in your application? Isn't that where the schema belongs? Each document represents a conceptual whole. It doesn't contain fields which have to be NULL simply because they weren't in previous versions of the schema. I've been an rdbms guy (datawarehousing/ETL) for a long time now, I've seen a lot of large databases which have been in production for considerable time. They get messy.…

I am a schemaless believer. BUT (big BUT) have you ever tried to program to a schema that has a XML blob? It's really difficult and painfully slow on the app side. The ops guys like it, because it's easy for them to maintain.

I find it much easier, but then I rely on a framework for serialization/de-serialization..

Re: A Year with MongoDB

#63

Post by lalmalang »

guywithabike wrote: post #59
aschobel wrote: post #38

We love MongoDB at Catch, it's been our primary backing store for all user data for over 20 months now. > Catch.com > Data Size: 50GB > Total Documents 27,000,000 > Operations per second: 450 (Create, reads, updates, etc.) > Lock % average 0% > CPU load average 0% Global Lock isn't ideal, but Mongo is so fast it hasn't been an issue for us. You need to keep on slow queries and design your schema and indexes correctly…

450 ops/sec is nothing . What's your breakdown between the operation types, and what kind of hardware are you on?

You'd think -- but the 10gen guys weren't surprised when we were struggling at this level (periodically), on a RS with two AWS large instances and relatively large objects. Absolute ops/sec in and of itself is relatively meaningless tbh.

Re: A Year with MongoDB

#64

Post by strlen »

aschobel wrote: post #38

We love MongoDB at Catch, it's been our primary backing store for all user data for over 20 months now. > Catch.com > Data Size: 50GB > Total Documents 27,000,000 > Operations per second: 450 (Create, reads, updates, etc.) > Lock % average 0% > CPU load average 0% Global Lock isn't ideal, but Mongo is so fast it hasn't been an issue for us. You need to keep on slow queries and design your schema and indexes correctly…

> Data Size: 50GB

I am sorry, to sound blunt, but that's an irrelevant data point. With a data set that fits comfortably into RAM (much less SSDs in RAID!), most any data store will work (including MySQL or Postgres).

> Operations per second: 450

Again, not a relevant data point. With a 10 ms seek time on a SATA disk, this is (again) well within the IOPS capacity of a single commodity machine (with RAID, a SAS drive, row cache, and operating system's elevator scheduling).

Re: A Year with MongoDB

#65

Post by brey »

Philadelphia wrote: post #33

The impression I got after hearing some of the 10gen developers speak at a conference is that MongoDB has the same essential problem as PHP. It was written by people without a lot of formal knowledge who, for whatever reason, aren't interested in researching what's been tried before, what works, and what doesn't. Because of that, they're always trying to reinvent the wheel, and make flawed design decisions that keep…

which conference? I was at the recent Stockholm conference, had a lengthy 1-1 with one of their kernel devs, and I picked up the opposite impression.

Re: A Year with MongoDB

#66

Post by mnutt »

LeafStorm wrote: post #5
jrussbowman wrote: post #3

From the beginning I've understood mongodb to be built with it's approach for scaling, performance, redundancy and backup to be horizontal scaling. They recently added journaling for single server durability, but before that replication was how you made sure you data was safe. It seems to me when I see complaints about mongodb it's because people don't want to horizontally scale it and instead believe vertical scalin…

While I agree that you should definitely use your tools in the best way you're capable of, I think for most people there's a baseline expectation that if you save data in a database, that data will be safe (at the very least, recoverable) unless something happens like the server catching on fire. Nearly every other major database has this as the default -- MySQL, PostgreSQL, CouchDB, and Berkeley DB to name a few. (R…

There's actually two separate concerns:

Mongo client's "safe" operator causes the client to see if the database threw an error and throw an error itself. Like someone mentioned, it mostly falls on the mongo clients to implement this. We mostly use fire-and-forget for our application, since it is just logging stats and speed is more important than losing an increment here or there. There should probably be better documentation telling people to always use the safe operations for important data.

There is also the durability issue. Early versions shipped with durability turned off by default and required replication to maintain durability. Mongo has had the journal feature since 1.8 and has it enabled by default since 1.9.2. (current version is 2.0)

So while mongo has definitely been unsafe in the past, both kinds of safety are now supported, and one is default. The other is either not that big a deal or egregious, depending on the way you view mongo.

Re: A Year with MongoDB

#67

Post by madworld »

jalons wrote: post #44

Am I missing something, or did they say they didn't want to scale mongo horizontally via sharding, then comment that they're doing so with riak, but faulting mongodb for requiring it?

What they are doing with Riak isn't sharding. Riak from the ground up was been designed as a distributed database. They didn't want to go horizontal when really they shouldn't have to with their datasize based on Mongo's claims. The problem is, Mongo lies about what their database can do, and the fact that Kiip figured that out is why they didn't want to bother scaling out with mongo as a band-aid for its problems. It was better for them to just use something made to scale. That's how I read it, based on that blog and by his comments on this post.

Re: A Year with MongoDB

#68

Post by raverbashing »

judofyr wrote: post #40
mitchellh wrote: post #15

Earlier quoted context omitted.

> Ugh, this sounds like a maintenance nightmare. How do you deal with adding extra field to the document? Do you ever feel the need of running on-the-fly migration of old versions? (But when you do, shouldn't running a migration for all documents a better idea?) Yes, we did on-the-fly migration as we loaded old data in. Doing full data migration was not really an option because querying from MongoDB on un-indexed dat…

So it's actually 100% the same as you would do with on-the-fly migrations in SQL: (1) Add column and add code moves the old data when you access it. Deploy. (2) Let it run for a while. Run a background job that migrates the rest (this might be done days or months later). (3) Remove the column and the custom code. The more I hear about "schemaless" the more I realize that it doesn't make any difference at all.

Absolutely not!!

It's more like this

You have old user table with for example: login and user name

In MongoDB this is a JSON object {login:'user', name:'User Name'}

You want to add 'shoe size'. So you add

1 - the shoe size to the signup/user editing form

2 - next user created is like this: {login:'user', name:'User Name', 'shoe_size': 7}

3 - Old users DON'T get a shoe_size added automatically to their document, but next time they login they get asked "What's your shoe size". It's dynamic. if (!user.shoe_size), done.

You add new columns on demand, and you almost never remove them (you don't need to add fields for migrating, except of course those you want to add)

You (almost never) "run the migration" for a while adding shoe_size to existing documents.

And it absolutely does make a huge difference, since you don't have to deal with the mess of relational DB migrations.

I'll never develop a system using a relational DB again if I can avoid.

Re: A Year with MongoDB

#69

Post by jasonmccay »

lalmalang wrote: post #53

We had a very similar situation ~300 writes per second on AWS. but I suspect some of this has to do with the fact that most people address scaling by adding a replica set, rather than the much hairier sharding setup ( http://www.mongodb.org/display/DOCS/Sharding+Introduction ), this seems natural b/c mongodb's 'scalability' is often touted. In reality though, because of the lock, RS dont really address the problem mu…

Again ... this is more an indictment on the poor IO performance of Amazon EBS vs. MongoDB as a solution. MongoDB can scale both vertically and horizontally, but as with anything you scale on Amazon infrastructure, you are going to have to really think through strategies for dealing with the unpredictable performance of EBS. There are blog posts galore addressing this fact.

I often think MongoDB has suffered more as a young technology because of the proliferation of the AWS Cloud and the expectations of EBS performance.

Re: A Year with MongoDB

#70

Post by lalmalang »

jasonmccay wrote: post #69
lalmalang wrote: post #53

We had a very similar situation ~300 writes per second on AWS. but I suspect some of this has to do with the fact that most people address scaling by adding a replica set, rather than the much hairier sharding setup ( http://www.mongodb.org/display/DOCS/Sharding+Introduction ), this seems natural b/c mongodb's 'scalability' is often touted. In reality though, because of the lock, RS dont really address the problem mu…

Again ... this is more an indictment on the poor IO performance of Amazon EBS vs. MongoDB as a solution. MongoDB can scale both vertically and horizontally, but as with anything you scale on Amazon infrastructure, you are going to have to really think through strategies for dealing with the unpredictable performance of EBS. There are blog posts galore addressing this fact. I often think MongoDB has suffered more as a…

In fact, I tried on-instance storage too -- this didnt help substantially. The reality is that many (most?) stacks these days need to be able live happily on AWS...
Post reply on HN