Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Year with MongoDB

blog.engineering.kiip.me

Post reply on HN
Print view View original

141–150 of 153 posts

Re: A Year with MongoDB

#141

Post by jeffdavis »

quadhome wrote: post #139
jeffdavis wrote: post #129

Earlier quoted context omitted.

> Thus, "build an api" is the best solution. I think there's an asymmetry in your argument. You are describing all of the problems with data management as though they were specific to schemas in a traditional RDBMS; but glossing over how "building an API" solves those same problems, and whether that method is better or worse. In other words, "build an API" is the problem at hand, not the solution. A traditional DBMS…

Yes, I agree, "build an API" is the problem. I specifically said the problem and the complexity inherent in solving it doesn't disappear with per-application databases. But, the application (the 'A' part) is where the most context around the information is to be found. Tying multiple applications together at the Data-side (rather than the Application-side) means you don't lose that context... and eliminate your chanc…

That's a good point about the context, but not very concrete.

At some point you'll need to go into the details, but I don't think a comment is the right format. Maybe a blog post to show exactly what you mean.

Re: A Year with MongoDB

#142

Post by [deleted] »

justinsb wrote: post #26
antirez wrote: post #11

Earlier quoted context omitted.

You are wrong about Redis. Cheers

Care to explain? I believe for Redis, "appendfsync everysec" is the default. The poster's point was that MySQL and Postgres both ship with something like "appendfsync always", and you have to opt-in to the the less safe mode if you want to get more performance. Redis ships with the less safe mode pre-selected, and so has higher performance out-of-the-box.

[deleted]

Re: A Year with MongoDB

#143

Post by antirez »

justinsb wrote: post #26
antirez wrote: post #11

Earlier quoted context omitted.

You are wrong about Redis. Cheers

Care to explain? I believe for Redis, "appendfsync everysec" is the default. The poster's point was that MySQL and Postgres both ship with something like "appendfsync always", and you have to opt-in to the the less safe mode if you want to get more performance. Redis ships with the less safe mode pre-selected, and so has higher performance out-of-the-box.

What I mean is simply that the default configuration file does not matter, but the point is, what are the guarantees and tradeoffs that a database system can provide? and Redis is no different than other non-in-memory DBs about this. To put Redis apart because in the default configuration file we just enable RDB persistence seems odd, it's like if with a one line commit tomorrow I "change Redis" because I alter the default configuration to AOF + fsync always. Obviously the system remains exactly the same, so systems should be compared by capabilities and not by default config files.

Re: A Year with MongoDB

#144

Post by reddiric »

raverbashing wrote: post #68
judofyr wrote: post #40

Earlier quoted context omitted.

So it's actually 100% the same as you would do with on-the-fly migrations in SQL: (1) Add column and add code moves the old data when you access it. Deploy. (2) Let it run for a while. Run a background job that migrates the rest (this might be done days or months later). (3) Remove the column and the custom code. The more I hear about "schemaless" the more I realize that it doesn't make any difference at all.

Absolutely not!! It's more like this You have old user table with for example: login and user name In MongoDB this is a JSON object {login:'user', name:'User Name'} You want to add 'shoe size'. So you add 1 - the shoe size to the signup/user editing form 2 - next user created is like this: {login:'user', name:'User Name', 'shoe_size': 7} 3 - Old users DON'T get a shoe_size added automatically to their document, but n…

How is that different than adding a nullable column shoe_size to the user table in your relational DB, and an if(user.shoe_size != null) in your application?

Re: A Year with MongoDB

#146

Post by gatesvp1 »

chx wrote: post #87
rdtsc wrote: post #10

> Safe off by default I think that is fixed now. But this the single most appalling design decision they could have made while also claiming their product was a "database". (And this has been discussed here before so just do a search if you will). This wasn't a bug, it was a deliberate design decision. Ok, that that would have been alright if they put a bright red warning on their front page. "We disabled durability…

Quote the manual" the option is --journal, and is on by default in version 1.9.2+ on 64-bit platforms"

So that's not actually "safe". If you issue an insert in the default "fire and forget" mode and that insert causes an error (say a duplicate key violation), no exception will be thrown.

Even with journaling on your code does not get an exception.

Journaling is a method for doing "fast recovery" and flushing to disk on a regular basis. "Write Safety" is method for controlling how / where the data has been written. So these are really two different things.

Re: A Year with MongoDB

#147

Post by gatesvp1 »

bitops wrote: post #24
sirn wrote: post #13

> We changed the structure of our heaviest used models a couple times in the past year, and instead of going back and updating millions of old documents, we simply added a “version” field to the document and the application handled the logic of reading both the old and new version. This flexibility was useful for both application developers and operations engineers. Ugh, this sounds like a maintenance nightmare. How…

> I'll admit I'm a non-believer, but every time I see "Schemaless" in MongoDB, I think "oh, so you're implementing schema in your application?" I think that is arguably one of the selling points of MongoDB. Yes, you do implement schema in your application, but should be doing that knowingly and embracing both the costs and benefits. The benefit is that you can very quickly change your "schema" since it's just however…

> A more "Mongo" approach is to migrate data as you need to; e.g. you pull in an older document and at that time add any missing fields.

I think this works when you're talking about adding fields. But it really breaks down if you need to make some change regarding "sub-objects" or arrays of "sub-objects".

If you have made a modeling mistake and you need to pull out a sub-object you generally have to do simply stop the system and migrate.

Re: A Year with MongoDB

#148

Post by gatesvp1 »

latch wrote: post #96
mrkurt wrote: post #45

Part of the lesson here is that if you're doing MongoDB on EC2, you should have more than enough RAM for your working set. EBS is pretty bad underlying IO for databases, so you should treat your drives more as a relatively cold storage engine. This is the primary reason we're moving the bulk of our database ops to real hardware with real arrays (and Fusion IO cards for the cool kids). We have a direct connect to Amaz…

I'm not sure why people are using EBS with their databases. If you already have replication properly set up, what does it buy you except for performance problems? Chris Westin, of 10gen, blogged about this a while ago: https://www.bookofbrilliantthings.com/blog/what-is-amazon-eb... In fairness though, 10gen's official stance is to use EBS. I think that's a mistake, and I think maybe they do it for extra safety.

The big thing here is cost.

> If you put the above rules together, you can see that the minimum MySQL deployment is four servers: two in each of two colos...

The ideal scenario is to have 4 "fully equipped" nodes, 2 in each data center. That means having 3 pieces of expensive "by the hour" hardware sitting around doing basically nothing. (and paying 4-5k / computer for MongoDB licenses)

In that scenario you can have everything on instance store and live with 4 copies on volatile storage.

Of course, no start-up wants to commit that many resources to a project. It's far cheaper just to use EBS and assume that the data there is "safe". Is it bad practice, would I avoid EBS like the plague? You bet!

But it's definitely cheaper and that's hard to beat.

Re: A Year with MongoDB

#149

Post by gatesvp1 »

aschobel wrote: post #38

We love MongoDB at Catch, it's been our primary backing store for all user data for over 20 months now. > Catch.com > Data Size: 50GB > Total Documents 27,000,000 > Operations per second: 450 (Create, reads, updates, etc.) > Lock % average 0% > CPU load average 0% Global Lock isn't ideal, but Mongo is so fast it hasn't been an issue for us. You need to keep on slow queries and design your schema and indexes correctly…

Not a personal attack as Catch is a neat product, but these numbers are basically irrelevant.

This type of load can easily be handled by a simple SQL box. We did these types of #s with a single SQL Server box 4 years ago, except that your "total documents" was our daily write load.

Re: A Year with MongoDB

#150

Post by gatesvp1 »

Jebus wrote: post #83

Uncompressed field names - If you store 1,000 documents with the key “foo”, then “foo” is stored 1,000 times in your data set Oh my god. I didn't know about this. And I hate short, meaningless and anti-intuitive field names. Please fix it mongodb devs!

This is a long-outstanding bug, over 2 years old now:

https://jira.mongodb.org/browse/SERVER-863

Obviously you can vote for them to fix the bug. But it's been two years, so I'm not sure it's really high on the priority list.

Post reply on HN