Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Year with MongoDB

blog.engineering.kiip.me

Post reply on HN
Print view View original

91–100 of 153 posts

Re: A Year with MongoDB

#91

Post by matthewcford »

trimbo wrote: post #86
radicalbyte wrote: post #25

Earlier quoted context omitted.

> oh, so you're implementing schema in your application? Isn't that where the schema belongs? Each document represents a conceptual whole. It doesn't contain fields which have to be NULL simply because they weren't in previous versions of the schema. I've been an rdbms guy (datawarehousing/ETL) for a long time now, I've seen a lot of large databases which have been in production for considerable time. They get messy.…

> isn't that where the schema belongs? [In the application] Well, unless you have, you know, multiple applications accessing said data. Then it's kind of important to keep it in sync, which is why RDBMS exist and operate the way they do. In my experience, on a long enough timeline, the probability of needing multi-application access for your data goes to 1.

build an api

Re: A Year with MongoDB

#92

Post by mbreese »

matthewcford wrote: post #91
trimbo wrote: post #86

Earlier quoted context omitted.

> isn't that where the schema belongs? [In the application] Well, unless you have, you know, multiple applications accessing said data. Then it's kind of important to keep it in sync, which is why RDBMS exist and operate the way they do. In my experience, on a long enough timeline, the probability of needing multi-application access for your data goes to 1.

build an api

So you want to put an API in place to access your schema-less database? So that multiple applications can access your database in a consistent manner? That makes sense.

You know what though... you may want to use multiple programming languages, and maintaining an API in multiple languages sucks. So, why not just make a service that each application can connect to, and then let it talk to your database. Then you just need to define a simple protocol that language can use to connect to your service.

Or, you could just skip a few steps and use an actual database to begin with.

Re: A Year with MongoDB

#93

Post by quadhome »

matthewcford wrote: post #91
trimbo wrote: post #86

Earlier quoted context omitted.

> isn't that where the schema belongs? [In the application] Well, unless you have, you know, multiple applications accessing said data. Then it's kind of important to keep it in sync, which is why RDBMS exist and operate the way they do. In my experience, on a long enough timeline, the probability of needing multi-application access for your data goes to 1.

build an api

It's a shame this comment is pithy, because I think it's dead-on.

There times to integrate at the database level. But, the default should be single-application databases.

The rationale is the same as the grandparent's rationale FOR database integration. The odds of needing to share data over time are 1.

Given that shared belief, the problem with database integration is that MANY applications need to share facets of the same data. The single database ends up having a huge surface area trying to satisfy every application's needs.

The resulting schema will have definitions relevant for applications A, B, and C but X, Y and Z.

But, worse, there are dependencies between each application's working schema. This means ensuring integrity becomes harder with every application that integrates.

Finally, integration points are the hardest to change after-the-fact. The more services that integrate, the less ability to make the inevitable model changes necessary to fix mistakes/scale/normalize/denormalize/change solutions.

Thus, "build an api" is the best solution. Well-defined APIs and data-flows between applications helps data and process locality and avoids most of the problems I just listed. The trade-off is you're now meta-programming at the process level— the complexity doesn't disappear, it's just reconceptualised.

Re: A Year with MongoDB

#94

Post by birken »

pbiggar wrote: post #78
bradgessler wrote: post #71

Earlier quoted context omitted.

That's still a migration, its just incremental.

It's the same concept, but it's very different in practice. For example, adding a field with a default value in postgres will lock the table during the migration, which may be killer. If you use postgres for big data sets, what you'll end up implementing for your migrations looks a lot like what schemaless gives you for free.

If you add a default value it locks the table and re-writes it.

However, if you don't add a default value postgres will perform the operation instantly. Old rows will be untouched, and new values will be added with the shoe_size of either NULL or whatever you set it to be... IE exactly the same outcome and performance as the MongoDB case mentioned above.

Adding a field in postgres while setting a default value would be exactly the same as adding a field in MongoDB and updating every existing row with that default value (except for the fact that postgres will only lock that one table, while MongoDB will lock your entire database).

Now I'm not anti-MongoDB, I'm just saying you shouldn't give it credit for something that relational database do just fine.

Re: A Year with MongoDB

#95

Post by latch »

Philadelphia wrote: post #33

The impression I got after hearing some of the 10gen developers speak at a conference is that MongoDB has the same essential problem as PHP. It was written by people without a lot of formal knowledge who, for whatever reason, aren't interested in researching what's been tried before, what works, and what doesn't. Because of that, they're always trying to reinvent the wheel, and make flawed design decisions that keep…

One of the silliest things I've read on HN, largely because of how insulting it is, yet how easy it is to verify.

Rather than making a comment that can be summed up as "i think it's written by people who don't know what they are doing", why not Google them and find out what their background is?

The developers at the conference might not have represented the kernel team (it's a surprisingly large company with a lot of different development branches (core, drivers, tools, support)).

Re: A Year with MongoDB

#96

Post by latch »

mrkurt wrote: post #45

Part of the lesson here is that if you're doing MongoDB on EC2, you should have more than enough RAM for your working set. EBS is pretty bad underlying IO for databases, so you should treat your drives more as a relatively cold storage engine. This is the primary reason we're moving the bulk of our database ops to real hardware with real arrays (and Fusion IO cards for the cool kids). We have a direct connect to Amaz…

I'm not sure why people are using EBS with their databases. If you already have replication properly set up, what does it buy you except for performance problems?

Chris Westin, of 10gen, blogged about this a while ago: https://www.bookofbrilliantthings.com/blog/what-is-amazon-eb...

In fairness though, 10gen's official stance is to use EBS. I think that's a mistake, and I think maybe they do it for extra safety.

Re: A Year with MongoDB

#97

Post by ericflo »

jasonmccay wrote: post #51
ericflo wrote: post #9

Earlier quoted context omitted.

Mongo was not designed with horizontal scaling in mind. Riak, Cassandra, HBase, Project Voldemort...these are the projects that were designed with horizontal scaling in mind (as evidenced by their architectures.) But not Mongo.

I have to respectfully disagree. Your comment is a bit sweeping and I really don't think that MongoDB's sharding solution is a bad one ... simply the strategies are different. There is a large set of nice features that makes Mongo, for most people, nice to use long before you even need to address sharding. The percentage of people that will need to shard is much lower than the percentage of people that can get consid…

I am definitely a Riak-lover :) And I also agree that there is a large set of nice features that make Mongo nice to use before sharding.

Re: A Year with MongoDB

#98

Post by verroq »

mbreese wrote: post #92
matthewcford wrote: post #91

Earlier quoted context omitted.

build an api

So you want to put an API in place to access your schema-less database? So that multiple applications can access your database in a consistent manner? That makes sense. You know what though... you may want to use multiple programming languages, and maintaining an API in multiple languages sucks. So, why not just make a service that each application can connect to, and then let it talk to your database. Then you just…

Building an API is still a great choice no matter how good your database is. Why would you not have an API if you are using the data store from 5 difference languages?

Re: A Year with MongoDB

#99

Post by pbiggar »

birken wrote: post #94
pbiggar wrote: post #78

Earlier quoted context omitted.

It's the same concept, but it's very different in practice. For example, adding a field with a default value in postgres will lock the table during the migration, which may be killer. If you use postgres for big data sets, what you'll end up implementing for your migrations looks a lot like what schemaless gives you for free.

If you add a default value it locks the table and re-writes it. However, if you don't add a default value postgres will perform the operation instantly. Old rows will be untouched, and new values will be added with the shoe_size of either NULL or whatever you set it to be... IE exactly the same outcome and performance as the MongoDB case mentioned above. Adding a field in postgres while setting a default value would…

> Adding a field in postgres while setting a default value would be exactly the same as adding a field

Not true. Postgres locks the entire table for the duration of the update (which could be hours or days for a large enough dataset). Mongo will lock your entire databased for tiny fractions of a second, lots and lots of times. The difference is huge. The postgres implementation takes down your site, the Mongo implementation doesn't.

At scale, people start using Postgres a lot like a NoSQL service. Check out the Reddit schema for example, or read IMVU's blog posts on the topic (with MySQL, but same point). Or Facebook's architecture. All those migrations strategies look a lot more like mongo than postgres, even though they all use SQL DBs.

> Now I'm not anti-MongoDB, I'm just saying you shouldn't give it credit for something that relational database do just fine.

Saying "I can do NoSQL migrations just fine in Postgres" is like saying "I can do OO just fine in assembly".

Re: A Year with MongoDB

#100

Post by willvarfar »

jabwork wrote: post #21
mitchellh wrote: post #7

Earlier quoted context omitted.

[Note: I wrote the blog post] I'm not at all against horizontally scaling. However, I don't believe that horizontally scaling should be necessary doing a mere 200 updates to per second to a data store that isn't even fsyncing writes to disk. Think of it in terms of cost per ops. Let's just say 200 update ops per second is the point at which you need to shard (not scientific, but let's just use that as a benchmark sin…

Can you give any sort of indication of the value of a schemaless database and the flexibility it provided as the team fleshed out the data model? Was this a mere convenience over traditional schema migration or something more?

And also, did you ever introduce a bug by getting a typo in a field name? ;)
Post reply on HN