Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a MongoDB Clone in Postgres

legitimatesounding.com

Post reply on HN
Print view View original

71–80 of 99 posts

Re: Building a MongoDB Clone in Postgres

#71

Post by mgkimsal »

rhizome wrote: post #29
randomdata wrote: post #19

Earlier quoted context omitted.

Wouldn't that kind of defeat the purpose of using a document database in the first place? Being able to throw unstructured data into the system, and then being able to query on that data once the space is better understood is where the document databases really shine. If you are able to start with rigid structure, you, in many cases, could have just used a relational database to begin with.

Wouldn't that kind of defeat the purpose of using a document database in the first place? Yes. Use the right tool for the job.

Defining 'right', and being able to predict 'right' for future iterations/versions of an app's life is where that mantra seems to fall down.

I'm not a huge nosql fan right now, given that I sometimes need document-style schemaless data storage, but I always (eventually) need adhoc reporting and relational querying capabilities on projects. With that (fore)knowledge, I may as well always choose a relational db.

Re: Building a MongoDB Clone in Postgres

#72

Post by qeorge »

randomdata wrote: post #19
rhizome wrote: post #15

Earlier quoted context omitted.

Isn't this a category error? If you need to query within serialized data, you don't want to serialize. Normalize your database for this.

Wouldn't that kind of defeat the purpose of using a document database in the first place? Being able to throw unstructured data into the system, and then being able to query on that data once the space is better understood is where the document databases really shine. If you are able to start with rigid structure, you, in many cases, could have just used a relational database to begin with.

In my (limited) experience with Mongo, its just not like that.

You can't just (quickly) query any key in the document - you have to define indexes anyway or you'll be doing the equivalent of a table scan or Map/Reduce.

Document databases shine when you're just not doing that kind of query anyway, and you've got a ton of data to store. If you need to do e.g., fulltext and/or geospatial search you should probably use a search server anyway (I like Sphinx), and then just lookup by PK in whatever datastore you're using.

Its been mentioned elsewhere on this thread, but I'm also quite fond of the old FriendFeed method: serialize your arbitrary data into a (MySQL) BLOB, and then run a background process to build "indexes" (really tables) when you find out you need them.

Its ridiculously simple to start doing that, and there's no new tech to learn.

Re: Building a MongoDB Clone in Postgres

#73

Post by peterhunt »

This is the right way to think about building a "NoSQL" datastore.

I think that schemaless, eventually-consistent data stores have a place and are useful. I just think that most of the current efforts are throwing away years of investment in SQL datastores. Rather than thinking of NoSQL as a brand-new paradigm shift that requires a ground-up reimplementation, we need to think of it as layer of abstraction on top of MySQL and memcache (or your preferred setup). Re-implementing all of the work that has gone into these projects is a bad idea and is contrary to The Unix Way.

Thinking of SQL databases as a storage engine rather than the kitchen sink is the key to building a scalable system.

Re: Building a MongoDB Clone in Postgres

#74

Post by nsanch »

moe wrote: post #67
leothekim wrote: post #52

Earlier quoted context omitted.

They had to put engineering effort into it. With Mongo, you don't.

With Mongo, you don't. Bullshit. The sharding impl in MongoDB still[1] crumbles pitifully[2] under load. Regardless of sharding MongoDB still halts the world[3] under write-load. Their map/reduce impl is a joke[4][5]. If you had done the slightest research you'd know that every single aspect that you need to scale out Mongo is either broken by design or so immature that you can't rely on it. MongoDB may be fine as lo…

[1] looks like an example where the data didn't fit in RAM. Mongo works best when data fits in RAM or if you use SSD's. Yes, it's sub-optimal.

[2] is from a year and a half ago. It doesn't belong in a sentence that includes the word "still." I work at foursquare, btw. Those outages happened on my first and second days at the company. I wasn't so keen on mongo then either. We've gotten much better at administering it. Basically all our data is in mongo, and it has its flaws, but I'm still glad we use it.

[3] is also from a year and a half ago. Mongo 2.2 will have a per-database write lock, which is at least progress, even though it's obviously not enough. Since 2.0 (or 1.8?) it's also gotten better at yielding during long writes.

I have no experience with their mapreduce impl and can't speak to it.

Re: Building a MongoDB Clone in Postgres

#75

Post by taligent »

moe wrote: post #67
leothekim wrote: post #52

Earlier quoted context omitted.

They had to put engineering effort into it. With Mongo, you don't.

With Mongo, you don't. Bullshit. The sharding impl in MongoDB still[1] crumbles pitifully[2] under load. Regardless of sharding MongoDB still halts the world[3] under write-load. Their map/reduce impl is a joke[4][5]. If you had done the slightest research you'd know that every single aspect that you need to scale out Mongo is either broken by design or so immature that you can't rely on it. MongoDB may be fine as lo…

Well that's just factually incorrect. MongoDB now has a per-database write lock and will have a per-collection write lock in the next version. So your halt under write-load statement is incorrect.

The map reduce implementation is quite new sure. But it is getting better and you can always link it up with Hadoop.

At the very least provide links that aren't nearly 2 years old.

Re: Building a MongoDB Clone in Postgres

#76

Post by hobb0001 »

bad_user wrote: post #33
mtkd wrote: post #25

I do encourage any SQL user, who hasn't already tried MongoDB, to fire it up and try it themselves. Mongoid in Ruby is fairly fast to get started. I've been using SQL since early 90s. For web apps and large collections I've started using MongoDB more recently. It's one of the most exciting technologies I've used in a long time. It takes a while to stop thinking SQL, but once you pass that it's really very primitive (…

Experimenting with new technologies is encouraged, especially since cross-pollination of ideas happen this way. However I would actually advise developers to stop and think if they really need MongoDB or the latest fad, because their current relational database, such as PostgreSQL, does a mighty fine job for most of their needs. Why? Because I have never seen angry opinions about PostgreSQL losing people's data. Or a…

> However I would actually advise developers to stop and think if they really need MongoDB or the latest fad, because their current relational database, such as PostgreSQL, does a mighty fine job for most of their needs.

Every time I've had to use an ORM, it always became a headache sooner or later. It got to the point where I stopped even trying to perform an automatic mapping; I reverted back to using explicit SQL in my code and avoided the "abstraction" altogether.

With a document store like MongoDB, though, the mapping between object and document is almost seamless. It's actually really quite comfortable.

Re: Building a MongoDB Clone in Postgres

#77

Post by mjallday »

hobb0001 wrote: post #76
bad_user wrote: post #33

Earlier quoted context omitted.

Experimenting with new technologies is encouraged, especially since cross-pollination of ideas happen this way. However I would actually advise developers to stop and think if they really need MongoDB or the latest fad, because their current relational database, such as PostgreSQL, does a mighty fine job for most of their needs. Why? Because I have never seen angry opinions about PostgreSQL losing people's data. Or a…

> However I would actually advise developers to stop and think if they really need MongoDB or the latest fad, because their current relational database, such as PostgreSQL, does a mighty fine job for most of their needs. Every time I've had to use an ORM, it always became a headache sooner or later. It got to the point where I stopped even trying to perform an automatic mapping; I reverted back to using explicit SQL…

Can you give an example? I don't generally have these problems and would like to see if it's my way of thinking or if you're working with vastly different data to me.

Re: Building a MongoDB Clone in Postgres

#78

Post by jeffdavis »

leothekim wrote: post #52
jeffdavis wrote: post #24

Earlier quoted context omitted.

"That said, if your site grows in some way you didn't originally anticipate and you get to a point where you need to shard, but can only do so by changing data stores, then it's sad." I think you're being too absolute. For instance, Instagram used sharding in postgres, and they didn't have to throw anything away or dedicate any huge engineering team to solve it.

They had to put engineering effort into it. With Mongo, you don't.

I attended a talk by Instagram post-buyout. That's where I got the impression that sharding was not a huge obstacle for them (though it was significant). Keep in mind their entire data management team was 2 people I think.

My point was that sharding is not an absolute "have it or not". Some features require major engineering efforts to get anywhere at all, but sharding is not one of them.

But if you think the overall effort is less with MongoDB then go for it.

Re: Building a MongoDB Clone in Postgres

#79

Post by dstorrs »

taligent wrote: post #64
gbog wrote: post #63

Earlier quoted context omitted.

Ahem, lists and maps seem a very easy thing to do in SQL. Where you get to the limits of the relational model is when three structure diversity is out of your hands, for instance a big bunch of parametrized messages.

Lists/Maps require extra tables which means more scripts, more migrations, more backups, more worry. With MongoDB all I have to do is add Map myMap to a Java class and that's it.

Actually, Postgres has had an Array data type since at least v8.0. http://www.postgresql.org/docs/8.0/static/arrays.html

But yes, you're right that maps require a join table.

Re: Building a MongoDB Clone in Postgres

#80

Post by cbsmith »

nsanch wrote: post #9
jerrysievert wrote: post #4

Earlier quoted context omitted.

while sharding is an important aspect of mongodb, i don't consider it the most important feature.

I don't know if it's the _most_ important feature, but I wouldn't build a serious site on top of anything that didn't have some sort of built-in sharding story. With postgres you have to roll your own. If you want to bridge the gap from postgres to mongo, I think that's where you have to start.

> I don't know if it's the _most_ important feature, but I wouldn't build a serious site on top of anything that didn't have some sort of built-in sharding story.

You know people say this, but in practice I find that simple hash bucketing with a redundant pair works surprisingly well, particularly in the cloud. Yes it isn't fancy, but it is trivial to manage and debug, and you can do a lot of optimizations given such a clear cut set of partitioning rules.

Your problems have to get really big before a more sophisticated mechanism really pays off in terms of avoiding headaches, and often the more sophisticated mechanisms actually cause more headaches before you get there.

Post reply on HN