Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a MongoDB Clone in Postgres

legitimatesounding.com

Post reply on HN
Print view View original

61–70 of 99 posts

Re: Building a MongoDB Clone in Postgres

#61

Post by nsanch »

bri3d wrote: post #12
nsanch wrote: post #3

Unless he's planning to build sharding on postgres too, I think he's missing the point.

Should be pretty trivial for him to shard collections across multiple databases with the same level of intelligence as MongoDB's automated sharding simply using the primary key, since MongoDB doesn't join. Not sure how sharding is the point of MongoDB, though - in most of the universe, sharding is a database architecture/schema-level thing, not a database-server level thing, and for good reason - it's pretty darn har…

I disagree that sharding can ever be a trivial problem if you're going to try to tackle moving data between shards while staying online. I'm not saying it's impossible, just that it's not trivial.

Re: Building a MongoDB Clone in Postgres

#62

Post by vampirechicken »

mtkd wrote: post #25

I do encourage any SQL user, who hasn't already tried MongoDB, to fire it up and try it themselves. Mongoid in Ruby is fairly fast to get started. I've been using SQL since early 90s. For web apps and large collections I've started using MongoDB more recently. It's one of the most exciting technologies I've used in a long time. It takes a while to stop thinking SQL, but once you pass that it's really very primitive (…

> SQL still has a place and MongoDB is no replacement Let's flip this: NoSQL is finding it's place in the database ecosystem.

SQL is the name of a database query language, NoSQL is a term that identifies a set of new, ostensibly non-relational, data storage engines that so implement an SQL parser despite SQL being the Standard Query Language, with twenty years of history behind it.

My opinion - They decided to call it NoSQL, not because SQL requires a relation database, but because writing a smart query optimizer is hard task. Much harder than writing the storage engine itself.

I'd much rather access data from Mongo, Riak, Redis, and Couch using the SELECT, INSERT, UPDATE and DELETE statements that I have known since 1988, instead of having to learn four new database APIs.

Re: Building a MongoDB Clone in Postgres

#63

Post by gbog »

taligent wrote: post #44
bad_user wrote: post #33

Earlier quoted context omitted.

Experimenting with new technologies is encouraged, especially since cross-pollination of ideas happen this way. However I would actually advise developers to stop and think if they really need MongoDB or the latest fad, because their current relational database, such as PostgreSQL, does a mighty fine job for most of their needs. Why? Because I have never seen angry opinions about PostgreSQL losing people's data. Or a…

I guess if only I "woke up" I would realise that PostgreSQL is the answer to all my problems. Guess what. PostgreSQL is just another SQL database. It's definitely one of the best ones but it still stuffers from all the same issues, limitations and frustrations. Many of which stem not from the database itself but from the relational modelling and tools to support it. I use Java + MongoDB and life is significantly bett…

Ahem, lists and maps seem a very easy thing to do in SQL. Where you get to the limits of the relational model is when three structure diversity is out of your hands, for instance a big bunch of parametrized messages.

Re: Building a MongoDB Clone in Postgres

#64

Post by taligent »

gbog wrote: post #63
taligent wrote: post #44

Earlier quoted context omitted.

I guess if only I "woke up" I would realise that PostgreSQL is the answer to all my problems. Guess what. PostgreSQL is just another SQL database. It's definitely one of the best ones but it still stuffers from all the same issues, limitations and frustrations. Many of which stem not from the database itself but from the relational modelling and tools to support it. I use Java + MongoDB and life is significantly bett…

Ahem, lists and maps seem a very easy thing to do in SQL. Where you get to the limits of the relational model is when three structure diversity is out of your hands, for instance a big bunch of parametrized messages.

Lists/Maps require extra tables which means more scripts, more migrations, more backups, more worry.

With MongoDB all I have to do is add Map myMap to a Java class and that's it.

Re: Building a MongoDB Clone in Postgres

#65

Post by bri3d »

nsanch wrote: post #61
bri3d wrote: post #12

Earlier quoted context omitted.

Should be pretty trivial for him to shard collections across multiple databases with the same level of intelligence as MongoDB's automated sharding simply using the primary key, since MongoDB doesn't join. Not sure how sharding is the point of MongoDB, though - in most of the universe, sharding is a database architecture/schema-level thing, not a database-server level thing, and for good reason - it's pretty darn har…

I disagree that sharding can ever be a trivial problem if you're going to try to tackle moving data between shards while staying online. I'm not saying it's impossible, just that it's not trivial.

MongoDB's relatively simple approach involves continuing to use the old shard as an authoritative source (and committing updates to it) while shipping data in the background, then pushing the additional changes across and marking the new shard as "master."

Such an approach wouldn't be horribly difficult to implement in SQL using a copy table and write triggers - almost identically to how SoundCloud's Large Hadron Migrator allows writes to occur over a MySQL InnoDB table that's locked for migration (but even simpler because the table schema can't conflict afterwords).

The entire problem is admittedly nontrivial, since if the application happens to be writing data to the shard under migration too quickly (or the shard being migrated to dies), the server can end up in a situation where the new shard is never able to catch up and become a master. However, the easy solution (give up and retry later) is Good Enough for most situations (and is pretty much how MongoDB works).

Re: Building a MongoDB Clone in Postgres

#66

Post by einhverfr »

taligent wrote: post #44
bad_user wrote: post #33

Earlier quoted context omitted.

Experimenting with new technologies is encouraged, especially since cross-pollination of ideas happen this way. However I would actually advise developers to stop and think if they really need MongoDB or the latest fad, because their current relational database, such as PostgreSQL, does a mighty fine job for most of their needs. Why? Because I have never seen angry opinions about PostgreSQL losing people's data. Or a…

I guess if only I "woke up" I would realise that PostgreSQL is the answer to all my problems. Guess what. PostgreSQL is just another SQL database. It's definitely one of the best ones but it still stuffers from all the same issues, limitations and frustrations. Many of which stem not from the database itself but from the relational modelling and tools to support it. I use Java + MongoDB and life is significantly bett…

I wouldn't call it just another SQL database. The extensible type system is actually really cool and you can do a lot with it, and the same goes for Listen/Notify. It's actually an application development platform in a box.

But I guess here's the flip side. PostgreSQL is another relational database and and that means fundamentally it operates on sets of tuples. For anything where set operations are helpful, the relational model brings something that no other models bring to the table. I doubt I will ever see a viable ERP suite written on a NoSQL system.

OTOH, there are plenty of areas where set operations are not that big of a deal. In this area, a networked, mostly reliable, multi-master replicated store of application state data is a really really cool thing. For example, imagine MongoDB as a backplane for an LDAP forest, replacing local BDB storage for something like OpenLDAP. The source data may be in a more mature data store somewhere and fed into MongoDB so if it gets lost it isn't the end of the world (just possibly some temporary business hicups). Same with, say, Root DNS root servers.

Re: Building a MongoDB Clone in Postgres

#67

Post by moe »

leothekim wrote: post #52
jeffdavis wrote: post #24

Earlier quoted context omitted.

"That said, if your site grows in some way you didn't originally anticipate and you get to a point where you need to shard, but can only do so by changing data stores, then it's sad." I think you're being too absolute. For instance, Instagram used sharding in postgres, and they didn't have to throw anything away or dedicate any huge engineering team to solve it.

They had to put engineering effort into it. With Mongo, you don't.

With Mongo, you don't.

Bullshit.

The sharding impl in MongoDB still[1] crumbles pitifully[2] under load. Regardless of sharding MongoDB still halts the world[3] under write-load.

Their map/reduce impl is a joke[4][5].

If you had done the slightest research you'd know that every single aspect that you need to scale out Mongo is either broken by design or so immature that you can't rely on it.

MongoDB may be fine as long as your working set fits into RAM on a single server. If you plan to go beyond that then you'd better start with a more stable foundation - or brace yourself for some serious pain.

[1] https://groups.google.com/group/mongodb-user/browse_thread/t...

[2] http://highscalability.com/blog/2010/10/15/troubles-with-sha...

[3] http://2.bp.blogspot.com/_VHQJkYQ5-dY/TUO3RAn8SNI/AAAAAAAABq...

[4] http://stackoverflow.com/a/3951871

[5] http://steveeichert.com/2010/03/31/data-analysis-using-mongo...

Re: Building a MongoDB Clone in Postgres

#68

Post by einhverfr »

vampirechicken wrote: post #62
mtkd wrote: post #25

I do encourage any SQL user, who hasn't already tried MongoDB, to fire it up and try it themselves. Mongoid in Ruby is fairly fast to get started. I've been using SQL since early 90s. For web apps and large collections I've started using MongoDB more recently. It's one of the most exciting technologies I've used in a long time. It takes a while to stop thinking SQL, but once you pass that it's really very primitive (…

> SQL still has a place and MongoDB is no replacement Let's flip this: NoSQL is finding it's place in the database ecosystem. SQL is the name of a database query language, NoSQL is a term that identifies a set of new, ostensibly non-relational, data storage engines that so implement an SQL parser despite SQL being the Standard Query Language, with twenty years of history behind it. My opinion - They decided to call i…

"I'd much rather access data from Mongo, Riak, Redis, and Couch using the SELECT, INSERT, UPDATE and DELETE statements that I have known since 1988, instead of having to learn four new database APIs."

Doesn't work because those are set operations which are meaningless outside the relational world. If you want to use set operations, use a relational database.

Re: Building a MongoDB Clone in Postgres

#70

Post by einhverfr »

nsanch wrote: post #9
jerrysievert wrote: post #4

Earlier quoted context omitted.

while sharding is an important aspect of mongodb, i don't consider it the most important feature.

I don't know if it's the _most_ important feature, but I wouldn't build a serious site on top of anything that didn't have some sort of built-in sharding story. With postgres you have to roll your own. If you want to bridge the gap from postgres to mongo, I think that's where you have to start.

Postgres-XC is also in beta. Basically it's sharded PostgreSQL with full RI enforcement between shards, and seamless query integration. I assume they are working off the 9.2 codebase (hence the beta being the same time as Pg 9.2) but maybe it's only 9.1 (i.e. no JSON).

Postgres-XC is probably the most exciting PostgreSQL-related project out there. It promises full write-extensibility across the cluster without sacrificing consistency.

Post reply on HN