Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a MongoDB Clone in Postgres

legitimatesounding.com

Post reply on HN
Print view View original

81–90 of 99 posts

Re: Building a MongoDB Clone in Postgres

#81

Post by k_bx »

roncohen wrote: post #32

If you just need a simple key/value, a good way to go is HSTORE in PostgreSQL. It allows only string key/values, and not complex structures. Use the JSON datatype if you need lists, nested objects etc. With HSTORE you get indexes on keys.

I wonder if postgres will adopt BSON instead of JSON and implement all the querying -- that would be so much better.

p.s.: guys who disagree -- why would you do that? I mean, BSON lets you quickly skip (embedded) documents you're not interested in, since it stores their size. Also it has type info and some additional types. So it's just "better JSON" for storing and navigating, and if postgres wants to have querying in JSON, they will either take BSON or invent the wheel for something similar.

Re: Building a MongoDB Clone in Postgres

#82

Post by shykes »

mtkd wrote: post #25

I do encourage any SQL user, who hasn't already tried MongoDB, to fire it up and try it themselves. Mongoid in Ruby is fairly fast to get started. I've been using SQL since early 90s. For web apps and large collections I've started using MongoDB more recently. It's one of the most exciting technologies I've used in a long time. It takes a while to stop thinking SQL, but once you pass that it's really very primitive (…

Here's a quick an easy way to deploy both Mongo and Postgres, and start comparing them:

    git clone http://github.com/shykes/sqlwars
    cd sqlwars
    pip install dotcloud
    dotcloud create sqlwars
    dotcloud push sqlwars
    
    # Open mongo shell
    dotcloud run sqlwars.mongo mongo

    # Open postgres shell
    dotcloud run sqlwars.postgres psql
Happy hacking!

Re: Building a MongoDB Clone in Postgres

#83

Post by moe »

taligent wrote: post #75
moe wrote: post #67

Earlier quoted context omitted.

With Mongo, you don't. Bullshit. The sharding impl in MongoDB still[1] crumbles pitifully[2] under load. Regardless of sharding MongoDB still halts the world[3] under write-load. Their map/reduce impl is a joke[4][5]. If you had done the slightest research you'd know that every single aspect that you need to scale out Mongo is either broken by design or so immature that you can't rely on it. MongoDB may be fine as lo…

Well that's just factually incorrect. MongoDB now has a per-database write lock and will have a per-collection write lock in the next version. So your halt under write-load statement is incorrect. The map reduce implementation is quite new sure. But it is getting better and you can always link it up with Hadoop. At the very least provide links that aren't nearly 2 years old.

MongoDB now has a per-database write lock and will have a per-collection write lock in the next version.

That doesn't help when you need to make a bulk-update on a busy collection. Busy collections have a tendency towards being in need for bulk updates occasionally.

At the very least provide links that aren't nearly 2 years old.

The first link is 1 month old.

The other links also still describe the state of the art. Feel free to correct them factually if that is not true.

Re: Building a MongoDB Clone in Postgres

#84

Post by moe »

nsanch wrote: post #74
moe wrote: post #67

Earlier quoted context omitted.

With Mongo, you don't. Bullshit. The sharding impl in MongoDB still[1] crumbles pitifully[2] under load. Regardless of sharding MongoDB still halts the world[3] under write-load. Their map/reduce impl is a joke[4][5]. If you had done the slightest research you'd know that every single aspect that you need to scale out Mongo is either broken by design or so immature that you can't rely on it. MongoDB may be fine as lo…

[1] looks like an example where the data didn't fit in RAM. Mongo works best when data fits in RAM or if you use SSD's. Yes, it's sub-optimal. [2] is from a year and a half ago. It doesn't belong in a sentence that includes the word "still." I work at foursquare, btw. Those outages happened on my first and second days at the company. I wasn't so keen on mongo then either. We've gotten much better at administering it.…

[2] When you look at [1] you'll notice that these exact problems are still prevalent.

I'd in fact be curious how exactly did you work around the sharding issues at 4square?

Remember I replied to someone who claimed it takes "no engineering effort" to scale MongoDB. That's not only obviously false, but last time I tried the sharding was so brittle that recommending it as a scaling path would border on malice.

I ran a few rather simple tests for common scenarios; high write-load, flapping mongod, kill -9/rejoin, temporary network partition, deliberate memory starvation. MongoDB failed terribly in every single one of them. The behavior would range from the cluster becoming unresponsive (temporary or terminally), over data-corruption (collection disappears or inaccessible with error), silent data-corruption (inconsistent query-results), to severe cluster imbalance, to crashes (segfault, "shard version not ok" and a whole range of other messages).

I didn't try very hard, it was terribly easy to trigger pathological behavior.

My take-home was that I most certainly don't want to be around when a large MongoDB deployment fails in production.

As such I'm a little disconcerted every time the Mongo scalability myth is reinstated on HN, usually by people who haven't even tried it beyond a single instance on their laptop.

Re: Building a MongoDB Clone in Postgres

#85

Post by hobb0001 »

mjallday wrote: post #77
hobb0001 wrote: post #76

Earlier quoted context omitted.

> However I would actually advise developers to stop and think if they really need MongoDB or the latest fad, because their current relational database, such as PostgreSQL, does a mighty fine job for most of their needs. Every time I've had to use an ORM, it always became a headache sooner or later. It got to the point where I stopped even trying to perform an automatic mapping; I reverted back to using explicit SQL…

Can you give an example? I don't generally have these problems and would like to see if it's my way of thinking or if you're working with vastly different data to me.

Since the relational model is very set-oriented, it has always had a particularly hard time dealing with ordered lists. But that's really a minor thing, and is actually something an ORM can do well to abstract.

My real headache with ORM has always been to get it to perform efficiently. For example, say I want to iterate through the GPA's of all the students. To do so, the ORM usually pulls in the full object model for each student - multiplying the cost of the desired operation by 10x or 100x. Now, each ORM tool usually has some tweaky knob or special declaration you can use to have it limit the fields queried, but by the time you figure out the right incantation you may as well have written the SQL yourself.

But wait! The ORM tool often saves you from this sort of overhead by caching the object model in memory. Alas, therein lies my biggest headache with ORM. "There are only two hard things in Computer Science: cache invalidation, naming things, and off-by-one errors."

Re: Building a MongoDB Clone in Postgres

#86

Post by hobb0001 »

hobb0001 wrote: post #85
mjallday wrote: post #77

Earlier quoted context omitted.

Can you give an example? I don't generally have these problems and would like to see if it's my way of thinking or if you're working with vastly different data to me.

Since the relational model is very set-oriented, it has always had a particularly hard time dealing with ordered lists. But that's really a minor thing, and is actually something an ORM can do well to abstract. My real headache with ORM has always been to get it to perform efficiently. For example, say I want to iterate through the GPA's of all the students. To do so, the ORM usually pulls in the full object model fo…

I should acknowledge that these issues apply just as much to a document store as to a relational database. However, you tend to interact with a document store differently, since there is no need for a mapping abstraction. You often just use the database primitives provided, instead of going through a 3rd party. It's akin to manually using SQL in your code, where the database naturally knows what you're asking for.

Re: Building a MongoDB Clone in Postgres

#87

Post by doronlevari »

jeltz wrote: post #43
nsanch wrote: post #16

Earlier quoted context omitted.

Fair, my statement was overly broad. Sites that are read-only or store blob data in something like S3 can often avoid sharding for quite a while and rely on machines to just get bigger over time. That said, if your site grows in some way you didn't originally anticipate and you get to a point where you need to shard, but can only do so by changing data stores, then it's sad.

I would say most sites will do just fine without sharding. You can get very far by just scaling up with a more expensive database server and caching the most common read operations. Some of the largest websites in the world do not need to do more than this.

Hi, disclaimer - I work for ScaleBase, giving a true automated transparent sharding, so I live and breath sharding for 4 years now...

The main problem is user/session concurrency. On one machine - it kills at some (near) point. A DB is doing much more for every write then reads (look at my blog here: http://database-scalability.blogspot.com/2012/05/were-in-big...). The limit is here and now, even 100 heavy writing sessions will choke the MySQL (or any SQL DB...) on any hardware.

Catch 22: Scale-out to repl slaves with R/W splitting? This can lower read load on the master DB, but read load can be better lowered by caching. The problem is writes and small supporting transactional reads, and slaves won't help. Distributing data (sharding?) is the only way to distribute write intensive load, and it also helps reads by putting them on smaller chunks, and parallelizing them is a sweet sweet bonus :)

As I see around (hundreds of medium-large sites) - there's no other way...

And one final word about the cloud: "one DB machine" is limited to a rather limited non-powerful virtualized compute and I/O space... In the cloud limits are here and now! Cloud is all about elasticity and scale-out.

Hope I helped! Doron

Re: Building a MongoDB Clone in Postgres

#88

Post by bad_user »

hobb0001 wrote: post #85
mjallday wrote: post #77

Earlier quoted context omitted.

Can you give an example? I don't generally have these problems and would like to see if it's my way of thinking or if you're working with vastly different data to me.

Since the relational model is very set-oriented, it has always had a particularly hard time dealing with ordered lists. But that's really a minor thing, and is actually something an ORM can do well to abstract. My real headache with ORM has always been to get it to perform efficiently. For example, say I want to iterate through the GPA's of all the students. To do so, the ORM usually pulls in the full object model fo…

I've used ORM tools such as the one in Django, Active::Record and DBIx::Class.

I do not have the issues you mentioned ... I always remember the API calls I need to make and performance has not been an issue (granted, I'm fairly familiar with all issues that can come up, so I know when or where to optimize in general).

    the ORM usually pulls in the full object model for each student
Here's how to do it in Django (and note this is off the top of my head):

    for row in Student.objects.values_list("gpa"):
        print row[0]
You're really talking about shitty ORMs or APIs you haven't had the patience to become familiar with.

    by the time you figure out the right
    incantation you may as well have written the 
    SQL yourself
That's not true. On complex filtering, you often want to add or subtract filters based on certain conditions. With plain SQL you end up doing really ugly string concatenations, whereas with a good ORM the queries are composable.

Re: Building a MongoDB Clone in Postgres

#89

Post by jeffdavis »

autarch wrote: post #41
jeffdavis wrote: post #14

Relational databases have a history of absorbing the advantages of other systems when they come along, particularly changes to the model (cf. object databases and XML databases). As the author shows, a similar thing will happen quite quickly for document models. Architectural changes are slower, but you can also start to see this happening in postgres with features like unlogged tables (i.e. don't write to the recove…

I'd phrase this differently. Once you implement Mongo in Postgres, what you have isn't a relational database. What you are taking advantage of is the incredibly solid underlying infrastructure of Pg, which provides reliability, scalability, transactions, replication, etc. But the data model itself is no longer relational from the application's perspective.

Every data model is just a degenerate special case of every other data model ;)

Re: Building a MongoDB Clone in Postgres

#90

Post by masklinn »

dstorrs wrote: post #79
taligent wrote: post #64

Earlier quoted context omitted.

Lists/Maps require extra tables which means more scripts, more migrations, more backups, more worry. With MongoDB all I have to do is add Map myMap to a Java class and that's it.

Actually, Postgres has had an Array data type since at least v8.0. http://www.postgresql.org/docs/8.0/static/arrays.html But yes, you're right that maps require a join table.

> But yes, you're right that maps require a join table.

Not in postgres: http://www.postgresql.org/docs/9.1/static/hstore.html

Post reply on HN