Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why MongoDB is a bad choice for storing our scraped data

blog.scrapinghub.com

Post reply on HN
Print view View original

91–100 of 121 posts

Re: Why MongoDB is a bad choice for storing our scraped data

#91

Post by papsosouid »

threeseed wrote: post #90
papsosouid wrote: post #57

Earlier quoted context omitted.

Could you actually go into some detail about these mythical problems with actual databases? Faux database apologists seem to really love claiming databases are so unusable, but I've never gotten an actual explanation as to what problems they are having. As both a developer and a sysadmin, postgresql is much less of a pain in the ass than mongodb. And I have no idea what "don't mix well with the kinds of programs.." i…

So the guys at Foursquare are driven by "fads" and don't have a clue about databases or scaling ?

You are reading something I didn't write.

Re: Why MongoDB is a bad choice for storing our scraped data

#92

Post by threeseed »

ownagefool wrote: post #78
lucisferre wrote: post #49

Earlier quoted context omitted.

In the real world though, both groups still need to use what works in practice. It's entirely possible for MongoDB to work sufficiently well for a certain group of people in a reasonably cost effective way. Exaggerating its problems (as bad as they are) doesn't add weight to your agrement. For example, global write-locking will not _inevitably_ lead to consistency or throughput unless the write frequencies are suffic…

SQL isn't the alternative, it's the standard and noSQL databases are supposed to offer extra value to cause you to migrate. According to these articles, MongoDB doesn't offer any real additional value, thus you shouldn't use it. It's not about elitism, it's about making good decisions. That said, given all the hype with companies that hire, protesting loudly might not be the best short term personal decision. Meh.

These articles are just one side of the picture which gets heavily upvoted on HN.

And of course it is about elitism. Listen to yourself. "It's about making good decisions".

I mean who are you to judge from the outside what technology a company should use for a specific use cases ?

Re: Why MongoDB is a bad choice for storing our scraped data

#93

Post by threeseed »

AlisdairO wrote: post #86
craigching wrote: post #69

Whenever I see the "You don't need Mongo DB, use an SQL database" and then in the flaming back and forth, I never see my key problem mentioned: MongoDB makes it easy to scale out (replica sets and sharding), where is the "easy to setup replicated and sharded open source SQL database?" I mean, I know that Postgres has replication (via Slony? honestly, it's been awhile since I looked at their solutions) but I don't rec…

Postgres has hot standby built in nowadays, and it works well. Sharding certainly isn't as easy - the technical compromises that mongo makes make it pretty trivial to implement, whereas it's relatively hard to make it work in an RDBMS while maintaining all the expected capabilities. It generally requires some application-level work on open source dbs. With that said, I really think many people grossly underestimate t…

Standby is a pretty poor solution compared to replica set let alone what Cassandra has to offer. Sharding is trivial on MongoDB/Cassandra and it is open source. So let's be accurate here. It is a problem inherent with the SQL databases.

And I think you underestimate the benefits of scaling out. If I want to ensure close to 100% uptime or have a server closer to my users than Cassandra or even MongoDB would be infinitely easier to setup and manage than Postgres. These are "very nice to haves" for even the tiniest startup.

Re: Why MongoDB is a bad choice for storing our scraped data

#94

Post by snaky »

kanja wrote: post #79
snaky wrote: post #61

Earlier quoted context omitted.

If you need to query JSON content , why to store it like plain string, to begin with? Parse it at application level (or even in stored procedure) and store like ordinary fields. Or use hstore.

Postgres additionally has a json field type.

That is for now, until PostgreSQL 9.3 released, have only one useful feature comparing to plain string - validation. That's not much help, and that original lmm's point was about.

Re: Why MongoDB is a bad choice for storing our scraped data

#95

Post by nasalgoat »

craigching wrote: post #69

Whenever I see the "You don't need Mongo DB, use an SQL database" and then in the flaming back and forth, I never see my key problem mentioned: MongoDB makes it easy to scale out (replica sets and sharding), where is the "easy to setup replicated and sharded open source SQL database?" I mean, I know that Postgres has replication (via Slony? honestly, it's been awhile since I looked at their solutions) but I don't rec…

That "easy to scale out" is a misnomer. Replica sets and sharding work in the technical sense, but the implementation isn't anywhere near what I would qualify as production ready.

For example, today my entire production MongoDB database was running 3x slower because a single replica in one shard was down, and their buggy PHP driver kept trying to talk to it despite it being marked down. I really enjoyed waking up at 2am to deal with that.

It relates back to the "easy to use" nature of their marketing. It really is super easy to use and develop on, but the minute you need to do anything important or serious, it breaks down.

You aren't doing yourself any favors going with it except as a proof-of-concept.

Re: Why MongoDB is a bad choice for storing our scraped data

#96

Post by nasalgoat »

nosespray wrote: post #74

People aren't complaining just to complain. When you can't even ctrl-c out of the shell there is a huge issue. It's a known bug, around since v1.8, 'minor' priority. Yeah, thanks for trapping me in the shell.

Ahh yes, the fun "Control-Z-kill-pid" trick. I do enjoy it so.

Re: Why MongoDB is a bad choice for storing our scraped data

#97

Post by craigching »

threeseed wrote: post #93
AlisdairO wrote: post #86

Earlier quoted context omitted.

Postgres has hot standby built in nowadays, and it works well. Sharding certainly isn't as easy - the technical compromises that mongo makes make it pretty trivial to implement, whereas it's relatively hard to make it work in an RDBMS while maintaining all the expected capabilities. It generally requires some application-level work on open source dbs. With that said, I really think many people grossly underestimate t…

Standby is a pretty poor solution compared to replica set let alone what Cassandra has to offer. Sharding is trivial on MongoDB/Cassandra and it is open source. So let's be accurate here. It is a problem inherent with the SQL databases. And I think you underestimate the benefits of scaling out. If I want to ensure close to 100% uptime or have a server closer to my users than Cassandra or even MongoDB would be infinit…

Thanks, that's very much what I'm talking about. Cassandra would be my ideal store, I absolutely love it except for the ability to index across nodes. My understanding, when last I looked at it, was that indexes were only local and didn't span Cassandra nodes. Does Cassandra now have properly distributed indexes?

When I was looking at implementing Cassandra instead of Mongo DB, it seemed like we had to create reverse column family (IIRC, been away from Cassandra for a bit now). Is that still the case?

Re: Why MongoDB is a bad choice for storing our scraped data

#98

Post by j-kidd »

threeseed wrote: post #87
pablohoffman wrote: post #20

Earlier quoted context omitted.

We are not plainly complaining about MongoDB, nor saying it's useless. We are just explaining why it's a poor choice for a specific use case: storing scraped data. FWIW, we still use Mongo in other internal applications, it's just not the right choice for our crawl data storage backend.

One issue is that many of these points are design characteristics of MongoDB and should have been known before hand. I am not criticising but it's almost like you did zero research before hand. Transactions for example have never existed in MongoDB and joins doesn't really make much sense.

Perhaps they did their research on MongoDB and knew all the limitations, but thought to themselves "meh, I can solve all that in the application code", and eventually found out it wasn't so easy to handle transactions and joins in the code?

After all, developers are rather susceptible to the "don't tell me I can't do that" behavior.

Re: Why MongoDB is a bad choice for storing our scraped data

#99

Post by craigching »

nasalgoat wrote: post #95
craigching wrote: post #69

Whenever I see the "You don't need Mongo DB, use an SQL database" and then in the flaming back and forth, I never see my key problem mentioned: MongoDB makes it easy to scale out (replica sets and sharding), where is the "easy to setup replicated and sharded open source SQL database?" I mean, I know that Postgres has replication (via Slony? honestly, it's been awhile since I looked at their solutions) but I don't rec…

That "easy to scale out" is a misnomer. Replica sets and sharding work in the technical sense, but the implementation isn't anywhere near what I would qualify as production ready. For example, today my entire production MongoDB database was running 3x slower because a single replica in one shard was down, and their buggy PHP driver kept trying to talk to it despite it being marked down. I really enjoyed waking up at…

But Mongo DB being buggy isn't a reason to need to use an SQL database vs. a NoSQL store. An SQL database could be buggy as well (I still use Postgres and comparing anything to that quality-wise is just going to bring sorrow for the thing you compare it to ;) ).

FWIW, it's been spotless for us so far. Our needs aren't web scale, but they're big enough to need scaling features.

Re: Why MongoDB is a bad choice for storing our scraped data

#100

Post by hderms »

threeseed wrote: post #93
AlisdairO wrote: post #86

Earlier quoted context omitted.

Postgres has hot standby built in nowadays, and it works well. Sharding certainly isn't as easy - the technical compromises that mongo makes make it pretty trivial to implement, whereas it's relatively hard to make it work in an RDBMS while maintaining all the expected capabilities. It generally requires some application-level work on open source dbs. With that said, I really think many people grossly underestimate t…

Standby is a pretty poor solution compared to replica set let alone what Cassandra has to offer. Sharding is trivial on MongoDB/Cassandra and it is open source. So let's be accurate here. It is a problem inherent with the SQL databases. And I think you underestimate the benefits of scaling out. If I want to ensure close to 100% uptime or have a server closer to my users than Cassandra or even MongoDB would be infinit…

ask yourself how facebook does it
Post reply on HN