Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why MongoDB is a bad choice for storing our scraped data

blog.scrapinghub.com

Post reply on HN
Print view View original

111–120 of 121 posts

Re: Why MongoDB is a bad choice for storing our scraped data

#111

Post by est »

rdtsc wrote: post #109
JulianMorrison wrote: post #11

I really don't understand why people use MongoDB. It seems like it's a elegant technological metaphor (lets use mmap, the OS is our cache and we can overwrite in place in RAM) that in practise turns out to be a terrible idea. Overwrite/mmap cannot be made reliable, requires blocking write-locks, wastes disk, and causes problems shuffling data around as it grows. Add other bad decisions (keys aren't interned, seriousl…

Marketing. They are also on my blacklist, forever. They used to ship with unacknowledged writes as the default option. Think about it for a little, a database that just throws your _data_ over the fence and prays for the best, proceeding without an write acknowledgement. There was not flashing warning on their front page about, no bold disclaimers, but there were sure plenty of "Oh look super fast benchmarks beating…

So can you tell us what's your easieast way to implement something like:

    books: [
        {id: 1, tags: ['a', 'b'], author: ['c', 'd'], count_read: 123, count_bought: 456},
    ]
where count_read and count_bought is atomic increment, tags are arbitrary string array and could be indexed for searching.

Yeah I knew Postgre could do that. But MongoDB is the most simple and direct way on the market. Stuff like tags makes MySQL m2m joins very inefficient.

Re: Why MongoDB is a bad choice for storing our scraped data

#112

Post by crypto5 »

craigching wrote: post #97
threeseed wrote: post #93

Earlier quoted context omitted.

Standby is a pretty poor solution compared to replica set let alone what Cassandra has to offer. Sharding is trivial on MongoDB/Cassandra and it is open source. So let's be accurate here. It is a problem inherent with the SQL databases. And I think you underestimate the benefits of scaling out. If I want to ensure close to 100% uptime or have a server closer to my users than Cassandra or even MongoDB would be infinit…

Thanks, that's very much what I'm talking about. Cassandra would be my ideal store, I absolutely love it except for the ability to index across nodes. My understanding, when last I looked at it, was that indexes were only local and didn't span Cassandra nodes. Does Cassandra now have properly distributed indexes? When I was looking at implementing Cassandra instead of Mongo DB, it seemed like we had to create reverse…

Just curious: did you get any real problem with local indexes? For me it works just fine.

Re: Why MongoDB is a bad choice for storing our scraped data

#113

Post by KaiserPro »

Now the interesting thing about this post is this, I can see why they wanted to use mongodb, and I can see why it bit them in the arse.

What interests me is why they would want to keep everything in the database? I'd assume that they need to aggregate and curate the scraped data. After the initial scrape the majority of actions surely are going to be on the metadata of the scraped content? (where is said data, when was it scraped, how big, relationship to other data, etc) This data is much smaller and can be stored in relational database, as its proper structured data with relationships.

This allows the nasty unstructured data to be kept on a plain boring filesystem. After all filesystems are exceptionally mature, universal, multilevel key-value stores.

Now people will say that filesystems don't scale, well that's not really true. ext4/ntfs on a single system won't scale, but something like lustre/gluster(although not as neat)/gpfs scales linearly with the amount of nodes you apply to it.

Re: Why MongoDB is a bad choice for storing our scraped data

#114

Post by shane42 »

KaiserPro wrote: post #113

Now the interesting thing about this post is this, I can see why they wanted to use mongodb, and I can see why it bit them in the arse. What interests me is why they would want to keep everything in the database? I'd assume that they need to aggregate and curate the scraped data. After the initial scrape the majority of actions surely are going to be on the metadata of the scraped content? (where is said data, when w…

This approach (scraped data on FS + metadata in DB) works well for storing scraped data. It was the first thing I prototyped when we started the project to move away from MongoDB. We've worked on similar designs in the past where the data is in S3, it's a common pattern.

We'd need to code the searching, filtering, paginating, (distributed?) job management ourselves while being careful to keep the DB & metadata consistent. It works best if each file is a reasonable 'chunk' of data (not too big, not tiny). None of this is a problem, and it scales very well as you said.

In the end, we went with HBase for crawl data in the new system. Of course, you can look at this as files on a filesystem (HDFS or others) :) It does a lot of what we would otherwise have to code ourselves and it's a good fit for applications we want to build on that data in future (e.g. storing other crawl datastructures, processing with hadoop). I'll provide more details on that in the next post.

Re: Why MongoDB is a bad choice for storing our scraped data

#115

Post by craigching »

crypto5 wrote: post #112
craigching wrote: post #97

Earlier quoted context omitted.

Thanks, that's very much what I'm talking about. Cassandra would be my ideal store, I absolutely love it except for the ability to index across nodes. My understanding, when last I looked at it, was that indexes were only local and didn't span Cassandra nodes. Does Cassandra now have properly distributed indexes? When I was looking at implementing Cassandra instead of Mongo DB, it seemed like we had to create reverse…

Just curious: did you get any real problem with local indexes? For me it works just fine.

How do you use a local index when your data is distributed across numerous nodes? Maybe I'm missing something fundamental, so I'd definitely like to understand.

Re: Why MongoDB is a bad choice for storing our scraped data

#116

Post by papsosouid »

gnaritas wrote: post #106
papsosouid wrote: post #80

Earlier quoted context omitted.

You made baseless assertions, and provided nothing to back them up. Your comment got precisely the response it deserves. Acting indignant does not support your assertions. The fact that you have some unspecified problem does not mean anyone else who does not have that problem is inexperienced. Given the complete lack of information available, it is just as reasonable to conclude that you are in fact lacking in experi…

And which assertion was that; the one that Mongo fits a niche which it obviously does, or the one that many people find relational databases a pain, which they obviously do. Neither of those require me to provide evidence, they are self evident facts to anyone with even moderate experience in the field. I don't have an unspecified problem, not once did I even mention a problem, so take your childish argumentative b.s…

>And which assertion was that

Both of them. You only wrote two sentences, it shouldn't be hard to find them.

>the one that many people find

You didn't say anything about "many people find". You said they are a pain in the ass, and don't mix well with the applications many people are writing. Those are both assertions, and you supported neither of them. Even after replying twice, you still haven't even given a hint as to what you might be referring to. That really makes it seem like you are just saying things out of ignorance.

Re: Why MongoDB is a bad choice for storing our scraped data

#117

Post by gnaritas »

papsosouid wrote: post #116
gnaritas wrote: post #106

Earlier quoted context omitted.

And which assertion was that; the one that Mongo fits a niche which it obviously does, or the one that many people find relational databases a pain, which they obviously do. Neither of those require me to provide evidence, they are self evident facts to anyone with even moderate experience in the field. I don't have an unspecified problem, not once did I even mention a problem, so take your childish argumentative b.s…

>And which assertion was that Both of them. You only wrote two sentences, it shouldn't be hard to find them. >the one that many people find You didn't say anything about "many people find". You said they are a pain in the ass, and don't mix well with the applications many people are writing. Those are both assertions, and you supported neither of them. Even after replying twice, you still haven't even given a hint as…

> You didn't say anything about "many people find". You said they are a pain in the ass

I said they were a pain in the ass for the kinds of programs many people want to write. I'm sorry you're too ignorant to grok my meaning without it being explained to you like a five year old child.

> Those are both assertions, and you supported neither of them.

They are self evident facts and don't require supporting evidence; the very fact that a community exists around these products should make that clear to you.

In any case, it's absolutely clear there's no value in conversing with you, good day.

Re: Why MongoDB is a bad choice for storing our scraped data

#118

Post by papsosouid »

gnaritas wrote: post #117
papsosouid wrote: post #116

Earlier quoted context omitted.

>And which assertion was that Both of them. You only wrote two sentences, it shouldn't be hard to find them. >the one that many people find You didn't say anything about "many people find". You said they are a pain in the ass, and don't mix well with the applications many people are writing. Those are both assertions, and you supported neither of them. Even after replying twice, you still haven't even given a hint as…

> You didn't say anything about "many people find". You said they are a pain in the ass I said they were a pain in the ass for the kinds of programs many people want to write. I'm sorry you're too ignorant to grok my meaning without it being explained to you like a five year old child. > Those are both assertions, and you supported neither of them. They are self evident facts and don't require supporting evidence; th…

>I said they were a pain in the ass for the kinds of programs many people want to write

And refuse to specify what those kinds of programs might be. You are inventing a "many" and giving them a problem to create a false impression of consensus, when it is actually just you making a singular, baseless assertion.

>the very fact that a community exists around these products should make that clear to you.

I address that in the first post. You keep responding purely to act like a petulant child, but provide absolutely nothing to support your claims. Do you really think that makes you appear to be the rational, logical party?

Re: Why MongoDB is a bad choice for storing our scraped data

#119

Post by crypto5 »

craigching wrote: post #115
crypto5 wrote: post #112

Earlier quoted context omitted.

Just curious: did you get any real problem with local indexes? For me it works just fine.

How do you use a local index when your data is distributed across numerous nodes? Maybe I'm missing something fundamental, so I'd definitely like to understand.

It's hidden from me behind client library API(astyanax in my case), I shouldn't know anything about index locality. I just send request(give me records for this index value), and get response and don't care if it's local index or distributed index. Astyanax takes care about everything, it queries all nodes.

Re: Why MongoDB is a bad choice for storing our scraped data

#120

Post by craigching »

crypto5 wrote: post #119
craigching wrote: post #115

Earlier quoted context omitted.

How do you use a local index when your data is distributed across numerous nodes? Maybe I'm missing something fundamental, so I'd definitely like to understand.

It's hidden from me behind client library API(astyanax in my case), I shouldn't know anything about index locality. I just send request(give me records for this index value), and get response and don't care if it's local index or distributed index. Astyanax takes care about everything, it queries all nodes.

I will check that out, thanks! I was looking at Hector and Pelops at the time and I don't believe they provided anything like that. But now that you mention it, I bet Hive could do what you describe as well. Need to dig a bit deeper now!
Post reply on HN