Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why MongoDB is a bad choice for storing our scraped data

blog.scrapinghub.com

Post reply on HN
Print view View original

31–40 of 121 posts

Re: Why MongoDB is a bad choice for storing our scraped data

#31

Post by wslh »

JulianMorrison wrote: post #11

I really don't understand why people use MongoDB. It seems like it's a elegant technological metaphor (lets use mmap, the OS is our cache and we can overwrite in place in RAM) that in practise turns out to be a terrible idea. Overwrite/mmap cannot be made reliable, requires blocking write-locks, wastes disk, and causes problems shuffling data around as it grows. Add other bad decisions (keys aren't interned, seriousl…

> I really don't understand why people use MongoDB.

Really? When I don't want to think in databases and only on persisting my native types in my favorite programming language MongoDB is my choice. I use it only for experimenting, so I don't care about all those scalability issues.

Re: Why MongoDB is a bad choice for storing our scraped data

#32

Post by kernel_sanders »

latchkey wrote: post #30

I use Mongo for storing a fairly large amount of scraped data and it works great. Some of the data I store is results from bike races that I want to display on my website in a better way than it is displayed elsewhere. The columns change which makes Mongo a great fit, but the data is pretty static. The real issue here is that it feels like the author has just 'discovered' these problems as if Mongo was hiding them al…

Got a link? I can't get enough bike.

Re: Why MongoDB is a bad choice for storing our scraped data

#33

Post by PavlovsCat »

mayank wrote: post #22
PavlovsCat wrote: post #14

Earlier quoted context omitted.

You don't "walk an index". It's an index If you have an address book, you don't have to walk through the city to find an address, but you do have to look through your address book in some way or other. Of course, you can have an index of the index ("C starts at page 7"), but then you have to look through the index of the index.

Not sure what you're getting at. "Walk" usually means a sequential scan. An index is sorted, so you can binary search.

See other reply. If the docs say MongoDB "walks", it's hilarious to complain that someone obviously didn't read the docs for saying "walk", too.

Re: Why MongoDB is a bad choice for storing our scraped data

#34

Post by brasetvik »

pablohoffman wrote: post #20
eknkc wrote: post #15

It's like people started complaining about MongoDB just for the sake of it. I guess it's the new trend? - Ordered data and skip / limit: These would run just fine on any database system. Given that you have appropriate indexes. It does not matter if there are a trillion items total, as long as you are seeking over an index and the result set is in reasonable size. - Restrictions: A lot of software has restrictions. F…

We are not plainly complaining about MongoDB, nor saying it's useless. We are just explaining why it's a poor choice for a specific use case: storing scraped data. FWIW, we still use Mongo in other internal applications, it's just not the right choice for our crawl data storage backend.

How was the evaluation process that led to using MongoDB in the first place?

At some point you must have compared it to, say, Postgres – which is what the section before the summary hints to.

Re: Why MongoDB is a bad choice for storing our scraped data

#35

Post by shane42 »

dkhenry wrote: post #3

you lost me here """ Ordered data Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting which is impractical when the number of records gets large. "" it requires _indexing_ and is quite feasable as I do it every day with stock ticker logs ( also required to be retrieved incrementially ) There are a few other flags that make me wonder about the exact l…

Take the example of crawl logs. Each log entry has a log level, timestamp and message. Typical use would be to view all ERROR (or higher) log levels, show all entries with a specific text in the message, or download the entire log. All of these should be in timestamp order. It's a shame that natural order is not insert order for non-capped collections.

It's a good point that some of this can be achieved with indexing, I should have given more details in the blog post.

Re: Why MongoDB is a bad choice for storing our scraped data

#36

Post by timr »

aroman wrote: post #24
JulianMorrison wrote: post #11

I really don't understand why people use MongoDB. It seems like it's a elegant technological metaphor (lets use mmap, the OS is our cache and we can overwrite in place in RAM) that in practise turns out to be a terrible idea. Overwrite/mmap cannot be made reliable, requires blocking write-locks, wastes disk, and causes problems shuffling data around as it grows. Add other bad decisions (keys aren't interned, seriousl…

Because when you're not operating at significant scale, or have certain specific use cases, it's a fantastically elegant solution and one that's very quick and easy to set up. I have used mongodb for a number of smaller projects, and I have had an excellent experience. It's not "a terrible idea in practice". It might be a terrible fit for what you want , but that doesn't mean it's bad technology.

"when you're not operating at significant scale, or have certain specific use cases, it's a fantastically elegant solution and one that's very quick and easy to set up."

When you're not operating at significant scale, you can use a relational database. They're easy and fast to set up, have nice write-safety guarantees, are more flexible than a key-value store, and will scale well beyond anything that mongo has ever achieved. You can even use them as a key-value store! The downside, of course, is that you have to a tiny bit of knowledge about set theory, and that's a deal breaker for most "developers" today.

The whole point of the GP was that Mongo isn't elegant or easy...it's just naive and short-sighted, and the architectural mistakes within it are fundamental and probably unfixable (at least, not without killing the speed advantages they claim). The real reason that people use mongo is that most webapp devs don't have a very good understanding of how computers work, and want everything to look like Javascript, because that's all they really know.

Re: Why MongoDB is a bad choice for storing our scraped data

#37

Post by lmm »

JulianMorrison wrote: post #26
aroman wrote: post #24

Earlier quoted context omitted.

Because when you're not operating at significant scale, or have certain specific use cases, it's a fantastically elegant solution and one that's very quick and easy to set up. I have used mongodb for a number of smaller projects, and I have had an excellent experience. It's not "a terrible idea in practice". It might be a terrible fit for what you want , but that doesn't mean it's bad technology.

Are you dead certain those projects aren't ever eating data, and nor are they about to crash tomorrow with an unrecoverably hosed DB? I still say it's bad technology. Use plain old SQL instead.

"plain old SQL" can require a lot of mangling one's data to fit its constraints. I refuse to believe that there isn't a better key-value store for the case where the values are json documents, even if mongodb isn't it.

Re: Why MongoDB is a bad choice for storing our scraped data

#38

Post by calpaterson »

latchkey wrote: post #30

I use Mongo for storing a fairly large amount of scraped data and it works great. Some of the data I store is results from bike races that I want to display on my website in a better way than it is displayed elsewhere. The columns change which makes Mongo a great fit, but the data is pretty static. The real issue here is that it feels like the author has just 'discovered' these problems as if Mongo was hiding them al…

Every time there is a mongodb retrospective or experience report posted to HN, the top comment is one along the lines of "Well, these issues are all well documented."

Firstly, the fact that some drawback is well documented does not excuse the fact that it is a drawback.

Second, while some drawbacks are documented some implications of these drawbacks are nuanced and only become obvious with experience. A good example of this is the implications of "schemaless" databases (more accurately: databases that do not check data against a schema). Not having to migrate tables is a boon for lots of development. It's also a giant pain if it turns out that bugs cause data integrity issues.

Third, this experience report is really useful since poorly structured scrape data is one of the areas that I would have considered to be ideal for mongodb.

Most people don't have perfect foresight. I don't fault the author on his lack of omniscience with respect to how mongodb would turn out for them. His original reasoning (given in paragraph 1, sentence 1) does not seem stupid.

Re: Why MongoDB is a bad choice for storing our scraped data

#39

Post by JulianMorrison »

wslh wrote: post #31
JulianMorrison wrote: post #11

I really don't understand why people use MongoDB. It seems like it's a elegant technological metaphor (lets use mmap, the OS is our cache and we can overwrite in place in RAM) that in practise turns out to be a terrible idea. Overwrite/mmap cannot be made reliable, requires blocking write-locks, wastes disk, and causes problems shuffling data around as it grows. Add other bad decisions (keys aren't interned, seriousl…

> I really don't understand why people use MongoDB. Really? When I don't want to think in databases and only on persisting my native types in my favorite programming language MongoDB is my choice. I use it only for experimenting, so I don't care about all those scalability issues.

Try Redis instead? Or rethinkdb (if you don't mind it being a bit new)?

Re: Why MongoDB is a bad choice for storing our scraped data

#40

Post by monstrado »

pablohoffman wrote: post #28
monstrado wrote: post #17

Earlier quoted context omitted.

Clouderan here! Glad to hear you guys went with HBase, I'm looking forward to your follow up post. Will you detail your key design / architectural setup? Did you guys roll your own HBase environment or did you go with the CDH? If you're using the CDH version and have any questions, feel free to shoot an email to cdh-user.

We are using CDH4.2 and have had a very positive experience so far. Cloudera has in fact been an inspiration for us to follow, you guys have really struck the right balance between open source and commercial support. We follow the same philosophy with Scrapy (an open source web crawling framework), as you do with Hadoop and its ecosystem.

That's really awesome to hear, thanks for your kind words. I'm looking forward to the follow up blog, depending on your key design you may be able to take advantage of Impala for ad-hoc queries using SQL.
Post reply on HN