Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why MongoDB is a bad choice for storing our scraped data

blog.scrapinghub.com

Post reply on HN
Print view View original

21–30 of 121 posts

Re: Why MongoDB is a bad choice for storing our scraped data

#21

Post by shane42 »

PavlovsCat wrote: post #14
mayank wrote: post #10

Earlier quoted context omitted.

No kidding. Without details, it really sounds like the author is a bit clueless. He mentions the lack of joins, but doesn't say a word about Mapreduce. "MongoDB needs to walk the index from the beginning to the offset..." You don't "walk an index". It's an index. "Too many databases" sounds a little suspicious. Why not add an indexed field to partition records? Complaining about a lack of schema, transactions and tri…

You don't "walk an index". It's an index If you have an address book, you don't have to walk through the city to find an address, but you do have to look through your address book in some way or other. Of course, you can have an index of the index ("C starts at page 7"), but then you have to look through the index of the index.

The problems with pagination are explained better on this SO post: http://stackoverflow.com/questions/7228169/slow-pagination-o... Mongo docs used the "walk" terminology

Re: Why MongoDB is a bad choice for storing our scraped data

#22

Post by mayank »

PavlovsCat wrote: post #14
mayank wrote: post #10

Earlier quoted context omitted.

No kidding. Without details, it really sounds like the author is a bit clueless. He mentions the lack of joins, but doesn't say a word about Mapreduce. "MongoDB needs to walk the index from the beginning to the offset..." You don't "walk an index". It's an index. "Too many databases" sounds a little suspicious. Why not add an indexed field to partition records? Complaining about a lack of schema, transactions and tri…

You don't "walk an index". It's an index If you have an address book, you don't have to walk through the city to find an address, but you do have to look through your address book in some way or other. Of course, you can have an index of the index ("C starts at page 7"), but then you have to look through the index of the index.

Not sure what you're getting at. "Walk" usually means a sequential scan. An index is sorted, so you can binary search.

Re: Why MongoDB is a bad choice for storing our scraped data

#23

Post by eknkc »

kalleboo wrote: post #18
eknkc wrote: post #15

It's like people started complaining about MongoDB just for the sake of it. I guess it's the new trend? - Ordered data and skip / limit: These would run just fine on any database system. Given that you have appropriate indexes. It does not matter if there are a trillion items total, as long as you are seeking over an index and the result set is in reasonable size. - Restrictions: A lot of software has restrictions. F…

Why are you taking this personally? They're just listing reasons why it's not a good fit for them. Useful information to others who are trying to pick a database for similar applications.

I'm sorry if it looks like I'm attacking the criticism. Nope, I would not use MongoDB ever again, after a year and a half with it. I have my reasons for this decision.

I just don't like people bashing something without valid reasons. It might just be a perfect solution for similar applications, this is not a good way to evaluate.

Re: Why MongoDB is a bad choice for storing our scraped data

#24

Post by aroman »

JulianMorrison wrote: post #11

I really don't understand why people use MongoDB. It seems like it's a elegant technological metaphor (lets use mmap, the OS is our cache and we can overwrite in place in RAM) that in practise turns out to be a terrible idea. Overwrite/mmap cannot be made reliable, requires blocking write-locks, wastes disk, and causes problems shuffling data around as it grows. Add other bad decisions (keys aren't interned, seriousl…

Because when you're not operating at significant scale, or have certain specific use cases, it's a fantastically elegant solution and one that's very quick and easy to set up.

I have used mongodb for a number of smaller projects, and I have had an excellent experience. It's not "a terrible idea in practice". It might be a terrible fit for what you want, but that doesn't mean it's bad technology.

Re: Why MongoDB is a bad choice for storing our scraped data

#25

Post by andrix »

kalleboo wrote: post #18
eknkc wrote: post #15

It's like people started complaining about MongoDB just for the sake of it. I guess it's the new trend? - Ordered data and skip / limit: These would run just fine on any database system. Given that you have appropriate indexes. It does not matter if there are a trillion items total, as long as you are seeking over an index and the result set is in reasonable size. - Restrictions: A lot of software has restrictions. F…

Why are you taking this personally? They're just listing reasons why it's not a good fit for them. Useful information to others who are trying to pick a database for similar applications.

Exactly. It just mention why MongoDB hasn't worked for us.

Re: Why MongoDB is a bad choice for storing our scraped data

#26

Post by JulianMorrison »

aroman wrote: post #24
JulianMorrison wrote: post #11

I really don't understand why people use MongoDB. It seems like it's a elegant technological metaphor (lets use mmap, the OS is our cache and we can overwrite in place in RAM) that in practise turns out to be a terrible idea. Overwrite/mmap cannot be made reliable, requires blocking write-locks, wastes disk, and causes problems shuffling data around as it grows. Add other bad decisions (keys aren't interned, seriousl…

Because when you're not operating at significant scale, or have certain specific use cases, it's a fantastically elegant solution and one that's very quick and easy to set up. I have used mongodb for a number of smaller projects, and I have had an excellent experience. It's not "a terrible idea in practice". It might be a terrible fit for what you want , but that doesn't mean it's bad technology.

Are you dead certain those projects aren't ever eating data, and nor are they about to crash tomorrow with an unrecoverably hosed DB?

I still say it's bad technology. Use plain old SQL instead.

Re: Why MongoDB is a bad choice for storing our scraped data

#27

Post by monstrado »

PavlovsCat wrote: post #14
mayank wrote: post #10

Earlier quoted context omitted.

No kidding. Without details, it really sounds like the author is a bit clueless. He mentions the lack of joins, but doesn't say a word about Mapreduce. "MongoDB needs to walk the index from the beginning to the offset..." You don't "walk an index". It's an index. "Too many databases" sounds a little suspicious. Why not add an indexed field to partition records? Complaining about a lack of schema, transactions and tri…

You don't "walk an index". It's an index If you have an address book, you don't have to walk through the city to find an address, but you do have to look through your address book in some way or other. Of course, you can have an index of the index ("C starts at page 7"), but then you have to look through the index of the index.

It looks like they went with HBase for the replacement which means they can scan an index range, keys are in lexicographical order, so it makes it pretty easy to scan over a series of data with a single RPC call.

Re: Why MongoDB is a bad choice for storing our scraped data

#28

Post by pablohoffman »

monstrado wrote: post #17
pablohoffman wrote: post #9

Earlier quoted context omitted.

We went with HBase. Cassandra would have been suitable too, but we already use Hadoop for data processing so it was a natural choice within the infrastructure ecosystem. We will write a followup about that.

Clouderan here! Glad to hear you guys went with HBase, I'm looking forward to your follow up post. Will you detail your key design / architectural setup? Did you guys roll your own HBase environment or did you go with the CDH? If you're using the CDH version and have any questions, feel free to shoot an email to cdh-user.

We are using CDH4.2 and have had a very positive experience so far.

Cloudera has in fact been an inspiration for us to follow, you guys have really struck the right balance between open source and commercial support. We follow the same philosophy with Scrapy (an open source web crawling framework), as you do with Hadoop and its ecosystem.

Re: Why MongoDB is a bad choice for storing our scraped data

#29

Post by leothekim »

I'm interested in hearing what the author's new storage system is. What would be compelling is to hear if the same hardware and storage with the new storage system performed better than mongo with some semblance of concrete metrics. There are a lot of complaints here about mongo -- all of them not new -- but no hard numbers.

Re: Why MongoDB is a bad choice for storing our scraped data

#30

Post by latchkey »

I use Mongo for storing a fairly large amount of scraped data and it works great. Some of the data I store is results from bike races that I want to display on my website in a better way than it is displayed elsewhere. The columns change which makes Mongo a great fit, but the data is pretty static.

The real issue here is that it feels like the author has just 'discovered' these problems as if Mongo was hiding them all along and after a long time using the system he just found them. The reality is that all of the things he brings up are well documented. It is fascinating to me how people pick a buzzword database and don't bother to think about how their application might run poorly on it over time.

Post reply on HN