Live data from Hacker News

Why MongoDB is a bad choice for storing our scraped data

blog.scrapinghub.com

11–20 of 121 posts

Re: Why MongoDB is a bad choice for storing our scraped data

#11
I really don't understand why people use MongoDB.

It seems like it's a elegant technological metaphor (lets use mmap, the OS is our cache and we can overwrite in place in RAM) that in practise turns out to be a terrible idea. Overwrite/mmap cannot be made reliable, requires blocking write-locks, wastes disk, and causes problems shuffling data around as it grows. Add other bad decisions (keys aren't interned, seriously?)and it's just a terrible limping monster.

Abandon it, walk away.

Re: Why MongoDB is a bad choice for storing our scraped data

#12
post #3

you lost me here """ Ordered data Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting which is impractical when the number of records gets large. "" it requires _indexing_ and is quite feasable as I do it every day with stock ticker logs ( also required to be retrieved incrementially ) There are a few other flags that make me wonder about the exact l…

With a database like HBase, it's already ordered lexicographically which makes it easy to grab a range of data in the order it was written in. You could have a key design like - which would allow quick scans over large amounts of data. ie..Scan from -1368536860 to -1368540450

HBase is multidimensional though, which allows you to keep N numbers of versions of a cell. By default you will get the latest version of the cell back, but you could also opt to receive N versions back, which is useful for time series use cases.

Re: Why MongoDB is a bad choice for storing our scraped data

#13
post #3

you lost me here """ Ordered data Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting which is impractical when the number of records gets large. "" it requires _indexing_ and is quite feasable as I do it every day with stock ticker logs ( also required to be retrieved incrementially ) There are a few other flags that make me wonder about the exact l…

Also, mongo has natural ordering which would do what the author wants without sorting.

Re: Why MongoDB is a bad choice for storing our scraped data

#14
post #10
post #3

you lost me here """ Ordered data Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting which is impractical when the number of records gets large. "" it requires _indexing_ and is quite feasable as I do it every day with stock ticker logs ( also required to be retrieved incrementially ) There are a few other flags that make me wonder about the exact l…

No kidding. Without details, it really sounds like the author is a bit clueless. He mentions the lack of joins, but doesn't say a word about Mapreduce. "MongoDB needs to walk the index from the beginning to the offset..." You don't "walk an index". It's an index. "Too many databases" sounds a little suspicious. Why not add an indexed field to partition records? Complaining about a lack of schema, transactions and tri…

You don't "walk an index". It's an index

If you have an address book, you don't have to walk through the city to find an address, but you do have to look through your address book in some way or other. Of course, you can have an index of the index ("C starts at page 7"), but then you have to look through the index of the index.

Re: Why MongoDB is a bad choice for storing our scraped data

#15
It's like people started complaining about MongoDB just for the sake of it. I guess it's the new trend?

- Ordered data and skip / limit: These would run just fine on any database system. Given that you have appropriate indexes. It does not matter if there are a trillion items total, as long as you are seeking over an index and the result set is in reasonable size.

- Restrictions: A lot of software has restrictions. Filesystems has file name limitations. RDBMSs have table / column name limitations. It's a fact of life. Why is this a con for MongoDB?

- Impossible to keep working set in memory: It is a fair argument that MongoDB has shitty memory management because it just delegates the responsibility to OS. However, this is a concern with any DBMS. Also, given that there are appropriate indexes, you don't need to keep the entire database on memory. This comes back to indexing problem.

- No transactions / lack of schema / no joins...: I don't remember mongoDB claiming to have such features. My car can't fly. I'm not complaining. (Well, sometimes)

- Locking: Fair point. Better I/O performance might come handy (like an SSD) or eventually sharding.

- Poor space efficiency: Fair point about fragmentation and field names. Compression can be achieved on the filesystem level. There was an article about that a couple of days ago. I'm not sure about pefroamnce though.

- Too many databases: This should not be a big issue. Mongo does not go ahead and allocate a couple gigagbytes for each db, it uses incremental file sizes.

- Silent failures: Yep.. There it fails miserably. Recent versions are better though.

Re: Why MongoDB is a bad choice for storing our scraped data

#16
post #15

It's like people started complaining about MongoDB just for the sake of it. I guess it's the new trend? - Ordered data and skip / limit: These would run just fine on any database system. Given that you have appropriate indexes. It does not matter if there are a trillion items total, as long as you are seeking over an index and the result set is in reasonable size. - Restrictions: A lot of software has restrictions. F…

It isn't new, i remember people complaining about MongoDB since its first releases.

Re: Why MongoDB is a bad choice for storing our scraped data

#17
post #2

I read the whole post waiting to see what they ended up using as we are having similar issues, only to find that it's another post I have to wait for..

We went with HBase. Cassandra would have been suitable too, but we already use Hadoop for data processing so it was a natural choice within the infrastructure ecosystem. We will write a followup about that.

Clouderan here! Glad to hear you guys went with HBase, I'm looking forward to your follow up post. Will you detail your key design / architectural setup?

Did you guys roll your own HBase environment or did you go with the CDH? If you're using the CDH version and have any questions, feel free to shoot an email to cdh-user.

Re: Why MongoDB is a bad choice for storing our scraped data

#18
post #15

It's like people started complaining about MongoDB just for the sake of it. I guess it's the new trend? - Ordered data and skip / limit: These would run just fine on any database system. Given that you have appropriate indexes. It does not matter if there are a trillion items total, as long as you are seeking over an index and the result set is in reasonable size. - Restrictions: A lot of software has restrictions. F…

Why are you taking this personally? They're just listing reasons why it's not a good fit for them. Useful information to others who are trying to pick a database for similar applications.

Re: Why MongoDB is a bad choice for storing our scraped data

#19

I really don't understand why people use MongoDB. It seems like it's a elegant technological metaphor (lets use mmap, the OS is our cache and we can overwrite in place in RAM) that in practise turns out to be a terrible idea. Overwrite/mmap cannot be made reliable, requires blocking write-locks, wastes disk, and causes problems shuffling data around as it grows. Add other bad decisions (keys aren't interned, seriousl…

[deleted]

Re: Why MongoDB is a bad choice for storing our scraped data

#20
post #15

It's like people started complaining about MongoDB just for the sake of it. I guess it's the new trend? - Ordered data and skip / limit: These would run just fine on any database system. Given that you have appropriate indexes. It does not matter if there are a trillion items total, as long as you are seeking over an index and the result set is in reasonable size. - Restrictions: A lot of software has restrictions. F…

We are not plainly complaining about MongoDB, nor saying it's useless. We are just explaining why it's a poor choice for a specific use case: storing scraped data.

FWIW, we still use Mongo in other internal applications, it's just not the right choice for our crawl data storage backend.

Post reply on HN