Why MongoDB is a bad choice for storing our scraped data
blog.scrapinghub.com
Why MongoDB is a bad choice for storing our scraped data
1–10 of 121 posts
Re: Why MongoDB is a bad choice for storing our scraped data
#2I read the whole post waiting to see what they ended up using as we are having similar issues, only to find that it's another post I have to wait for..
Re: Why MongoDB is a bad choice for storing our scraped data
#3you lost me here
""" Ordered data
Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting which is impractical when the number of records gets large. ""
it requires _indexing_ and is quite feasable as I do it every day with stock ticker logs ( also required to be retrieved incrementially )
There are a few other flags that make me wonder about the exact limitations you found, but I will be anticipating your follow up post to see what your fix was since some of those issues are very common.
Re: Why MongoDB is a bad choice for storing our scraped data
#4I read the whole post waiting to see what they ended up using as we are having similar issues, only to find that it's another post I have to wait for..
Me too, except I skipped straight to the end :)
Re: Why MongoDB is a bad choice for storing our scraped data
#5I read the whole post waiting to see what they ended up using as we are having similar issues, only to find that it's another post I have to wait for..
Just a plain old RDMS?
Re: Why MongoDB is a bad choice for storing our scraped data
#6I read the whole post waiting to see what they ended up using as we are having similar issues, only to find that it's another post I have to wait for..
Look into Cassandra. Very fast writes and scales linearly (although there is some elbow grease involved in keyspace distribution). Our analytics platform guys are very happy with it, after flirting with several other options (including big, beefy RDBMS). Great for large, flat, denormalized tables.
Re: Why MongoDB is a bad choice for storing our scraped data
#7I read the whole post waiting to see what they ended up using as we are having similar issues, only to find that it's another post I have to wait for..
Based on their use case, I'd expect they went with either HBase or Cassandra. I'm quite partial to HBase, it's insanely scalable and has a lot of pretty amazing features, but at the cost of knowing exactly what you want to do with your application beforehand.
I'm not too familiar with Cassandra, but the scalability of an HBase table is almost entirely dependent on your key design. Judging from their use case and requirements, they would likely use a incremental key design which would allow for super fast range scans, of course, this leads to region server hotspotting, which may or not may not be a big deal to them.
Re: Why MongoDB is a bad choice for storing our scraped data
#8you lost me here """ Ordered data Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting which is impractical when the number of records gets large. "" it requires _indexing_ and is quite feasable as I do it every day with stock ticker logs ( also required to be retrieved incrementially ) There are a few other flags that make me wonder about the exact l…
[deleted]
Re: Why MongoDB is a bad choice for storing our scraped data
#9I read the whole post waiting to see what they ended up using as we are having similar issues, only to find that it's another post I have to wait for..
We went with HBase. Cassandra would have been suitable too, but we already use Hadoop for data processing so it was a natural choice within the infrastructure ecosystem. We will write a followup about that.
Re: Why MongoDB is a bad choice for storing our scraped data
#10you lost me here """ Ordered data Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting which is impractical when the number of records gets large. "" it requires _indexing_ and is quite feasable as I do it every day with stock ticker logs ( also required to be retrieved incrementially ) There are a few other flags that make me wonder about the exact l…
No kidding. Without details, it really sounds like the author is a bit clueless.
He mentions the lack of joins, but doesn't say a word about Mapreduce.
"MongoDB needs to walk the index from the beginning to the offset..." You don't "walk an index". It's an index.
"Too many databases" sounds a little suspicious. Why not add an indexed field to partition records?
Complaining about a lack of schema, transactions and triggers? Really? Did you read the docs at all before starting?
MongoDB is not without its problems, but friend, I think you wanted either Postgres or Hadoop.