Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why MongoDB is a bad choice for storing our scraped data

blog.scrapinghub.com

Post reply on HN
Print view View original

81–90 of 121 posts

Re: Why MongoDB is a bad choice for storing our scraped data

#81

Post by ams6110 »

JulianMorrison wrote: post #11

I really don't understand why people use MongoDB. It seems like it's a elegant technological metaphor (lets use mmap, the OS is our cache and we can overwrite in place in RAM) that in practise turns out to be a terrible idea. Overwrite/mmap cannot be made reliable, requires blocking write-locks, wastes disk, and causes problems shuffling data around as it grows. Add other bad decisions (keys aren't interned, seriousl…

I think the author summed it up well: "There is a niche where MongoDB can work well."

s/MongoDB/Any technology/g

The size of the niche varies. The lesson is to be sure the choices you make are appropriate for your situation, and be aware that things may change if new requirements emerge or scale needs to go beyond what you projected. These concerns are not specific to MongoDB. It is a rare project that goes from prototype to small scale to large scale on its original implementation technology choices.

Re: Why MongoDB is a bad choice for storing our scraped data

#82

Post by foobar1988 »

wiremine wrote: post #71

Quote from the original author in the comments. TLDR: it was human error that got us into this situation: "The lack of joins & transactions of course did factor into the original decision. My point (which perhaps could be clearer) was that MongoDB ended up being used outside of the area in which we originally intended to use it. There was some reluctance to add another technology when we could get by with what we had…

Proof that the OP was just looking for more Hacker News cred so they wrote about why MongoDB sucks but in reality wanted to discuss their "human error" :p Nice marketing guys

http://blog.serverdensity.com/does-everyone-hate-mongodb/

Re: Why MongoDB is a bad choice for storing our scraped data

#83

Post by dvt »

timr wrote: post #59
raganwald wrote: post #53

Earlier quoted context omitted.

we've confused the ability to build something with actually knowing anything of value Am I reading this correctly? It seems to imply that the ability to build something is somehow orthogonal to knowledge of value. I don't know about throwing mud into a heap and then calling it sculpture, but if we are talking about the subset of "things" that have value in and of themselves, the ability to build them does imply some…

"Am I reading this correctly? It seems to imply that the ability to build something is somehow orthogonal to knowledge of value." Not only are you reading it correctly, that is in fact (part of) what I'm saying. Building something doesn't automatically create value. We've confused the two.

Orthogonality generally implies mutual exclusiveness (which I don't think is what you're trying to say).

Re: Why MongoDB is a bad choice for storing our scraped data

#84

Post by IdahoEv »

timr wrote: post #43
lucisferre wrote: post #41

Earlier quoted context omitted.

You lost me when you quoted the word "developers". Edit: And this is downvoted for calling out the fact that people on HN can't discuss a freakin' database without hurling insults.

I quoted "developers", because we need a term to distinguish people who know basic computer science from people who know just enough to install software and piece together APIs. The latter group tends not to realize that things like overwriting your working set in memory and global write-locking lead inevitably to consistency and throughput issues. The primary problem in software today is that we've confused the abil…

OTOH, I've experienced a lot of people who know basic computer science but don't grasp any software engineering. Nor do they even realize that it's a thing.

Those sorts of people tend to be very focussed on clever algorithms and data structures, and frequently miss the larger picture and coding best practices. I've seen far too much code that had extensive CS cleverness at the root but was spaghettified, untested, undocumented, poorly performant, and not even tracked in a version control system. Such people often don't value writing code that other developers can read and maintain. On a team, clarity matters more than cleverness.

Good developers need both sets of skills.

Re: Why MongoDB is a bad choice for storing our scraped data

#85

Post by timr »

dvt wrote: post #83
timr wrote: post #59

Earlier quoted context omitted.

"Am I reading this correctly? It seems to imply that the ability to build something is somehow orthogonal to knowledge of value." Not only are you reading it correctly, that is in fact (part of) what I'm saying. Building something doesn't automatically create value. We've confused the two.

Orthogonality generally implies mutual exclusiveness (which I don't think is what you're trying to say).

No, it doesn't. Orthogonal means "at right angles", or "independent". The latter meaning applies.

Re: Why MongoDB is a bad choice for storing our scraped data

#86

Post by AlisdairO »

craigching wrote: post #69

Whenever I see the "You don't need Mongo DB, use an SQL database" and then in the flaming back and forth, I never see my key problem mentioned: MongoDB makes it easy to scale out (replica sets and sharding), where is the "easy to setup replicated and sharded open source SQL database?" I mean, I know that Postgres has replication (via Slony? honestly, it's been awhile since I looked at their solutions) but I don't rec…

Postgres has hot standby built in nowadays, and it works well.

Sharding certainly isn't as easy - the technical compromises that mongo makes make it pretty trivial to implement, whereas it's relatively hard to make it work in an RDBMS while maintaining all the expected capabilities. It generally requires some application-level work on open source dbs.

With that said, I really think many people grossly underestimate the effectiveness of scale-up. It's worth remembering that Stack Overflow (for example) is still running on a single pair of master/hot standby database machines.

Re: Why MongoDB is a bad choice for storing our scraped data

#87

Post by threeseed »

pablohoffman wrote: post #20
eknkc wrote: post #15

It's like people started complaining about MongoDB just for the sake of it. I guess it's the new trend? - Ordered data and skip / limit: These would run just fine on any database system. Given that you have appropriate indexes. It does not matter if there are a trillion items total, as long as you are seeking over an index and the result set is in reasonable size. - Restrictions: A lot of software has restrictions. F…

We are not plainly complaining about MongoDB, nor saying it's useless. We are just explaining why it's a poor choice for a specific use case: storing scraped data. FWIW, we still use Mongo in other internal applications, it's just not the right choice for our crawl data storage backend.

One issue is that many of these points are design characteristics of MongoDB and should have been known before hand. I am not criticising but it's almost like you did zero research before hand.

Transactions for example have never existed in MongoDB and joins doesn't really make much sense.

Re: Why MongoDB is a bad choice for storing our scraped data

#88

Post by dkhenry »

shane42 wrote: post #35
dkhenry wrote: post #3

you lost me here """ Ordered data Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting which is impractical when the number of records gets large. "" it requires _indexing_ and is quite feasable as I do it every day with stock ticker logs ( also required to be retrieved incrementially ) There are a few other flags that make me wonder about the exact l…

Take the example of crawl logs. Each log entry has a log level, timestamp and message. Typical use would be to view all ERROR (or higher) log levels, show all entries with a specific text in the message, or download the entire log. All of these should be in timestamp order. It's a shame that natural order is not insert order for non-capped collections. It's a good point that some of this can be achieved with indexing…

Right for that example ,not knowing the specifics, I would index on timestamp and then I could sort by timestamp which once indexed should be a relativly fast operation. I could even go one step better and make the _id a construct of { ,, } Then Inserting would be done in order and I would get a magic index for free (depending on if I often query by loglevel I might leave it out) This gives locality of timestamps and helps keep "Hot" sections in memory.

Re: Why MongoDB is a bad choice for storing our scraped data

#89

Post by threeseed »

timr wrote: post #43
lucisferre wrote: post #41

Earlier quoted context omitted.

You lost me when you quoted the word "developers". Edit: And this is downvoted for calling out the fact that people on HN can't discuss a freakin' database without hurling insults.

I quoted "developers", because we need a term to distinguish people who know basic computer science from people who know just enough to install software and piece together APIs. The latter group tends not to realize that things like overwriting your working set in memory and global write-locking lead inevitably to consistency and throughput issues. The primary problem in software today is that we've confused the abil…

Sorry but I don't think you know what you're talking about here.

Write locking definitely leads to throughput issues but it results in better consistency not less.

Re: Why MongoDB is a bad choice for storing our scraped data

#90

Post by threeseed »

papsosouid wrote: post #57
gnaritas wrote: post #54

Earlier quoted context omitted.

> In the real world, people have been using relational databases to solve problems for years. They work, they're understood, they scale. And they're a pain the ass and don't mix well with the kinds of programs many want to write. Mongo clearly fills a niche that relational databases don't serve well; if it didn't, no one would use it.

Could you actually go into some detail about these mythical problems with actual databases? Faux database apologists seem to really love claiming databases are so unusable, but I've never gotten an actual explanation as to what problems they are having. As both a developer and a sysadmin, postgresql is much less of a pain in the ass than mongodb. And I have no idea what "don't mix well with the kinds of programs.." i…

So the guys at Foursquare are driven by "fads" and don't have a clue about databases or scaling ?
Post reply on HN