Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why MongoDB is a bad choice for storing our scraped data

blog.scrapinghub.com

Post reply on HN
Print view View original

71–80 of 121 posts

Re: Why MongoDB is a bad choice for storing our scraped data

#71

Post by wiremine »

Quote from the original author in the comments. TLDR: it was human error that got us into this situation:

"The lack of joins & transactions of course did factor into the original decision. My point (which perhaps could be clearer) was that MongoDB ended up being used outside of the area in which we originally intended to use it. There was some reluctance to add another technology when we could get by with what we had for what was (initially) only a small use. Additionally, some limitations were not always well understood by web developers (who were new to mongo and enthusiastic to try it). I see this as our mistake. With hindsight, it’s clear we should have introduced an RDBMS immediately and kept MongoDB for managing the crawl data."

Re: Why MongoDB is a bad choice for storing our scraped data

#72

Post by nramirezuy »

JulianMorrison wrote: post #11

I really don't understand why people use MongoDB. It seems like it's a elegant technological metaphor (lets use mmap, the OS is our cache and we can overwrite in place in RAM) that in practise turns out to be a terrible idea. Overwrite/mmap cannot be made reliable, requires blocking write-locks, wastes disk, and causes problems shuffling data around as it grows. Add other bad decisions (keys aren't interned, seriousl…

Because is easy: * Easy to learn * Easy to use * Easy to deploy

All easy things get us in troubles when we want high performance. Cheers;)

Re: Why MongoDB is a bad choice for storing our scraped data

#73

Post by jeffdavis »

wiremine wrote: post #71

Quote from the original author in the comments. TLDR: it was human error that got us into this situation: "The lack of joins & transactions of course did factor into the original decision. My point (which perhaps could be clearer) was that MongoDB ended up being used outside of the area in which we originally intended to use it. There was some reluctance to add another technology when we could get by with what we had…

Databases almost always grow outside of their original scope, much more so than applications. And there is a good reason: data is more valuable when it's combined with other data.

So I think it's reasonable to be cautious about using a system that can't effectively grow outside of its initial special purpose.

Re: Why MongoDB is a bad choice for storing our scraped data

#76

Post by Jaigus »

eknkc wrote: post #23
kalleboo wrote: post #18

Earlier quoted context omitted.

Why are you taking this personally? They're just listing reasons why it's not a good fit for them. Useful information to others who are trying to pick a database for similar applications.

I'm sorry if it looks like I'm attacking the criticism. Nope, I would not use MongoDB ever again, after a year and a half with it. I have my reasons for this decision. I just don't like people bashing something without valid reasons. It might just be a perfect solution for similar applications, this is not a good way to evaluate.

>I just don't like people bashing something without valid reasons

That really doesn't seem to be the case here. Like you, the article's author(and several others here) have had issues with it for their particular use-case, and the reasons are clearly listed in a well organized paragraph by paragraph summary explanation in the article. Others here who've had a similar experience at least stated they had issues with it as well, even if they didn't go into much detail about it.

And speaking of the lack of valid reasons, to be fair, many relatively new technologies like these often get significant praise/hype without many valid reasons as well, other than [X]startup/company is using it, so it should be able to work for me, or it must be an awesome technology to use.

Re: Why MongoDB is a bad choice for storing our scraped data

#77

Post by mratzloff »

nosespray wrote: post #74

People aren't complaining just to complain. When you can't even ctrl-c out of the shell there is a huge issue. It's a known bug, around since v1.8, 'minor' priority. Yeah, thanks for trapping me in the shell.

Apparently this is the ticket being referred to:

https://jira.mongodb.org/browse/SERVER-2986?page=com.atlassi...

Re: Why MongoDB is a bad choice for storing our scraped data

#78

Post by ownagefool »

lucisferre wrote: post #49
timr wrote: post #43

Earlier quoted context omitted.

I quoted "developers", because we need a term to distinguish people who know basic computer science from people who know just enough to install software and piece together APIs. The latter group tends not to realize that things like overwriting your working set in memory and global write-locking lead inevitably to consistency and throughput issues. The primary problem in software today is that we've confused the abil…

In the real world though, both groups still need to use what works in practice. It's entirely possible for MongoDB to work sufficiently well for a certain group of people in a reasonably cost effective way. Exaggerating its problems (as bad as they are) doesn't add weight to your agrement. For example, global write-locking will not _inevitably_ lead to consistency or throughput unless the write frequencies are suffic…

SQL isn't the alternative, it's the standard and noSQL databases are supposed to offer extra value to cause you to migrate. According to these articles, MongoDB doesn't offer any real additional value, thus you shouldn't use it.

It's not about elitism, it's about making good decisions. That said, given all the hype with companies that hire, protesting loudly might not be the best short term personal decision. Meh.

Re: Why MongoDB is a bad choice for storing our scraped data

#79

Post by kanja »

snaky wrote: post #61
lmm wrote: post #48

Earlier quoted context omitted.

It is not nice to have to manipulate JSON as a plain string (you miss out on validation, have to manually construct like expressions for queries, and I don't even want to think about what you'd have to do to update part of a document), at least using "plain old SQL". (PostgreSQL's native JSON support would make it quite easy, but that supports my point)

If you need to query JSON content , why to store it like plain string, to begin with? Parse it at application level (or even in stored procedure) and store like ordinary fields. Or use hstore.

Postgres additionally has a json field type.

Re: Why MongoDB is a bad choice for storing our scraped data

#80

Post by papsosouid »

gnaritas wrote: post #70
papsosouid wrote: post #57

Earlier quoted context omitted.

Could you actually go into some detail about these mythical problems with actual databases? Faux database apologists seem to really love claiming databases are so unusable, but I've never gotten an actual explanation as to what problems they are having. As both a developer and a sysadmin, postgresql is much less of a pain in the ass than mongodb. And I have no idea what "don't mix well with the kinds of programs.." i…

> Faux database apologists seem to really love claiming databases are so unusable You really think tossing out insults like that is a way to have a reasoned conversation? I think not, come back when you can converse like an adult. > And I have no idea what "don't mix well with the kinds of programs.." is supposed to mean. Then you need more experience as a programmer perhaps. I'm a programer and a SQL guy, and I'm fu…

You made baseless assertions, and provided nothing to back them up. Your comment got precisely the response it deserves. Acting indignant does not support your assertions.

The fact that you have some unspecified problem does not mean anyone else who does not have that problem is inexperienced. Given the complete lack of information available, it is just as reasonable to conclude that you are in fact lacking in experience which allows others to solve the problem you continue to refuse to define.

Post reply on HN