Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why MongoDB is a bad choice for storing our scraped data

blog.scrapinghub.com

Post reply on HN
Print view View original

51–60 of 121 posts

Re: Why MongoDB is a bad choice for storing our scraped data

#51

Post by joydeepdg »

timr wrote: post #36
aroman wrote: post #24

Earlier quoted context omitted.

Because when you're not operating at significant scale, or have certain specific use cases, it's a fantastically elegant solution and one that's very quick and easy to set up. I have used mongodb for a number of smaller projects, and I have had an excellent experience. It's not "a terrible idea in practice". It might be a terrible fit for what you want , but that doesn't mean it's bad technology.

"when you're not operating at significant scale, or have certain specific use cases, it's a fantastically elegant solution and one that's very quick and easy to set up." When you're not operating at significant scale, you can use a relational database. They're easy and fast to set up, have nice write-safety guarantees, are more flexible than a key-value store, and will scale well beyond anything that mongo has ever a…

We use MongoDB in production for a couple of use cases: 1> e-Commerce Product Catalog 2> Home grown CMS for our news/editorial site

Both these applications have been in production for about a year without a single problem. Both are using the same MongoDB instance - data size is about 200 GB (RAM on the Mongo machine is 16 GB)

Both these applications were previously on Oracle and were a pain to maintain. The Mongo schema is simpler and far more maintainable then the RDBMS schema. Backups/Monitoring/Replication on Mongo has never given us any problems.

Now, since you claim that an RDBMS can do anything better than MongoDB, can you point me to a simple/elegant/maintainable RDBMS schema for an e-Commerce Product Catalog? I would love to see one.

The original post, like most of the 'Why we moved away from MongoDB' posts displays a shocking lack of due-diligence on the part of the development team / tech lead at these firms. All the points under the 'Data that should be good, ends up bad!' section are known facts about MongoDB. All of them are covered in the manual. If you are not fine with any of these points - please don't use MongoDB at all. Don't put it in production. It baffles me how these firms can put MongoDB into production and 'discover' these things later. Instead of ranting at MongoDB, the CTO's of all these firms deserve the sack for lack of due-diligence and putting data at risk.

One last point:

> ..more flexible than a key-value store

MongoDB is not a key-value store.

Re: Why MongoDB is a bad choice for storing our scraped data

#52

Post by timr »

lucisferre wrote: post #49
timr wrote: post #43

Earlier quoted context omitted.

I quoted "developers", because we need a term to distinguish people who know basic computer science from people who know just enough to install software and piece together APIs. The latter group tends not to realize that things like overwriting your working set in memory and global write-locking lead inevitably to consistency and throughput issues. The primary problem in software today is that we've confused the abil…

In the real world though, both groups still need to use what works in practice. It's entirely possible for MongoDB to work sufficiently well for a certain group of people in a reasonably cost effective way. Exaggerating its problems (as bad as they are) doesn't add weight to your agrement. For example, global write-locking will not _inevitably_ lead to consistency or throughput unless the write frequencies are suffic…

"In the real world though, both groups still need use what works in practice."

In the real world, people have been using relational databases to solve problems for years. They work, they're understood, they scale.

"global write-locking will not _inevitably_ lead to consistency or throughput unless the write frequencies are sufficient to cause and require that"

In which case, you can just as easily use a relational database and avoid the chance of problems altogether.

Re: Why MongoDB is a bad choice for storing our scraped data

#53

Post by raganwald »

timr wrote: post #43
lucisferre wrote: post #41

Earlier quoted context omitted.

You lost me when you quoted the word "developers". Edit: And this is downvoted for calling out the fact that people on HN can't discuss a freakin' database without hurling insults.

I quoted "developers", because we need a term to distinguish people who know basic computer science from people who know just enough to install software and piece together APIs. The latter group tends not to realize that things like overwriting your working set in memory and global write-locking lead inevitably to consistency and throughput issues. The primary problem in software today is that we've confused the abil…

we've confused the ability to build something with actually knowing anything of value

Am I reading this correctly? It seems to imply that the ability to build something is somehow orthogonal to knowledge of value.

I don't know about throwing mud into a heap and then calling it sculpture, but if we are talking about the subset of "things" that have value in and of themselves, the ability to build them does imply some knowledge of value.

Now, the relative value of knowing how to put together simple web sites using jQuery vs. the knowledge to discover Chaitin's Constant is very much worthy of discussion. But likewise, the knowledge of how to construct a true but unproveable statement in a toy system vs the knowledge of how to build VisiCalc and revolutionize programming is worthy of discussion as well.

Re: Why MongoDB is a bad choice for storing our scraped data

#54

Post by gnaritas »

timr wrote: post #52
lucisferre wrote: post #49

Earlier quoted context omitted.

In the real world though, both groups still need to use what works in practice. It's entirely possible for MongoDB to work sufficiently well for a certain group of people in a reasonably cost effective way. Exaggerating its problems (as bad as they are) doesn't add weight to your agrement. For example, global write-locking will not _inevitably_ lead to consistency or throughput unless the write frequencies are suffic…

"In the real world though, both groups still need use what works in practice." In the real world, people have been using relational databases to solve problems for years. They work, they're understood, they scale. "global write-locking will not _inevitably_ lead to consistency or throughput unless the write frequencies are sufficient to cause and require that" In which case, you can just as easily use a relational da…

> In the real world, people have been using relational databases to solve problems for years. They work, they're understood, they scale.

And they're a pain the ass and don't mix well with the kinds of programs many want to write. Mongo clearly fills a niche that relational databases don't serve well; if it didn't, no one would use it.

Re: Why MongoDB is a bad choice for storing our scraped data

#55

Post by bengillies »

lucisferre wrote: post #41
timr wrote: post #36

Earlier quoted context omitted.

"when you're not operating at significant scale, or have certain specific use cases, it's a fantastically elegant solution and one that's very quick and easy to set up." When you're not operating at significant scale, you can use a relational database. They're easy and fast to set up, have nice write-safety guarantees, are more flexible than a key-value store, and will scale well beyond anything that mongo has ever a…

You lost me when you quoted the word "developers". Edit: And this is downvoted for calling out the fact that people on HN can't discuss a freakin' database without hurling insults.

> Edit: And this is downvoted for calling out the fact that people on HN can't discuss a freakin' database without hurling insults.

The original comment you're replying to aside, it was likely because stating that you dismissed the entire comment without giving an actual objection to it added nothing at all to the discussion.

Re: Why MongoDB is a bad choice for storing our scraped data

#56

Post by andrix »

gnaritas wrote: post #54
timr wrote: post #52

Earlier quoted context omitted.

"In the real world though, both groups still need use what works in practice." In the real world, people have been using relational databases to solve problems for years. They work, they're understood, they scale. "global write-locking will not _inevitably_ lead to consistency or throughput unless the write frequencies are sufficient to cause and require that" In which case, you can just as easily use a relational da…

> In the real world, people have been using relational databases to solve problems for years. They work, they're understood, they scale. And they're a pain the ass and don't mix well with the kinds of programs many want to write. Mongo clearly fills a niche that relational databases don't serve well; if it didn't, no one would use it.

I think it's not a fight. RDBMS has been there for many years and they have proved to work in many areas. NoSQL databases born for new needs people was asking to have in their new projects. Both would probably works perfect for many cases, but nosql databases are very suitable for scenarios when you do not require an strict schema, and also they are simple to setup.

I still think MongoDB is great for many applications as many companies are using it for their data needs (like Foresquare), and the same with RDBMS like MySQL, that lot of big fishes use them for different parts of their architecture (facebook, twitter, etc). In the end, each option has pros/cons, but one will be better for your use case.

Re: Why MongoDB is a bad choice for storing our scraped data

#57

Post by papsosouid »

gnaritas wrote: post #54
timr wrote: post #52

Earlier quoted context omitted.

"In the real world though, both groups still need use what works in practice." In the real world, people have been using relational databases to solve problems for years. They work, they're understood, they scale. "global write-locking will not _inevitably_ lead to consistency or throughput unless the write frequencies are sufficient to cause and require that" In which case, you can just as easily use a relational da…

> In the real world, people have been using relational databases to solve problems for years. They work, they're understood, they scale. And they're a pain the ass and don't mix well with the kinds of programs many want to write. Mongo clearly fills a niche that relational databases don't serve well; if it didn't, no one would use it.

Could you actually go into some detail about these mythical problems with actual databases? Faux database apologists seem to really love claiming databases are so unusable, but I've never gotten an actual explanation as to what problems they are having. As both a developer and a sysadmin, postgresql is much less of a pain in the ass than mongodb. And I have no idea what "don't mix well with the kinds of programs.." is supposed to mean.

The idea that "it must be good for something or people wouldn't use it" is absurd. People do the wrong thing all the time. People make technical decisions based on fads constantly. Mongodb is one of the prime examples of fad driven development choices, where people choose it because "it is web scale" while having no idea what they are even supposed to be comparing it to.

Re: Why MongoDB is a bad choice for storing our scraped data

#58

Post by snaky »

JulianMorrison wrote: post #11

I really don't understand why people use MongoDB. It seems like it's a elegant technological metaphor (lets use mmap, the OS is our cache and we can overwrite in place in RAM) that in practise turns out to be a terrible idea. Overwrite/mmap cannot be made reliable, requires blocking write-locks, wastes disk, and causes problems shuffling data around as it grows. Add other bad decisions (keys aren't interned, seriousl…

Because the koolaid is so sweet.

MongoDB is every developer's wet dream. With it's expressive query syntax and extreme ease of use, everyone wants to drink the koolaid. This is a huge problem, because mongodb as a database is dangerous

[..]

I have developers begging me to let them use it. This time to collect logs from our servers for analysis later. I cave in, and give my go ahead, with a warning saying that no critical data can enter that section. Mongo processes were crashing. Several times per day. About 20% of the crashes yielded a completely corrupted database. This programmers wet dream quickly shows itself to be a serious operations nightmare.

http://hackingdistributed.com/2013/02/07/10gen-response/#com...

Re: Why MongoDB is a bad choice for storing our scraped data

#59

Post by timr »

raganwald wrote: post #53
timr wrote: post #43

Earlier quoted context omitted.

I quoted "developers", because we need a term to distinguish people who know basic computer science from people who know just enough to install software and piece together APIs. The latter group tends not to realize that things like overwriting your working set in memory and global write-locking lead inevitably to consistency and throughput issues. The primary problem in software today is that we've confused the abil…

we've confused the ability to build something with actually knowing anything of value Am I reading this correctly? It seems to imply that the ability to build something is somehow orthogonal to knowledge of value. I don't know about throwing mud into a heap and then calling it sculpture, but if we are talking about the subset of "things" that have value in and of themselves, the ability to build them does imply some…

"Am I reading this correctly? It seems to imply that the ability to build something is somehow orthogonal to knowledge of value."

Not only are you reading it correctly, that is in fact (part of) what I'm saying. Building something doesn't automatically create value. We've confused the two.

Re: Why MongoDB is a bad choice for storing our scraped data

#60

Post by snaky »

timr wrote: post #36
aroman wrote: post #24

Earlier quoted context omitted.

Because when you're not operating at significant scale, or have certain specific use cases, it's a fantastically elegant solution and one that's very quick and easy to set up. I have used mongodb for a number of smaller projects, and I have had an excellent experience. It's not "a terrible idea in practice". It might be a terrible fit for what you want , but that doesn't mean it's bad technology.

"when you're not operating at significant scale, or have certain specific use cases, it's a fantastically elegant solution and one that's very quick and easy to set up." When you're not operating at significant scale, you can use a relational database. They're easy and fast to set up, have nice write-safety guarantees, are more flexible than a key-value store, and will scale well beyond anything that mongo has ever a…

When you're not operating at significant scale, you can use a relational database.

Yes, if you know how to do it. But posts like https://news.ycombinator.com/item?id=5675902 and questions 'Should I learn SQL?' here and there make me think that's not required knowledge these days.

Post reply on HN