Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why MongoDB is a bad choice for storing our scraped data

blog.scrapinghub.com

Post reply on HN
Print view View original

101–110 of 121 posts

Re: Why MongoDB is a bad choice for storing our scraped data

#101

Post by AlisdairO »

threeseed wrote: post #93
AlisdairO wrote: post #86

Earlier quoted context omitted.

Postgres has hot standby built in nowadays, and it works well. Sharding certainly isn't as easy - the technical compromises that mongo makes make it pretty trivial to implement, whereas it's relatively hard to make it work in an RDBMS while maintaining all the expected capabilities. It generally requires some application-level work on open source dbs. With that said, I really think many people grossly underestimate t…

Standby is a pretty poor solution compared to replica set let alone what Cassandra has to offer. Sharding is trivial on MongoDB/Cassandra and it is open source. So let's be accurate here. It is a problem inherent with the SQL databases. And I think you underestimate the benefits of scaling out. If I want to ensure close to 100% uptime or have a server closer to my users than Cassandra or even MongoDB would be infinit…

Sharding is, of course, trivial on those systems - after all, they're extremely feature-poor, and have given up those features specifically to support trivial sharding.

You can replicate much of this behaviour using open source RDBMSs, but yeah, it's not what they're designed for and it's harder. If you want quality replication/clustering you're currently looking at paid-for DBs.

Being able to scale out is absolutely a nice-to-have. I'm not sure it's a nice-to-have on the scale of giving up all of the features an RDBMS provides for most people's use-cases. Further, you might find that the relative lack of data headaches you get with an RDBMS more than makes up for a little extra time setting up hot standby.

Finally, if 100% uptime is that important you're probably not relying on a relatively niche NoSQL database. If uptime on the level of Stack Overflow is good enough (which for most people it probably is, let's face it), then you'll probably find replicated postgres good enough.

Re: Why MongoDB is a bad choice for storing our scraped data

#102

Post by Zenst »

gnaritas wrote: post #70
papsosouid wrote: post #57

Earlier quoted context omitted.

Could you actually go into some detail about these mythical problems with actual databases? Faux database apologists seem to really love claiming databases are so unusable, but I've never gotten an actual explanation as to what problems they are having. As both a developer and a sysadmin, postgresql is much less of a pain in the ass than mongodb. And I have no idea what "don't mix well with the kinds of programs.." i…

> Faux database apologists seem to really love claiming databases are so unusable You really think tossing out insults like that is a way to have a reasoned conversation? I think not, come back when you can converse like an adult. > And I have no idea what "don't mix well with the kinds of programs.." is supposed to mean. Then you need more experience as a programmer perhaps. I'm a programer and a SQL guy, and I'm fu…

It depends upon the application, He is right and you also are right for the feilds of programming and applications you do, gets down to pro's and cons of types of interface to the database depending upon the application at hand. We all know the variations of those. Though perhaps could of been toned right. You are just going to argue over differing types of application without saying which type.

In short you are both right, applications and also use of said applications and requirments make the difference as to what interface is best. Don't need to argue over that without even stating it. You both win, this is the internet - now laugh :).

Re: Why MongoDB is a bad choice for storing our scraped data

#103

Post by Zenst »

papsosouid wrote: post #80
gnaritas wrote: post #70

Earlier quoted context omitted.

> Faux database apologists seem to really love claiming databases are so unusable You really think tossing out insults like that is a way to have a reasoned conversation? I think not, come back when you can converse like an adult. > And I have no idea what "don't mix well with the kinds of programs.." is supposed to mean. Then you need more experience as a programmer perhaps. I'm a programer and a SQL guy, and I'm fu…

You made baseless assertions, and provided nothing to back them up. Your comment got precisely the response it deserves. Acting indignant does not support your assertions. The fact that you have some unspecified problem does not mean anyone else who does not have that problem is inexperienced. Given the complete lack of information available, it is just as reasonable to conclude that you are in fact lacking in experi…

+1 though your both right depending on the applicatiton and with that you will go from your experience and both be right without specifying in detail an example and neither of you want to go down to doing specs on a forum to win a argument that you both will win and end up arguing about the applicaiton and specs.

I say payroll databases at dawn, 50 paces each, turn and shoot. Go :-)

Re: Why MongoDB is a bad choice for storing our scraped data

#104

Post by JulianMorrison »

ams6110 wrote: post #81
JulianMorrison wrote: post #11

I really don't understand why people use MongoDB. It seems like it's a elegant technological metaphor (lets use mmap, the OS is our cache and we can overwrite in place in RAM) that in practise turns out to be a terrible idea. Overwrite/mmap cannot be made reliable, requires blocking write-locks, wastes disk, and causes problems shuffling data around as it grows. Add other bad decisions (keys aren't interned, seriousl…

I think the author summed it up well: "There is a niche where MongoDB can work well." s/MongoDB/Any technology/g The size of the niche varies. The lesson is to be sure the choices you make are appropriate for your situation, and be aware that things may change if new requirements emerge or scale needs to go beyond what you projected. These concerns are not specific to MongoDB. It is a rare project that goes from prot…

MongoDB is the Pinto of databases. It is not safe at any scale.

Re: Why MongoDB is a bad choice for storing our scraped data

#105

Post by Luyt »

papsosouid wrote: post #57
gnaritas wrote: post #54

Earlier quoted context omitted.

> In the real world, people have been using relational databases to solve problems for years. They work, they're understood, they scale. And they're a pain the ass and don't mix well with the kinds of programs many want to write. Mongo clearly fills a niche that relational databases don't serve well; if it didn't, no one would use it.

Could you actually go into some detail about these mythical problems with actual databases? Faux database apologists seem to really love claiming databases are so unusable, but I've never gotten an actual explanation as to what problems they are having. As both a developer and a sysadmin, postgresql is much less of a pain in the ass than mongodb. And I have no idea what "don't mix well with the kinds of programs.." i…

people choose it because "it is web scale" while having no idea what they are even supposed to be comparing it to

Some compare it to relational databases... http://www.mongodb-is-web-scale.com/

Re: Why MongoDB is a bad choice for storing our scraped data

#106

Post by gnaritas »

papsosouid wrote: post #80
gnaritas wrote: post #70

Earlier quoted context omitted.

> Faux database apologists seem to really love claiming databases are so unusable You really think tossing out insults like that is a way to have a reasoned conversation? I think not, come back when you can converse like an adult. > And I have no idea what "don't mix well with the kinds of programs.." is supposed to mean. Then you need more experience as a programmer perhaps. I'm a programer and a SQL guy, and I'm fu…

You made baseless assertions, and provided nothing to back them up. Your comment got precisely the response it deserves. Acting indignant does not support your assertions. The fact that you have some unspecified problem does not mean anyone else who does not have that problem is inexperienced. Given the complete lack of information available, it is just as reasonable to conclude that you are in fact lacking in experi…

And which assertion was that; the one that Mongo fits a niche which it obviously does, or the one that many people find relational databases a pain, which they obviously do. Neither of those require me to provide evidence, they are self evident facts to anyone with even moderate experience in the field. I don't have an unspecified problem, not once did I even mention a problem, so take your childish argumentative b.s. somewhere else.

Re: Why MongoDB is a bad choice for storing our scraped data

#107

Post by nasalgoat »

craigching wrote: post #99
nasalgoat wrote: post #95

Earlier quoted context omitted.

That "easy to scale out" is a misnomer. Replica sets and sharding work in the technical sense, but the implementation isn't anywhere near what I would qualify as production ready. For example, today my entire production MongoDB database was running 3x slower because a single replica in one shard was down, and their buggy PHP driver kept trying to talk to it despite it being marked down. I really enjoyed waking up at…

But Mongo DB being buggy isn't a reason to need to use an SQL database vs. a NoSQL store. An SQL database could be buggy as well (I still use Postgres and comparing anything to that quality-wise is just going to bring sorrow for the thing you compare it to ;) ). FWIW, it's been spotless for us so far. Our needs aren't web scale, but they're big enough to need scaling features.

The main issue with MongoDB is that it's so easy to use and seems like it scales, but soon you're invested in it to the point of refactoring being a serious engineering effort, and you're stuck with something that doesn't actually offer real scaling features.

So, it's less "nosql vs. sql" and more just "don't use mongodb".

Re: Why MongoDB is a bad choice for storing our scraped data

#108

Post by ownagefool »

threeseed wrote: post #92
ownagefool wrote: post #78

Earlier quoted context omitted.

SQL isn't the alternative, it's the standard and noSQL databases are supposed to offer extra value to cause you to migrate. According to these articles, MongoDB doesn't offer any real additional value, thus you shouldn't use it. It's not about elitism, it's about making good decisions. That said, given all the hype with companies that hire, protesting loudly might not be the best short term personal decision. Meh.

These articles are just one side of the picture which gets heavily upvoted on HN. And of course it is about elitism. Listen to yourself. "It's about making good decisions". I mean who are you to judge from the outside what technology a company should use for a specific use cases ?

I wasn't specifically talking about the articles, just in general point that'll I'm happy to stand behind. I'll happily repeat it: unless you have a specfic use case, RDBMS is the default and it should be so. Now if you have a specific use case, fair enough, but the aforementioned examples don't honestly seem to be valid reason to throw our the advantages of RDBMS.

Re: Why MongoDB is a bad choice for storing our scraped data

#109

Post by rdtsc »

JulianMorrison wrote: post #11

I really don't understand why people use MongoDB. It seems like it's a elegant technological metaphor (lets use mmap, the OS is our cache and we can overwrite in place in RAM) that in practise turns out to be a terrible idea. Overwrite/mmap cannot be made reliable, requires blocking write-locks, wastes disk, and causes problems shuffling data around as it grows. Add other bad decisions (keys aren't interned, seriousl…

Marketing.

They are also on my blacklist, forever.

They used to ship with unacknowledged writes as the default option. Think about it for a little, a database that just throws your _data_ over the fence and prays for the best, proceeding without an write acknowledgement.

There was not flashing warning on their front page about, no bold disclaimers, but there were sure plenty of "Oh look super fast benchmarks beating SQL and other NoSQL database, albeit created by fanboys".

That decision told me the story of who they are and what kind of principles they use to build their product. It wasn't a mistake it was a deliberate shady tactic employed.

(Yes, I know I have written about it at least 3 times before and will mention it every time I see MongoDB mentioned )

Re: Why MongoDB is a bad choice for storing our scraped data

#110

Post by rdtsc »

eknkc wrote: post #15

It's like people started complaining about MongoDB just for the sake of it. I guess it's the new trend? - Ordered data and skip / limit: These would run just fine on any database system. Given that you have appropriate indexes. It does not matter if there are a trillion items total, as long as you are seeking over an index and the result set is in reasonable size. - Restrictions: A lot of software has restrictions. F…

> It's like people started complaining about MongoDB just for the sake of it. I guess it's the new trend?

All the HN crowd went insane at once or, who knows, maybe they found out one product that is relatively heavily marketed is mostly blowing smoke up everyone's asses.

The complaining is vis-a-vis the marketing and perceived fan-boyism. It might also not be completely bad news as it means people are still using it.

Post reply on HN