Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why MongoDB is a bad choice for storing our scraped data

blog.scrapinghub.com

Post reply on HN
Print view View original

41–50 of 121 posts

Re: Why MongoDB is a bad choice for storing our scraped data

#41

Post by lucisferre »

timr wrote: post #36
aroman wrote: post #24

Earlier quoted context omitted.

Because when you're not operating at significant scale, or have certain specific use cases, it's a fantastically elegant solution and one that's very quick and easy to set up. I have used mongodb for a number of smaller projects, and I have had an excellent experience. It's not "a terrible idea in practice". It might be a terrible fit for what you want , but that doesn't mean it's bad technology.

"when you're not operating at significant scale, or have certain specific use cases, it's a fantastically elegant solution and one that's very quick and easy to set up." When you're not operating at significant scale, you can use a relational database. They're easy and fast to set up, have nice write-safety guarantees, are more flexible than a key-value store, and will scale well beyond anything that mongo has ever a…

You lost me when you quoted the word "developers".

Edit: And this is downvoted for calling out the fact that people on HN can't discuss a freakin' database without hurling insults.

Re: Why MongoDB is a bad choice for storing our scraped data

#42

Post by gaius »

lmm wrote: post #37
JulianMorrison wrote: post #26

Earlier quoted context omitted.

Are you dead certain those projects aren't ever eating data, and nor are they about to crash tomorrow with an unrecoverably hosed DB? I still say it's bad technology. Use plain old SQL instead.

"plain old SQL" can require a lot of mangling one's data to fit its constraints. I refuse to believe that there isn't a better key-value store for the case where the values are json documents, even if mongodb isn't it.

Except it isn't "SQL" it's first-order predicate calculus - a provably sound way to store and query your data.

Re: Why MongoDB is a bad choice for storing our scraped data

#43

Post by timr »

lucisferre wrote: post #41
timr wrote: post #36

Earlier quoted context omitted.

"when you're not operating at significant scale, or have certain specific use cases, it's a fantastically elegant solution and one that's very quick and easy to set up." When you're not operating at significant scale, you can use a relational database. They're easy and fast to set up, have nice write-safety guarantees, are more flexible than a key-value store, and will scale well beyond anything that mongo has ever a…

You lost me when you quoted the word "developers". Edit: And this is downvoted for calling out the fact that people on HN can't discuss a freakin' database without hurling insults.

I quoted "developers", because we need a term to distinguish people who know basic computer science from people who know just enough to install software and piece together APIs. The latter group tends not to realize that things like overwriting your working set in memory and global write-locking lead inevitably to consistency and throughput issues.

The primary problem in software today is that we've confused the ability to build something with actually knowing anything of value.

Re: Why MongoDB is a bad choice for storing our scraped data

#44

Post by mratzloff »

JulianMorrison wrote: post #11

I really don't understand why people use MongoDB. It seems like it's a elegant technological metaphor (lets use mmap, the OS is our cache and we can overwrite in place in RAM) that in practise turns out to be a terrible idea. Overwrite/mmap cannot be made reliable, requires blocking write-locks, wastes disk, and causes problems shuffling data around as it grows. Add other bad decisions (keys aren't interned, seriousl…

I was required to use MongoDB in a recent large-scale analytics project. It is a disaster. I am in the process of replacing the most important metrics with Redis; I can't ditch Mongo fast enough.

Re: Why MongoDB is a bad choice for storing our scraped data

#45

Post by swah »

monstrado wrote: post #12
dkhenry wrote: post #3

you lost me here """ Ordered data Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting which is impractical when the number of records gets large. "" it requires _indexing_ and is quite feasable as I do it every day with stock ticker logs ( also required to be retrieved incrementially ) There are a few other flags that make me wonder about the exact l…

With a database like HBase, it's already ordered lexicographically which makes it easy to grab a range of data in the order it was written in. You could have a key design like - which would allow quick scans over large amounts of data. ie..Scan from -1368536860 to -1368540450 HBase is multidimensional though, which allows you to keep N numbers of versions of a cell. By default you will get the latest version of the c…

The last time I looked there weren't many resources explaining how to design NoSQL databases (how to compose your keys, when to avoid normalization, etc). Has this improved?

Re: Why MongoDB is a bad choice for storing our scraped data

#46

Post by pwg »

lmm wrote: post #37
JulianMorrison wrote: post #26

Earlier quoted context omitted.

Are you dead certain those projects aren't ever eating data, and nor are they about to crash tomorrow with an unrecoverably hosed DB? I still say it's bad technology. Use plain old SQL instead.

"plain old SQL" can require a lot of mangling one's data to fit its constraints. I refuse to believe that there isn't a better key-value store for the case where the values are json documents, even if mongodb isn't it.

But if one is going to use a key-value store anyway, where is the "mangling"? Building a key-value store in an SQL database is trivial (this below is for PostgreSQL):

    create table keyvalue (key text, value text);
    create index keyvalue_idx on keyvalue(key);
And use is trivial as well:

    insert into keyvalue (key, value) values ('key', 'value');

    select value from keyvalue where key = 'key';
[edit] formatting fix.

Re: Why MongoDB is a bad choice for storing our scraped data

#47

Post by wslh »

JulianMorrison wrote: post #39
wslh wrote: post #31

Earlier quoted context omitted.

> I really don't understand why people use MongoDB. Really? When I don't want to think in databases and only on persisting my native types in my favorite programming language MongoDB is my choice. I use it only for experimenting, so I don't care about all those scalability issues.

Try Redis instead? Or rethinkdb (if you don't mind it being a bit new)?

No. The Redis interfaces doesn't abstract data types. For example Python types are quasi transparently interfaced with MongoDB.

Was your comment a little bit ironic?

Re: Why MongoDB is a bad choice for storing our scraped data

#48

Post by lmm »

pwg wrote: post #46
lmm wrote: post #37

Earlier quoted context omitted.

"plain old SQL" can require a lot of mangling one's data to fit its constraints. I refuse to believe that there isn't a better key-value store for the case where the values are json documents, even if mongodb isn't it.

But if one is going to use a key-value store anyway, where is the "mangling"? Building a key-value store in an SQL database is trivial (this below is for PostgreSQL): create table keyvalue (key text, value text); create index keyvalue_idx on keyvalue(key); And use is trivial as well: insert into keyvalue (key, value) values ('key', 'value'); select value from keyvalue where key = 'key'; [edit] formatting fix.

It is not nice to have to manipulate JSON as a plain string (you miss out on validation, have to manually construct like expressions for queries, and I don't even want to think about what you'd have to do to update part of a document), at least using "plain old SQL". (PostgreSQL's native JSON support would make it quite easy, but that supports my point)

Re: Why MongoDB is a bad choice for storing our scraped data

#49

Post by lucisferre »

timr wrote: post #43
lucisferre wrote: post #41

Earlier quoted context omitted.

You lost me when you quoted the word "developers". Edit: And this is downvoted for calling out the fact that people on HN can't discuss a freakin' database without hurling insults.

I quoted "developers", because we need a term to distinguish people who know basic computer science from people who know just enough to install software and piece together APIs. The latter group tends not to realize that things like overwriting your working set in memory and global write-locking lead inevitably to consistency and throughput issues. The primary problem in software today is that we've confused the abil…

In the real world though, both groups still need to use what works in practice. It's entirely possible for MongoDB to work sufficiently well for a certain group of people in a reasonably cost effective way. Exaggerating its problems (as bad as they are) doesn't add weight to your agrement. For example, global write-locking will not _inevitably_ lead to consistency or throughput unless the write frequencies are sufficient to cause and require that. Lots of data problems aren't "big data", they're barely a medium.

I'll agree MongoDB is not a terribly well engineered database however I don't agree that SQL is always the best alternative in these simpler scenarios. There are lots of things wrong with using SQL to solve every data problem. I also don't agree that knowing SQL is equivalent to understanding "set theory". I've known plenty of DBAs who don't know the first thing about set theory and really just know just enough to install the software and piece together the APIs. The fact that one has chosen SQL doesn't make them good at working with data any more than choosing MongoDB makes someone bad, or implies they don't understand "set theory".

What concerns me mostly though isn't the MongoDB issue but that we can't discus the issue in a professional way, without elitism and distain dripping through. You seem to have confused "computer science" with "anything of value". I happen to believe there are more things worth knowing that are of value to practical software development than just the computer science (not to minimize that of course).

Re: Why MongoDB is a bad choice for storing our scraped data

#50

Post by [deleted] »

dkhenry wrote: post #3

you lost me here """ Ordered data Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting which is impractical when the number of records gets large. "" it requires _indexing_ and is quite feasable as I do it every day with stock ticker logs ( also required to be retrieved incrementially ) There are a few other flags that make me wonder about the exact l…

[deleted]
Post reply on HN