Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why MongoDB is a bad choice for storing our scraped data

blog.scrapinghub.com

Post reply on HN
Print view View original

61–70 of 121 posts

Re: Why MongoDB is a bad choice for storing our scraped data

#61

Post by snaky »

lmm wrote: post #48
pwg wrote: post #46

Earlier quoted context omitted.

But if one is going to use a key-value store anyway, where is the "mangling"? Building a key-value store in an SQL database is trivial (this below is for PostgreSQL): create table keyvalue (key text, value text); create index keyvalue_idx on keyvalue(key); And use is trivial as well: insert into keyvalue (key, value) values ('key', 'value'); select value from keyvalue where key = 'key'; [edit] formatting fix.

It is not nice to have to manipulate JSON as a plain string (you miss out on validation, have to manually construct like expressions for queries, and I don't even want to think about what you'd have to do to update part of a document), at least using "plain old SQL". (PostgreSQL's native JSON support would make it quite easy, but that supports my point)

If you need to query JSON content, why to store it like plain string, to begin with? Parse it at application level (or even in stored procedure) and store like ordinary fields. Or use hstore.

Re: Why MongoDB is a bad choice for storing our scraped data

#62

Post by [deleted] »

snaky wrote: post #58
JulianMorrison wrote: post #11

I really don't understand why people use MongoDB. It seems like it's a elegant technological metaphor (lets use mmap, the OS is our cache and we can overwrite in place in RAM) that in practise turns out to be a terrible idea. Overwrite/mmap cannot be made reliable, requires blocking write-locks, wastes disk, and causes problems shuffling data around as it grows. Add other bad decisions (keys aren't interned, seriousl…

Because the koolaid is so sweet. MongoDB is every developer's wet dream. With it's expressive query syntax and extreme ease of use, everyone wants to drink the koolaid. This is a huge problem, because mongodb as a database is dangerous [..] I have developers begging me to let them use it. This time to collect logs from our servers for analysis later. I cave in, and give my go ahead, with a warning saying that no crit…

[deleted]

Re: Why MongoDB is a bad choice for storing our scraped data

#63

Post by JulianMorrison »

lmm wrote: post #37
JulianMorrison wrote: post #26

Earlier quoted context omitted.

Are you dead certain those projects aren't ever eating data, and nor are they about to crash tomorrow with an unrecoverably hosed DB? I still say it's bad technology. Use plain old SQL instead.

"plain old SQL" can require a lot of mangling one's data to fit its constraints. I refuse to believe that there isn't a better key-value store for the case where the values are json documents, even if mongodb isn't it.

Redis. Riak. Rethinkdb. Cassandra (with a mapping layer). SQL (with a mapping layer). SQL, in a JSON-typed column (PostgreSQL supports it). SQL, in a blob field, plus indexes. I could go on.

Re: Why MongoDB is a bad choice for storing our scraped data

#64

Post by lucisferre »

bengillies wrote: post #55
lucisferre wrote: post #41

Earlier quoted context omitted.

You lost me when you quoted the word "developers". Edit: And this is downvoted for calling out the fact that people on HN can't discuss a freakin' database without hurling insults.

> Edit: And this is downvoted for calling out the fact that people on HN can't discuss a freakin' database without hurling insults. The original comment you're replying to aside, it was likely because stating that you dismissed the entire comment without giving an actual objection to it added nothing at all to the discussion.

Tone. Which is an actual objection.

Re: Why MongoDB is a bad choice for storing our scraped data

#65

Post by berito »

leothekim wrote: post #13
dkhenry wrote: post #3

you lost me here """ Ordered data Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting which is impractical when the number of records gets large. "" it requires _indexing_ and is quite feasable as I do it every day with stock ticker logs ( also required to be retrieved incrementially ) There are a few other flags that make me wonder about the exact l…

Also, mongo has natural ordering which would do what the author wants without sorting.

No mongo natural order is just order on the disk. It is not always in reverse insertion order

Re: Why MongoDB is a bad choice for storing our scraped data

#66

Post by berito »

leothekim wrote: post #13
dkhenry wrote: post #3

you lost me here """ Ordered data Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting which is impractical when the number of records gets large. "" it requires _indexing_ and is quite feasable as I do it every day with stock ticker logs ( also required to be retrieved incrementially ) There are a few other flags that make me wonder about the exact l…

Also, mongo has natural ordering which would do what the author wants without sorting.

No mongo natural order is just order on the disk. It is not always in reverse insertion order

Re: Why MongoDB is a bad choice for storing our scraped data

#67

Post by monstrado »

swah wrote: post #45
monstrado wrote: post #12

Earlier quoted context omitted.

With a database like HBase, it's already ordered lexicographically which makes it easy to grab a range of data in the order it was written in. You could have a key design like - which would allow quick scans over large amounts of data. ie..Scan from -1368536860 to -1368540450 HBase is multidimensional though, which allows you to keep N numbers of versions of a cell. By default you will get the latest version of the c…

The last time I looked there weren't many resources explaining how to design NoSQL databases (how to compose your keys, when to avoid normalization, etc). Has this improved?

Well, NoSQL databases is a pretty broad term. Not all NoSQL databases are created alike, for example, MongoDB is a "document orientated database" where as HBase is a "column-oriented store" based on the Google BigTable whitepaper.

As far as I know, key design is not an important aspect with MongoDB but I could be mistaken. HBase has a pretty awesome book (http://www.hbasebook.com/), which has an entire chapter dedicated to key design. Lars (the author) also has a pretty in depth 1 hour video on key design (http://www.youtube.com/watch?v=_HLoH_PgrLk).

HBase is pretty widely used, I've seen 1200+ node clusters running production tables.

Re: Why MongoDB is a bad choice for storing our scraped data

#68

Post by wildster »

timr wrote: post #43
lucisferre wrote: post #41

Earlier quoted context omitted.

You lost me when you quoted the word "developers". Edit: And this is downvoted for calling out the fact that people on HN can't discuss a freakin' database without hurling insults.

I quoted "developers", because we need a term to distinguish people who know basic computer science from people who know just enough to install software and piece together APIs. The latter group tends not to realize that things like overwriting your working set in memory and global write-locking lead inevitably to consistency and throughput issues. The primary problem in software today is that we've confused the abil…

This issue is well covered in Joel Spolsky's article The Perils of JavaSchools http://www.joelonsoftware.com/articles/ThePerilsofJavaSchool...

Saying that I think it is a bit elitist to think you need a new term, who gets to say who can use that term?

Re: Why MongoDB is a bad choice for storing our scraped data

#69

Post by craigching »

Whenever I see the "You don't need Mongo DB, use an SQL database" and then in the flaming back and forth, I never see my key problem mentioned:

MongoDB makes it easy to scale out (replica sets and sharding), where is the "easy to setup replicated and sharded open source SQL database?"

I mean, I know that Postgres has replication (via Slony? honestly, it's been awhile since I looked at their solutions) but I don't recall it being as dead simple to set up.

For me, setting up replication needs to be easy because we redistribute the store as part of our product and we need scalability (both replication for redundancy and sharding for scaling).

So I'm honestly asking here, where is the easy to use sharded and replicated open source SQL store that I've been missing?

Re: Why MongoDB is a bad choice for storing our scraped data

#70

Post by gnaritas »

papsosouid wrote: post #57
gnaritas wrote: post #54

Earlier quoted context omitted.

> In the real world, people have been using relational databases to solve problems for years. They work, they're understood, they scale. And they're a pain the ass and don't mix well with the kinds of programs many want to write. Mongo clearly fills a niche that relational databases don't serve well; if it didn't, no one would use it.

Could you actually go into some detail about these mythical problems with actual databases? Faux database apologists seem to really love claiming databases are so unusable, but I've never gotten an actual explanation as to what problems they are having. As both a developer and a sysadmin, postgresql is much less of a pain in the ass than mongodb. And I have no idea what "don't mix well with the kinds of programs.." i…

> Faux database apologists seem to really love claiming databases are so unusable

You really think tossing out insults like that is a way to have a reasoned conversation? I think not, come back when you can converse like an adult.

> And I have no idea what "don't mix well with the kinds of programs.." is supposed to mean.

Then you need more experience as a programmer perhaps. I'm a programer and a SQL guy, and I'm fully aware of what a pain SQL can be in an application and if you don't see the ease of programming things like NoSQL databases or Mongo bring to programming, you aren't paying attention or you're lying to yourself about how well SQL fits with code.

Post reply on HN