Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't use Hadoop when your data isn't that big

chrisstucchio.com

Post reply on HN
Print view View original

31–40 of 235 posts

Re: Don't use Hadoop when your data isn't that big

#31

Post by eksith »

daxelrod wrote: post #19
eksith wrote: post #3

"A 2 terabyte hard drive costs $94.99, 4 terabytes is $169.99. Buy one and stick it in a desktop computer or server. Then install Postgres on it." Done! Although with more drives and a backup server. Right now, we're pushing 15Tb with no loss in performance.

Do you use standard off-the-shelf consumer hard drives at those prices? Most companies I've worked for have shelled out quite a bit more for "enterprise" class hard drives. I've always struggled to understand what these bring to the table, and my understanding is that it's some combination of greater reliability and a service agreement. It's always seemed to me, in my software-centric naivete, that it would be more c…

That's totally understandable. Honestly, we haven't seen any appreciable difference in performance or failure rates between "enterprise" class drives and "consumer" drives (usually WD). Matching RPMs and cache sizes, there's just no way to justify the extra cost for them.

The one area we didn't penny-pinch on was the tape backup. Glacially slow compared to HDDs, obviously, but absolutely necessary.

Re: Don't use Hadoop when your data isn't that big

#32

Post by yummyfajitas »

antonmks wrote: post #25

Big advantage of Hadoop is that you do not need to pay license fees which in case of relational databases can reach into tens of thousands dollars. The big disadvantage is that Hadoop is two orders of magnitude slower than relational databases. Also, Hadoop clusters are not what one would call a "green" solution. More like a terrible waste of computing resources.

Postgres is free.

Re: Don't use Hadoop when your data isn't that big

#33

Post by rplnt »

daxelrod wrote: post #19
eksith wrote: post #3

"A 2 terabyte hard drive costs $94.99, 4 terabytes is $169.99. Buy one and stick it in a desktop computer or server. Then install Postgres on it." Done! Although with more drives and a backup server. Right now, we're pushing 15Tb with no loss in performance.

Do you use standard off-the-shelf consumer hard drives at those prices? Most companies I've worked for have shelled out quite a bit more for "enterprise" class hard drives. I've always struggled to understand what these bring to the table, and my understanding is that it's some combination of greater reliability and a service agreement. It's always seemed to me, in my software-centric naivete, that it would be more c…

MTTF should (and probably is) higher. Perhaps some other things like reallocated sectors count can be higher without affecting performance/stability much.

But from what I've seen, the most important thing is the service agreement, i.e. having new disk ready the next day without any questions asked (or money spent for a new drive).

Re: Don't use Hadoop when your data isn't that big

#34

Post by bunderbunder »

dbecker wrote: post #11

I'd love to understood how the Hadoop hype and marketing team generated so much unwarranted interest in Hadoop. I'm witnessing a feeding frenzy for Hadoop talent in situations where there's absolutely no need for Hadoop, and I can't recall anything like this for any other software.

I think it goes like this:

Certain very popular companies that everyone wants to emulate, who have truly enormous needs in the initial data crunching (e.g., ETL) department, were running into bottlenecks related to raw I/O bandwidth. They hit on a "let the mountain come to Mohammed" insight that helped them get past that problem, so that they could do their ETL jobs in less time and ultimately keep their workhorse databases (e.g., web search indexes) better-fed.

Simultaneously, a whole lot of people who were not having the same problems, but who want to believe that they are like companies who have those problems (because who doesn't want to be Google?) started also running into problems with handling large amounts of data. Unfortunately, the most popular mistake in applying Feynman's Algorithm[1] is to skip the first step. Rather than investigating their problems and recognizing that the issue was poor tooling or inefficient implementations and they weren't actually coming anywhere close to any true limits of the kind that the companies that came up with Big Data were trying to get around, they instead just went, "Hey, X company that we look up to is also having problems that look cosmetically similar to ours, and they use Y technology - let's give that a try!" and proceeded to dive straight into constructing bamboo control towers and coconut radios without ever looking back.

After that, well, I think it's a tragedy of mostly-rational behavior. Managers don't understand these technologies well enough to take programmers' advice skeptically, so they have to listen to their engineers. Engineers want to make their CV's look nice and impress their managers, so they've got every reason to come up with excuses to use $HOT_NEW_TOY. As usual, everyone individually acting in accordance with their rational self-interest is not the same thing as everyone collectively acting in a way that produces ideal results for the parties involved.

[1] http://c2.com/cgi/wiki?FeynmanAlgorithm

Re: Don't use Hadoop when your data isn't that big

#35

Post by kilburn »

daxelrod wrote: post #19
eksith wrote: post #3

"A 2 terabyte hard drive costs $94.99, 4 terabytes is $169.99. Buy one and stick it in a desktop computer or server. Then install Postgres on it." Done! Although with more drives and a backup server. Right now, we're pushing 15Tb with no loss in performance.

Do you use standard off-the-shelf consumer hard drives at those prices? Most companies I've worked for have shelled out quite a bit more for "enterprise" class hard drives. I've always struggled to understand what these bring to the table, and my understanding is that it's some combination of greater reliability and a service agreement. It's always seemed to me, in my software-centric naivete, that it would be more c…

Hard drives are tricky little beasts.

On the one hand, cheap disks are much more prone to start operating slowly when they begin to fail, instead of transitioning directly from working to not working (or setting themselves as failed).

On the other hand, RAID write speeds are roughly that of the slowest drive in the set. The more drives you have, the higher the probability of a malfunctioning hard drive making the whole thing go slow. Slow drives mean I/O bottlenecks, rendering the whole server unusable.

Pair both things and you have actually increased your effective downtime instead of decreasing it. Now use expensive disks (that work more reliably and either work or don't with a higher probability) and just calculate wether the price increase is lower than the cost of the downtime/slowdowns when running cheap disks.

Oftentimes the price premium is worth it.

Re: Don't use Hadoop when your data isn't that big

#36

Post by eksith »

raverbashing wrote: post #17
eksith wrote: post #3

"A 2 terabyte hard drive costs $94.99, 4 terabytes is $169.99. Buy one and stick it in a desktop computer or server. Then install Postgres on it." Done! Although with more drives and a backup server. Right now, we're pushing 15Tb with no loss in performance.

You would probably want some RAID + Backups, (and a very generous amount of RAM there) but this is ok. In fact, I wouldn't doubt that for several "big data" users even SQLite would be enough.

I wouldn't doubt that either.

We do use RAID 6 (I think) + tape backups and the backup server itself is a hot standby. If you're curious, this is the chassis : http://www.supermicro.com/products/chassis/3U/836/SC836A-R12...

The motherboard is also Supermicro running dual AMD Opterons ( single processor on the backup server ) with 256Gb RAM.

Re: Don't use Hadoop when your data isn't that big

#37

Post by mattjaynes »

Novelty Driven Development (NDD)

Chris points out a great example of NDD here with Hadoop.

I do a lot of client work and I see this mistake CONSTANTLY. So often in fact, that I recently wrote up a story to illustrate the problem. Rather than use a tech example, I use a restaurant and plumbing to drive the point home. When the same scenario is put into the context of something more concrete like physical plumbing, it shows how ridiculous NDD really is.

http://devopsu.com/blog/boring-systems-build-badass-business...

Re: Don't use Hadoop when your data isn't that big

#38

Post by mrcactu5 »

I remember starting to grasp what "big data" meant when I had a phone interview with Twitter.

@ Imagine you have some numbers spread over some computers -- too many to fit in one computer find the median.

▪ Uhh, sort them?

@ Can you find the median on a single computer without sorting them.

▪ :-(

@ We'll call you back tomorrow.

I was promptly rejected, but it set the tone for my later studies.

The criterion for Big Data seems to be that it fits on thousands of computers, perhaps several TB or a PB. Then I had to think of some examples:

* A million YouTube Videos

* All the tweets in the US in the past 15 minutes

* All US tax records

I still think the map-reduce philosophy is really cool. And I know at that scale there are special counting algorithms (like Bloom Filters) that may lead to some improvements at the GB or MB scales.

Re: Don't use Hadoop when your data isn't that big

#39

Post by pjbringer »

mightybyte wrote: post #27

I agree with the general thrust of this article. But hadoop isn't just for scaling up the absolute size of the data set. It is also useful for scaling up the absolute amount of CPU power you can throw at a problem. If I have 1 GB data set, but the computations that I need to do on that data set are complex enough that it would take a single machine a long time to do them, then hadoop is still useful. I gain tremendou…

In that case, your IO problem is easy, because it's small with regard to CPU time. You can get away with putting the whole data in sql databases, and/or making multiple copies of your data. Then you can use as many workers as you want, with usually simple partitioning logic.

Re: Don't use Hadoop when your data isn't that big

#40

Post by acdha »

rcavezza wrote: post #4

I understand and agree with the author's main point that many companies that use big data do not need to use these technologies. I do not agree that the tools are inferior to Sql. Hive is really close to sql and Pig is extremely powerful. I would take a look at a few of the recent updates to these tools before declaring them inferior to Sql.

There's a key nuance missing: SQL is mature, while even core Hadoop is struggling to get there. You can simply install Postgres, MySQL, etc. in a couple minutes and start working on your data (i.e. the actual work) and not spend hours dealing with … mixed quality … documentation, extensive configuration on multiple nodes, and writing code to provide what are built-in features in most databases. For anything not set in stone with massive data volumes, that overhead adds up quickly.

The other cost is interactivity: Hive takes a LONG time to return results compared to a SQL database even if you don't have massive data. If an analyst is working on a query interactively this is a significant impediment, particularly given the gap in ease of monitoring and performance optimization advice. Again, if you have enough data Hadoop might still be worth it but, particularly in the post-SSD big memory era, the time-to-correct result gap is massive.

Post reply on HN