Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't use Hadoop when your data isn't that big

chrisstucchio.com

Post reply on HN
Print view View original

171–180 of 235 posts

Re: Don't use Hadoop when your data isn't that big

#171

Post by bhauer »

superuser2 wrote: post #151
bhauer wrote: post #69

The point of the article that resonates with me is how frequently a technology that is poorly fit with a problem domain is selected because of conventional wisdom rather than data. Related, it is remarkable how we developers routinely cite Knuth's advice about premature optimization to justify our decision when the shoe fits, and then turn around and flatly ignore the advice when it doesn't fit. Selecting Hadoop befo…

> the same developers who believe it's reasonable for a web application with modest functionality to require dozens of application nodes to service concurrent request load measured in the mere thousands I'm assuming this is a dig at Rails and Django? What are you suggesting instead?

Just about anything on the JVM (Dropwizard, Play, Finagle, Scalatra, Rest-Express, Rest-Easy, Compojure, Unfiltered, Jersey, Vert.x, Spark), Go (Gorilla, Beego, Revel), Lua (Lapis), Haskell, Erlang...

Re: Don't use Hadoop when your data isn't that big

#172

Post by sitkack »

Hadoop is the problem, MapReduce is not the problem. Having used both Hadoop and Disco, I can say that Disco is by far a net positive on all projects I used it on. And the overhead to coding it in Disco vs single node is about an extra 30 minutes. You can start with working single node and go multinode w/o much effort.

http://discoproject.org

Hadoop on the other hand is a huge, massive pain in the ass. And I am a Hadoop consultant. I recommend that most customers NOT use it.

Re: Don't use Hadoop when your data isn't that big

#173

Post by boomzilla »

dekhn wrote: post #153
w_t_payne wrote: post #147

Earlier quoted context omitted.

1.5 PB is pretty huge for most people outside of supercomputing. The biggest dataset I ever worked with fell some way short of that (It was approaching 1PB, I think, although we were not too sure how much data we had exactly, and it might have been slightly over, depending on how you measured). Even handling small(er) working sets could be a challenge in organisation, since we only had a small compute cluster (& asso…

"Supercomputing" has never really been about big data. I used to work in supercomputing (NERSC) and have used most of the major machines in the field in the past. The supercomputer centers claim they're moving lots of data around, but it you look closely, it's almost entirely message passing during a calculation. And in most cases, the calculations they are doing are just wasted resources- simulating a protein over o…

On the other hand, Hadoop also serves a lot more than a computing platform for "big data". The OP probably never worked in a mid/big sized company where sharing data is big issue. HDFS can easily replace a bunch of high maintenance and under performing NFSes. Actually, I think someone should write a Samba clone for HDFS.

The next huge thing in the Hadoop ecosystem is Hive. It's like having an unlimited Postgres server that hundreds of people can work on at the same time. Hive's got its quirks and developers need to be educated on how to write the Hive jobs properly, but it gives a lot more back by allowing joins between a lot of datasets that might not be able to fit in one single machine.

There is a reason why Hadoop is so popular, especially in enterprises. I think that's why Cloudera got such a high valuation.

Re: Don't use Hadoop when your data isn't that big

#174

Post by davidmr »

dekhn wrote: post #153
w_t_payne wrote: post #147

Earlier quoted context omitted.

1.5 PB is pretty huge for most people outside of supercomputing. The biggest dataset I ever worked with fell some way short of that (It was approaching 1PB, I think, although we were not too sure how much data we had exactly, and it might have been slightly over, depending on how you measured). Even handling small(er) working sets could be a challenge in organisation, since we only had a small compute cluster (& asso…

"Supercomputing" has never really been about big data. I used to work in supercomputing (NERSC) and have used most of the major machines in the field in the past. The supercomputer centers claim they're moving lots of data around, but it you look closely, it's almost entirely message passing during a calculation. And in most cases, the calculations they are doing are just wasted resources- simulating a protein over o…

I wrote a different reply but deleted it. I'm curious about something: if, as you claim, supercomputers are wasted money, why are there so many of them? Have all the world's top supercomputing sites somehow colluded to convince all the world's largest governments that they're useful?

Re: Don't use Hadoop when your data isn't that big

#175

Post by davidmr »

harrytuttle wrote: post #44
w_t_payne wrote: post #2

Hooray! Some sense at last. I have worked for at least 3 different employers that claimed to be using "Big Data". Only one of them was really telling the truth. All of them wanted to feel like they were doing something special. The sad thing is, they were all special, each in their own particular way, but none of what made each company magic and special had anything to do with the size of the data that they were hand…

Yes I've worked for a couple of outfits that did this over the years. It's usually down to who sold them the solution though. It's worst if the salesperson is external. The funniest has to be the engineering company which I used to work with (not for thank fuck) back in the 90's (when 100Gb was big data!). They managed to bag 2x full 42U racks full of HP N-Class HPUX UNIX kit, PDU systems and separate disk arrays for…

After I moved from academia/gov't labs to private industry, I was absolutely flabbergasted at the waste. Many companies literally just buy whatever their vendors throw at them to solve a problem, and if a reasonably intelligent techie spent just an hour or two thinking through the problem, they'd realize that the solution was either far less complicated than the proposed kit or far more; either way, what's proposed is almost never right.

Another thing I haven't seen much of is acceptance testing. This is actually more surprising to me than the reliance on vendors for problem solving. It's an absolute no-brainer to say "I have problem X, which will be solved by solution Y, and the success will be measured by the set of tests Z. Unless the above happens, we will not pay the bill and you will take the hardware back." The vendors looked at me like I was from Mars when I started suggesting these.

I have not since complained about government waste in research computing budgets.

Re: Don't use Hadoop when your data isn't that big

#176

Post by stingraycharles »

davidmr wrote: post #174
dekhn wrote: post #153

Earlier quoted context omitted.

"Supercomputing" has never really been about big data. I used to work in supercomputing (NERSC) and have used most of the major machines in the field in the past. The supercomputer centers claim they're moving lots of data around, but it you look closely, it's almost entirely message passing during a calculation. And in most cases, the calculations they are doing are just wasted resources- simulating a protein over o…

I wrote a different reply but deleted it. I'm curious about something: if, as you claim, supercomputers are wasted money, why are there so many of them? Have all the world's top supercomputing sites somehow colluded to convince all the world's largest governments that they're useful?

I'm not disagreeing with you, but the argument of "people are using it and the government keeps paying for them" is hardly convincing.

Re: Don't use Hadoop when your data isn't that big

#177

Post by dekhn »

cshimmin wrote: post #167
dekhn wrote: post #154

Earlier quoted context omitted.

CERN and its associated teams (CMS and ATLAS) are awesome. They are the only outfit outside of Google that seems to be handling data management and computing properly.

ATLAS guy here. I agree with @toomuchtodo, we're stuck in the stone age. It's basically just batch processing and lots of copying data around for no reason.

You could say the same thing about google's backend (batch processing and copying data), but it seems to be working for them as well.

Re: Don't use Hadoop when your data isn't that big

#178

Post by deathflute »

pwang wrote: post #168
deathflute wrote: post #111

Well written article. I think most people who do not have a background in data are unaware of the various options out there and fall for the marketing behind hadoop like tools. I would urge people doing analytics to take a look at kdb+ from kx. Unless you have ridiculously large amounts of data(> 200 TB), I can bet that you would be better off with kdb. The only downside is that it costs a lot of money which is a pit…

You'd have to hire a team of people that can write good K. Since they're so highly in demand in the niche area of finance, you'll be paying a very pretty penny for them.

I have been part of a team which used kdb heavily and this is not really true. The best part of kdb is that it requires almost no administration and at the end of the day becomes just another programming environment/tool for you to use. Hiring a team of people who just do kdb is only done by banks or other unimaginative large teams.

Re: Don't use Hadoop when your data isn't that big

#179

Post by davidmr »

stingraycharles wrote: post #176
davidmr wrote: post #174

Earlier quoted context omitted.

I wrote a different reply but deleted it. I'm curious about something: if, as you claim, supercomputers are wasted money, why are there so many of them? Have all the world's top supercomputing sites somehow colluded to convince all the world's largest governments that they're useful?

I'm not disagreeing with you, but the argument of "people are using it and the government keeps paying for them" is hardly convincing.

It's not just the government that buy these things. Oil and gas, heavy manufacturing, automotive, genomics, financial, etc. all have them. The poster I was replying to appeared to be under the gross misconception that all parallel problems were more effectively solved with high throughput clusters.

Re: Don't use Hadoop when your data isn't that big

#180

Post by vidarh »

davidmr wrote: post #57

While I couldn't agree more with the general point of the article, I have some small additional comments. Just as a bit of background, I think that Chris would very much agree that I am not the intended recipient of this advice, and so my comments probably aren't keeping in the spirit of the article. I've spent the last 10+ years exclusively in very large HPC environments where the average size of the problem set is…

> We'll be generous and say that 4TB can do 150MB/s. A single run through the data at maximum efficiency will cost you ~8 hours. Since we've restricted ourselves to a single box, we're also not going to be able to keep the data in memory for subsequent calculations/simulations/whatever.

I agree with your overall point, but of course the actual limits for a fairly cost effective single box are fairly high these days. E.g. from my preferred (UK) vendor, 4x PCIe based 960GB SSD's (each card is really 4 SSD's + controllers) adds up to an additional $7500. We often get 800MB/sec with a single card.

Your point still holds - the actual thresholds are just likely to be quite a bit higher than what the performance of a single 4TB hd might imply.

Tack on 40+ cores, and 1TB RAM, and the cost to purchase a server like that is somewhere around the $60k mark in the UK (without shopping around or building it yourself), but that adds up to "only" about $2k/month in leasing costs. That makes a single server solution viable for quite a lot larger jobs than people often think.

Post reply on HN