Don't use Hadoop when your data isn't that big • Hacker News Board

Re: Don't use Hadoop when your data isn't that big

#211

Post by jacques_chester » Wed, Sep 18, 2013, 12:40 PM UTC

davidmr wrote: post #175

harrytuttle wrote: post #44

Earlier quoted context omitted.

Yes I've worked for a couple of outfits that did this over the years. It's usually down to who sold them the solution though. It's worst if the salesperson is external. The funniest has to be the engineering company which I used to work with (not for thank fuck) back in the 90's (when 100Gb was big data!). They managed to bag 2x full 42U racks full of HP N-Class HPUX UNIX kit, PDU systems and separate disk arrays for…

After I moved from academia/gov't labs to private industry, I was absolutely flabbergasted at the waste. Many companies literally just buy whatever their vendors throw at them to solve a problem, and if a reasonably intelligent techie spent just an hour or two thinking through the problem, they'd realize that the solution was either far less complicated than the proposed kit or far more; either way, what's proposed i…

Outsourcing your own teams means there are no in-house experts to perform the oversight role you've described. Thus the principal-agent is amplified.

Re: Don't use Hadoop when your data isn't that big

#212

Post by hhuuggoo » Wed, Sep 18, 2013, 1:14 PM UTC

sitkack wrote: post #172

Hadoop is the problem, MapReduce is not the problem. Having used both Hadoop and Disco, I can say that Disco is by far a net positive on all projects I used it on. And the overhead to coding it in Disco vs single node is about an extra 30 minutes. You can start with working single node and go multinode w/o much effort. http://discoproject.org Hadoop on the other hand is a huge, massive pain in the ass. And I am a Had…

agreed, disco is really great becuase you can easily get access to the file descriptor which has your data, then you can load it however way you want

Re: Don't use Hadoop when your data isn't that big

#213

Post by jacques_chester » Wed, Sep 18, 2013, 1:14 PM UTC

ivanprado wrote: post #86

You are right that it should be used the proper tool for each particular problem. And Hadoop world is harder than single machine systems (like pandas). So, you shouldn't user Hadoop if you can do the job with simpler systems. But I have something to add. Hadoop is not only introducing new techniques for distributed storage and computation. Hadoop is also proposing a methodological change in the way a data project is…

> That is not possible with stateful systems, like those based in doing updates over a database.

Could you elaborate? This sounds like the NoSQL argument that relational databases are "not agile", which usually means relational databases complain that you have records that won't logically fit the changes you just made.

Re: Don't use Hadoop when your data isn't that big

#214

Post by acdha » Wed, Sep 18, 2013, 1:30 PM UTC

dekhn wrote: post #156

acdha wrote: post #139

Earlier quoted context omitted.

This assumes that it's easier to manage 100 servers using Hadoop than, say, using the same 100 EC2 servers with a normal grid scheduler, task queue, or even something like GNU parallel. It's orders of magnitude easier to start 100 servers and reuse basic Unix skills than it is to setup and manage Hadoop on the same infrastructure.

does your grid scheduler checkpoint intermediate work automatically, so that node failures have small impact?

Yes, although you have control over that kind of thing: this is a problem HPC groups have been tackling for decades so there's a ton of prior art as well as options for whatever makes sense for your application (e.g. if you're doing some sort of shared memory simulation you have to restart every instance, not just the one which failed).

Re: Don't use Hadoop when your data isn't that big

#215

Post by dekhn » Wed, Sep 18, 2013, 1:41 PM UTC

davidmr wrote: post #179

stingraycharles wrote: post #176

Earlier quoted context omitted.

I'm not disagreeing with you, but the argument of "people are using it and the government keeps paying for them" is hardly convincing.

It's not just the government that buy these things. Oil and gas, heavy manufacturing, automotive, genomics, financial, etc. all have them. The poster I was replying to appeared to be under the gross misconception that all parallel problems were more effectively solved with high throughput clusters.

Can you point to a genomics company that uses a supercomputer? Adn really claims that its unique capabilities are necessary>

Remember, the human genome was assembled on two different architectures: a Dec Alpha cluster with large memory with fast network (for the day) and a bunch of cheap linux machines with slow network. It turns out you didn't actually need the former, although the people at Celera insisted you did.

Re: Don't use Hadoop when your data isn't that big

#216

Post by dekhn » Wed, Sep 18, 2013, 2:03 PM UTC

stingraycharles wrote: post #160

dekhn wrote: post #153

Earlier quoted context omitted.

"Supercomputing" has never really been about big data. I used to work in supercomputing (NERSC) and have used most of the major machines in the field in the past. The supercomputer centers claim they're moving lots of data around, but it you look closely, it's almost entirely message passing during a calculation. And in most cases, the calculations they are doing are just wasted resources- simulating a protein over o…

It's not a waste of resources, it's just a different approach to solving a problem. Hadoop / "big data" clusters make the problem harder to solve (and probably even restrict the types of problems that can be solved) in exchange for cheap hardware. Supercomputers give engineers the ability to solve problems in a traditional manner, while moving the costs over to the hardware.

Nobody solves problems on supercomputers "in the traditional manner". You have to completely rewrite your application around message passing.

Re: Don't use Hadoop when your data isn't that big

#217

Post by tfgg » Wed, Sep 18, 2013, 2:58 PM UTC

dekhn wrote: post #182

davidmr wrote: post #174

Earlier quoted context omitted.

I wrote a different reply but deleted it. I'm curious about something: if, as you claim, supercomputers are wasted money, why are there so many of them? Have all the world's top supercomputing sites somehow colluded to convince all the world's largest governments that they're useful?

At this point, large capability supercomputers that solve problems we don't need solved are mostly kept around as trophy pieces with a few "show me" calculations. Note that when the Chinese obtained the leadership in TOP500 a lot of people got worried (as they did when Japan did it with the Earth Simulator). And so the US may spend a bunch of money to get on top of that list again. Big deal. It's LINPACK. Fortunately…

As far as I understand, you're saying that high-speed interconnects are largely a waste of time? (and apparently that doing protein MD is a waste of time, but that's a scientific question that I'm going to disagree with you on). I can imagine that might be the case for some simulations, but how I do, for example, a parallel FFT without significant communication?

I personally see it more as there's a pretty limited set of computational problems which are low-communication, often requiring rather gross approximations, even with the "clever computer scientists" looking at the problem, and they're often not the problems scientists actually want to solve, and so dismissing supercomputing is rather ridiculous.

Re: Don't use Hadoop when your data isn't that big

#218

Post by dekhn » Wed, Sep 18, 2013, 3:21 PM UTC

tfgg wrote: post #217

dekhn wrote: post #182

Earlier quoted context omitted.

At this point, large capability supercomputers that solve problems we don't need solved are mostly kept around as trophy pieces with a few "show me" calculations. Note that when the Chinese obtained the leadership in TOP500 a lot of people got worried (as they did when Japan did it with the Earth Simulator). And so the US may spend a bunch of money to get on top of that list again. Big deal. It's LINPACK. Fortunately…

As far as I understand, you're saying that high-speed interconnects are largely a waste of time? (and apparently that doing protein MD is a waste of time, but that's a scientific question that I'm going to disagree with you on). I can imagine that might be the case for some simulations, but how I do, for example, a parallel FFT without significant communication? I personally see it more as there's a pretty limited se…

High speed interconnects are not worth the extra cost. They just compensate for poor programming.

With regards to protein MD, no, I'm not saying protein MD is a waste of time. In fact, I run the Exacycle program at Google which has run the largest MD simulations (many milliseconds) ever done, and the results are quite good. You would never have gotten results as good as ours on a supercomputer- even the world's largest, with the most highly scaling MD codes.

I speak from experience- I used to code for supercomputers, in fact working on protein (and DNA/RNA, my interest) structural dynamics, with some of the leading MD codes.

I helped port a better approach to Google's infrastructure some time ago. It's not a clever hack or gross approximation: https://simtk.org/home/msmbuilder it's a distinctly better way of modelling protein dynamics than running long single trajectories on supercomputers.

Re: Don't use Hadoop when your data isn't that big

#219

Post by tfgg » Wed, Sep 18, 2013, 3:57 PM UTC

dekhn wrote: post #218

tfgg wrote: post #217

Earlier quoted context omitted.

As far as I understand, you're saying that high-speed interconnects are largely a waste of time? (and apparently that doing protein MD is a waste of time, but that's a scientific question that I'm going to disagree with you on). I can imagine that might be the case for some simulations, but how I do, for example, a parallel FFT without significant communication? I personally see it more as there's a pretty limited se…

High speed interconnects are not worth the extra cost. They just compensate for poor programming. With regards to protein MD, no, I'm not saying protein MD is a waste of time. In fact, I run the Exacycle program at Google which has run the largest MD simulations (many milliseconds) ever done, and the results are quite good. You would never have gotten results as good as ours on a supercomputer- even the world's large…

> High speed interconnects are not worth the extra cost. They just compensate for poor programming.

Sorry, that's complete rubbish. Some problems just can't be made embarrassingly parallel or near so. You might be able to swap a problem for a similarish problem with better characteristics, but that's not the same as being a "poor programmer" since you're now using a different model rather than optimizing the implementation of the existing one. You still haven't explained how I do a parallel FFT without high speed interconnects -- I suspect the answer is "don't". I've tried running the stuff I do (DFT) over a 10GigE network on my cluster and it ran at about 10% of the speed of an Infiniband-enabled calculation. There are methods for getting better scaling, but they all (as I said) involve rather crass approximations which you don't always want to do. Those methods will probably increasingly be used more on large supercomputers due to their superior scaling characteristics, even with their nice high-speed interconnects, but you're still making a sacrifice in accuracy.

It might be possible to get good low-communication scaling for some models, as you were in the case of your sampling system (which typically parallelizes nicely, but only if the individual sampling jobs can fit on a single node), but you can't extrapolate that to assume that everyone can. As I said, exchanging a poorly-scaling model for a different, non-equivalent well-scaling model is a scientific question with tradeoffs, not a programming one.

Re: Don't use Hadoop when your data isn't that big

#220

Post by mindcrime » Wed, Sep 18, 2013, 4:36 PM UTC

sgt101 wrote: post #165

mindcrime wrote: post #101

On a related note, I find it amusing that people seem to think that "Big Data == Hadoop". In actuality, there are plenty of other approaches to scaling out clusters to handle large jobs, including MPI and OpenMP, as well as BSP (Bulk Synchronous Parallel)[3] frameworks. [1]: http://en.wikipedia.org/wiki/Message_Passing_Interface [2]: http://en.wikipedia.org/wiki/OpenMP [3]: http://en.wikipedia.org/wiki/Bulk_synchrono…

Isn't YARN / Giraffe BSP ?

Giraph was definitely inspired by BSP and resembles it to some degree. I'm not an expert on Giraph, so I'm not sure exactly how closely related they are.

I don't know much about YARN, so can't really speak to that.