Don't use Hadoop when your data isn't that big • Hacker News Board

Re: Don't use Hadoop when your data isn't that big

#231

Post by dekhn » Thu, Sep 19, 2013, 3:44 PM UTC

xtreme wrote: post #227

dekhn wrote: post #222

Earlier quoted context omitted.

Are you sure you need to do an FFT? Are you sure there is not a network friendly FFT algorithm? I used to think very differently about computing before I read Google's MR, BigTable, and GFS papers. After joining Google and working on problems like this, I can assure you that FFTs can indeed scale quite nicely. note that infiniband isn't a high speed interconnect. it's a cheap commodity interconnect- in fact, per port…

> note that infiniband isn't a high speed interconnect. it's a cheap commodity interconnect- in fact, per port, it can be cheaper than 10GB. Why do you think that an interconnect can't be both cheap and high-speed? As you said, there has been a lot of work to make MPI efficient on infiniband, but that has been possible because inifiniband offers much lower (~10 times) latency than 10GbE, not because no one has optimi…

actually, wire latencies for 10GbE are much better than you think- they are basically the same.

Anyway, my point was that infiniband is not a high speed interconnect, and it's cheaper than 10GbE. The challenge is to deliver scaled infiniband, which is far harder than scaled ethernet.

Re: Don't use Hadoop when your data isn't that big

#232

Post by dekhn » Thu, Sep 19, 2013, 3:45 PM UTC

jerven wrote: post #230

dekhn wrote: post #215

Earlier quoted context omitted.

Can you point to a genomics company that uses a supercomputer? Adn really claims that its unique capabilities are necessary> Remember, the human genome was assembled on two different architectures: a Dec Alpha cluster with large memory with fast network (for the day) and a bunch of cheap linux machines with slow network. It turns out you didn't actually need the former, although the people at Celera insisted you did.

Genetics is normally very very data parallel. And often the ideal case for throughput computing. But there are rare times when doing modelling that a lot of chatter is required over interconnects. But even here people normally just get 4-6TB RAM machines instead of having lots of machines.

yes, and the people buying those machines often tell me that their science is limited by how much RAM they could buy in a single machine.

Which is poppycock.

Re: Don't use Hadoop when your data isn't that big

#233

Post by sgt101 » Sun, Sep 22, 2013, 7:03 AM UTC

mindcrime wrote: post #220

sgt101 wrote: post #165

Earlier quoted context omitted.

Isn't YARN / Giraffe BSP ?

Giraph was definitely inspired by BSP and resembles it to some degree. I'm not an expert on Giraph, so I'm not sure exactly how closely related they are. I don't know much about YARN, so can't really speak to that.

Doh - got the spelling wrong!

Hmm I should go off and get to the bottom of this!

Re: Don't use Hadoop when your data isn't that big

#234

Post by wglb » Tue, Sep 24, 2013, 1:24 PM UTC

strictfp wrote: post #228

bhauer wrote: post #171

Earlier quoted context omitted.

Just about anything on the JVM (Dropwizard, Play, Finagle, Scalatra, Rest-Express, Rest-Easy, Compojure, Unfiltered, Jersey, Vert.x, Spark), Go (Gorilla, Beego, Revel), Lua (Lapis), Haskell, Erlang...

Eh, I just got bitten by a Python framework which could only do about 400 rps of "Hello World" on my laptop. Pure python could be pushed to 14000 with gevent and 4 processes, while a pure Java servlet easily got 100krps, async java around 140krps, haskell 200k with 4 workers. Out of the box Java servlet was better than fastcgi nginx. So I don't see what your'e getting at.

Did you happen to try with C?

Re: Don't use Hadoop when your data isn't that big

#235

Post by jerven » Tue, Oct 01, 2013, 7:30 PM UTC

dekhn wrote: post #232

jerven wrote: post #230

Earlier quoted context omitted.

Genetics is normally very very data parallel. And often the ideal case for throughput computing. But there are rare times when doing modelling that a lot of chatter is required over interconnects. But even here people normally just get 4-6TB RAM machines instead of having lots of machines.

yes, and the people buying those machines often tell me that their science is limited by how much RAM they could buy in a single machine. Which is poppycock.

Oh, you could make this run using MPI etc... no problem. But then all you are really doing is going from high density to low density ram. And that is just not as large a saving as you would think it is. Because you are expanding rapidly the number of cpu's and racks you need to compensate for the less dense RAM. A good sequencing lab does not need more RAM than that per assembly. If you have a reference even less.

Plus the science is hard enough that throwing mulitnode programming in the middle is tricky. Remember most of this code is written by a PhD student on his first real programming job. Its hard enough to go from getting it to run on something else than the PhD students laptop, never mind robustly parallelize this across nodes.