Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't use Hadoop when your data isn't that big

chrisstucchio.com

Post reply on HN
Print view View original

221–230 of 235 posts

Re: Don't use Hadoop when your data isn't that big

#221

Post by pnathan »

NathanRice wrote: post #88
mrcactu5 wrote: post #38

I remember starting to grasp what "big data" meant when I had a phone interview with Twitter. @ Imagine you have some numbers spread over some computers -- too many to fit in one computer find the median. ▪ Uhh, sort them? @ Can you find the median on a single computer without sorting them. ▪ :-( @ We'll call you back tomorrow. I was promptly rejected, but it set the tone for my later studies. The criterion for Big D…

That second question is kind of dirty. Pretty much all algorithms to find the median will perform a partial sort. Without any sorting at all, the only thing I can think of is some kind of statistical approximation based on sampling.

Here's an interesting probabilistic way to sample up to the median: http://blog.aggregateknowledge.com/2013/09/16/sketch-of-the-...

Re: Don't use Hadoop when your data isn't that big

#222

Post by dekhn »

tfgg wrote: post #219
dekhn wrote: post #218

Earlier quoted context omitted.

High speed interconnects are not worth the extra cost. They just compensate for poor programming. With regards to protein MD, no, I'm not saying protein MD is a waste of time. In fact, I run the Exacycle program at Google which has run the largest MD simulations (many milliseconds) ever done, and the results are quite good. You would never have gotten results as good as ours on a supercomputer- even the world's large…

> High speed interconnects are not worth the extra cost. They just compensate for poor programming. Sorry, that's complete rubbish. Some problems just can't be made embarrassingly parallel or near so. You might be able to swap a problem for a similarish problem with better characteristics, but that's not the same as being a "poor programmer" since you're now using a different model rather than optimizing the implemen…

Are you sure you need to do an FFT?

Are you sure there is not a network friendly FFT algorithm?

I used to think very differently about computing before I read Google's MR, BigTable, and GFS papers. After joining Google and working on problems like this, I can assure you that FFTs can indeed scale quite nicely.

note that infiniband isn't a high speed interconnect. it's a cheap commodity interconnect- in fact, per port, it can be cheaper than 10GB. but lots of effort has been put into making the MPI libraries work efficiently over it, compared to 10GBe, so codes run more effficiently (as you observed).

DFT seems like the next thing that's going to not need supercomputers. I see some nice new algorithms coming on line designed for Amazon cloud infrastructure.

Re: Don't use Hadoop when your data isn't that big

#223

Post by epistasis »

boomzilla wrote: post #173
dekhn wrote: post #153

Earlier quoted context omitted.

"Supercomputing" has never really been about big data. I used to work in supercomputing (NERSC) and have used most of the major machines in the field in the past. The supercomputer centers claim they're moving lots of data around, but it you look closely, it's almost entirely message passing during a calculation. And in most cases, the calculations they are doing are just wasted resources- simulating a protein over o…

On the other hand, Hadoop also serves a lot more than a computing platform for "big data". The OP probably never worked in a mid/big sized company where sharing data is big issue. HDFS can easily replace a bunch of high maintenance and under performing NFSes. Actually, I think someone should write a Samba clone for HDFS. The next huge thing in the Hadoop ecosystem is Hive. It's like having an unlimited Postgres serve…

There's a connector between Hadoop and the Gluster network file systems [1], giving a unified view of a large, distributed, POSIX compatible filesystem, which can be served up through Samba. We vastly prefer it to NFS, which we find to be far more flaky, slower, and when NFS fails it fails in really really painful ways. (Though looking again, I'm not sure if you meant NFS or NAS...)

[1] http://gluster.org/community/documentation/index.php/Hadoop

Re: Don't use Hadoop when your data isn't that big

#224

Post by dekhn »

JackFr wrote: post #138
mattjaynes wrote: post #37

Novelty Driven Development (NDD) Chris points out a great example of NDD here with Hadoop. I do a lot of client work and I see this mistake CONSTANTLY. So often in fact, that I recently wrote up a story to illustrate the problem. Rather than use a tech example, I use a restaurant and plumbing to drive the point home. When the same scenario is put into the context of something more concrete like physical plumbing, it…

I get your point of course, but indoor plumbing is orders of magnitude more complicated than a well and outhouses. Replacing wells with fountains and eliminating outhouses are driven by population density and public health concerns -- but once available they are preferable to the alternative, despite their complexity.

actually in the developing world people prefer cellphones to indoor plumbing. i expect this to continue to be the case.

Re: Don't use Hadoop when your data isn't that big

#225

Post by [deleted] »

dekhn wrote: post #222
tfgg wrote: post #219

Earlier quoted context omitted.

> High speed interconnects are not worth the extra cost. They just compensate for poor programming. Sorry, that's complete rubbish. Some problems just can't be made embarrassingly parallel or near so. You might be able to swap a problem for a similarish problem with better characteristics, but that's not the same as being a "poor programmer" since you're now using a different model rather than optimizing the implemen…

Are you sure you need to do an FFT? Are you sure there is not a network friendly FFT algorithm? I used to think very differently about computing before I read Google's MR, BigTable, and GFS papers. After joining Google and working on problems like this, I can assure you that FFTs can indeed scale quite nicely. note that infiniband isn't a high speed interconnect. it's a cheap commodity interconnect- in fact, per port…

[deleted]

Re: Don't use Hadoop when your data isn't that big

#226

Post by mindcrime »

gfodor wrote: post #195
mindcrime wrote: post #184

Earlier quoted context omitted.

Sure, and I'm not saying "MPI > Hadoop" or anything. Just pointing out that Hadoop is not the only game in town, and is hardly the only way to deal with "big data". I'm a Hadoop fan myself, but I also did a lot of MPI stuff in the past, and I believe there are still scenarios where MPI makes a lot of sense. I have less experience with OpenMP, but I think anybody planning a "big data" project would be well served to a…

Often times choosing the framework du jour is the best choice just because it's the framework du jour. Support, training, books, an active ecosystem, a rich base of developers to hire from with experience, corporations incentivized to fund further development, etc, etc all act as a hedge against "slight technical mismatch between our requirements and the technology vs more esoteric ones."

But what you're talking about here is still a considered decision, based on actual analysis and thought. It's not choosing a framework only because it's the framework du jour, but because of the second order effects of it being so.

Re: Don't use Hadoop when your data isn't that big

#227

Post by xtreme »

dekhn wrote: post #222
tfgg wrote: post #219

Earlier quoted context omitted.

> High speed interconnects are not worth the extra cost. They just compensate for poor programming. Sorry, that's complete rubbish. Some problems just can't be made embarrassingly parallel or near so. You might be able to swap a problem for a similarish problem with better characteristics, but that's not the same as being a "poor programmer" since you're now using a different model rather than optimizing the implemen…

Are you sure you need to do an FFT? Are you sure there is not a network friendly FFT algorithm? I used to think very differently about computing before I read Google's MR, BigTable, and GFS papers. After joining Google and working on problems like this, I can assure you that FFTs can indeed scale quite nicely. note that infiniband isn't a high speed interconnect. it's a cheap commodity interconnect- in fact, per port…

> note that infiniband isn't a high speed interconnect. it's a cheap commodity interconnect- in fact, per port, it can be cheaper than 10GB.

Why do you think that an interconnect can't be both cheap and high-speed? As you said, there has been a lot of work to make MPI efficient on infiniband, but that has been possible because inifiniband offers much lower (~10 times) latency than 10GbE, not because no one has optimized MPI for ethernet. In fact, standards like RoCE and iWarp have been devised by Ethernet working group to compete with infiniband on this particular metric.

Re: Don't use Hadoop when your data isn't that big

#228

Post by strictfp »

bhauer wrote: post #171
superuser2 wrote: post #151

Earlier quoted context omitted.

> the same developers who believe it's reasonable for a web application with modest functionality to require dozens of application nodes to service concurrent request load measured in the mere thousands I'm assuming this is a dig at Rails and Django? What are you suggesting instead?

Just about anything on the JVM (Dropwizard, Play, Finagle, Scalatra, Rest-Express, Rest-Easy, Compojure, Unfiltered, Jersey, Vert.x, Spark), Go (Gorilla, Beego, Revel), Lua (Lapis), Haskell, Erlang...

Eh, I just got bitten by a Python framework which could only do about 400 rps of "Hello World" on my laptop. Pure python could be pushed to 14000 with gevent and 4 processes, while a pure Java servlet easily got 100krps, async java around 140krps, haskell 200k with 4 workers. Out of the box Java servlet was better than fastcgi nginx. So I don't see what your'e getting at.

Re: Don't use Hadoop when your data isn't that big

#229

Post by Scarblac »

rdtsc wrote: post #13

"We don't have big data" or "our data is rather small" -- said no dev team ever. "Big data" is like "cloud" it is a cool label everyone applying to their system. Just like OO was in its time. Well once they applied the label they feel they need to live up to it so well "we gotta use what big data companies use" and they pick Hadoop. I've heard hadoop used when MySQL, SQLite or even flat files would have worked.

Also, although every new project starts out tiny, everyone _hopes_ to become really big some day. And to make that clear to everyone on the team, the architecture is immediately designed to handle the wildest success imaginable. Which ends up costing so much to implement that the first paying customer never comes...

Re: Don't use Hadoop when your data isn't that big

#230

Post by jerven »

dekhn wrote: post #215
davidmr wrote: post #179

Earlier quoted context omitted.

It's not just the government that buy these things. Oil and gas, heavy manufacturing, automotive, genomics, financial, etc. all have them. The poster I was replying to appeared to be under the gross misconception that all parallel problems were more effectively solved with high throughput clusters.

Can you point to a genomics company that uses a supercomputer? Adn really claims that its unique capabilities are necessary> Remember, the human genome was assembled on two different architectures: a Dec Alpha cluster with large memory with fast network (for the day) and a bunch of cheap linux machines with slow network. It turns out you didn't actually need the former, although the people at Celera insisted you did.

Genetics is normally very very data parallel. And often the ideal case for throughput computing. But there are rare times when doing modelling that a lot of chatter is required over interconnects. But even here people normally just get 4-6TB RAM machines instead of having lots of machines.
Post reply on HN