Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't use Hadoop when your data isn't that big

chrisstucchio.com

Post reply on HN
Print view View original

151–160 of 235 posts

Re: Don't use Hadoop when your data isn't that big

#151

Post by superuser2 »

bhauer wrote: post #69

The point of the article that resonates with me is how frequently a technology that is poorly fit with a problem domain is selected because of conventional wisdom rather than data. Related, it is remarkable how we developers routinely cite Knuth's advice about premature optimization to justify our decision when the shoe fits, and then turn around and flatly ignore the advice when it doesn't fit. Selecting Hadoop befo…

> the same developers who believe it's reasonable for a web application with modest functionality to require dozens of application nodes to service concurrent request load measured in the mere thousands

I'm assuming this is a dig at Rails and Django? What are you suggesting instead?

Re: Don't use Hadoop when your data isn't that big

#152

Post by w_t_payne »

progx wrote: post #64

Thx! Like always: use the right tool for the job. But many customers dont understand this simple rule.

The people who sign the cheques tend to be managers, who tend to be the sort of person who got to where they are because they are busy and aggressive, and as a result tend to be burdened with a packed; stressful; attention-depleting schedule.

Regardless of the intelligence that they were born with; as a result of all the testosterone-fuelled busyness, they lack the cognitive resources to think deeply or in detail about the decisions that they are making (And, Ironically, as a result also lack the cognitive resources to realise just how intellectually compromised they are).

The upshot of all of this stress and fatigue is that we end up with decisions that are dominated by groupthink; buzzwords and the sales pitch of the latest vendor to stick his shoe in the door. Oh, and by whatever HBR & McKinsey said last week.

Re: Don't use Hadoop when your data isn't that big

#153

Post by dekhn »

w_t_payne wrote: post #147
jedbrown wrote: post #84

Earlier quoted context omitted.

"Big Data starts at 1.5 PB [because that's what fits in memory on Blue Waters]" -- Bill Gropp On today's top HPC installations, it currently takes about an hour to read or write the contents of memory from/to global storage. If the workflow has broad dependencies (PDEs are especially bad, but lots of network analysis also fits the bill), it's much better to use more parallelism to run for a shorter period of time wit…

1.5 PB is pretty huge for most people outside of supercomputing. The biggest dataset I ever worked with fell some way short of that (It was approaching 1PB, I think, although we were not too sure how much data we had exactly, and it might have been slightly over, depending on how you measured). Even handling small(er) working sets could be a challenge in organisation, since we only had a small compute cluster (& asso…

"Supercomputing" has never really been about big data. I used to work in supercomputing (NERSC) and have used most of the major machines in the field in the past. The supercomputer centers claim they're moving lots of data around, but it you look closely, it's almost entirely message passing during a calculation. And in most cases, the calculations they are doing are just wasted resources- simulating a protein over one long trajectory. Supercomputers, as compared to high throughput clusters, are just wasted money.

Re: Don't use Hadoop when your data isn't that big

#154

Post by dekhn »

toomuchtodo wrote: post #144
dalke wrote: post #136

Earlier quoted context omitted.

You and me both. It's made even worse when you learn that the first-level trigger removes 90% of the input stream before passing it to 2,000 computers, which in turn select only 0.2% of that data for storage and further analysis. https://lhcb-public.web.cern.ch/lhcb-public/en/Data%20Collec...

Fermilab in Batavia takes data from CERN over a 40Gb/s optical link (CMS Tier1, CERN CMS is Tier0); when I was there several years ago, we were staging the data to ~500TB of spinning disk in Nexans and cold storage were 2-3 Storagetek tape libraries the size of school busses. Computing and particle physics is where awesome meet.

CERN and its associated teams (CMS and ATLAS) are awesome. They are the only outfit outside of Google that seems to be handling data management and computing properly.

Re: Don't use Hadoop when your data isn't that big

#155

Post by dekhn »

t1m wrote: post #77
mrcactu5 wrote: post #38

I remember starting to grasp what "big data" meant when I had a phone interview with Twitter. @ Imagine you have some numbers spread over some computers -- too many to fit in one computer find the median. ▪ Uhh, sort them? @ Can you find the median on a single computer without sorting them. ▪ :-( @ We'll call you back tomorrow. I was promptly rejected, but it set the tone for my later studies. The criterion for Big D…

Twitter often gets singled out for it's big dataness, though the latest numbers I've seen are only about 400 million tweets per day. Even allowing 1K/tweet this is a rather manageable 400GB uncompressed. 15 minutes of US tweets would fit on your phone :-)

There's another important dimension: the fan-out factor when a person with many followers tweets.

Re: Don't use Hadoop when your data isn't that big

#156

Post by dekhn »

acdha wrote: post #139
mightybyte wrote: post #27

I agree with the general thrust of this article. But hadoop isn't just for scaling up the absolute size of the data set. It is also useful for scaling up the absolute amount of CPU power you can throw at a problem. If I have 1 GB data set, but the computations that I need to do on that data set are complex enough that it would take a single machine a long time to do them, then hadoop is still useful. I gain tremendou…

This assumes that it's easier to manage 100 servers using Hadoop than, say, using the same 100 EC2 servers with a normal grid scheduler, task queue, or even something like GNU parallel. It's orders of magnitude easier to start 100 servers and reuse basic Unix skills than it is to setup and manage Hadoop on the same infrastructure.

does your grid scheduler checkpoint intermediate work automatically, so that node failures have small impact?

Re: Don't use Hadoop when your data isn't that big

#157

Post by VladRussian2 »

bhauer wrote: post #69

The point of the article that resonates with me is how frequently a technology that is poorly fit with a problem domain is selected because of conventional wisdom rather than data. Related, it is remarkable how we developers routinely cite Knuth's advice about premature optimization to justify our decision when the shoe fits, and then turn around and flatly ignore the advice when it doesn't fit. Selecting Hadoop befo…

>Selecting Hadoop before you have a specific and concrete need for it--or see that need approaching rapidly on the horizon

the great achievement of Hadoop isn't performance, throughput, etc... (and in many cases it can actually be worse than alternatives).

The great achievement is that it is an easy (i.e. on the orders of magnitude) accessible (i.e. easy installable, configurable, scalable, supportable, programmable) cluster for "mere mortal" enterprise. Before Hadoop, any clusterable software (where even laughable by Hadoop standards 2 nodes were already priced at the cluster level) would cost arm and leg and 4 or 8 nodes would be considered a huge cluster. Hadoop gives a chance to an enterprise (not to CERN :) to taste an unconstrained (loose use of term) distributed computing, and the next generation of software is already coming - Impala, etc...

And obligatory - "nobody gets fired for using Hadoop".

Re: Don't use Hadoop when your data isn't that big

#158

Post by dekhn »

mindcrime wrote: post #146
gfodor wrote: post #112

Earlier quoted context omitted.

Why do you find this amusing?

It's basically what w_t_payne said. It's amusing (in a sense) because it betrays such a lack of understanding of what's actually going on, and such a willingness to simply accept the "received wisdom" and not question it and think independently.

Some of us have backgrounds in MPI, BSP and other distributed computing paradigms. I switched to high throughput computing and I can tell you that I would never go back to MPI. People who use MPI almost always use it as a crutch to avoid solving the hard problem that would make their computation a lot easier.

As an example I used to write MPI code for molecular dynamics simulators. The goal was to use a large supercomputer to scale the computation up to longer trajectories. But then, with cheap linux boxes it made more sense to just run many simulations in parallel, because low latency MPI class networks cost much more than the machines. We and others made the switch to other approaches which didn't need MPI- we ran many sims in parallel and did only loosely coupled exchanges of data. In fact the coupling was so loose we would run for weeks, dumping output in a large storage system and processing the data with MapReduce.

When I look back at the people who still do the MPI simulations- they are doing dinky stuff and can't even process the data they generate because they spend all their time scaling a code using MPI that didn't need to be scaled with MPI.

Re: Don't use Hadoop when your data isn't that big

#159

Post by zenbowman »

noelwelsh wrote: post #132

In these discussions it is mandatory to quote this paper: http://research.microsoft.com/pubs/163083/hotcbp12%20final.p... : "We completely agree that Hadoop on a cluster is the right solution for jobs where the input data is multi-terabyte or larger. However, in this position paper we ask if this is the right path for general purpose data analytics? Evidence suggests that many MapReduce-like jobs process relatively s…

Even if all jobs take less than 14GB data as input, but the sum total of your data is far greater, it is easier to use Hadoop as a unified single filesystem for storing data rather than relying on multiple different machines that you manage by hand. Even if you treat Hadoop simply as a multi-machine filesystem, it automates the job of managing which files are where, that you inevitably have to manage by hand if you go with storing your data on multiple machines sans some kind of filesystem interface.

Re: Don't use Hadoop when your data isn't that big

#160

Post by stingraycharles »

dekhn wrote: post #153
w_t_payne wrote: post #147

Earlier quoted context omitted.

1.5 PB is pretty huge for most people outside of supercomputing. The biggest dataset I ever worked with fell some way short of that (It was approaching 1PB, I think, although we were not too sure how much data we had exactly, and it might have been slightly over, depending on how you measured). Even handling small(er) working sets could be a challenge in organisation, since we only had a small compute cluster (& asso…

"Supercomputing" has never really been about big data. I used to work in supercomputing (NERSC) and have used most of the major machines in the field in the past. The supercomputer centers claim they're moving lots of data around, but it you look closely, it's almost entirely message passing during a calculation. And in most cases, the calculations they are doing are just wasted resources- simulating a protein over o…

It's not a waste of resources, it's just a different approach to solving a problem. Hadoop / "big data" clusters make the problem harder to solve (and probably even restrict the types of problems that can be solved) in exchange for cheap hardware. Supercomputers give engineers the ability to solve problems in a traditional manner, while moving the costs over to the hardware.
Post reply on HN