Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't use Hadoop when your data isn't that big

chrisstucchio.com

Post reply on HN
Print view View original

181–190 of 235 posts

Re: Don't use Hadoop when your data isn't that big

#181

Post by frozenport »

jedbrown wrote: post #84
davidmr wrote: post #57

While I couldn't agree more with the general point of the article, I have some small additional comments. Just as a bit of background, I think that Chris would very much agree that I am not the intended recipient of this advice, and so my comments probably aren't keeping in the spirit of the article. I've spent the last 10+ years exclusively in very large HPC environments where the average size of the problem set is…

"Big Data starts at 1.5 PB [because that's what fits in memory on Blue Waters]" -- Bill Gropp On today's top HPC installations, it currently takes about an hour to read or write the contents of memory from/to global storage. If the workflow has broad dependencies (PDEs are especially bad, but lots of network analysis also fits the bill), it's much better to use more parallelism to run for a shorter period of time wit…

PDEs are kind compared to graphic codes. Molecular dynamics problems that are characterized by PDEs are some of the largest scaling codes - ever.

Re: Don't use Hadoop when your data isn't that big

#182

Post by dekhn »

davidmr wrote: post #174
dekhn wrote: post #153

Earlier quoted context omitted.

"Supercomputing" has never really been about big data. I used to work in supercomputing (NERSC) and have used most of the major machines in the field in the past. The supercomputer centers claim they're moving lots of data around, but it you look closely, it's almost entirely message passing during a calculation. And in most cases, the calculations they are doing are just wasted resources- simulating a protein over o…

I wrote a different reply but deleted it. I'm curious about something: if, as you claim, supercomputers are wasted money, why are there so many of them? Have all the world's top supercomputing sites somehow colluded to convince all the world's largest governments that they're useful?

At this point, large capability supercomputers that solve problems we don't need solved are mostly kept around as trophy pieces with a few "show me" calculations. Note that when the Chinese obtained the leadership in TOP500 a lot of people got worried (as they did when Japan did it with the Earth Simulator). And so the US may spend a bunch of money to get on top of that list again. Big deal. It's LINPACK. Fortunately, TOP500 should eventually introduce better codes to benchmark.

I'm sure there are scientists who put their all into getting their code to run on a super computer, then apply for a pittance of time and sit in a queue for weeks, but they're not publishing as many (or as interesting) papers as the ones who are pulling out their credit card and building mini-supercomputers on Amazon that rely on conventional interconnects and better algorithms and data processing tools.

Don't assume I'm ignorant about what supercomputers are used for. I used to work on supercomputers; that includes writing, running and evaluating codes, and selecting proposals on some of the largest machines in the world (at the time).

But these new approaches to doing science and engineering on large computer systems have obsoleted conventional supercomputing for all but an extremely limited set of computational problems. And that set of problems becomes smaller when clever computer scientists figure out smarter ways to run things on cheaper architectures. For example, page rank is a classic eigenvector problem; you can solve it by building a big matrix on your supercomputer and doing the appropriate calculations, using 50+ years of numerical optimization (but not really very good support for nodes failing between checkpoints). However, you can also implement it as an iterative mapreduce. The mapreduce checkpoints every little bit of map work, and along the way, handles those failures quite well. It can also handle data sets larger than the sum of RAM on the machines quite well.

Guess which one works better operationally, scales to a larger data set size, and ports to a lot of architectures cheaply?

Re: Don't use Hadoop when your data isn't that big

#184

Post by mindcrime »

dekhn wrote: post #158
mindcrime wrote: post #146

Earlier quoted context omitted.

It's basically what w_t_payne said. It's amusing (in a sense) because it betrays such a lack of understanding of what's actually going on, and such a willingness to simply accept the "received wisdom" and not question it and think independently.

Some of us have backgrounds in MPI, BSP and other distributed computing paradigms. I switched to high throughput computing and I can tell you that I would never go back to MPI. People who use MPI almost always use it as a crutch to avoid solving the hard problem that would make their computation a lot easier. As an example I used to write MPI code for molecular dynamics simulators. The goal was to use a large superco…

Sure, and I'm not saying "MPI > Hadoop" or anything. Just pointing out that Hadoop is not the only game in town, and is hardly the only way to deal with "big data". I'm a Hadoop fan myself, but I also did a lot of MPI stuff in the past, and I believe there are still scenarios where MPI makes a lot of sense. I have less experience with OpenMP, but I think anybody planning a "big data" project would be well served to at least go out, and study up on all of Hadoop (and other M/R platforms), MPI, OpenMP, BSP, etc., before simply choosing Hadoop because it's the framework du jour.

Re: Don't use Hadoop when your data isn't that big

#185

Post by edraferi »

dalke wrote: post #14
Blahah wrote: post #9

Earlier quoted context omitted.

There's nothing boring about pandas

It's even more exciting mixing pandas with the IPython notebook.

heh. Until you actually have to repeat something! It's quite useful for iterative refinements like tweaking pyplot graphics, but it's all to easy to lose track of what versions of which blocks executed to create the current state.

Maybe I just need more discipline..

Re: Don't use Hadoop when your data isn't that big

#186

Post by edraferi »

sitkack wrote: post #172

Hadoop is the problem, MapReduce is not the problem. Having used both Hadoop and Disco, I can say that Disco is by far a net positive on all projects I used it on. And the overhead to coding it in Disco vs single node is about an extra 30 minutes. You can start with working single node and go multinode w/o much effort. http://discoproject.org Hadoop on the other hand is a huge, massive pain in the ass. And I am a Had…

Cool project, thanks for the link.

Re: Don't use Hadoop when your data isn't that big

#187

Post by edraferi »

w_t_payne wrote: post #149
acidity wrote: post #113

I have been looking into building simple recommendation engine (i have at max million data rows) using Python. I looked into Crab ( https://github.com/muricoca/crab ) and it seems to not have updated for 2 years. Any suggestions for libraries or just use basic numpy/scipy and implement the algorithms?

Just implement the damn algorithms. Anything else is just another library to learn, and another tool to babysit.

Agreed. Take the Machine Learning class on Coursera if you need an introduction to them.

Re: Don't use Hadoop when your data isn't that big

#188

Post by nl »

w_t_payne wrote: post #148
harrytuttle wrote: post #44

Earlier quoted context omitted.

Yes I've worked for a couple of outfits that did this over the years. It's usually down to who sold them the solution though. It's worst if the salesperson is external. The funniest has to be the engineering company which I used to work with (not for thank fuck) back in the 90's (when 100Gb was big data!). They managed to bag 2x full 42U racks full of HP N-Class HPUX UNIX kit, PDU systems and separate disk arrays for…

Yeah... it looks like my employer is going down the same route with an EMC Isilon NAS system. We really just want high throughput for linear scans; possibly with a small amount of parallelism, which would argue for a fairly simple solution with as little cacheing as possible (ideally none whatsoever) ... but I cannot even tell (from the website & the literature that I can access) how these things are supposed to oper…

Isilon is nice - expensive (it is EMC), but it gives good performance and is reliable.

Re: Don't use Hadoop when your data isn't that big

#189

Post by [deleted] »

acidity wrote: post #113

I have been looking into building simple recommendation engine (i have at max million data rows) using Python. I looked into Crab ( https://github.com/muricoca/crab ) and it seems to not have updated for 2 years. Any suggestions for libraries or just use basic numpy/scipy and implement the algorithms?

[deleted]

Re: Don't use Hadoop when your data isn't that big

#190

Post by acidity »

edraferi wrote: post #187
w_t_payne wrote: post #149

Earlier quoted context omitted.

Just implement the damn algorithms. Anything else is just another library to learn, and another tool to babysit.

Agreed. Take the Machine Learning class on Coursera if you need an introduction to them.

Okay. I already have read about the algorithms and even implemented them crudely while studying them.

Just not a big fan of NHI. But if in this case, thats the best way - let it be :)

Post reply on HN