Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Democratizing Big Data - Is Hadoop Our Only Hope?

gigaom.com

Post reply on HN
Print view View original

11–16 of 16 posts

Re: Democratizing Big Data - Is Hadoop Our Only Hope?

#11

Post by helwr »

helwr wrote: post #4

A good list of hadoop alternatives: http://www.quora.com/What-are-some-promising-open-source-alt... my personal favorite is BashReduce (~120 lines shell script vs ~600k lines of java code in hadoop): http://blog.last.fm/2009/04/06/mapreduce-bash-script If you're in bioinformatics you might be interested in this talk on handling ridiculous amounts of data (PyCon 2011): http://blip.tv/pycon-us-videos-2009-2010-2011/pyc…

Also see GraphLab, a New Parallel Framework for Machine Learning: http://www.graphlab.ml.cmu.edu/

Re: Democratizing Big Data - Is Hadoop Our Only Hope?

#12

Post by cdavid »

martincmartin wrote: post #7

I think the real question is "Is MapReduce Our Only Hope?" MapReduce has a lot of limitations. It doesn't have a query language, instead, you need to figure out the sequence of map and reduce steps and implement those in your favourite low level language yourself. And it can't do efficient joins. That means you need to visit each and every row for each and every map-reduce stage. There's no b-tree or other "lookup st…

I don't understand the claim that adding new data force you to recompute everything. What requires re-computation will depend on the algorithm, but for most simples cases I can think of, at least the map part will not require computation if you record its result. I believe that's how couchdb view work, for example

Re: Democratizing Big Data - Is Hadoop Our Only Hope?

#13

Post by yummyfajitas »

cdavid wrote: post #12
martincmartin wrote: post #7

I think the real question is "Is MapReduce Our Only Hope?" MapReduce has a lot of limitations. It doesn't have a query language, instead, you need to figure out the sequence of map and reduce steps and implement those in your favourite low level language yourself. And it can't do efficient joins. That means you need to visit each and every row for each and every map-reduce stage. There's no b-tree or other "lookup st…

I don't understand the claim that adding new data force you to recompute everything. What requires re-computation will depend on the algorithm, but for most simples cases I can think of, at least the map part will not require computation if you record its result. I believe that's how couchdb view work, for example

Hadoop is a fairly low level framework. So it requires the programmer to write logic to incrementally calculate the map (i.e., input_data_may -> map_results_may, repeat for june).

Similarly, it's up to the programmer to write a reducer which allows incremental additions of data. And even then, you still need to make a pass over all the data.

Re: Democratizing Big Data - Is Hadoop Our Only Hope?

#14

Post by cdavid »

yummyfajitas wrote: post #13
cdavid wrote: post #12

Earlier quoted context omitted.

I don't understand the claim that adding new data force you to recompute everything. What requires re-computation will depend on the algorithm, but for most simples cases I can think of, at least the map part will not require computation if you record its result. I believe that's how couchdb view work, for example

Hadoop is a fairly low level framework. So it requires the programmer to write logic to incrementally calculate the map (i.e., input_data_may -> map_results_may, repeat for june). Similarly, it's up to the programmer to write a reducer which allows incremental additions of data. And even then, you still need to make a pass over all the data.

That's a limitation of hadoop, though, not the MR idea by itself.

Re: Democratizing Big Data - Is Hadoop Our Only Hope?

#15

Post by yummyfajitas »

cdavid wrote: post #14
yummyfajitas wrote: post #13

Earlier quoted context omitted.

Hadoop is a fairly low level framework. So it requires the programmer to write logic to incrementally calculate the map (i.e., input_data_may -> map_results_may, repeat for june). Similarly, it's up to the programmer to write a reducer which allows incremental additions of data. And even then, you still need to make a pass over all the data.

That's a limitation of hadoop, though, not the MR idea by itself.

The limitation on the map end is only a limitation for hadoop.

But the limit on the reducer is fundamental. Some reduce functions are not associative, and some don't even have type [T x U] -> T x U. In those cases, there is nothing to be done but redo the reduce.

Re: Democratizing Big Data - Is Hadoop Our Only Hope?

#16

Post by cdavid »

yummyfajitas wrote: post #15
cdavid wrote: post #14

Earlier quoted context omitted.

That's a limitation of hadoop, though, not the MR idea by itself.

The limitation on the map end is only a limitation for hadoop. But the limit on the reducer is fundamental. Some reduce functions are not associative, and some don't even have type [T x U] -> T x U. In those cases, there is nothing to be done but redo the reduce.

Indeed, reduce is the difficult part. OTOH, I think this limitation is seen in many algorithms at a fairly fundamental level, and not just an artefact of MR. The only alternative framework I can think of for dealing with really large datasets in a distributed manner is sampling-based methods, with one-pass algorithms (or mostly one pass algorithm).
Post reply on HN