Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Key/value is dead. Long live tuples: Pangool for Hadoop

datasalt.com

Post reply on HN
Print view View original

11–20 of 30 posts

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#11

Post by avibryant »

ivanprado wrote: post #7
avibryant wrote: post #4

Earlier quoted context omitted.

Can you give an example of a job that would be difficult or impossible to perform efficiently with Cascading, but Pangool gives an advantage over raw MapReduce?

Hi avibryant, According to our initial benchmark ( http://pangool.net/benchmark.html ), secondary sorting in Cascading is slow ( http://bit.ly/wTKOxo ), showing a 243% performance overhead compared to an efficient implementation in MapReduce. The implementation in MapReduce has a lot of lines ( http://bit.ly/yYGnGe ) whereas Pangool's implementation is quite simple ( http://bit.ly/x9U7Yj ). A common application of se…

Ok, so Cascading has a slow implementation of secondary sort, but is there any reason you believe that couldn't be improved? I don't think you're really comparing architectures there, just how well optimized particular implementations are.

I'm asking because in my experience the extra level of abstraction provided by Cascading, Crunch etc is a huge advantage, and if you're making a conscious choice to operate at a lower level, you better be getting something significant in return; it's not clear to me yet what that is.

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#13

Post by ferrerabertran »

haberman wrote: post #9
ivanprado wrote: post #3

Earlier quoted context omitted.

Hi, I'm one of the developers of Pangool. The idea of Pangool is not to be yet another higher level API on top of Hadoop but rather to pose a replacement for the low-level Hadoop Java MapReduce API. Pangool has the same performance and flexibility than that of the Java MapReduce API although it makes several things a lot easier and convenient. There is no tradeoff, just advantages. There will be cases where you'd wan…

> There is no tradeoff, just advantages. Though I don't have deep expertise in Hadoop, I find this claim highly suspect. High-level APIs achieve user-friendliness by making decisions/assumptions about the way a lower-level API will be used. I would be very surprised if there was no use case for which your API does impose a trade-off vs. the low-level Hadoop API. I feel much more confident using a high-level API if it…

Hi haberman, I'm one of the developers of Pangool. Let me try to clarify why we stated that. I understand it may sound aggresive.

Pangool is based on an extension of the MapReduce model we suggest and call "Tuple MapReduce". This is explained in detail in this post: http://www.datasalt.com/2012/02/tuple-mapreduce-beyond-the-c...

What this means is that in Pangool, if you worked with 2-sized Tuples, you would be able to do exactly the same that you do now with Java MapReduce - That includes custom RawComparators and arbitrary business logic in any place of the MapReduce chain (Mapper, Combiner, Reducer). Using n-sized Tuples together with Pangool's group & sort by, reduce-side join API will only mean less code, easier code at no loss of performance or flexibility.

Realize that Pangool is still a MapReduce API so it doesn't add any level of abstraction.

We designed Pangool with the aim of offering it as a replacement of the current MapReduce API. Therefore we are not labelling it as a "higher-level API" but as comparable low-level API.

On the other hand we are also benchmarking Pangool to show it doesn't impose a performance overhead: http://pangool.net/benchmark.html

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#14

Post by ferrerabertran »

rjurney wrote: post #12

So it sounds like this slots in like so, in order of abstraction: HIVE -> Pig -> Pangool -> Cascading -> MapReduce Nice addition!

Hi rjurney. I would say "Hive, Pig, Cascading" are on the higher level API side and "Pangool, MapReduce" on the low-level side. Pangool is a MapReduce API that aims to make MapReduce simpler. We explain this better in our FAQ: http://pangool.net/faq.html

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#15

Post by scott_s »

ferrerabertran wrote: post #13
haberman wrote: post #9

Earlier quoted context omitted.

> There is no tradeoff, just advantages. Though I don't have deep expertise in Hadoop, I find this claim highly suspect. High-level APIs achieve user-friendliness by making decisions/assumptions about the way a lower-level API will be used. I would be very surprised if there was no use case for which your API does impose a trade-off vs. the low-level Hadoop API. I feel much more confident using a high-level API if it…

Hi haberman, I'm one of the developers of Pangool. Let me try to clarify why we stated that. I understand it may sound aggresive. Pangool is based on an extension of the MapReduce model we suggest and call "Tuple MapReduce". This is explained in detail in this post: http://www.datasalt.com/2012/02/tuple-mapreduce-beyond-the-c... What this means is that in Pangool, if you worked with 2-sized Tuples, you would be able…

The tradeoff, then, is that if someone's current problem maps exactly to the current API, then your API is more complex than needed.

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#18

Post by tim_h »

scott_s wrote: post #15
ferrerabertran wrote: post #13

Earlier quoted context omitted.

Hi haberman, I'm one of the developers of Pangool. Let me try to clarify why we stated that. I understand it may sound aggresive. Pangool is based on an extension of the MapReduce model we suggest and call "Tuple MapReduce". This is explained in detail in this post: http://www.datasalt.com/2012/02/tuple-mapreduce-beyond-the-c... What this means is that in Pangool, if you worked with 2-sized Tuples, you would be able…

The tradeoff, then, is that if someone's current problem maps exactly to the current API, then your API is more complex than needed.

Pangool actually seems like a generalization of Hadoop. This doesn't necessarily make it more complex. If a problem maps exactly to the Hadoop API, then it should also map exactly to the Pangool API by setting m=2 (in the extended map reduce model described at http://www.datasalt.com/2012/02/tuple-mapreduce-beyond-the-c...).

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#19

Post by kellenfujimoto »

philwelch wrote: post #17
jasonkolb wrote: post #10

I just popped in to say that I'm tired of the "X is dead" linkbait headlines. They demonstrate a myopic view of the world. Visual Basic and COBOL are still around.

Dead is relative. Dead usually means "dead to me".

Or in the case of VB, "better off dead".

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#20

Post by scott_s »

tim_h wrote: post #18
scott_s wrote: post #15

Earlier quoted context omitted.

The tradeoff, then, is that if someone's current problem maps exactly to the current API, then your API is more complex than needed.

Pangool actually seems like a generalization of Hadoop. This doesn't necessarily make it more complex. If a problem maps exactly to the Hadoop API, then it should also map exactly to the Pangool API by setting m=2 (in the extended map reduce model described at http://www.datasalt.com/2012/02/tuple-mapreduce-beyond-the-c... ).

I agree with your first sentence, but disagree with the second. That you can find an exact mapping does not prevent the underlying API from being more complex than what you need. That you had to realize "Oh, m=2" is more complexity.

I'm not arguing this is a terrible thing. In fact, I think this is an acceptable level of additional complexity for the power it buys you. But if we're going to make an honest evaluation of the trade-offs, I think we must mention this.

It may be relevant to the discussion to point out that I work on a tuple-based streaming system. Product: http://www-01.ibm.com/software/data/infosphere/streams/ Academic: http://dl.acm.org/citation.cfm?id=1890754.1890761, http://dl.acm.org/citation.cfm?id=1645953.1646061

Post reply on HN