Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Key/value is dead. Long live tuples: Pangool for Hadoop

datasalt.com

Post reply on HN
Print view View original

1–10 of 30 posts

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#3

Post by ivanprado »

silssilsssil wrote: post #2

I'm wondering what's the need for this when we already have Apache Pig, etc?

Hi, I'm one of the developers of Pangool. The idea of Pangool is not to be yet another higher level API on top of Hadoop but rather to pose a replacement for the low-level Hadoop Java MapReduce API. Pangool has the same performance and flexibility than that of the Java MapReduce API although it makes several things a lot easier and convenient. There is no tradeoff, just advantages. There will be cases where you'd want to use Pig or Cascading. There will be some other cases where you'd want the flexibility and efficiency of MapReduce. For those cases we conceived Pangool. Nowadays only very advanced Hadoop users could write efficiently-performing MapReduce Jobs. Pangool hides all the advanced boilerplate code needed for writing highly efficient MapReduce jobs, making things like secondary sorting or reduce-side joins extremely easy.

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#4

Post by avibryant »

ivanprado wrote: post #3
silssilsssil wrote: post #2

I'm wondering what's the need for this when we already have Apache Pig, etc?

Hi, I'm one of the developers of Pangool. The idea of Pangool is not to be yet another higher level API on top of Hadoop but rather to pose a replacement for the low-level Hadoop Java MapReduce API. Pangool has the same performance and flexibility than that of the Java MapReduce API although it makes several things a lot easier and convenient. There is no tradeoff, just advantages. There will be cases where you'd wan…

Can you give an example of a job that would be difficult or impossible to perform efficiently with Cascading, but Pangool gives an advantage over raw MapReduce?

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#6

Post by fruchtose »

silssilsssil wrote: post #2

I'm wondering what's the need for this when we already have Apache Pig, etc?

Apache Pig is a much different beast than this project, from what I can tell reading the documentation for Pangool. While they both operate on tuples and work at a higher level than pure Hadoop, they accomplish their goals much differently.Pig uses its own language called Pig Latin (http://pig.apache.org/docs/r0.9.2/basic.html), which is then compiled down into code that interfaces with the Hadoop library. Pangool is much closer to Hadoop, in that you are writing Java. If you look at one of their examples (http://pangool.net/introduction.html), I get the sense that the developers aim to make Hadoop easier to use, while Pig aims to make data analysis easier to use.

These goals are greatly divergent. In Pig, Java code is written to create new functions that can be used for analysis--i.e. Java is written in support of Pig Latin. Pangool focuses instead on extending Hadoop by making the Java code easier to write. This means Pig could potentially be implemented in Pangool, if Pangool were to satisfy the requirements for the task. (Not that I am suggesting that Pig actually be written--it might just be possible, depending on the technical requirements.)

Having used Hadoop in the past, I would be more inclined to use Pangool. Parts of Hadoop are poorly written--especially the reliance on singletons--and anything that makes it easier to write code that runs on a Hadoop cluster is a desirable goal in my eyes. I look forward to seeing how this project shapes up.

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#7

Post by ivanprado »

avibryant wrote: post #4
ivanprado wrote: post #3

Earlier quoted context omitted.

Hi, I'm one of the developers of Pangool. The idea of Pangool is not to be yet another higher level API on top of Hadoop but rather to pose a replacement for the low-level Hadoop Java MapReduce API. Pangool has the same performance and flexibility than that of the Java MapReduce API although it makes several things a lot easier and convenient. There is no tradeoff, just advantages. There will be cases where you'd wan…

Can you give an example of a job that would be difficult or impossible to perform efficiently with Cascading, but Pangool gives an advantage over raw MapReduce?

Hi avibryant, According to our initial benchmark (http://pangool.net/benchmark.html), secondary sorting in Cascading is slow (http://bit.ly/wTKOxo), showing a 243% performance overhead compared to an efficient implementation in MapReduce. The implementation in MapReduce has a lot of lines (http://bit.ly/yYGnGe) whereas Pangool's implementation is quite simple (http://bit.ly/x9U7Yj). A common application of secondary sort is calculating moving averages, for instance.

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#8

Post by bobbyi »

fruchtose wrote: post #6
silssilsssil wrote: post #2

I'm wondering what's the need for this when we already have Apache Pig, etc?

Apache Pig is a much different beast than this project, from what I can tell reading the documentation for Pangool. While they both operate on tuples and work at a higher level than pure Hadoop, they accomplish their goals much differently.Pig uses its own language called Pig Latin ( http://pig.apache.org/docs/r0.9.2/basic.html ), which is then compiled down into code that interfaces with the Hadoop library. Pangool…

> In Pig, Java code is written to create new functions that can be used for analysis--i.e. Java is written in support of Pig Latin.

It doesn't change your larger point, but Pig also allows writing UDFs (user-defined functions) in Python via Jython: https://cwiki.apache.org/confluence/display/PIG/UDFsUsingScr...

We are using this in production and it works quite well.

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#9

Post by haberman »

ivanprado wrote: post #3
silssilsssil wrote: post #2

I'm wondering what's the need for this when we already have Apache Pig, etc?

Hi, I'm one of the developers of Pangool. The idea of Pangool is not to be yet another higher level API on top of Hadoop but rather to pose a replacement for the low-level Hadoop Java MapReduce API. Pangool has the same performance and flexibility than that of the Java MapReduce API although it makes several things a lot easier and convenient. There is no tradeoff, just advantages. There will be cases where you'd wan…

> There is no tradeoff, just advantages.

Though I don't have deep expertise in Hadoop, I find this claim highly suspect. High-level APIs achieve user-friendliness by making decisions/assumptions about the way a lower-level API will be used. I would be very surprised if there was no use case for which your API does impose a trade-off vs. the low-level Hadoop API.

I feel much more confident using a high-level API if its author is up-front about what assumptions it's making. If the claim is that there is no trade-off vs. the low-level API, I generally conclude that the author doesn't understand the problem space well enough to know what those trade-offs are.

I could be wrong, but this is my bias/experience.

Post reply on HN