Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Key/value is dead. Long live tuples: Pangool for Hadoop

datasalt.com

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#21

Post by ivanprado »

avibryant wrote: post #11
ivanprado wrote: post #7

Earlier quoted context omitted.

Hi avibryant, According to our initial benchmark ( http://pangool.net/benchmark.html ), secondary sorting in Cascading is slow ( http://bit.ly/wTKOxo ), showing a 243% performance overhead compared to an efficient implementation in MapReduce. The implementation in MapReduce has a lot of lines ( http://bit.ly/yYGnGe ) whereas Pangool's implementation is quite simple ( http://bit.ly/x9U7Yj ). A common application of se…

Ok, so Cascading has a slow implementation of secondary sort, but is there any reason you believe that couldn't be improved? I don't think you're really comparing architectures there, just how well optimized particular implementations are. I'm asking because in my experience the extra level of abstraction provided by Cascading, Crunch etc is a huge advantage, and if you're making a conscious choice to operate at a lo…

Pangool is not an alternative for Cascading. For example, at this point, Pangool does not help you managing workflows. If you are starting a MapReduce application, it is probably the best option to start using higher level abstractions: Cascading, Hive, Pig, etc.

But if you are thinking about learning Hadoop using the standard Hadoop API, or if you need for some particular reason to use it for your project, we recommend you to use Pangool instead.

Or if you are considering to implement another abstraction on top of Hadoop, probably using Pangool for it would also be a good idea.

In fact, what we believe is that the default Hadoop API should look like Pangool.

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#23

Post by haberman »

ferrerabertran wrote: post #13
haberman wrote: post #9

Earlier quoted context omitted.

> There is no tradeoff, just advantages. Though I don't have deep expertise in Hadoop, I find this claim highly suspect. High-level APIs achieve user-friendliness by making decisions/assumptions about the way a lower-level API will be used. I would be very surprised if there was no use case for which your API does impose a trade-off vs. the low-level Hadoop API. I feel much more confident using a high-level API if it…

Hi haberman, I'm one of the developers of Pangool. Let me try to clarify why we stated that. I understand it may sound aggresive. Pangool is based on an extension of the MapReduce model we suggest and call "Tuple MapReduce". This is explained in detail in this post: http://www.datasalt.com/2012/02/tuple-mapreduce-beyond-the-c... What this means is that in Pangool, if you worked with 2-sized Tuples, you would be able…

It sounds like you are implementing an in-memory data structure (Tuple) and serialization of that data structure on top of the raw strings provided by the Hadoop API. While I can believe that the overall overhead of this would be small in many cases, you would observe it most severely in cases where your data was natively key/value pairs of very short strings, or where you had lots of tuples with very short payloads. Do any of your performance tests cover this case? I would expect Pangool to display more than negligible CPU and memory overhead in this case.

Also, since the data model is more complicated and provides more features, it takes more code and a more complex implementation. This could be significant if you were trying to port the model to another language or implementation, or were trying to formally things about the code or mathematical model, etc.

I'm not saying it's not cool; I actually think it's a good and powerful abstraction -- I just object to the characterization of "all features and no tradeoffs".

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#24

Post by alatkins »

protomyth wrote: post #5

a couple of links that came to mind with this: http://en.wikipedia.org/wiki/Tuple_space http://en.wikipedia.org/wiki/Linda_(coordination_language) http://www.amazon.com/Mirror-Worlds-Software-Universe-Shoebo...

And for a slightly more modern take on the tuple space, check out Java Spaces [1] or Gigaspaces [2]. There's still plenty of active research on the topic too [3] (disclaimer: I did my PhD thesis on distributed tuple spaces).

I've long contended that a tuple space was basically a generalised key-value store, so it's nice to see projects like this one crop up.

[1] http://java.net/projects/jini/

[2] http://www.gigaspaces.com/

[3] http://eprints.utas.edu.au/9996/

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#25

Post by squarecog »

ivanprado wrote: post #7
avibryant wrote: post #4

Earlier quoted context omitted.

Can you give an example of a job that would be difficult or impossible to perform efficiently with Cascading, but Pangool gives an advantage over raw MapReduce?

Hi avibryant, According to our initial benchmark ( http://pangool.net/benchmark.html ), secondary sorting in Cascading is slow ( http://bit.ly/wTKOxo ), showing a 243% performance overhead compared to an efficient implementation in MapReduce. The implementation in MapReduce has a lot of lines ( http://bit.ly/yYGnGe ) whereas Pangool's implementation is quite simple ( http://bit.ly/x9U7Yj ). A common application of se…

You are doing regex matching in the Cascading code, but splitting on a character in the pangool code. The latter is obviously much faster. I don't know that that's the reason for the difference you observe, but it certainly can't hurt to fix that and make the user-supplied code more comparable.

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#26

Post by ferrerabertran »

squarecog wrote: post #25
ivanprado wrote: post #7

Earlier quoted context omitted.

Hi avibryant, According to our initial benchmark ( http://pangool.net/benchmark.html ), secondary sorting in Cascading is slow ( http://bit.ly/wTKOxo ), showing a 243% performance overhead compared to an efficient implementation in MapReduce. The implementation in MapReduce has a lot of lines ( http://bit.ly/yYGnGe ) whereas Pangool's implementation is quite simple ( http://bit.ly/x9U7Yj ). A common application of se…

You are doing regex matching in the Cascading code, but splitting on a character in the pangool code. The latter is obviously much faster. I don't know that that's the reason for the difference you observe, but it certainly can't hurt to fix that and make the user-supplied code more comparable.

Indeed that regex was problematic because it had a bug itself. We replaced that line by RegexSplitter and updated the benchmark page. Please shout if you notice something else wrong. Thanks.

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#27

Post by ivanprado »

squarecog wrote: post #25
ivanprado wrote: post #7

Earlier quoted context omitted.

Hi avibryant, According to our initial benchmark ( http://pangool.net/benchmark.html ), secondary sorting in Cascading is slow ( http://bit.ly/wTKOxo ), showing a 243% performance overhead compared to an efficient implementation in MapReduce. The implementation in MapReduce has a lot of lines ( http://bit.ly/yYGnGe ) whereas Pangool's implementation is quite simple ( http://bit.ly/x9U7Yj ). A common application of se…

You are doing regex matching in the Cascading code, but splitting on a character in the pangool code. The latter is obviously much faster. I don't know that that's the reason for the difference you observe, but it certainly can't hurt to fix that and make the user-supplied code more comparable.

Just for clarify, split() java function is using regexp for the split as well. The code of String.split() is:

return Pattern.compile(regex).split(this, limit);

The benchmark seems fair to me.

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#28

Post by dredmorbius »

jasonkolb wrote: post #10

I just popped in to say that I'm tired of the "X is dead" linkbait headlines. They demonstrate a myopic view of the world. Visual Basic and COBOL are still around.

So, what you're saying is "'X is dead' is considered harmful"?

Re: Key/value is dead. Long live tuples: Pangool for Hadoop

#29

Post by rjurney »

ferrerabertran wrote: post #14
rjurney wrote: post #12

So it sounds like this slots in like so, in order of abstraction: HIVE -> Pig -> Pangool -> Cascading -> MapReduce Nice addition!

Hi rjurney. I would say "Hive, Pig, Cascading" are on the higher level API side and "Pangool, MapReduce" on the low-level side. Pangool is a MapReduce API that aims to make MapReduce simpler. We explain this better in our FAQ: http://pangool.net/faq.html

HIVE -> Pig -> Cascading -> Pangool -> MapReduce ?
Post reply on HN