Live data from Hacker News

Clojure on Hadoop: A New Hope

blog.factual.com

11–20 of 29 posts

Re: Clojure on Hadoop: A New Hope

#11
post #9

My experience with Hadoop tells me that its great for all counting tasks. Makes a lot of sense that it was designed at Google to construct posting lists for their search index. Beyond this sweet spot, it gets really tricky to map your solution to a map-reduce task. The programmer has to rely more on the fact that map/reduce are java black boxes to express everything he needs. Hadoop's big victory is the scale it oper…

I work on Streams at IBM Research. Our solution to handling large amounts of data is called stream programming, and its more general than MapReduce. For a sample of what our language looks like, check out this example: http://publib.boulder.ibm.com/infocenter/streams/v2r0/index.... And a pdf version of the same: http://publib.boulder.ibm.com/infocenter/streams/v2r0/topic/...

isn't stream programming what storm is supposed to be?

Re: Clojure on Hadoop: A New Hope

#12

My experience with Hadoop tells me that its great for all counting tasks. Makes a lot of sense that it was designed at Google to construct posting lists for their search index. Beyond this sweet spot, it gets really tricky to map your solution to a map-reduce task. The programmer has to rely more on the fact that map/reduce are java black boxes to express everything he needs. Hadoop's big victory is the scale it oper…

> "SQL-like-declarative-query-language + Hadoop"

It already exists, is free, and is widely deployed. It's also actively maintained, mainly by a lot of Facebook folks, and I've heard that they know a thing or two about scale:

http://hive.apache.org/

Re: Clojure on Hadoop: A New Hope

#13
post #10

This is an entertaining comparison of 8 different map-reduce languages for Hadoop, including a flame-tastic take on Cascalog/Clojure: http://www.dataspora.com/2011/04/pigs-bees-and-elephants-a-c...

"LISP has been around some 50-odd years without taking off despite several attempts at its revival... I suspect something is wrong with it" I laughed out loud at that.

The guy is a genuinely funny writer, incredibly rare in tech. If you find anything else written by him, let me know.

Re: Clojure on Hadoop: A New Hope

#14
post #9

Earlier quoted context omitted.

I work on Streams at IBM Research. Our solution to handling large amounts of data is called stream programming, and its more general than MapReduce. For a sample of what our language looks like, check out this example: http://publib.boulder.ibm.com/infocenter/streams/v2r0/index.... And a pdf version of the same: http://publib.boulder.ibm.com/infocenter/streams/v2r0/topic/...

isn't stream programming what storm is supposed to be?

Yes, it is a similar programming model. Some differences (please put "to the best of my knowledge" in front of all of these):

- Storm does not allow arbitrary state in operators (what Nathan Marz calls "bolts"). This makes implementing the runtime easier, such as being able to replay tuple sends for fault tolerance, but it limits what kinds of applications one can make. Yes, I'm on board with the idea that we should avoid mutable state as much as possible, but people who build real applications want it. Yet, fault tolerance in our system requires more work, so it's a trade-off.

- Storm programs are implemented in Java. Streams applications are implemented in our programming language, which has the rather pedestrian name Streams Programming Language, but usually just SPL. This may seem minor, but it's a big deal. Marz is working on a higher level language in Clojure. Implementing programs in a higher level language enables developers to abstract away many issues related to high performance, distributed systems. I compare it to the difference between writing assembly code and writing C code. (Or the difference between writing Python code and writing C code.) The code that we generate is similar in principle to how one writes a Storm application. Which brings me to...

- Storm runs on the JVM, we generate C++ code which gets compiled.

Neither Storm or Streams are the first or only in this area. Stream programming is also popular for hardware, but that is usually synchronous and if there's state, it's shared-memory. Storm and Streams are distributed and asynchronous. There are academic distributed streaming systems such as Borealis. The research name for Streams is System S, and there are many academic papers about it, or that use it as a platform for other research: http://dl.acm.org/results.cfm?h=1&cfid=66087472&cfto...

And for the record, I am impressed with Storm.

Re: Clojure on Hadoop: A New Hope

#15

My experience with Hadoop tells me that its great for all counting tasks. Makes a lot of sense that it was designed at Google to construct posting lists for their search index. Beyond this sweet spot, it gets really tricky to map your solution to a map-reduce task. The programmer has to rely more on the fact that map/reduce are java black boxes to express everything he needs. Hadoop's big victory is the scale it oper…

Actually, Hadoop was designed at Yahoo!, based on Google's MapReduce. Yahoo! doesn't get nearly enough credit for what they do get right.

Re: Clojure on Hadoop: A New Hope

#16
post #14

Earlier quoted context omitted.

isn't stream programming what storm is supposed to be?

Yes, it is a similar programming model. Some differences (please put "to the best of my knowledge" in front of all of these): - Storm does not allow arbitrary state in operators (what Nathan Marz calls "bolts"). This makes implementing the runtime easier, such as being able to replay tuple sends for fault tolerance, but it limits what kinds of applications one can make. Yes, I'm on board with the idea that we should…

Also, Storm is freely available as open-source software with a permissive license.

Re: Clojure on Hadoop: A New Hope

#17
post #12

My experience with Hadoop tells me that its great for all counting tasks. Makes a lot of sense that it was designed at Google to construct posting lists for their search index. Beyond this sweet spot, it gets really tricky to map your solution to a map-reduce task. The programmer has to rely more on the fact that map/reduce are java black boxes to express everything he needs. Hadoop's big victory is the scale it oper…

> "SQL-like-declarative-query-language + Hadoop" It already exists, is free, and is widely deployed. It's also actively maintained, mainly by a lot of Facebook folks, and I've heard that they know a thing or two about scale: http://hive.apache.org/

Academics are more ambitious. They want the entire power of relational dbs at the hadoop scale. With consistency/transactions/complex joins etc..

Re: Clojure on Hadoop: A New Hope

#18
post #14

Earlier quoted context omitted.

isn't stream programming what storm is supposed to be?

Yes, it is a similar programming model. Some differences (please put "to the best of my knowledge" in front of all of these): - Storm does not allow arbitrary state in operators (what Nathan Marz calls "bolts"). This makes implementing the runtime easier, such as being able to replay tuple sends for fault tolerance, but it limits what kinds of applications one can make. Yes, I'm on board with the idea that we should…

Very cool. Is it possible to add operators to Stream? I commonly run into the problem of batch resizing a lot of images. If there was an easy way to integrate imagemagick as an operator into a system which can push this task to different cores, that would be a big big win.

Re: Clojure on Hadoop: A New Hope

#19
post #16
post #14

Earlier quoted context omitted.

Yes, it is a similar programming model. Some differences (please put "to the best of my knowledge" in front of all of these): - Storm does not allow arbitrary state in operators (what Nathan Marz calls "bolts"). This makes implementing the runtime easier, such as being able to replay tuple sends for fault tolerance, but it limits what kinds of applications one can make. Yes, I'm on board with the idea that we should…

Also, Storm is freely available as open-source software with a permissive license.

Yes, I meant to mention this. Streams is software that IBM sells. But there is a free academic license.

Re: Clojure on Hadoop: A New Hope

#20
post #14

Earlier quoted context omitted.

Yes, it is a similar programming model. Some differences (please put "to the best of my knowledge" in front of all of these): - Storm does not allow arbitrary state in operators (what Nathan Marz calls "bolts"). This makes implementing the runtime easier, such as being able to replay tuple sends for fault tolerance, but it limits what kinds of applications one can make. Yes, I'm on board with the idea that we should…

Very cool. Is it possible to add operators to Stream? I commonly run into the problem of batch resizing a lot of images. If there was an easy way to integrate imagemagick as an operator into a system which can push this task to different cores, that would be a big big win.

Yes, user-defined operators are a big part of the design of the system. But, as noted below, this is software that IBM sells, and right now we're targeting large companies.
Post reply on HN