Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Clojure on Hadoop: A New Hope

blog.factual.com

Post reply on HN
Print view View original

21–29 of 29 posts

Re: Clojure on Hadoop: A New Hope

#21

Post by samstokes »

scott_s wrote: post #14
toisanji wrote: post #11

Earlier quoted context omitted.

isn't stream programming what storm is supposed to be?

Yes, it is a similar programming model. Some differences (please put "to the best of my knowledge" in front of all of these): - Storm does not allow arbitrary state in operators (what Nathan Marz calls "bolts"). This makes implementing the runtime easier, such as being able to replay tuple sends for fault tolerance, but it limits what kinds of applications one can make. Yes, I'm on board with the idea that we should…

Storm does not allow arbitrary state in operators (what Nathan Marz calls "bolts").

Maybe I misunderstand your point, but Storm does allow state in Bolts - a Bolt is just a Java object, so it can have member variables. That's how aggregation (e.g. counting events per user) is done. Of course, if you want a Bolt that scales horizontally, you need to account for the state being split across several instances of the Bolt class; and if you need the state to survive a restart, you need to keep it in an external database instead of in the object's memory.

Re: Clojure on Hadoop: A New Hope

#22

Post by scott_s »

samstokes wrote: post #21
scott_s wrote: post #14

Earlier quoted context omitted.

Yes, it is a similar programming model. Some differences (please put "to the best of my knowledge" in front of all of these): - Storm does not allow arbitrary state in operators (what Nathan Marz calls "bolts"). This makes implementing the runtime easier, such as being able to replay tuple sends for fault tolerance, but it limits what kinds of applications one can make. Yes, I'm on board with the idea that we should…

Storm does not allow arbitrary state in operators (what Nathan Marz calls "bolts"). Maybe I misunderstand your point, but Storm does allow state in Bolts - a Bolt is just a Java object, so it can have member variables. That's how aggregation (e.g. counting events per user) is done. Of course, if you want a Bolt that scales horizontally, you need to account for the state being split across several instances of the Bol…

Then I stand corrected. Is there any means to declare partitioned state that the runtime then handles for the user, or does the user always have to manage it themselves? This is one of those things that is, I think, easier to do declaratively one level up in abstraction. (You can see examples of this by looking at invocations of our Aggregate operator in the above documents.)

I had assumed there was no arbitrary state because of the replay semantics. Let's say bolt A sends tuples to bolt B. B has internal state. A sends tuples t1, t2, t3 and t4. A receives acknowledgements that t1, t3 and t4 were processed. So t2 needs to be replayed. But the semantics of what that means is undefined - B has internal state that already incorporates, for certain, t3 and t4, and maybe t2. (While it's unlikely, you never know where a tuple got lost.) So replaying t2 is problematic - do you just blindly replay it, and allow potentially broken semantics? The alternative is to do rollback, which is quite hairy.

Re: Clojure on Hadoop: A New Hope

#23

Post by earl »

wicknicks wrote: post #7

My experience with Hadoop tells me that its great for all counting tasks. Makes a lot of sense that it was designed at Google to construct posting lists for their search index. Beyond this sweet spot, it gets really tricky to map your solution to a map-reduce task. The programmer has to rely more on the fact that map/reduce are java black boxes to express everything he needs. Hadoop's big victory is the scale it oper…

Obviously the ease of programming any given task in MR corresponds, to invent a word, with how cross-talky the task is. If it is no-communication parallelizable, then it's very easy to do. The more communication or the more data striping you do, the more interesting it becomes.

In any case, tons of people would like a simple sql like language for hadoop. There already exist some examples: pig, sawzall, etc. Unfortunately, the inefficiencies hurt. Say pig takes 2x as much data processing as hand coding java. At large scale, that can eat you alive. To get some intuition about scale, review Ron Bodkin's, former vp eng at quantcast, slides: [1], page 9 and on. Obviously if a 2x penalty means going from 4 to 8 machines in your cluster, it's not such a big deal. But if you buy clusters a datacenter's cage worth at a time or more, its painful. We haven't escaped the tradeoff between programmer time and computer cost.

People would also love something as easy to program as R or matlab that magically scales to large data. Nobody has written such a thing despite quite a lot of demand, which makes me think it's even harder than I thought it as, and I believe it to be a quite hard task.

For the tools: I'm not a qc spokesperson and none of this represents the opinion of my employer and wasn't endorsed by them. If you want qc's position on anything, ask our spokesperson.

[1] http://qconsf.com/dl/qcon-sanfran-2010/slides/RonBodkin_Larg...

Re: Clojure on Hadoop: A New Hope

#24

Post by nathanmarz »

scott_s wrote: post #14
toisanji wrote: post #11

Earlier quoted context omitted.

isn't stream programming what storm is supposed to be?

Yes, it is a similar programming model. Some differences (please put "to the best of my knowledge" in front of all of these): - Storm does not allow arbitrary state in operators (what Nathan Marz calls "bolts"). This makes implementing the runtime easier, such as being able to replay tuple sends for fault tolerance, but it limits what kinds of applications one can make. Yes, I'm on board with the idea that we should…

You can have any state you want in bolts. What Storm does not provide is a persistence mechanism for that state. For that, you can just use an external database that knows how to handle distributed state and the associated tradeoffs (such as Riak or Cassandra).

The "state spout" abstraction, a future feature for Storm, will alleviate the performance problems with using an external database. Although in the time being, smart use of batching/checkpointing is sufficient for most applications.

Also, Storm topologies can be written in any language. Storm has great multi-language support.

I do agree though that higher level abstractions are important. That will come later, once we're confident that we've mastered the primitives for doing fault-tolerant realtime computation.

Re: Clojure on Hadoop: A New Hope

#25

Post by benatkin »

swannodette wrote: post #3
alexatkeplar wrote: post #2

This is an entertaining comparison of 8 different map-reduce languages for Hadoop, including a flame-tastic take on Cascalog/Clojure: http://www.dataspora.com/2011/04/pigs-bees-and-elephants-a-c...

That's not flame-tastic. It's just inaccurate.

He got Nathan Marz' name wrong, too.

Re: Clojure on Hadoop: A New Hope

#26

Post by samstokes »

scott_s wrote: post #22
samstokes wrote: post #21

Earlier quoted context omitted.

Storm does not allow arbitrary state in operators (what Nathan Marz calls "bolts"). Maybe I misunderstand your point, but Storm does allow state in Bolts - a Bolt is just a Java object, so it can have member variables. That's how aggregation (e.g. counting events per user) is done. Of course, if you want a Bolt that scales horizontally, you need to account for the state being split across several instances of the Bol…

Then I stand corrected. Is there any means to declare partitioned state that the runtime then handles for the user, or does the user always have to manage it themselves? This is one of those things that is, I think, easier to do declaratively one level up in abstraction. (You can see examples of this by looking at invocations of our Aggregate operator in the above documents.) I had assumed there was no arbitrary stat…

partitioned state that the runtime then handles for the user

Yes, when "wiring up" a bolt to a spout or another bolt, you can say that it should receive tuples grouped by one or more of the tuple fields. (e.g. tuples [user, num_events] grouped by "user".) Then Storm takes care of the consistent hashing needed to ensure that each instance of the bolt receives all of the tuples in a given group (e.g. instance 1 gets all events for user 42, instance 2 gets all events for user 53).

I had assumed there was no arbitrary state because of the replay semantics.

Storm itself doesn't implement replay; it relies on the ability of an external event source (e.g. a message queue) to replay tuples if needed. It just provides the ability to notify the source of whether replay is needed (by tracking the tuples as they flow through the topology). So whether or not you replay tuples is an application choice.

But yes, if you have stateful bolts and replay then you do need to make sure that processing a given tuple is idempotent.

Re: Clojure on Hadoop: A New Hope

#27

Post by adatta02 »

wicknicks wrote: post #7

My experience with Hadoop tells me that its great for all counting tasks. Makes a lot of sense that it was designed at Google to construct posting lists for their search index. Beyond this sweet spot, it gets really tricky to map your solution to a map-reduce task. The programmer has to rely more on the fact that map/reduce are java black boxes to express everything he needs. Hadoop's big victory is the scale it oper…

Check out Hive - https://cwiki.apache.org/confluence/display/Hive/Home - it uses a "SQL like" syntax to let you run ad-hoc queries across data in a Hadoop+HDFS cluster. We're using it to run reporting nightly on ~10gb of data and are pretty happy with it.

Re: Clojure on Hadoop: A New Hope

#28

Post by bitemyapp »

earl wrote: post #23
wicknicks wrote: post #7

My experience with Hadoop tells me that its great for all counting tasks. Makes a lot of sense that it was designed at Google to construct posting lists for their search index. Beyond this sweet spot, it gets really tricky to map your solution to a map-reduce task. The programmer has to rely more on the fact that map/reduce are java black boxes to express everything he needs. Hadoop's big victory is the scale it oper…

Obviously the ease of programming any given task in MR corresponds, to invent a word, with how cross-talky the task is. If it is no-communication parallelizable, then it's very easy to do. The more communication or the more data striping you do, the more interesting it becomes. In any case, tons of people would like a simple sql like language for hadoop. There already exist some examples: pig, sawzall, etc. Unfortuna…

Not to be snarky but: http://hive.apache.org/

Re: Clojure on Hadoop: A New Hope

#29

Post by moomin »

ekoontz wrote: post #10
alexatkeplar wrote: post #2

This is an entertaining comparison of 8 different map-reduce languages for Hadoop, including a flame-tastic take on Cascalog/Clojure: http://www.dataspora.com/2011/04/pigs-bees-and-elephants-a-c...

"LISP has been around some 50-odd years without taking off despite several attempts at its revival... I suspect something is wrong with it" I laughed out loud at that.

The problem is, I'm not sure that line was actually meant to be funny.
Post reply on HN