Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yahoo Open Sources S4 Real-Time MapReduce Framework

github.com

Post reply on HN
Print view View original

11–16 of 16 posts

Re: Yahoo Open Sources S4 Real-Time MapReduce Framework

#11

Post by swah »

beagle3 wrote: post #10
swah wrote: post #8

It's kind of funny how almost no one is advised to write code in Java these days, and on the other hand really cool infrastructure projects like this or Cassandra are almost always written in Java. I understand they have to perform better than the average webapp, but still...funny.

My recent experience with Java code is that efficiency conscious Java code is ridiculously under-performing compared to efficiency conscious C code. By virtue of Java's structure and culture (as codified in its libraries), you have to spend approximately twice as much physical memory as you would in C. E.g., I had a 100,000,000 record array that I needed to sort. That's great -- Java has an Array.Utils.sort() method;…

I was actually thinking of Ruby/Python folks, perhaps there is a group that will skip Java/C/C++ entirely.

Against C there is always the "Java makes it more difficult for you to mess things up" argument...

Re: Yahoo Open Sources S4 Real-Time MapReduce Framework

#12

Post by atarashi »

rektide wrote: post #4

the entire point was that it's not mapreduce, it's a stream processing system. try http://wiki.s4.io/Manual/S4Overview#toc1 for a " what is S4 " intro. The architecture resembles the Actors model, providing semantics of encapsulation and location transparency, thus allowing applications to be massively concurrent while exposing a simple programming interface to application developers. and S4 is a general-purpose, dis…

Indeed, it's not really MapReduce (as in Hadoop or Google's MapReduce), but Yahoo Labs did call it "Real-Time MapReduce."

http://labs.yahoo.com/event/99

Re: Yahoo Open Sources S4 Real-Time MapReduce Framework

#13

Post by schumihan »

DanielRibeiro wrote: post #6
rektide wrote: post #4

the entire point was that it's not mapreduce, it's a stream processing system. try http://wiki.s4.io/Manual/S4Overview#toc1 for a " what is S4 " intro. The architecture resembles the Actors model, providing semantics of encapsulation and location transparency, thus allowing applications to be massively concurrent while exposing a simple programming interface to application developers. and S4 is a general-purpose, dis…

Seems very much like Percolator from google: http://www.infoq.com/news/2010/10/google-percolator

I don't think so. Percolator does not support stream operators, and it supports transaction semantic. Streaming process engine like s4 will never handle the race conditions brought by multiple writers.

Re: Yahoo Open Sources S4 Real-Time MapReduce Framework

#14

Post by schumihan »

beagle3 wrote: post #10
swah wrote: post #8

It's kind of funny how almost no one is advised to write code in Java these days, and on the other hand really cool infrastructure projects like this or Cassandra are almost always written in Java. I understand they have to perform better than the average webapp, but still...funny.

My recent experience with Java code is that efficiency conscious Java code is ridiculously under-performing compared to efficiency conscious C code. By virtue of Java's structure and culture (as codified in its libraries), you have to spend approximately twice as much physical memory as you would in C. E.g., I had a 100,000,000 record array that I needed to sort. That's great -- Java has an Array.Utils.sort() method;…

You can work with much higher productivity in java...

Re: Yahoo Open Sources S4 Real-Time MapReduce Framework

#15

Post by beagle3 »

schumihan wrote: post #14
beagle3 wrote: post #10

Earlier quoted context omitted.

My recent experience with Java code is that efficiency conscious Java code is ridiculously under-performing compared to efficiency conscious C code. By virtue of Java's structure and culture (as codified in its libraries), you have to spend approximately twice as much physical memory as you would in C. E.g., I had a 100,000,000 record array that I needed to sort. That's great -- Java has an Array.Utils.sort() method;…

You can work with much higher productivity in java...

That's perhaps true if that's the language you gained most of your experience in.

I don't think there's a measure of "absolute productivity", but if there is, Python will probably take world cup in a variety of settings.

Java is an excellent language when you need to use Java. If that's not a constraint, then it isn't such a good choice.

Re: Yahoo Open Sources S4 Real-Time MapReduce Framework

#16

Post by beagle3 »

swah wrote: post #11
beagle3 wrote: post #10

Earlier quoted context omitted.

My recent experience with Java code is that efficiency conscious Java code is ridiculously under-performing compared to efficiency conscious C code. By virtue of Java's structure and culture (as codified in its libraries), you have to spend approximately twice as much physical memory as you would in C. E.g., I had a 100,000,000 record array that I needed to sort. That's great -- Java has an Array.Utils.sort() method;…

I was actually thinking of Ruby/Python folks, perhaps there is a group that will skip Java/C/C++ entirely. Against C there is always the "Java makes it more difficult for you to mess things up" argument...

Yep. It's much harder to mess things up when you use regular expression.

Between C, Python and Jython, I don't think there's a single reason left to use Java.

Post reply on HN