Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S4

s4.io

Post reply on HN
Print view View original

11–20 of 62 posts

Re: S4

#11

Post by barrkel »

It sounds like a slightly more structured and distributed Unix shell pipeline; but from looking at the twitter example, a lot more awkward to use, owing to being structured around Java.

I imagine a composition language (DSL) wrapped around it could improve its usability - especially ad-hoc experimentation - greatly; at least one better than Spring IoC xml.

Re: S4

#12

Post by sabat »

olalonde wrote: post #9
sabat wrote: post #3

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. I'm sure this is cool and useful technology. At this moment, from the marketing-speak, I have no idea what it does except that it has something to do with volumes of streaming data. Whose data? Is it a service? (Maybe n…

Up voted the story just so more people read your comment. This kind of "marketing-speak" focused on technical details is well too widespread. The project may be technical by nature but there's got to be a higher level way of describing it.

It's frustrating, isn't it? S4 sounds like it does something cool and useful, and I'd like to know what this is, just for reference at least. But I don't have time to go and figure it out on my own; I'm too busy trying to stay productive.

Re: S4

#14

Post by samratjp »

sabat wrote: post #3

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. I'm sure this is cool and useful technology. At this moment, from the marketing-speak, I have no idea what it does except that it has something to do with volumes of streaming data. Whose data? Is it a service? (Maybe n…

"At Yahoo! Labs we design algorithms that are primarily driven by large scale applications for data mining and machine learning in a production environment. We show that the S4 design is surprisingly flexible and lends itself to run in large clusters built with commodity hardware."

Via http://labs.yahoo.com/event/99

I could've sworn there was a blurb there about where they are using and I recall using "real time map-reduce jobs" for things live bidding on ADs and such; another use case would be with stock market data.

You're right that it's marketing speak, this project has gotten too much attention on HN in the last few days even when all the git repo had was initial commit. It's too bad they don't have proper explanation, maybe it's because they weren't probably expecting all this attention yet.

Re: S4

#16

Post by th0ma5 »

olalonde wrote: post #9
sabat wrote: post #3

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. I'm sure this is cool and useful technology. At this moment, from the marketing-speak, I have no idea what it does except that it has something to do with volumes of streaming data. Whose data? Is it a service? (Maybe n…

Up voted the story just so more people read your comment. This kind of "marketing-speak" focused on technical details is well too widespread. The project may be technical by nature but there's got to be a higher level way of describing it.

could just be we're not the audience yet, they see this as something they want to break out within businesses, so they have to communicate it to business types that want to know practically and conceptually how this new thing is better than what they have, but don't care about the actual code.

Re: S4

#17

Post by jamii »

chr15 wrote: post #6

The Github repo has an example application: https://github.com/s4/examples/tree/master/twittertopiccount... It's a twitter topic counter: "This application detects popular hashtags on Twitter by listening to the Twitter gardenhose." From http://labs.yahoo.com/event/99 : "S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications…

The model is very similar to the one argued for in this paper:

http://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf

That paper was a big inspiration when we were redesigning the betting exchange at smarkets. It's a very well reasoned exposition of why this is the only sensible architecture for large scale distributed systems.

Re: S4

#18

Post by protomyth »

chr15 wrote: post #6

The Github repo has an example application: https://github.com/s4/examples/tree/master/twittertopiccount... It's a twitter topic counter: "This application detects popular hashtags on Twitter by listening to the Twitter gardenhose." From http://labs.yahoo.com/event/99 : "S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications…

I always thought it would very interesting to try an build a language that had an Actor focus but used simple objects for basic stuff (a pure model seems a little odd - I guess C++ for actors). It just seems like a natural way to organize a big system. We talk about certain objects as doing things to other objects and it would provide a simpler concurrency.

Actors, linda, tuples..... hum....

Re: S4

#19

Post by jamii »

sabat wrote: post #3

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. I'm sure this is cool and useful technology. At this moment, from the marketing-speak, I have no idea what it does except that it has something to do with volumes of streaming data. Whose data? Is it a service? (Maybe n…

As I understand it the processing model is similar to Yahoo pipes or good old unix pipes, except with easy support for parallel processing, distribution and fault tolerance.

Re: S4

#20

Post by igrekel »

sabat wrote: post #3

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. I'm sure this is cool and useful technology. At this moment, from the marketing-speak, I have no idea what it does except that it has something to do with volumes of streaming data. Whose data? Is it a service? (Maybe n…

Just looked a little at the documentation. It seems like its an engine for stream processing. Think of it of data mining up front. You figure out what information you want and collect it as the data comes in instead of storing all the data and mining for what you want from the accumulated pile.
Post reply on HN