Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S4

s4.io

Post reply on HN
Print view View original

21–30 of 62 posts

Re: S4

#21

Post by allertonm »

sabat wrote: post #3

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. I'm sure this is cool and useful technology. At this moment, from the marketing-speak, I have no idea what it does except that it has something to do with volumes of streaming data. Whose data? Is it a service? (Maybe n…

One way of looking at it is that S4 is to map-reduce what Streambase or Coral8 are to SQL.

In the enterprise software world, this is what's called CEP - Complex Event Processing: http://en.wikipedia.org/wiki/Complex_event_processing

Re: S4

#22

Post by scott_s »

sabat wrote: post #3

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. I'm sure this is cool and useful technology. At this moment, from the marketing-speak, I have no idea what it does except that it has something to do with volumes of streaming data. Whose data? Is it a service? (Maybe n…

It's not marketing speak, it's research speak. I have worked on a similar project (and will work on it again in the future), and I know exactly what they mean by those things.

General purpose: in the same way C is a "general purpose" language. It can handle arbitrary problems.

Distributed: designed to be used across multiple compute nodes.

Scalable: they've made the effort to ensure that performances increases as they increase the number of compute nodes.

Partially fault-tolerant: node failure does not mean the results of the computation are lost. "Partially," I assume, implies they can't guarantee this completely.

Continuous unbound streams of data: think sensors that are constantly sending more data. Or a stock market ticker. Or radio telescopes constantly monitoring the sky. Or a medical patient's various monitors.

The reason these terms don't resonate with you is that these type of applications - this type of programming - is something you're not familiar with.

Re: S4

#23

Post by LiveTheDream »

sabat wrote: post #3

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. I'm sure this is cool and useful technology. At this moment, from the marketing-speak, I have no idea what it does except that it has something to do with volumes of streaming data. Whose data? Is it a service? (Maybe n…

> Whose data? Anyone's data that you can stuff into the system: "The drivers to read from and write to the platform can be implemented in any language making it possible to integrate with legacy data sources and systems."

> Is it a service? No, it's a platform. You could turn it into a Platform-as-a-Service, like Amazon does with various technologies. "S4 is a ... platform"

> What could it do for me (in simple terms)? What's a basic use case? My first thought would be real-time trending calculations. You have a massive, never-ending stream of data...how do you extract real-time insights from that?

> Why do people assume that we can mind-read? Perhaps because after being immersed in a project for a long time, it's easy to forget what is obvious to you, but non-obvious to others.

Re: S4

#25

Post by olalonde »

sabat wrote: post #3

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. I'm sure this is cool and useful technology. At this moment, from the marketing-speak, I have no idea what it does except that it has something to do with volumes of streaming data. Whose data? Is it a service? (Maybe n…

My attempt at a more friendly description of S4 (based on my very limited understanding of it - please correct me if I'm wrong):

When processing large amounts of streaming data, you have to process the data as fast as it comes in or else you can't keep up. You probably wonder: "Why not simply delay the processing of new data while the old data is being processed?". The problem lies in the fact we're dealing with a stream which always brings in new data. Eventually, the virtual line up of delayed data will occupy all available memory.

A solution to this is to dispatch the data between multiple computers which can each independently process the data they receive and then send back the result of their processing to a central computer whose job is to put back the results together. Can't keep up with the stream? Simply add a new computer! That's more or less what S3 does.

Re: S4

#26

Post by devmonk »

scott_s wrote: post #22
sabat wrote: post #3

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. I'm sure this is cool and useful technology. At this moment, from the marketing-speak, I have no idea what it does except that it has something to do with volumes of streaming data. Whose data? Is it a service? (Maybe n…

It's not marketing speak, it's research speak. I have worked on a similar project (and will work on it again in the future), and I know exactly what they mean by those things. General purpose: in the same way C is a "general purpose" language. It can handle arbitrary problems. Distributed: designed to be used across multiple compute nodes. Scalable: they've made the effort to ensure that performances increases as the…

No, I think what sabat is saying is that the front page documentation is geared towards the wrong audience and needs to have less general information on usage.

I think if the following actually had examples of use, I'd try it out:

http://wiki.s4.io/Cookbook/Cookbook

Currently it only has:

UsingEclipse

   1. Check out sources from git
   2. Create Eclipse configurations: mvn eclipse:eclipse
   3. Import project into Eclipse
   4. set variable M2_REPO to the local Maven repository: e.g. ~/.m2/repository
   5. Set up formatting:
          * Spaces for indentation
          * Tab width = 4
I completely understand that this isn't there yet, though. It is a new project.

Re: S4

#27

Post by ithkuil »

great hype but still not yet clear what is this good for. Does anybody know about any other use case except the twitter topic count example?

Re: S4

#28

Post by LiveTheDream »

samratjp wrote: post #14
sabat wrote: post #3

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. I'm sure this is cool and useful technology. At this moment, from the marketing-speak, I have no idea what it does except that it has something to do with volumes of streaming data. Whose data? Is it a service? (Maybe n…

"At Yahoo! Labs we design algorithms that are primarily driven by large scale applications for data mining and machine learning in a production environment. We show that the S4 design is surprisingly flexible and lends itself to run in large clusters built with commodity hardware." Via http://labs.yahoo.com/event/99 I could've sworn there was a blurb there about where they are using and I recall using "real time map-…

Lets be clear. S4 is not real-time MapReduce, its a stream processing system.[1]

[1] http://twitter.com/s4project/status/29611855285

Re: S4

#29

Post by scott_s »

olalonde wrote: post #25
sabat wrote: post #3

S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. I'm sure this is cool and useful technology. At this moment, from the marketing-speak, I have no idea what it does except that it has something to do with volumes of streaming data. Whose data? Is it a service? (Maybe n…

My attempt at a more friendly description of S4 (based on my very limited understanding of it - please correct me if I'm wrong): When processing large amounts of streaming data, you have to process the data as fast as it comes in or else you can't keep up. You probably wonder: "Why not simply delay the processing of new data while the old data is being processed?". The problem lies in the fact we're dealing with a st…

Good, except stream processing does not imply that there needs to be a master node - it can truly be distributed. I don't know if S4 is set up like this, though.
Post reply on HN