My first reaction is that this sounds similar to SQL Server Stream Insight (in terms of processing continuous streams of data)
S4
41–50 of 62 posts
Re: S4
#42S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. I'm sure this is cool and useful technology. At this moment, from the marketing-speak, I have no idea what it does except that it has something to do with volumes of streaming data. Whose data? Is it a service? (Maybe n…
As I understand it the processing model is similar to Yahoo pipes or good old unix pipes, except with easy support for parallel processing, distribution and fault tolerance.
So... It's like Unix pipes with GNU Parallel?
Re: S4
#43Does anyone have any real life examples of what this could be used for? I get what it does, just not quite sure where it fits in.
For example, do I push data into S4, does S4 poll for data. Is this like a distributed task system, where I distribute my tasks evenly across multiple servers seamlessly?
Re: S4
#44The Github repo has an example application: https://github.com/s4/examples/tree/master/twittertopiccount... It's a twitter topic counter: "This application detects popular hashtags on Twitter by listening to the Twitter gardenhose." From http://labs.yahoo.com/event/99 : "S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications…
The model is very similar to the one argued for in this paper: http://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf That paper was a big inspiration when we were redesigning the betting exchange at smarkets. It's a very well reasoned exposition of why this is the only sensible architecture for large scale distributed systems.
Does your version have "full" fault tolerance or only the "partial" fault tolerance noted on the s4 site.
Re: S4
#45Earlier quoted context omitted.
The model is very similar to the one argued for in this paper: http://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf That paper was a big inspiration when we were redesigning the betting exchange at smarkets. It's a very well reasoned exposition of why this is the only sensible architecture for large scale distributed systems.
Does your version have "full" fault tolerance or only the "partial" fault tolerance noted on the s4 site.
I guess it depends how you define 'full' fault tolerance. In the case of a machine failure it would lose a couple of seconds worth of transactions. In a data-processing scenario thats not an issue - just re-run the data that hasn't been processed yet. In an exchange the recovery delay is a bit more painful and there is always the potential to permanently lose transactions. Unfortunately there doesn't seem to be a way around that - we must have consistency and performance so machine failure is always going to cause some interruption to availability
Re: S4
#46website sure looks a lot like http://mootools.net
Re: S4
#47S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. I'm sure this is cool and useful technology. At this moment, from the marketing-speak, I have no idea what it does except that it has something to do with volumes of streaming data. Whose data? Is it a service? (Maybe n…
One way of looking at it is that S4 is to map-reduce what Streambase or Coral8 are to SQL. In the enterprise software world, this is what's called CEP - Complex Event Processing: http://en.wikipedia.org/wiki/Complex_event_processing
Yeah, I had gotten the impression it was distributed CEP.
Re: S4
#48Earlier quoted context omitted.
It's not marketing speak, it's research speak. I have worked on a similar project (and will work on it again in the future), and I know exactly what they mean by those things. General purpose: in the same way C is a "general purpose" language. It can handle arbitrary problems. Distributed: designed to be used across multiple compute nodes. Scalable: they've made the effort to ensure that performances increases as the…
No, I think what sabat is saying is that the front page documentation is geared towards the wrong audience and needs to have less general information on usage. I think if the following actually had examples of use, I'd try it out: http://wiki.s4.io/Cookbook/Cookbook Currently it only has: UsingEclipse 1. Check out sources from git 2. Create Eclipse configurations: mvn eclipse:eclipse 3. Import project into Eclipse 4.…
Please start at http://wiki.s4.io/Tutorials/GettingStarted
Re: S4
#49S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. I'm sure this is cool and useful technology. At this moment, from the marketing-speak, I have no idea what it does except that it has something to do with volumes of streaming data. Whose data? Is it a service? (Maybe n…
If this were relevant to you, you'd know what most of that means, particularly stream processing. Why exactly is yahoo faulted for your lack of understanding of the technical vocabulary? I'm mostly annoyed that you think that using the usual vocabulary to discuss a problem means that people have to mind-read.
Re: S4
#50Earlier quoted context omitted.
Thank you for this. Yahoo! should copy and paste your comment onto the S4 homepage.
it's opensource, anyone could do that, not just Yahoo!
Are you confusing open source with a wiki editable by everyone?
Open source just means that you can fork it and do whatever you want with your own version, e.g. fix it. The original owner's don't have to accept your fix.