[flagged]
Indeed... we sure wish we could have nabbed that crate name, but it was not to be. Our Rust SDK is here https://lib.rs/crates/streamstore
Introducing S2
11–20 of 206 posts
Re: Introducing S2
#12In the long-term, how different do you want to be from Apache Pulsar? At the moment, many differences are obvious, e.g., Pulsar offers transactions, queues and durable timers.
(Founder) We want S2 to be focussed on the stream primitive (log if you prefer). There is a lot that can be built on top, which we mostly want to do as open source layers. For example, Kafka compatibility, or queue semantics.
Re: Introducing S2
#13Can someone tell me what does this do? And why its better.
(Founder) There is a table on the landing page https://s2.dev/ which hopefully gives a nice overview :) It's like S3, but for streams. Cheap appends, and instead of dealing with blocks of data and byte ranges, you work with records. S2 takes care of ordering records, and letting you read from anywhere in the stream. This is an alternative to systems like Kafka which don't do great at giving a serverless experience.
Could you clarify the Kafka difference further?
Or more generally, when is it better to choose S2 vs services like SQS or Kinesis?
S2 sounds like an ordered queue to me, but those exist?
Re: Introducing S2
#14Can someone tell me what does this do? And why its better.
[deleted]
Re: Introducing S2
#15[flagged]
[deleted]
Re: Introducing S2
#16These folks knowingly chose to spend the rest of their careers explaining that they are not, in fact, S3.
Re: Introducing S2
#17Earlier quoted context omitted.
(Founder) There is a table on the landing page https://s2.dev/ which hopefully gives a nice overview :) It's like S3, but for streams. Cheap appends, and instead of dealing with blocks of data and byte ranges, you work with records. S2 takes care of ordering records, and letting you read from anywhere in the stream. This is an alternative to systems like Kafka which don't do great at giving a serverless experience.
Could you clarify the Kafka difference further? Or more generally, when is it better to choose S2 vs services like SQS or Kinesis? S2 sounds like an ordered queue to me, but those exist?
(Founder here) Managed cloud offerings for streaming limit ordered throughput pretty low, e.g. Kinesis at 1 MiBps, Redpanda serverless at 1 MiBps, Confluent's even higher-end clusters at 10-20 MiBps IIRC. If you really need ordering, this can indeed be a limit. S2 lets you push 125 MiBps currently, and we may grow that.
Another factor is how many ordered streams you can have. Typically a few thousand at most with those systems. We take the serverless spirit of S3 here, when did you have to worry about the number of objects in a bucket?
We are also able to offer latency comparable to disk-based streaming like Confluent's Kora and Kinesis, with our Express storage class (under 50 milliseconds end-to-end latency for client in the same cloud region) - while being backed by S3 with regional durability! Not a disk in the system.
We want people to be able to build safe distributed data systems on top of S2, so we also allow concurrency control mechanisms on the stream like fencing. Kafka or Kinesis won't let you do that. This is the approach AWS takes internally (https://brooker.co.za/blog/2024/04/25/memorydb.html), but they don't have that as a service. We want to democratize the pattern.
ED: on throughtputs, to clarify, I am talking about ordered throughput, i.e. per Kafka partition or Kinesis shard. WarpStream also does well here because of their architectural approach to separate ordering, but at a latency cost.
Re: Introducing S2
#18These folks knowingly chose to spend the rest of their careers explaining that they are not, in fact, S3.
(Founder) well 50% of our name is different
Re: Introducing S2
#19These folks knowingly chose to spend the rest of their careers explaining that they are not, in fact, S3.
How many of these letter-number storage services are there now? S3, B2, R2, S2...
Re: Introducing S2
#20Help me understand - you build on top of AWS, which charges $0.09/GB for egress to the Internet, yet you're charging $0.05/GB for egress to the Internet? Sounds like you're subsidizing egress from AWS? Or do you have access to non-public egress pricing?