Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing S2

s2.dev

Post reply on HN
Print view View original

91–100 of 206 posts

Re: Introducing S2

#91

Post by shikhar »

bawolff wrote: post #84

In terms of a pitch, i'm not sure i understand how this differs from existing solutions. Is the core value proposition a simpler api?

(Founder) Besides simple API,

- Unlimited streams. Current cloud systems limit to a few thousand. With dedicated clusters, few hundred K? If you want a stream per user, you are now dealing with multiple clusters.

- Elastic throughput per stream (i.e. a partition in Kafka) to 125 MiBps append / 500 MiBps realtime read / unlimited in aggregate for catching up. Current systems will have you at tens. And we may grow that limit yet. We are able to live migrate streams in milliseconds while keeping pipelined writes flowing, which gives us a lot of flexibility.

- Concurrency control mechanisms (https://s2.dev/docs/stream#concurrency-control)

Re: Introducing S2

#92

Post by philjohn »

sensodine wrote: post #66
Scaevolus wrote: post #45

This is a very useful service model, but I'm confused about the value proposition given how every write is persisted to S3 before being acknowledged. I suppose the writers could batch a group of records before writing them out as a larger blob, with background processes performing compaction, but it's still an object-backed streaming service, right? AWS has shown their willingness to implement mostly-protocol compati…

(S2 team member here) > I suppose the writers could batch a group of records before writing them out as a larger blob, with background processes performing compaction, but it's still an object-backed streaming service, right? This is how it works essentially, yes. Architecting the system so that chunks that are written to object storage (before we acknowledge a write) are multi-tenant, and contain records from differ…

Wait, data from multiple tenants is stored in the same place. Do you have per-tenant encryption key, or how else are you ensuring no bugs allow tenants to read others data?

Re: Introducing S2

#94

Post by shikhar »

philjohn wrote: post #92
sensodine wrote: post #66

Earlier quoted context omitted.

(S2 team member here) > I suppose the writers could batch a group of records before writing them out as a larger blob, with background processes performing compaction, but it's still an object-backed streaming service, right? This is how it works essentially, yes. Architecting the system so that chunks that are written to object storage (before we acknowledge a write) are multi-tenant, and contain records from differ…

Wait, data from multiple tenants is stored in the same place. Do you have per-tenant encryption key, or how else are you ensuring no bugs allow tenants to read others data?

(Founder) We will be using authenticated encryption with per-basin (our term for bucket) or per-stream keys, but we don't have this yet. This is noted on https://s2.dev/docs/security#encryption

Re: Introducing S2

#95

Post by shikhar »

joenot443 wrote: post #64
shikhar wrote: post #18

Earlier quoted context omitted.

(Founder) well 50% of our name is different

I like it. I see it as ostensibly a product for engineers and so when I see a name like S2 it's immediately clear that it's a product led and conceived by engineers. I also see that on your pricing page - "We are building the S3 experience for streaming data, and that includes pricing transparency" Love the simple and earnest copy. One can imagine what an LLM would cook up instead, I find the brevity way preferable.

(Founder) Thank you for the kind comment!

Yes we are not trying to confuse S2 with S3, we just think S3 is the best damn serverless experience out there, and we aspire to that greatness. We borrowed the structure of that name to reflect that aspiration, as have other services inspired by S3 like Cloudflare's object store R2.

Re: Introducing S2

#97

Post by shikhar »

shikhar wrote: post #91
bawolff wrote: post #84

In terms of a pitch, i'm not sure i understand how this differs from existing solutions. Is the core value proposition a simpler api?

(Founder) Besides simple API, - Unlimited streams. Current cloud systems limit to a few thousand. With dedicated clusters, few hundred K? If you want a stream per user, you are now dealing with multiple clusters. - Elastic throughput per stream (i.e. a partition in Kafka) to 125 MiBps append / 500 MiBps realtime read / unlimited in aggregate for catching up. Current systems will have you at tens. And we may grow that…

Forgot to mention storage classes to tune your latency vs cost tradeoff. That you can even reconfigure - soon we will make that a live migration.

Re: Introducing S2

#100

Post by shikhar »

johnrob wrote: post #37

This is a very interesting abstraction (and service). I can’t help but feature creep and ask for something like Athena, which runs PrestoDB (map reduce) over S3 files. It could be superior in theory because anyone using that pattern must shoehorn their data stream (almost everything is really a stream) into an S3 file system. Fragmentation and file packing become requirements that degrade transactional qualities.

(Founder) There are definitely some interesting possibilities. Pretty hyped about S3 Table (Iceberg) buckets. S2 stream to buffer small writes so you can flush decent size Parquet into the table, and avoid compaction costs.
Post reply on HN