191–200 of 206 posts
Earlier quoted context omitted.
Too late, name's taken for something else: https://incubator.apache.org/projects/s4.html
And don't forget the other S4: http://www.supersimplestorageservice.com/ It's like S3, except better because, by focusing on being a write-only data store, they can manage much more throughput and efficiency, plus your data is far more secure at rest than it is in S3.
1. Do you support compression for data stored in segments? 2. Does the choice of storage class only affect chunks or also segments? To me the best solution seem like combining storing writes on EBS (or even NVMe) initially to minimize the time until writes can be acknowledged, and creating a chunk on S3 standard every second or so. But I assume that would require significant engineering effort for applications that r…
1) Storage is priced on uncompressed data. We don't currently compress segments.
2) It only affects chunk storage. We do have a 'Native' chunk store in mind, the sketch involves introducing NVMe disks (as a separate service the core depends on) - so we can offer under 5 millisecond end-to-end tail latencies.
3) The append ack latency and end-to-end latency with a tailing reader is largely equivalent for us since latest writes are in memory for a brief period after acknowledgment. If you try the CLI ping command (see GIF on landing page) from the same cloud region as us (AWS us-east-1 only currently), you'll see end-to-end and append ack latency as basically the same. TTFB for older data is ~ TTFB to get a segment data range from object storage, so it can be a few hundred milliseconds.
4) We have a deadline to free chunks, so we we PUT a tiny segment if we have to.
IANAL,but naming your product S2 and mentioning in the intro that AWS S3 is the tech you are enhancing is probably looking for a branding/copyright claim from Amazon. Same vertical & definitely will cause consumer confusion. I'm sure you've done the research about whether a trademark has been registered. https://tsdr.uspto.gov/#caseNumber=98324800&caseSearchType=U...
Fun fact: S2 and EC2 sound exactly the same in Spanish - both are "ese dos". Add that to EC2 and S3 already being confusing to tell apart by ear
Roughly ten years ago, I started Gazette [0]. Gazette is in an architectural middle-ground between Kafka and WarpStream (and S2). It offers unbounded byte-oriented log streams which are backed by S3, but brokers use local scratch disks for initial replication / durability guarantees and to lower latency for appends and reads (p99 500ms), while guaranteeing all files make it to S3 with niceties like configurable targe…
ED: I appreciate where you are coming from, and understand the challenges ahead. Thank you for the advice.
Earlier quoted context omitted.
And don't forget the other S4: http://www.supersimplestorageservice.com/ It's like S3, except better because, by focusing on being a write-only data store, they can manage much more throughput and efficiency, plus your data is far more secure at rest than it is in S3.
I would really like to know how many people send money through that paypal link
Roughly ten years ago, I started Gazette [0]. Gazette is in an architectural middle-ground between Kafka and WarpStream (and S2). It offers unbounded byte-oriented log streams which are backed by S3, but brokers use local scratch disks for initial replication / durability guarantees and to lower latency for appends and reads (p99 500ms), while guaranteeing all files make it to S3 with niceties like configurable targe…
(S2 Founder) Congrats on the success with Estuary! You are not the first person to tell me there is no/tiny market for this. Clearly _you_ thought there was something to it, when you looked to HN for validation. We may do a lot more on top of S2, like offering Kafka compatibility, but the core primitive matters. I have wanted it. It gets reinvented in all kinds of contexts and reused sub-optimally in the form of syst…
In my opinion, the key is to find a value prop and positioning which lets prospects try your service while spending a minimum of their own risk capital / reputation points within their own org.
That makes it hard to go after core storage, because it's such a widely used, fundamental, and reliable part of most every company's infrastructure. You and I may agree that conventions of incremental files in S3 are a less-than-ideal primitive for representing streams, but plenty of companies are doing it this way just fine and don't feel that it's broken.
WarpStream, on the other hand, leaned in to the perceived complexity of running Kafka and the share of users who wanted a Kafka solution with the operational profile of using S3. Internal champions can sell trying their service because the prospect's existing thing is already understood to be a pain in the butt.
For what it's worth, if I were entering the space anew today I'd be thinking carefully about the Iceberg standard and what I might be able to do with it.
Earlier quoted context omitted.
(S2 Founder) Congrats on the success with Estuary! You are not the first person to tell me there is no/tiny market for this. Clearly _you_ thought there was something to it, when you looked to HN for validation. We may do a lot more on top of S2, like offering Kafka compatibility, but the core primitive matters. I have wanted it. It gets reinvented in all kinds of contexts and reused sub-optimally in the form of syst…
The market is gobsmackingly huge, it's just the go-to-market entry points which are narrow. In my opinion, the key is to find a value prop and positioning which lets prospects try your service while spending a minimum of their own risk capital / reputation points within their own org. That makes it hard to go after core storage, because it's such a widely used, fundamental, and reliable part of most every company's i…
Earlier quoted context omitted.
OR Amazon just builds the same thing, calls it S3 Streams, and doesn’t care about S2. Maybe they make a buyout offer. I highly doubt they would sue.
Trademark law encourages companies to defend their marks. If they don’t, they may lose the trademark. So Amazon has to write these guys a letter if it wants to defend the s3 trademark.
Earlier quoted context omitted.
Hey congrats! Looks like a really cool idea. Looks like you're pushing for the throughput angle - that could be important but IMO it's not often you come across devs who need this level of throughput without dealing with large scale problem. My feedback is the lack of per-tenant encryption is a big deal breaker here since you're mixing up data of tenants within one objects. Plus your security section talks very littl…
(Founder) It's a number of dimensions. I get excited about the ordered throughput angle because I have personally cared about this in the past, and yeah a lot of folks may not need that :) Simple API, reasonable pricing, latency flexibility, unlimited streams, _and_ elastic to high throughputs. All adding up to a great serverless experience. Re: the data colocation. This is how most multi-tenant systems - including S…