Apparently this is “S2, a new S3 competitor” not “S2, the spatial index system based on heirarchical qaudrilaterals”.
Introducing S2
181–190 of 206 posts
Re: Introducing S2
#182Earlier quoted context omitted.
Too late, name's taken for something else: https://incubator.apache.org/projects/s4.html
And don't forget the other S4: http://www.supersimplestorageservice.com/ It's like S3, except better because, by focusing on being a write-only data store, they can manage much more throughput and efficiency, plus your data is far more secure at rest than it is in S3.
How about the other other S4 https://adv-r.hadley.nz/s4.html
Re: Introducing S2
#183IANAL,but naming your product S2 and mentioning in the intro that AWS S3 is the tech you are enhancing is probably looking for a branding/copyright claim from Amazon. Same vertical & definitely will cause consumer confusion. I'm sure you've done the research about whether a trademark has been registered. https://tsdr.uspto.gov/#caseNumber=98324800&caseSearchType=U...
Yep, letter S and a number is copyrighted, can't do that
S3. But trademark law prevents subtle variations.
E.g. creating a product called “Gooogle”
Re: Introducing S2
#184Earlier quoted context omitted.
People keep making the same argument against Aptible ( https://aptible.com ) and it is still a very successful PaaS over a decade later.
I had never heard of this company so I took a look and the main pitch was compelling and then I went to the pricing page and saw the pricing goes from $0 to $500 a month once you want to go to “production”. i’m clearly not the target market, which makes sense why I’ve never heard it.
It’s popular for security sensitive (e.g. healthcare) stuff
Re: Introducing S2
#185Earlier quoted context omitted.
If you ever figure it out, LMK. I don't think I've ever looked at logs more than about 24 hours old. Persistence and durability is not something I care about. Errors, OTOH, I need a week or two of. But I consider these 2 different things. Logs are kind of a last resort when you really can't figure out what's going on in prod.
Here "log" means "append-only stream of small records". This isn't just about traditional logs (including http request logs and error logs). You could use it to store events for an event-sourced application, and even as the Write-Ahead-Log (WAL) for a database. A distributed, but still consistent and durable log is a great building block for higher level abstractions.
That makes more sense. I suppose an audit log would also fit. I guess append-only backups wouldn't fit the "small" requirement though.
Re: Introducing S2
#186Roughly ten years ago, I started Gazette [0]. Gazette is in an architectural middle-ground between Kafka and WarpStream (and S2). It offers unbounded byte-oriented log streams which are backed by S3, but brokers use local scratch disks for initial replication / durability guarantees and to lower latency for appends and reads (p99 500ms), while guaranteeing all files make it to S3 with niceties like configurable target sizes / compression / latency bounds. Clients doing historical reads pull content directly from S3, and then switch to live tailing of very recent appends.
Gazette started as an internal tool in my previous startup (AdTech related). When forming our current business, we very briefly considered offering it as a raw service [1] before moving on to a holistic data movement platform that uses Gazette as an internal detail [2].
My feedback is: the market positioning for a service like this is extremely narrow. You basically have to make it API compatible with a thing that your target customer is already using so that trying it is zero friction (WarpStream nailed this), or you have to move further up to the application stack and more-directly address the problems your target customers are trying to solve (as we have). Good luck!
[0]: https://gazette.readthedocs.io/en/latest/ [1]: https://news.ycombinator.com/item?id=21464300 [2]: https://estuary.dev
Re: Introducing S2
#187Earlier quoted context omitted.
Here "log" means "append-only stream of small records". This isn't just about traditional logs (including http request logs and error logs). You could use it to store events for an event-sourced application, and even as the Write-Ahead-Log (WAL) for a database. A distributed, but still consistent and durable log is a great building block for higher level abstractions.
That makes more sense. I suppose an audit log would also fit. I guess append-only backups wouldn't fit the "small" requirement though.
"Small" means 1MiB per record here. But a higher level abstraction could split one logical operation into multiple records. Just like FoundationDB has severe limits on its transaction size, while higher level databases built on top of it work around that limit.
The OP's blog post linked to this article, which explains some scenarios where this storage primitive would be helpful: https://engineering.linkedin.com/distributed-systems/log-wha...
This product offers two advantages over S3: 1) Appending a small amount of data is cheap 2) Writes are forced into a consistent order (so you don't need to implement Paxos or RAFT yourself). Neither of these are useful for backups. Raw S3 already works well for that usage-case, especially now that Amazon added support for pre-conditions.
Re: Introducing S2
#1881. Do you support compression for data stored in segments? 2. Does the choice of storage class only affect chunks or also segments? To me the best solution seem like combining storing writes on EBS (or even NVMe) initially to minimize the time until writes can be acknowledged, and creating a chunk on S3 standard every second or so. But I assume that would require significant engineering effort for applications that r…
> To me the best solution seem like combining storing writes on EBS (or even NVMe) initially to minimize the time until writes can be acknowledged, and creating a chunk on S3 standard every second or so.
Yep, this is approximately Gazette's architecture (https://github.com/gazette/core). It buys the latency profile of flash storage, with the unbounded storage and durability of S3.
An addendum is there's no need to flush to S3 quite that frequently, if readers instead tail ACK'd content from local disk. Another neat thing you can do is hand bulk historical readers pre-signed URLs to files in cloud storage, so those bytes don't need to proxy through brokers.
Re: Introducing S2
#189IANAL,but naming your product S2 and mentioning in the intro that AWS S3 is the tech you are enhancing is probably looking for a branding/copyright claim from Amazon. Same vertical & definitely will cause consumer confusion. I'm sure you've done the research about whether a trademark has been registered. https://tsdr.uspto.gov/#caseNumber=98324800&caseSearchType=U...
OR
Amazon just builds the same thing, calls it S3 Streams, and doesn’t care about S2.
Maybe they make a buyout offer.
I highly doubt they would sue.
Re: Introducing S2
#190IANAL,but naming your product S2 and mentioning in the intro that AWS S3 is the tech you are enhancing is probably looking for a branding/copyright claim from Amazon. Same vertical & definitely will cause consumer confusion. I'm sure you've done the research about whether a trademark has been registered. https://tsdr.uspto.gov/#caseNumber=98324800&caseSearchType=U...
OR Amazon just builds the same thing, calls it S3 Streams, and doesn’t care about S2. Maybe they make a buyout offer. I highly doubt they would sue.
Trademark law encourages companies to defend their marks. If they don’t, they may lose the trademark. So Amazon has to write these guys a letter if it wants to defend the s3 trademark.