Many S3 implementations appear to simply be transparent downloads to disk rather than a true "use the network as a disk".
S3 Express Is All You Need
51–60 of 87 posts
Re: S3 Express Is All You Need
#52If I'm not wrong, this is the low latency S3 that is written in Rust. Finally launched after years in the making.
Surely being written in a non-Rust language is not responsible for an extra 40ms of latency, right? Or is rust really that magic?
Of course not, it's designed differently from the original S3. AWS came out with this to compete with Azure premium blob storage, which has very good first byte latency, and Azure had it 4 years ago..
https://azure.microsoft.com/en-us/blog/premium-block-blob-st...
Re: S3 Express Is All You Need
#53If I'm not wrong, this is the low latency S3 that is written in Rust. Finally launched after years in the making.
ShardStore? (More info: https://www.thestack.technology/aws-shardstore-s3/ ) it seems that it was deployed years ago.
Re: S3 Express Is All You Need
#54If I'm not wrong, this is the low latency S3 that is written in Rust. Finally launched after years in the making.
ShardStore? (More info: https://www.thestack.technology/aws-shardstore-s3/ ) it seems that it was deployed years ago.
Nah, that looks very different, one of the stated goals of S3 Express is to minimize latency, which is the only thing about the Rust S3 that I remember.
Re: S3 Express Is All You Need
#55> “Of course the AWS S3 Express storage costs are still 8x higher than S3 standard, but that’s a non issue for any modern data storage system. Data can be trivially landed into low latency S3 Express buckets, and then compacted out to S3 Standard buckets asynchronously. Most modern data systems already have a form of compaction anyways, so this “storage tiering” is effectively free.”
This is key insight. The data storage cost essentially becomes negligible and latency goes down by a magnitude by making S3 Express as a buffer storage then moving data to standard S3. I see a future where most data-intensive apps would use S3 as main storage layer.
Re: S3 Express Is All You Need
#56Has anyone here a usecase which would perform better with this new S3 Express Tier? And a second question, would it be worth the 8x times surcharge?
I'm going to set up sccache [0] to use it tomorrow. We use MSVC, so EFS is off the cards.
Re: S3 Express Is All You Need
#57> “Of course the AWS S3 Express storage costs are still 8x higher than S3 standard, but that’s a non issue for any modern data storage system. Data can be trivially landed into low latency S3 Express buckets, and then compacted out to S3 Standard buckets asynchronously. Most modern data systems already have a form of compaction anyways, so this “storage tiering” is effectively free.” This is key insight. The data sto…
sounds a bit like CPU caches and main memory
Re: S3 Express Is All You Need
#58I don't understand why EFS never gets major shout outs - it's way better than S3: systems can mount it as a drive, shared across systems, already has had super low latency... Not sure what s3 express is really useful for if EFS already exists.
i'm sure there's some cases where mounting storage as a disk is desirable, but from my perspective "systems can mount it as a drive" is a negative, not a positive.
treating storage as an application-controlled thing that doesn't need systems management is a good thing. i want "put this file in this spot" logic in my application code, not "put this file in this spot on the filesystem, and hope that location is backed by the correct storage layer"
Re: S3 Express Is All You Need
#59I saw "X is all you Need" with the "Attention is all you need" paper [1], which launched the Transformer upon the world. Is it the first instance of that phrase?
Re: S3 Express Is All You Need
#60I saw "X is all you Need" with the "Attention is all you need" paper [1], which launched the Transformer upon the world. Is it the first instance of that phrase? [1] https://arxiv.org/abs/1706.03762
The Beatles might claim prior art.