21–30 of 87 posts
Has anyone here a usecase which would perform better with this new S3 Express Tier? And a second question, would it be worth the 8x times surcharge?
A cache with large blobs (images, etc)
Earlier quoted context omitted.
A cache with large blobs (images, etc)
If it's only a cache it should be on EBS, which is still way faster and 2x less expensive. I started a migration to s3 for such a project (container image caching) but then stopped when I realized what I was doing.
2. Variable throughput may be a concern.
3. You may have availability concerns.
> However, the new storage class does open up an exciting new opportunity for all modern data infrastructure: the ability to tune an individual workload for low latency and higher cost or higher latency and lower cost with the exact same architecture and code. I get it, but at the same time that is also what you lost when you locked yourself in with a particular vendor.
> I get it, but at the same time that is also what you lost when you locked yourself in with a particular vendor. What are other viable practical alternative solution(s)?
If you have a big use case and you really understand your needs, it's very doable.
Most production storage systems/databases built on top of S3 spend a significant amount of effort building an SSD/memory caching tier to make them performant enough for production (e.g. on top of RocksDB). But it's not easy to keep it in sync with blob... Even with the cache, the cold query latency lower-bound to S3 is subject to ~50ms roundtrips [0]. To build a performant system, you have to tightly control roundtri…
NVMe is what is changing the equation, not SSD. NVMe disks now have up to 8 GB/s, although the crap in the cloud providers barely goes to 2 GB/s - and only for expensive instances. So, instead of 40X better throughput than S3, we can get like 10X. Right now, these workloads are much better on-premises on the cheapest m.2 NVMe disks ($200 for 4TB with 4 GB/s read/write) backed by a S3 object store like Scality.
Earlier quoted context omitted.
A cache with large blobs (images, etc)
If it's only a cache it should be on EBS, which is still way faster and 2x less expensive. I started a migration to s3 for such a project (container image caching) but then stopped when I realized what I was doing.
[1] Yes, I am aware of multi-attach but this introduces a scaling bottleneck and requires a fairly exotic setup.
I don't understand why EFS never gets major shout outs - it's way better than S3: systems can mount it as a drive, shared across systems, already has had super low latency... Not sure what s3 express is really useful for if EFS already exists.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/mountp...
Earlier quoted context omitted.
Storage adapter to talk S3 compatible to target, assuming you're not relying on vendor specific extensions or behavior (ie this). Off the top of my head, Backblaze B2, Cloudflare R2, etc are S3 compatible, and Minio locally. https://www.google.com/search?q=s3+compatible
There are no vendor specific extensions or behavior here, are there? Isn't it just a different billing structure?
Earlier quoted context omitted.
Storage adapter to talk S3 compatible to target, assuming you're not relying on vendor specific extensions or behavior (ie this). Off the top of my head, Backblaze B2, Cloudflare R2, etc are S3 compatible, and Minio locally. https://www.google.com/search?q=s3+compatible
There are no vendor specific extensions or behavior here, are there? Isn't it just a different billing structure?
IMO warpstream is a really cool product and this new S3 offering makes them even better
I don't understand why EFS never gets major shout outs - it's way better than S3: systems can mount it as a drive, shared across systems, already has had super low latency... Not sure what s3 express is really useful for if EFS already exists.
For example a container or EC2 instance might only need a tiny bit of your storage and with s3 can just download what it needs when it needs it.
As opposed to EFS where the container or instance needs to load in the entire datastore on startup which can add minutes to startup time if the EFS drive is large.