If I'm not wrong, this is the low latency S3 that is written in Rust. Finally launched after years in the making.
S3 Express Is All You Need
41–50 of 87 posts
Re: S3 Express Is All You Need
#42We tested S3 Express for our search engine quickwit [0] a couple of weeks ago.
While this was really satisfying on the performance side, we were a bit disappointed by the price, and I mostly agree with the article on this matter.
I can see some very specific use cases where the pricing should be OK but currently, I would say most of our users will just stay on the classic S3 and add some local SSD caching if they have a lot of requests.
Re: S3 Express Is All You Need
#43> However, the new storage class does open up an exciting new opportunity for all modern data infrastructure: the ability to tune an individual workload for low latency and higher cost or higher latency and lower cost with the exact same architecture and code. I get it, but at the same time that is also what you lost when you locked yourself in with a particular vendor.
You can use a different vendor any time, it's all S3 compatible. You just don't get the same performance and billing.
Re: S3 Express Is All You Need
#44If I'm not wrong, this is the low latency S3 that is written in Rust. Finally launched after years in the making.
Do you have any sources for that? Very interested to know more about this.
Re: S3 Express Is All You Need
#45some additional context here is that warpstream is building a Kakfa compatible streaming system that uses s3 as the object store. This allows them to leverage cheap zone transfer costs for redundancy + automatic storage tiering to cut down on the costs of running and maintaining these systems. This has previously come at the cost of latency due to s3's read/write speeds but with S3 this makes them more competitive wi…
I am eager to hear how it will affect their latency numbers:
> Engineering is about trade-offs, and we’ve made a significant one with WarpStream: latency. The current implementation has a P99 of ~400ms for Produce requests because we never acknowledge data until it has been durably persisted in S3 and committed to our cloud control plane. In addition, our current P99 latency of data end-to-end from producer-to-consumer is around 1s
via https://www.warpstream.com/blog/kafka-is-dead-long-live-kafk...
Re: S3 Express Is All You Need
#46If I'm not wrong, this is the low latency S3 that is written in Rust. Finally launched after years in the making.
Do you have any sources for that? Very interested to know more about this.
Unfortunately I don't, this is already internal information that I don't know if I should say here. I never worked on S3 and I no longer work at AWS so someone from within would have to weigh in.
Re: S3 Express Is All You Need
#47Earlier quoted context omitted.
EFS is really expensive and has terrible latency with small files in my experience
Do you have any more details you can share about the performance of EFS? I've never met anyone who has actually used it in anger.
Yes, I built a moderately large system on it that used lots of small shared files. The performance was fairly terrible. There's weird little niggles with it--we had random slowdowns, throughput issues, and things just didn't work quite right.
It was an ok solution for what we were doing, but several times I came really close to just dumping it and standing up an NFS server using EBS volumes.
I also used it a couple of times to store webroots and that was a complete disaster with systems that had lots of small files (Drupal I'm looking at you).
Re: S3 Express Is All You Need
#48Has anyone here a usecase which would perform better with this new S3 Express Tier? And a second question, would it be worth the 8x times surcharge?
This will work great with the s3 mount point that AWS recently released. This will outperform EFS if your application does not require full POSIX compatibility.
Re: S3 Express Is All You Need
#49I don't understand why EFS never gets major shout outs - it's way better than S3: systems can mount it as a drive, shared across systems, already has had super low latency... Not sure what s3 express is really useful for if EFS already exists.
EFS exists if you don't care much about spend and performance while having to forklift a POSIX compliant use case into AWS for persistent data.
Thats basically how we were using it. It could have been worse.
Re: S3 Express Is All You Need
#50If I'm not wrong, this is the low latency S3 that is written in Rust. Finally launched after years in the making.
Surely being written in a non-Rust language is not responsible for an extra 40ms of latency, right?
Or is rust really that magic?