Will this improve running sqlite on s3?
S3 Express Is All You Need
61–70 of 87 posts
Re: S3 Express Is All You Need
#62I saw "X is all you Need" with the "Attention is all you need" paper [1], which launched the Transformer upon the world. Is it the first instance of that phrase? [1] https://arxiv.org/abs/1706.03762
The Beatles might claim prior art.
Yes, I just realised and started hearing "all you need is love".
Re: S3 Express Is All You Need
#63If I'm not wrong, this is the low latency S3 that is written in Rust. Finally launched after years in the making.
S3 Standard has slow first byte latency for three reasons:
1. All data is stored on old school spinning HDDs with multi millisecond seek times
2. There's still Java (and garbage collection) on the request path. There has been a multi-year effort to move the request path entirely to Rust to eliminate GC but Java still remains.
3. To reduce storage costs, objects are erasure-coded "wide", which means many hosts are involved in servicing a request. This means only one such sub-request has to be slow to slow your request down.
The new storage class is SSD-backed, presumably doesn't use Java anywhere, and doesn't stripe your data across as many hosts. It's more expensive because SSDs are more expensive than HDDs and narrow erasure codes are more costly than wide erasure codes.
Source: Used to work on S3.
Re: S3 Express Is All You Need
#64“All You Need Considered Harmful” - most cliche title?
Re: S3 Express Is All You Need
#65Most production storage systems/databases built on top of S3 spend a significant amount of effort building an SSD/memory caching tier to make them performant enough for production (e.g. on top of RocksDB). But it's not easy to keep it in sync with blob... Even with the cache, the cold query latency lower-bound to S3 is subject to ~50ms roundtrips [0]. To build a performant system, you have to tightly control roundtri…
Very excited about being able to build scalable vector databases on DiskANN like turbopuffer or lancedb. These changes in latency are game changing. The best server is no server. The capability a low latency vector database application that runs in lambda and S3 and is dirt cheap is pretty amazing.
Re: S3 Express Is All You Need
#66Earlier quoted context omitted.
There's not much to the S3 API, and data import/export even at massive scale is available with Snowball. Sure, there's many other AWS services that aren't available at other vendors, but blob storage is commodified at this point.
Exporting data from S3 is ludicrously expensive, even with Snowball it’s $30/TB just for network egress.
If I can get all of my data out of S3 for $30, then whatever the cost is for an enterprise with 1000TB, it must be within reason.
Re: S3 Express Is All You Need
#67Earlier quoted context omitted.
In my experience the biggest drawback with EFS is startup time for systems that mount it in. For example a container or EC2 instance might only need a tiny bit of your storage and with s3 can just download what it needs when it needs it. As opposed to EFS where the container or instance needs to load in the entire datastore on startup which can add minutes to startup time if the EFS drive is large.
My understanding is that EFS is exposed as an NFS share. I haven't used it personally, but NFS mounting is generally fast, nearly instant. What does "load in the entire datastore" mean?
Many servers start up, load a ton of data from storage into RAM, and then happily serve that data for a long time. The latency of the server when starting up before it can service its first request is entirely based on the throughput of the data load.
Often these servers will load 128+GB of data into RAM (crazy, huh?) and even if you have 1GB/sec it's still two minutes for the server to start up.
Re: S3 Express Is All You Need
#68I solved this problem locally. When uploading a file to the server before going to S3 it is cached in redis. Whenever the codebase needs to use the file, it checks redis, and if it is not there it fetches it and caches it again.
So you store the actual image data in redis? That's interesting, no issues with storing binary data?
I ask because I've always been taught to not store files in a database. This use case sounds like an interesting exception.
Re: S3 Express Is All You Need
#69We tested S3 Express for our search engine quickwit [0] a couple of weeks ago. While this was really satisfying on the performance side, we were a bit disappointed by the price, and I mostly agree with the article on this matter. I can see some very specific use cases where the pricing should be OK but currently, I would say most of our users will just stay on the classic S3 and add some local SSD caching if they hav…
I'd be fascinated if you could share your insights from using this. Where does the pricing fall down? And is the latency/throughput a big improvement for this use case? (ie. externalizing a search index).
Re: S3 Express Is All You Need
#70> “Of course the AWS S3 Express storage costs are still 8x higher than S3 standard, but that’s a non issue for any modern data storage system. Data can be trivially landed into low latency S3 Express buckets, and then compacted out to S3 Standard buckets asynchronously. Most modern data systems already have a form of compaction anyways, so this “storage tiering” is effectively free.” This is key insight. The data sto…
sounds a bit like CPU caches and main memory
Or like SSD’s vs spinning disks…