tldr: this caches your S3 data in EFS. we run datalakes using DuckLake and this sounds really useful. GCP should follow suit quickly.
I am curious about this use case
How do you see it helping with DuckLake?
S3 Files
41–50 of 128 posts
Re: S3 Files
#42Dumb Q: what would happen if you used this to store a SQLite database? Would it just... work?
My guess is this would only enable a read-replica and not backups as Litestream currently does?
Re: S3 Files
#43Terrible day for people who sloppily use filesystem vocabulary when referring to S3 objects and prefixes.
Re: S3 Files
#44tldr: this caches your S3 data in EFS. we run datalakes using DuckLake and this sounds really useful. GCP should follow suit quickly.
I was thinking of using it with Duckdb as well but seems it would be of limited benefit. Parquet objects are in MBs, so they would be streamed directly from S3. With raw parquet objects, it might help with S3 listing if you have a lot of them (shave off a couple of seconds from the query). If you are already on Ducklake, Duckdb will use that for getting the list of relevant objects anyway.
Re: S3 Files
#45I cannot 100% confirm this, but I believe AWS insisted a lot in NOT using S3 as a file system. Why the change now?
People and by people I mean architects and lead devs at big account orgs ( $$$ ) have been using S3 as a filesystem as one of the backbones of their usually wacky mega complex projects.
So there always been a pressure to AWS make it work like that. I suspect the amount of support tickets AWS receives related to "My S3 backed project is slow/fails sometimes/run into AWS limits (like the max number of buckets per account)" and "Why don't.." questions in the design phase which many times AWS people are in the room, serve as enough of a long applied pressure to overcome technical limitations of S3.
I'm not a fan of this type of "let's put a fresh coat on top of it and pretend it's something that fundamentally is not" abstractions. But I suspect here is a case of social pressure turbo charged by $$$.
Re: S3 Files
#46This is essentially S3FS using EFS (AWS's managed NFS service) as a cache layer for active data and small random accesses. Unfortunately, this also means that it comes with some of EFS's eye-watering pricing: — All writes cost $0.06/GB, since everything is first written to the EFS cache. For write-heavy applications, this could be a dealbreaker. — Reads hitting the cache get billed at $0.03/GB. Large reads (>128kB) g…
> Large reads (>128kB) get directly streamed from the underlying S3 bucket, which is free. Always uncached? S3 has pretty bad latency.
The threshold at which the cache gets used is configurable, with 128kB the default. The assumption is that any read larger than the threshold will be a long sustained read, for which latency doesn't matter too much. My question is, do reads 128kB get saved to the cache, or is it only used for files whose overall size is under the threshold? Frequent random access to large files is a textbook use case for a caching layer like this, but its cost will be substantial in this system.
Re: S3 Files
#47This is very close to its first official release: https://fiberfs.io/ Built in cache, CDN compatible, JSON metadata, concurrency safe and it targets all S3 compatible storage systems.
How would you compare this to Amazon's own FUSE implementation? I think it's on its 3rd major reincarnation now
Re: S3 Files
#48I wish they offered some managed bridging to local NVMe storage. AWS NVMe is super fast compared to EBS, and EBS (node-exclusive access as block device) is faster than EFS (multi-node access). I imagine this can go fast if you put some kind of further-cache-to-NVMe FS on top, but a completely vertically integrated option would be much better.
Since EFS is just an NFS mount, I wonder if you could do this yourself by attaching an NVMe volume to your instance and setting up something like cachefilesd on the NFS mount, pointed to the NVMe.
Would
mkfs.ext4 /dev/nvme0n1 && \
mount /dev/nvme0n1 /var/cache/fscache && \
mount -t s3files -o fsc fs-0aa860d05df9afdfe:/ /home/ec2-user/s3files
Re: S3 Files
#49I cannot 100% confirm this, but I believe AWS insisted a lot in NOT using S3 as a file system. Why the change now?
I think it opens them up to a huge customer base of less technically apt people who just downloaded some random "S3asYourFS.exe" program but also opens them up to needing to support that functionality and field support calls from less technically apt people. I don't know if that business decision makes sense (since AWS already lacks the CS infrastructure to even deal with professional clients) but the idea that you could get everyone and their brother paying monthly fees to AWS is likely too tempting of a fruit to pass up.
Re: S3 Files
#50Earlier quoted context omitted.
> Large reads (>128kB) get directly streamed from the underlying S3 bucket, which is free. Always uncached? S3 has pretty bad latency.
The threshold at which the cache gets used is configurable, with 128kB the default. The assumption is that any read larger than the threshold will be a long sustained read, for which latency doesn't matter too much. My question is, do reads 128kB get saved to the cache, or is it only used for files whose overall size is under the threshold? Frequent random access to large files is a textbook use case for a caching la…
NVMe read latency is in the 10-100µs range for 128kB blocks. S3 is about 100ms. That's 3-4 OOMs.
The threshold where the total read duration starts to dominate latency would be somewhere in the dozens to hundreds of megabytes, not kilobytes.