Hugging Face Buckets also recently added support for mounting Buckets as a filesystem: https://huggingface.co/changelog/hf-mount
S3 Files
51–60 of 128 posts
Re: S3 Files
#52Earlier quoted context omitted.
The threshold at which the cache gets used is configurable, with 128kB the default. The assumption is that any read larger than the threshold will be a long sustained read, for which latency doesn't matter too much. My question is, do reads 128kB get saved to the cache, or is it only used for files whose overall size is under the threshold? Frequent random access to large files is a textbook use case for a caching la…
NVMe read latency is in the 10-100µs range for 128kB blocks. S3 is about 100ms. That's 3-4 OOMs. The threshold where the total read duration starts to dominate latency would be somewhere in the dozens to hundreds of megabytes, not kilobytes.
I agree, it's an oddly low threshold. The latency differential of NFS vs. S3 is a couple OOMs, so a threshold of ~10MB seems more appropriate to me. Perhaps it's set intentionally low to avoid racking up immense EFS bills? Setting it higher would effectively mean getting billed $0.03/GB for a huge fraction of reads, which is untenable for most people's applications.
Re: S3 Files
#53[dead]
Re: S3 Files
#54Zero mention of s3fs which already did this for decades.
Yeah, that blog post was written as if sliced bread has been invented again.
Reading through it, I was only thinking "is this distinguished engineer TOC 2M aware that people have been doing this since forever?".
Re: S3 Files
#55Earlier quoted context omitted.
The threshold at which the cache gets used is configurable, with 128kB the default. The assumption is that any read larger than the threshold will be a long sustained read, for which latency doesn't matter too much. My question is, do reads 128kB get saved to the cache, or is it only used for files whose overall size is under the threshold? Frequent random access to large files is a textbook use case for a caching la…
NVMe read latency is in the 10-100µs range for 128kB blocks. S3 is about 100ms. That's 3-4 OOMs. The threshold where the total read duration starts to dominate latency would be somewhere in the dozens to hundreds of megabytes, not kilobytes.
Aren't you comparing local in-process latency to network latency? That's multiple OOM right there.
Re: S3 Files
#56Dumb Q: what would happen if you used this to store a SQLite database? Would it just... work? My guess is this would only enable a read-replica and not backups as Litestream currently does?
SQLite’s locking is not NFS safe so this would not work.
Re: S3 Files
#57"NFS provides the semantics your applications expect" is one of the funniest things I have ever read.
Do your applications not expect any network hiccup to cause them to block indefinitely in a system call making them effectively unkillable and making the filesystem unmountable?
Re: S3 Files
#58Earlier quoted context omitted.
NVMe read latency is in the 10-100µs range for 128kB blocks. S3 is about 100ms. That's 3-4 OOMs. The threshold where the total read duration starts to dominate latency would be somewhere in the dozens to hundreds of megabytes, not kilobytes.
Aren't you comparing local in-process latency to network latency? That's multiple OOM right there.
No, within the same DC network latency does not add that much. After all EFS also manages 600µs average latency.
It's really just S3 that's slow. I assume some large fraction of S3 is spread over HDDs, not SSDs.
Re: S3 Files
#59tldr: this caches your S3 data in EFS. we run datalakes using DuckLake and this sounds really useful. GCP should follow suit quickly.
I was thinking of using it with Duckdb as well but seems it would be of limited benefit. Parquet objects are in MBs, so they would be streamed directly from S3. With raw parquet objects, it might help with S3 listing if you have a lot of them (shave off a couple of seconds from the query). If you are already on Ducklake, Duckdb will use that for getting the list of relevant objects anyway.
Maybe the OP is thinking of reading/writing to DuckDB native format files. Those require filesystem semantics for writing. Unfortunately, even NFS or SMB are not sufficiently FS-like for DuckDB.
Parquet is static append only, so DuckDB has no problems with those living on S3.
Re: S3 Files
#60Zero mention of s3fs which already did this for decades.
There's also https://github.com/kahing/goofys, a Go equivalent. A bit of a dead project these days.