21–30 of 128 posts
I cannot 100% confirm this, but I believe AWS insisted a lot in NOT using S3 as a file system. Why the change now?
If there is ever a post that needs a TLDR or an AI summary it is that one. Sell the benefits. I have around 9 TB in 21m files on S3. How does this change benefit me?
edited slightly ... i really need to turn 10 minute post delay back on.
Zero mention of s3fs which already did this for decades.
This is pretty different than s3fs. s3fs is a FUSE file system that is backed by S3. This means that all of the non-atomic operations that you might want to do on S3 (including edits to the middle of files, renames, etc) are run on the machine running S3fs. As a result, if your machine crashes, it's not clear what's going to show up in your S3 bucket or if would corrupt things. As a result, S3fs is also slow because…
— All writes cost $0.06/GB, since everything is first written to the EFS cache. For write-heavy applications, this could be a dealbreaker.
— Reads hitting the cache get billed at $0.03/GB. Large reads (>128kB) get directly streamed from the underlying S3 bucket, which is free.
— Cache is charged at $0.30/GB/month. Even though everything is written to the cache (for consistency purposes), it seems like it's only used for persistent storage of small files (<128kB), so this shouldn't cost too much.
> For example, suppose you edit /mnt/s3files/report.csv through the file system. Before S3 Files synchronizes your changes back to the S3 bucket, another application uploads a new version of report.csv directly to the S3 bucket. When S3 Files detects the conflict, it moves your version of report.csv to the lost and found directory and replaces it with the version from the S3 bucket.
> The lost and found directory is located in your file system's root directory under the name .s3files-lost+found-file-system-id.
Zero mention of s3fs which already did this for decades.
A more solid (especially when it comes to caching) solution would be appreciated. I thought that would be their https://github.com/awslabs/mountpoint-s3 . But no mention about this one either. S3 files does have the advantage of having a "shared" cache via EFS, but then that would probably also make the cache slower.