Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

S3 Files

allthingsdistributed.com

Post reply on HN
Print view View original

61–70 of 128 posts

Re: S3 Files

#61

Post by [deleted] »

the8472 wrote: post #50
MontyCarloHall wrote: post #46

Earlier quoted context omitted.

The threshold at which the cache gets used is configurable, with 128kB the default. The assumption is that any read larger than the threshold will be a long sustained read, for which latency doesn't matter too much. My question is, do reads 128kB get saved to the cache, or is it only used for files whose overall size is under the threshold? Frequent random access to large files is a textbook use case for a caching la…

NVMe read latency is in the 10-100µs range for 128kB blocks. S3 is about 100ms. That's 3-4 OOMs. The threshold where the total read duration starts to dominate latency would be somewhere in the dozens to hundreds of megabytes, not kilobytes.

[deleted]

Re: S3 Files

#62

Post by dabinat »

The problem with using S3 as a filesystem is that it’s immutable, and that hasn’t changed with S3 Files. So if I have a large file and change 1 byte of it, or even just rename it, it needs to upload the entire file all over again. This seems most useful for read-heavy workflows of files that are small enough to fit in the cache.

Re: S3 Files

#63

Post by boulos »

wbl wrote: post #39

"NFS provides the semantics your applications expect" is one of the funniest things I have ever read.

Compared to roll-your-own with S3 or GCS it does :)

Re: S3 Files

#64

Post by wolttam »

dabinat wrote: post #62

The problem with using S3 as a filesystem is that it’s immutable, and that hasn’t changed with S3 Files. So if I have a large file and change 1 byte of it, or even just rename it, it needs to upload the entire file all over again. This seems most useful for read-heavy workflows of files that are small enough to fit in the cache.

That’s not that different than CoW filesystems - there is no rule that files must map 1:1 to objects; you can (transparently) divide a file into smaller chunks to enable more fine grained edits.

Re: S3 Files

#66

Post by jitl »

MontyCarloHall wrote: post #48
jitl wrote: post #33

I wish they offered some managed bridging to local NVMe storage. AWS NVMe is super fast compared to EBS, and EBS (node-exclusive access as block device) is faster than EFS (multi-node access). I imagine this can go fast if you put some kind of further-cache-to-NVMe FS on top, but a completely vertically integrated option would be much better.

Since EFS is just an NFS mount, I wonder if you could do this yourself by attaching an NVMe volume to your instance and setting up something like cachefilesd on the NFS mount, pointed to the NVMe. Would mkfs.ext4 /dev/nvme0n1 && \ mount /dev/nvme0n1 /var/cache/fscache && \ mount -t s3files -o fsc fs-0aa860d05df9afdfe:/ /home/ec2-user/s3files work out of the box? It does for EFS. It hardly seems worth it to offer a ma…

AWS's [docs on EFS performance](https://docs.aws.amazon.com/efs/latest/ug/performance-tips.h...) say:

> Don't use the following mount options:

> - fsc – This option enables local file caching, but does not change NFS cache coherency, and does not reduce latencies.

If the S3 Files sync logic ran client-side, we could almost entirely avoid file access latency for cached files and paying for new expensive EFS disks. I already pay for a lot of NVMe disks, let me just use those!

Re: S3 Files

#67

Post by [deleted] »

wolttam wrote: post #64
dabinat wrote: post #62

The problem with using S3 as a filesystem is that it’s immutable, and that hasn’t changed with S3 Files. So if I have a large file and change 1 byte of it, or even just rename it, it needs to upload the entire file all over again. This seems most useful for read-heavy workflows of files that are small enough to fit in the cache.

That’s not that different than CoW filesystems - there is no rule that files must map 1:1 to objects; you can (transparently) divide a file into smaller chunks to enable more fine grained edits.

[deleted]

Re: S3 Files

#68

Post by karmasimida »

gonzalohm wrote: post #13

I cannot 100% confirm this, but I believe AWS insisted a lot in NOT using S3 as a file system. Why the change now?

This is how tech people think, but Customer still want this, so it will be built, eventually

Re: S3 Files

#69

Post by MontyCarloHall »

jitl wrote: post #66
MontyCarloHall wrote: post #48

Earlier quoted context omitted.

Since EFS is just an NFS mount, I wonder if you could do this yourself by attaching an NVMe volume to your instance and setting up something like cachefilesd on the NFS mount, pointed to the NVMe. Would mkfs.ext4 /dev/nvme0n1 && \ mount /dev/nvme0n1 /var/cache/fscache && \ mount -t s3files -o fsc fs-0aa860d05df9afdfe:/ /home/ec2-user/s3files work out of the box? It does for EFS. It hardly seems worth it to offer a ma…

AWS's [docs on EFS performance]( https://docs.aws.amazon.com/efs/latest/ug/performance-tips.h... ) say: > Don't use the following mount options: > - fsc – This option enables local file caching, but does not change NFS cache coherency, and does not reduce latencies. If the S3 Files sync logic ran client-side, we could almost entirely avoid file access latency for cached files and paying for new expensive EFS disks. I…

>This option enables local file caching, but does not change NFS cache coherency, and does not reduce latencies.

That's true for any NFS setup, not just EFS. The benefit of local NFS caching is to speed up reads of large, immutable files, where latency is relatively negligible. I'm not sure why AWS specifically dissuades users from enabling caching, since it's not like bandwidth to an EFS volume is even in the ballpark of EBS/NVMe bandwidth.

Re: S3 Files

#70

Post by ktimespi »

MontyCarloHall wrote: post #26

This is essentially S3FS using EFS (AWS's managed NFS service) as a cache layer for active data and small random accesses. Unfortunately, this also means that it comes with some of EFS's eye-watering pricing: — All writes cost $0.06/GB, since everything is first written to the EFS cache. For write-heavy applications, this could be a dealbreaker. — Reads hitting the cache get billed at $0.03/GB. Large reads (>128kB) g…

This was my concern too. The whole point of using S3 as a file system instead of EBS / EFS (for me at least) is to minimize cost and I don't really see why I would use this instead of s3fs.
Post reply on HN