Couldn't tell from the README, does this do any sort of cache management or LRU type thing? In other words, does it fetch the underlying S3 object in real time, and then eventually eject them from memory and/or the backing FS when they haven't been used for a while?
Mountpoint – file client for S3 written in Rust, from AWS
11–20 of 102 posts
Re: Mountpoint – file client for S3 written in Rust, from AWS
#12I want a better client for Google Cloud Storage, too, while we’re at it. The Python gcloud / gsutil stuff is mediocre on the best of days.
In theory, you can just use this library since GC Storage supports S3 protocol. But in practice, I’m not sure
Re: Mountpoint – file client for S3 written in Rust, from AWS
#13Couldn't tell from the README, does this do any sort of cache management or LRU type thing? In other words, does it fetch the underlying S3 object in real time, and then eventually eject them from memory and/or the backing FS when they haven't been used for a while?
`catfs` is a FUSE FS that can do this for you. You'll need some changes to make it work well. I'll have a friend upstream them soon, but they're easy to make yourself.
Could replace `goofys` with this and then stick `catfs` in front.
Re: Mountpoint – file client for S3 written in Rust, from AWS
#14There's a similar project under awslabs for using S3 as a FileSystem within the Java JVM: https://github.com/awslabs/aws-java-nio-spi-for-s3
Re: Mountpoint – file client for S3 written in Rust, from AWS
#15This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out:
Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations:
Writes will only be supported to new files, and must be done sequentially.
Modifying existing files will not be supported.
Truncation will not be supported.
[1] https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMAN...
[2] https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuove...
[3] https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObje...
Re: Mountpoint – file client for S3 written in Rust, from AWS
#16After teaching customers for years that S3 shouldn't be mounted as a filesystem because of its whole object-or-nothing semantics, and even offering a paid solution named "storage gateway" to prevent issues between FS and S3 semantics, it's rather interesting they'd release a product like this. Amazon should really just fix the underlying issue of semantics by providing a PatchObjectPart API call that overwrites a par…
Distributed patching becomes hell. You need transactional semantics and files are not laid out well to help you define invariants that should reject the transaction.
Re: Mountpoint – file client for S3 written in Rust, from AWS
#17This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new fil…
[deleted]
Re: Mountpoint – file client for S3 written in Rust, from AWS
#18This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new fil…
Forgive the question but I never quite understood the point of S3. It seems it’s a terrible protocol but it’s designed for bandwidth. Why couldn’t they have used something like, say, 9P or Ceph? Surely I’m missing something fundamental.
EDIT: In my personal experience with S3 it’s always been super slow.
Re: Mountpoint – file client for S3 written in Rust, from AWS
#19They should benchmark it against rclone
Re: Mountpoint – file client for S3 written in Rust, from AWS
#20After teaching customers for years that S3 shouldn't be mounted as a filesystem because of its whole object-or-nothing semantics, and even offering a paid solution named "storage gateway" to prevent issues between FS and S3 semantics, it's rather interesting they'd release a product like this. Amazon should really just fix the underlying issue of semantics by providing a PatchObjectPart API call that overwrites a par…
Distributed patching becomes hell. You need transactional semantics and files are not laid out well to help you define invariants that should reject the transaction.
There is no reason why the descriptor of objects can’t be updated with a new value that has all of the old chunks and a new one, since S3 doesn’t do deduplication anyway the other chunks may be resized internally with an asynchronous process that gets rid of the excess data corresponding to the now overridden chunk.