For anyone looking to mount S3 as file system, I will suggest giving rclone a shot. It can mount, copy and do all file operations not just on s3 but on a wide range of cloud providers, you can also declare a remote as encrypted so it does client side encryption
Mountpoint – file client for S3 written in Rust, from AWS
51–60 of 102 posts
Re: Mountpoint – file client for S3 written in Rust, from AWS
#52I want a better client for Google Cloud Storage, too, while we’re at it. The Python gcloud / gsutil stuff is mediocre on the best of days.
`gsutil ...` was pretty bad (it is python like you say, and based on a very outdated fork of boto2).
I've had really good luck with `gcloud storage ...` though, which takes essentially the same CLI args. It's much faster and IIRC written in golang.
Re: Mountpoint – file client for S3 written in Rust, from AWS
#53JungleDisk was backup software I used ~2009 that allowed mounting S3. They were bought by Rackspace and the product wasn't updated. Seems to be called/part of Cyberfortress now. Later I used Panic's Transmit Disk but they removed the feature. Recently I'd been looking at s3fs-fuse to use with gocryptfs but haven't actually installed it yet! https://github.com/s3fs-fuse/s3fs-fuse https://github.com/rfjakob/gocryptfs
> Later I used Panic's Transmit Disk but they removed the feature.
BTW, Panic seemingly intends to re-build Transmit Disk. Hopefully it'll be part of Transmit 6: https://help.panic.com/transmit/transmit5/transmit-disk/#tec...
A supported macOS option appears to be Mountain Duck: https://mountainduck.io/
Re: Mountpoint – file client for S3 written in Rust, from AWS
#54This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new fil…
Forgive the question but I never quite understood the point of S3. It seems it’s a terrible protocol but it’s designed for bandwidth. Why couldn’t they have used something like, say, 9P or Ceph? Surely I’m missing something fundamental. EDIT: In my personal experience with S3 it’s always been super slow.
Here are reasons I'm using S3 in some projects:
1. Cost. It might vary depending on vendor, but generally S3 is much cheaper than block storage, at the same time with some welcome guarantees (like 3 copies).
2. Pay for what you use.
3. Very easy to hand off URL to client rather than creating some kind of file server. Also works with uploads AFAIR.
4. Offloads traffic. Big files often are the main source of traffic on many websites. Using S3 allows to remove that burden. And S3 usually served by multiple servers which further increases speed.
5. Provider-independent. I think that every mature cloud offers S3 API.
I think that there are more reasons. Encryption, multi-region and so on. I didn't use those features. Of course you can implement everything with your own software, but reusing good implementation is a good idea for most projects. You don't rewrite postgres, so you don't rewrite S3.
Re: Mountpoint – file client for S3 written in Rust, from AWS
#55Earlier quoted context omitted.
They have NFS (called EFS), but it's about 10x more expensive.
Good luck mounting EFS in Windows.
Apparently the nfs client in windows only supports nfs v3 - while efs only supports v4. The closest I found was:
http://citi.umich.edu/projects/nfsv4/windows/readme.html
Seems odd that there are no commercial nfs v4 clients for windows? Might now be possible to mount via wsl?
I see ractos has nfs client - but could not figure out which version...
Re: Mountpoint – file client for S3 written in Rust, from AWS
#56I would suggest changing the title to "Mountpoint-S3 - ..." as that's the project name to avoid confusion with mountpoint(1): https://man7.org/linux/man-pages/man1/mountpoint.1.html
Re: Mountpoint – file client for S3 written in Rust, from AWS
#57How does this compare to rclone, performance wise? https://rclone.org
And s4cmd...
https://github.com/bloomreach/s4cmd
Re: Mountpoint – file client for S3 written in Rust, from AWS
#58I want a better client for Google Cloud Storage, too, while we’re at it. The Python gcloud / gsutil stuff is mediocre on the best of days.
In theory, you can just use this library since GC Storage supports S3 protocol. But in practice, I’m not sure
Or https://clone.org ?
Re: Mountpoint – file client for S3 written in Rust, from AWS
#59This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new fil…
I use a WebDAV server for storing backups (Fastmail Files). The server allows 10GB usage, but max file size is 250MB, and in any case WebDAV does not support partial writes. So writing a file requires reuploading it, which is the same situation as S3.
What I did is:
1. Create 10000 files, each of 1MB size, so that the total usage is 10GB.
2. Mount each file as a loopback block device using `losetup`.
3. Create a RAID device over the 10000 loopback devices with `mdadm --build --level=linear`. This RAID device appears as a single block device of 10GB size. `--level=linear` means the RAID device is just a concatenation of the underlying devices. `--build` means that mdadm does not store metadata blocks in the devices, unlike `--create` which does. Not only would metadata blocks use up a significant portion of the 1MB device size, but also I don't really need mdadm to "discover" this device automatically, and also the metadata superblock does not support 10000 devices anyway (the max is 2000 IIRC).
4. From here the 10GB block device can be used as any other block device. In my case I created a LUKS device on top of this, then an XFS filesystem on the top of the LUKS device, then that XFS filesystem is my backup directory.
So any modification of files in the XFS layer eventually results in some of the 1MB blocks at the lowest layer being modified, and only those modified 1MB blocks need to be synced to the WebDAV server.
(Note: SI units. 1KB == 1000B, 1MB == 1000KB, 1GB == 1000MB.)
Re: Mountpoint – file client for S3 written in Rust, from AWS
#60I want a better client for Google Cloud Storage, too, while we’re at it. The Python gcloud / gsutil stuff is mediocre on the best of days.
I'm just about to start using it. I'd love to know what issues you've encountered. Thanks!