Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mountpoint – file client for S3 written in Rust, from AWS

github.com

Post reply on HN
Print view View original

71–80 of 102 posts

Re: Mountpoint – file client for S3 written in Rust, from AWS

#71

Post by Arnavion »

hardwaresofton wrote: post #70
Arnavion wrote: post #69

Earlier quoted context omitted.

I did create the block files as sparse originally (using `truncate`), but at some point in the process they became realized on disk. Don't know if it was the losetup or the mdadm or the cryptsetup. I didn't really worry about it, since the block files need to be synced to the WebDAV server in full anyway.

Ahh OK, I think I see -- since the block files are synced in full, you are always swapping blocks and doing ~1MB of writing no matter what. > I use a WebDAV server for storing backups (Fastmail Files). The server allows 10GB usage, but max file size is 250MB, *and in any case WebDAV does not support partial writes*. So writing a file requires reuploading it, which is the same situation as S3. This is the part I absol…

The stack is XFS inside cryptsetup inside mdraid on top of losetup. The directory containing the losetup block files could be `rclone mount`'d from the WebDAV server, but that would make the setup unavailable if I didn't have network access. So instead I chose to have the block files in a regular directory, and I make sure to `rclone sync` that directory to the WebDAV server when I make changes in the XFS layer. Manually syncing also lets me run `sync` and watch the `rclone sync` output, which gives me greater confidence that all the layers have synced successfully.

>Ahh OK, I think I see -- since the block files are synced in full, you are always swapping blocks and doing ~1MB of writing no matter what.

Right. Let's say I update two files in the XFS layer. Those writes eventually result in three blocks in the lowest layer being modified. So now the `rclone sync` will need to do a `PUT` request to replace those three blocks on the WebDAV server, which means it'll upload 3MB of data to the server.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#72

Post by hardwaresofton »

Arnavion wrote: post #71
hardwaresofton wrote: post #70

Earlier quoted context omitted.

Ahh OK, I think I see -- since the block files are synced in full, you are always swapping blocks and doing ~1MB of writing no matter what. > I use a WebDAV server for storing backups (Fastmail Files). The server allows 10GB usage, but max file size is 250MB, *and in any case WebDAV does not support partial writes*. So writing a file requires reuploading it, which is the same situation as S3. This is the part I absol…

The stack is XFS inside cryptsetup inside mdraid on top of losetup. The directory containing the losetup block files could be `rclone mount`'d from the WebDAV server, but that would make the setup unavailable if I didn't have network access. So instead I chose to have the block files in a regular directory, and I make sure to `rclone sync` that directory to the WebDAV server when I make changes in the XFS layer. Manu…

Thanks for the explanation, this makes perfect sense now, didn't realize the syncing was manual/separate.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#73

Post by nerpderp82 »

Arnavion wrote: post #59
ary wrote: post #15

This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new fil…

I use a WebDAV server for storing backups (Fastmail Files). The server allows 10GB usage, but max file size is 250MB, and in any case WebDAV does not support partial writes. So writing a file requires reuploading it, which is the same situation as S3. What I did is: 1. Create 10000 files, each of 1MB size, so that the total usage is 10GB. 2. Mount each file as a loopback block device using `losetup`. 3. Create a RAID…

Are you using davfs2 to mount the 1MB files from the WebDAV server?

Re: Mountpoint – file client for S3 written in Rust, from AWS

#74

Post by nerpderp82 »

hardwaresofton wrote: post #68
operator-name wrote: post #67

Earlier quoted context omitted.

If they're using LUKS then I think trimming/discard won't be possible.

My immediate instinct was that LUKS could issue trim/discard. It looks like there's some anecdotal evidence out there that LUKS can discard https://superuser.com/questions/124310/does-luks-encryption-... https://unix.stackexchange.com/questions/341442/luks-discard... My question is more for the mdraid at the bottom of the stack than anything. I'm also a little curious about performance of something webdav vs. samba v…

Wouldn't the blocks all be cached locally for the most part? WebDAV is being used as a write behind log/backup. It should be as fast as local access through a file system created over mdraid loopback block devices ...

Re: Mountpoint – file client for S3 written in Rust, from AWS

#76

Post by FullyFunctional »

aseipp wrote: post #31
FullyFunctional wrote: post #18

Earlier quoted context omitted.

Forgive the question but I never quite understood the point of S3. It seems it’s a terrible protocol but it’s designed for bandwidth. Why couldn’t they have used something like, say, 9P or Ceph? Surely I’m missing something fundamental. EDIT: In my personal experience with S3 it’s always been super slow.

Because you don't have to allocate any fixed amount up front, and it's pay as you go. At the time when the best storage options you could get were fixed-size hard drives from VPS providers, this was a big change, especially on both the "very small" and "very large" ends of the spectrum. It has always spoken HTTP with a relatively straightforward request-signing scheme for security, so integration at the basic levels…

Thanks, it see now. Essentially I lacked the original context. I got many excellent answers and can’t reply everyone.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#77

Post by FullyFunctional »

vbezhenar wrote: post #54
FullyFunctional wrote: post #18

Earlier quoted context omitted.

Forgive the question but I never quite understood the point of S3. It seems it’s a terrible protocol but it’s designed for bandwidth. Why couldn’t they have used something like, say, 9P or Ceph? Surely I’m missing something fundamental. EDIT: In my personal experience with S3 it’s always been super slow.

Here are reasons I'm using S3 in some projects: 1. Cost. It might vary depending on vendor, but generally S3 is much cheaper than block storage, at the same time with some welcome guarantees (like 3 copies). 2. Pay for what you use. 3. Very easy to hand off URL to client rather than creating some kind of file server. Also works with uploads AFAIR. 4. Offloads traffic. Big files often are the main source of traffic on…

Thanks, I was unclear and meant only the protocol S3 not the service, but I see now that as a KV store it makes sense.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#78

Post by kccqzy »

Arnavion wrote: post #59
ary wrote: post #15

This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new fil…

I use a WebDAV server for storing backups (Fastmail Files). The server allows 10GB usage, but max file size is 250MB, and in any case WebDAV does not support partial writes. So writing a file requires reuploading it, which is the same situation as S3. What I did is: 1. Create 10000 files, each of 1MB size, so that the total usage is 10GB. 2. Mount each file as a loopback block device using `losetup`. 3. Create a RAID…

FWIW this is similar to Apple's "sparse image bundle" feature, where you can create a disk image that internally is stored in 1MB chunks (the chunk size is probably only customizable via the command line `hdiutil` not the UI). You can encrypt it and put a filesystem on top of it.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#79

Post by hardwaresofton »

nerpderp82 wrote: post #74
hardwaresofton wrote: post #68

Earlier quoted context omitted.

My immediate instinct was that LUKS could issue trim/discard. It looks like there's some anecdotal evidence out there that LUKS can discard https://superuser.com/questions/124310/does-luks-encryption-... https://unix.stackexchange.com/questions/341442/luks-discard... My question is more for the mdraid at the bottom of the stack than anything. I'm also a little curious about performance of something webdav vs. samba v…

Wouldn't the blocks all be cached locally for the most part? WebDAV is being used as a write behind log/backup. It should be as fast as local access through a file system created over mdraid loopback block devices ...

you're right (see the sibling comment chain), I didn't realize this was just being done on local disk with periodic backup, thought webdav was below it all!

Re: Mountpoint – file client for S3 written in Rust, from AWS

#80

Post by grogenaut »

arretevad wrote: post #62

I find all the “written in Rust” qualifier in all these post title to just be a distraction. It doesn’t feel like it’s telling me anything.

Amazon is starting to invest in rust internally, strategically, since joining some of the leadership (https://aws.amazon.com/blogs/opensource/why-aws-loves-rust-a...).

It's considered a good replacement for C++, and like go, is really good for releasing tools. Tools like the AWS CLI... that work great when you can plop a single exe down as your install story, as opposed to say a python install and app (aws cli).

But it's also still new. Releasing a tool like this is likely a big deal in the area, and they're likely quite prod of it due to the effort of things like getting legal approval, marketing, etc, let alone the cool nerd factor of a filesystem, who doesn't want to show off by having written a filesystem or hell a fuse plugin... file system over dns anyone?

Post reply on HN