Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mountpoint – file client for S3 written in Rust, from AWS

github.com

Post reply on HN
Print view View original

81–90 of 102 posts

Re: Mountpoint – file client for S3 written in Rust, from AWS

#81

Post by Arnavion »

nerpderp82 wrote: post #73
Arnavion wrote: post #59

Earlier quoted context omitted.

I use a WebDAV server for storing backups (Fastmail Files). The server allows 10GB usage, but max file size is 250MB, and in any case WebDAV does not support partial writes. So writing a file requires reuploading it, which is the same situation as S3. What I did is: 1. Create 10000 files, each of 1MB size, so that the total usage is 10GB. 2. Mount each file as a loopback block device using `losetup`. 3. Create a RAID…

Are you using davfs2 to mount the 1MB files from the WebDAV server?

I started out with davfs2 but it was a) very slow at uploading for some reason, b) there was no way to explicitly sync it so I had to either wait a minute for some internal timer to trigger the sync or to unmount it, and c) it implements writes by writing to a cache directory in /var/cache, which was just a redundant 10GB copy of the data I already have.

I use `rclone`. Currently rclone doesn't support the SHA1 checksums that Fastmail Files implements. I have a PR for that: https://github.com/rclone/rclone/pull/6839

Re: Mountpoint – file client for S3 written in Rust, from AWS

#82

Post by legorobot »

BrianHenryIE wrote: post #29

JungleDisk was backup software I used ~2009 that allowed mounting S3. They were bought by Rackspace and the product wasn't updated. Seems to be called/part of Cyberfortress now. Later I used Panic's Transmit Disk but they removed the feature. Recently I'd been looking at s3fs-fuse to use with gocryptfs but haven't actually installed it yet! https://github.com/s3fs-fuse/s3fs-fuse https://github.com/rfjakob/gocryptfs

We've used the s3fs-fuse library for a while at work for SFTP/FTP server alternatives (AWS wants you to pay $150+/server/month last I checked!) and it's worked like a dream. We scripted the setup of new users via a simple bash script and the S3 CloudWatch events for file uploads is a dream. Its been pretty seamless and hasn't caused many headaches.

We've had to perform occasional maintenance but its operated for years with no major issues. 99% are solved with a server restart + a startup script to auto-re-mount s3fs-fuse in all the appropriate places.

Give them a try, I recommend it!

Re: Mountpoint – file client for S3 written in Rust, from AWS

#83

Post by FullyFunctional »

awill wrote: post #61
cyberax wrote: post #40

Earlier quoted context omitted.

That depends on what you consider "fast". EFS (the "serverless" NFS) has sub-millisecond operation latency. S3 is more in the 10-20ms range for most operations, with occasional spikes. BTW, if you need a pure Go client for NFSv4 (including AWS EFS), feel free to check my: https://github.com/Cyberax/go-nfs-client

fast is an overloaded word. Could mean throughput, or latency. S3 throughput is incredible. note, I worked on Amazon at S3 2015-2017.

I could have worded it better. I was just trying to understand the why of the protocol. My experience was probably irrelevant as we were just using it for storage and interfacing with an FS translation (rclone or similar). We have long stopped using AWS for cost reasons though.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#84

Post by nerpderp82 »

Arnavion wrote: post #81
nerpderp82 wrote: post #73

Earlier quoted context omitted.

Are you using davfs2 to mount the 1MB files from the WebDAV server?

I started out with davfs2 but it was a) very slow at uploading for some reason, b) there was no way to explicitly sync it so I had to either wait a minute for some internal timer to trigger the sync or to unmount it, and c) it implements writes by writing to a cache directory in /var/cache, which was just a redundant 10GB copy of the data I already have. I use `rclone`. Currently rclone doesn't support the SHA1 check…

Thanks for the response.

So you are using rclone sync to periodically push changes locally up to the webdav server?

Re: Mountpoint – file client for S3 written in Rust, from AWS

#85

Post by Arnavion »

nerpderp82 wrote: post #84
Arnavion wrote: post #81

Earlier quoted context omitted.

I started out with davfs2 but it was a) very slow at uploading for some reason, b) there was no way to explicitly sync it so I had to either wait a minute for some internal timer to trigger the sync or to unmount it, and c) it implements writes by writing to a cache directory in /var/cache, which was just a redundant 10GB copy of the data I already have. I use `rclone`. Currently rclone doesn't support the SHA1 check…

Thanks for the response. So you are using rclone sync to periodically push changes locally up to the webdav server?

Yes. https://news.ycombinator.com/item?id=35163090

Re: Mountpoint – file client for S3 written in Rust, from AWS

#86

Post by Arnavion »

Arnavion wrote: post #59
ary wrote: post #15

This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new fil…

I use a WebDAV server for storing backups (Fastmail Files). The server allows 10GB usage, but max file size is 250MB, and in any case WebDAV does not support partial writes. So writing a file requires reuploading it, which is the same situation as S3. What I did is: 1. Create 10000 files, each of 1MB size, so that the total usage is 10GB. 2. Mount each file as a loopback block device using `losetup`. 3. Create a RAID…

Of course, despite working on this for a week I only now discovered this... dm_linear is an easier way than mdadm to concatenate the loopback devices into a single device. Setting up the table input to `dmsetup create`'s stdin is more complicated than just `mdadm --build ... /dev/loop1{0000..9999}`, but it's all scripted anyway so it doesn't matter. And `mdadm --stop` blocks for multiple minutes for some unexplained reason, whereas `dmcreate remove` is almost instantaneous.

One caveat is that my 1MB (actually 999936B) block devices have 1953 sectors (999936B / 512B) but mdadm had silently only used 1920 sectors from each. In my first attempt at replacing mdadm with dm_linear I used 1953 as the number of sectors, which led to garbage when decrypted with dm_crypt. I discovered mdadm's behavior by inspecting the first two loopback devices and the RAID device in xxd. Using 1920 as the number of sectors fixed that, though I'll probably just nuke the LUKS partition and rebuild it on top of dm_linear with 1953 sectors each.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#87

Post by otterley »

netfortius wrote: post #23
the8472 wrote: post #22

Earlier quoted context omitted.

They have NFS (called EFS), but it's about 10x more expensive.

Good luck mounting EFS in Windows.

AWS also offers FSx for Windows File Server, and FSx for ONTAP if you need remote Windows file service.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#89

Post by orf »

chucky_z wrote: post #49
mlhpdx wrote: post #41

Earlier quoted context omitted.

We can write vast numbers and volume of objects to S3 per second using concurrent processes (spawn 1000 lambda invocations and try it). As long as I have the network bandwidth, I can push stuff essentially as fast as I want. Is that true for EFS? Handle limits. Network interface limits. Protocol limits. I’m not saying that S3 is perfect or even good for most workloads. However, it is most excellent when the workload…

Yea it kind of is! I've used EFS in real-world scenarios with more than 1,000 concurrent readers/writers. EFS's costs are just otherworldly compared to S3. If you need that interface though, it's a good (albeit expensive) choice.

At one point we had a ~560tb EFS disk that ran a variety of mixed workloads (large and small files). It was untenable - raw reading/writing IO is OK, but metadata IO hits a brick wall and destroys the performance of the whole disk for all connections (not just ones accessing a particular partition/tree/whatever).

In order to migrate off it and onto s3 I had to build a custom tool in rust that used libnfs directly to list the contents of the disk. We then launched a large number of lambdas to copy individual files to s3.

It was fun, but in my experience EFS is only good if you have a very homogenous workload and are able to carefully optimise metadata IO. I wouldn’t recommend it - s3 is just cheaper, faster and better.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#90

Post by drcongo »

CharlesW wrote: post #53
BrianHenryIE wrote: post #29

JungleDisk was backup software I used ~2009 that allowed mounting S3. They were bought by Rackspace and the product wasn't updated. Seems to be called/part of Cyberfortress now. Later I used Panic's Transmit Disk but they removed the feature. Recently I'd been looking at s3fs-fuse to use with gocryptfs but haven't actually installed it yet! https://github.com/s3fs-fuse/s3fs-fuse https://github.com/rfjakob/gocryptfs

> Later I used Panic's Transmit Disk but they removed the feature. BTW, Panic seemingly intends to re-build Transmit Disk. Hopefully it'll be part of Transmit 6: https://help.panic.com/transmit/transmit5/transmit-disk/#tec... A supported macOS option appears to be Mountain Duck: https://mountainduck.io/

ForkLift also lets you mount S3 as a drive. https://binarynights.com
Post reply on HN