Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mountpoint – file client for S3 written in Rust, from AWS

github.com

Post reply on HN
Print view View original

21–30 of 102 posts

Re: Mountpoint – file client for S3 written in Rust, from AWS

#21

Post by Scarbutt »

FullyFunctional wrote: post #18
ary wrote: post #15

This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new fil…

Forgive the question but I never quite understood the point of S3. It seems it’s a terrible protocol but it’s designed for bandwidth. Why couldn’t they have used something like, say, 9P or Ceph? Surely I’m missing something fundamental. EDIT: In my personal experience with S3 it’s always been super slow.

S3 is slow but at the same time low cost, if you want fast AWS has other alternatives but pricier.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#22

Post by the8472 »

FullyFunctional wrote: post #18
ary wrote: post #15

This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new fil…

Forgive the question but I never quite understood the point of S3. It seems it’s a terrible protocol but it’s designed for bandwidth. Why couldn’t they have used something like, say, 9P or Ceph? Surely I’m missing something fundamental. EDIT: In my personal experience with S3 it’s always been super slow.

They have NFS (called EFS), but it's about 10x more expensive.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#23

Post by netfortius »

the8472 wrote: post #22
FullyFunctional wrote: post #18

Earlier quoted context omitted.

Forgive the question but I never quite understood the point of S3. It seems it’s a terrible protocol but it’s designed for bandwidth. Why couldn’t they have used something like, say, 9P or Ceph? Surely I’m missing something fundamental. EDIT: In my personal experience with S3 it’s always been super slow.

They have NFS (called EFS), but it's about 10x more expensive.

Good luck mounting EFS in Windows.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#24

Post by ary »

FullyFunctional wrote: post #18
ary wrote: post #15

This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new fil…

Forgive the question but I never quite understood the point of S3. It seems it’s a terrible protocol but it’s designed for bandwidth. Why couldn’t they have used something like, say, 9P or Ceph? Surely I’m missing something fundamental. EDIT: In my personal experience with S3 it’s always been super slow.

When S3 was released the Internet was very different. Two of the things that stood out were:

1. It offered a resilient key/object store over HTTP.

2. By the standards of the day for bandwidth and storage it was (and to a certain extent still is) very inexpensive.

Since then much of AWS has been built on the foundation of S3 and so its importance has changed from merely being a tool to basically a pervasive dependency of the AWS stack. Also, it very much is designed for objects larger than 1KB and for applications that need durable storage of many, many large objects.

The key benefit, at least according to AWS marketing, is that you don't have to host it yourself.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#25

Post by supriyo-biswas »

FullyFunctional wrote: post #18
ary wrote: post #15

This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new fil…

Forgive the question but I never quite understood the point of S3. It seems it’s a terrible protocol but it’s designed for bandwidth. Why couldn’t they have used something like, say, 9P or Ceph? Surely I’m missing something fundamental. EDIT: In my personal experience with S3 it’s always been super slow.

The problem is S3 is just a lexicographically ordered key value store with (what I suspect is) key-range partitions[1] for the key part and Reed-Solomon encoded blobs for the value part. In other words, it’s a glorified NoSQL database with no semantics that you’d typically expect of a file system, and therefore repeated writes are slow because any modification to an object involves writing a new version of the key along with its new object.

[1] https://martinfowler.com/articles/patterns-of-distributed-sy...

Re: Mountpoint – file client for S3 written in Rust, from AWS

#26

Post by acdha »

the8472 wrote: post #22
FullyFunctional wrote: post #18

Earlier quoted context omitted.

Forgive the question but I never quite understood the point of S3. It seems it’s a terrible protocol but it’s designed for bandwidth. Why couldn’t they have used something like, say, 9P or Ceph? Surely I’m missing something fundamental. EDIT: In my personal experience with S3 it’s always been super slow.

They have NFS (called EFS), but it's about 10x more expensive.

I wouldn’t give a number because the pricing models are fairly different and the real cost will depend on how you’re using it and how easy it is to shift your access patterns. On my apps using EFA, that 10x is more like .8-1.1x — an easy call versus rewriting a bunch of code.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#27

Post by rakoo »

FullyFunctional wrote: post #18
ary wrote: post #15

This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new fil…

Forgive the question but I never quite understood the point of S3. It seems it’s a terrible protocol but it’s designed for bandwidth. Why couldn’t they have used something like, say, 9P or Ceph? Surely I’m missing something fundamental. EDIT: In my personal experience with S3 it’s always been super slow.

S3 is straight HTTP, the most widespread API. It can be directly used on the browser, has libraries in pretty much every language, and can reuse the mountain of available software and frameworks for load-balancing, redirections, auth, distributed storage etc

Re: Mountpoint – file client for S3 written in Rust, from AWS

#28

Post by spmurrayzzz »

FullyFunctional wrote: post #18
ary wrote: post #15

This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new fil…

Forgive the question but I never quite understood the point of S3. It seems it’s a terrible protocol but it’s designed for bandwidth. Why couldn’t they have used something like, say, 9P or Ceph? Surely I’m missing something fundamental. EDIT: In my personal experience with S3 it’s always been super slow.

I think theres an interesting story in software ecosystems where there are two flavors of applications (which coexist) that prefer object stores over filesystems and vice versa. Good reference point for this I think exists in many modern video transcoding infrastructures.

Using something like FSx [1] gives you a performant option for the use cases when the tooling involved prefers filesystem semantics.

[1] https://aws.amazon.com/fsx/lustre/

Re: Mountpoint – file client for S3 written in Rust, from AWS

#29

Post by BrianHenryIE »

JungleDisk was backup software I used ~2009 that allowed mounting S3. They were bought by Rackspace and the product wasn't updated. Seems to be called/part of Cyberfortress now.

Later I used Panic's Transmit Disk but they removed the feature.

Recently I'd been looking at s3fs-fuse to use with gocryptfs but haven't actually installed it yet!

https://github.com/s3fs-fuse/s3fs-fuse

https://github.com/rfjakob/gocryptfs

Re: Mountpoint – file client for S3 written in Rust, from AWS

#30

Post by mNovak »

favourable wrote: post #9

Something similar that I've been using for a while now for an S3 filesystem: Cyberduck[0] [0] https://cyberduck.io/s3/

In a similar vein, I've been using ExpanDrive [0] for a while. Though admittedly it's only suitable for infrequent access / long term storage type use.

[0] https://www.expandrive.com/desktop/

Post reply on HN