Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mountpoint – file client for S3 written in Rust, from AWS

github.com

Post reply on HN
Print view View original

41–50 of 102 posts

Re: Mountpoint – file client for S3 written in Rust, from AWS

#41

Post by mlhpdx »

cyberax wrote: post #40
mlhpdx wrote: post #36

Earlier quoted context omitted.

This is misleading. S3 is also incredibly fast. The former when you’re sequentially writing (or reading) objects and the latter when concurrently writing (or reading) vast numbers of them.

That depends on what you consider "fast". EFS (the "serverless" NFS) has sub-millisecond operation latency. S3 is more in the 10-20ms range for most operations, with occasional spikes. BTW, if you need a pure Go client for NFSv4 (including AWS EFS), feel free to check my: https://github.com/Cyberax/go-nfs-client

We can write vast numbers and volume of objects to S3 per second using concurrent processes (spawn 1000 lambda invocations and try it). As long as I have the network bandwidth, I can push stuff essentially as fast as I want. Is that true for EFS? Handle limits. Network interface limits. Protocol limits.

I’m not saying that S3 is perfect or even good for most workloads. However, it is most excellent when the workload fits.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#43

Post by znpy »

wmf wrote: post #34
supriyo-biswas wrote: post #10

After teaching customers for years that S3 shouldn't be mounted as a filesystem because of its whole object-or-nothing semantics, and even offering a paid solution named "storage gateway" to prevent issues between FS and S3 semantics, it's rather interesting they'd release a product like this. Amazon should really just fix the underlying issue of semantics by providing a PatchObjectPart API call that overwrites a par…

Eventually AWS always gives customers what they want even if it's a "bad idea".

Bad ideas are very relative.

Some bad ideas work extremely well if they fit your use case, you understand very well the tradeoffs and you’re building safeguards (disaster recovery).

Some other companies try to convince (force?) you into a workflow or into a specific solution. Aws just gives you the tools and some guidance on how to use them best.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#45

Post by [deleted] »

simooooo wrote: post #39
mlhpdx wrote: post #35

Earlier quoted context omitted.

I think you’re spot on: using multipart uploads, different sections of the ultimate object can be created out of order. Unfortunately, though, that’s subject to restrictions that require you to ensure all but the last part are sufficiently sized. I’m a little disappointed that this library (which is supposed to be “read optimized”) doesn’t take advantage of S3 Range requests to optimize read after seek. The simple ex…

Wouldn’t you be maintaining your own list of what is in the zip offline at this point?

[deleted]

Re: Mountpoint – file client for S3 written in Rust, from AWS

#47

Post by ignoramous »

FullyFunctional wrote: post #18
ary wrote: post #15

This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new fil…

Forgive the question but I never quite understood the point of S3. It seems it’s a terrible protocol but it’s designed for bandwidth. Why couldn’t they have used something like, say, 9P or Ceph? Surely I’m missing something fundamental. EDIT: In my personal experience with S3 it’s always been super slow.

> Forgive the question but I never quite understood the point of S3.

S3 and DynamoDB are essentially a decoupled BigTable; in that both are KV databases: One is used for high performance, small obj workloads; the other for high throughput, large obj workloads.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#48

Post by beebmam »

FullyFunctional wrote: post #18
ary wrote: post #15

This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new fil…

Forgive the question but I never quite understood the point of S3. It seems it’s a terrible protocol but it’s designed for bandwidth. Why couldn’t they have used something like, say, 9P or Ceph? Surely I’m missing something fundamental. EDIT: In my personal experience with S3 it’s always been super slow.

Same with my experience. Not a fan

Re: Mountpoint – file client for S3 written in Rust, from AWS

#49

Post by chucky_z »

mlhpdx wrote: post #41
cyberax wrote: post #40

Earlier quoted context omitted.

That depends on what you consider "fast". EFS (the "serverless" NFS) has sub-millisecond operation latency. S3 is more in the 10-20ms range for most operations, with occasional spikes. BTW, if you need a pure Go client for NFSv4 (including AWS EFS), feel free to check my: https://github.com/Cyberax/go-nfs-client

We can write vast numbers and volume of objects to S3 per second using concurrent processes (spawn 1000 lambda invocations and try it). As long as I have the network bandwidth, I can push stuff essentially as fast as I want. Is that true for EFS? Handle limits. Network interface limits. Protocol limits. I’m not saying that S3 is perfect or even good for most workloads. However, it is most excellent when the workload…

Yea it kind of is! I've used EFS in real-world scenarios with more than 1,000 concurrent readers/writers. EFS's costs are just otherworldly compared to S3. If you need that interface though, it's a good (albeit expensive) choice.

Re: Mountpoint – file client for S3 written in Rust, from AWS

#50

Post by fnordpiglet »

FullyFunctional wrote: post #18
ary wrote: post #15

This is really interesting and something I've been thinking about for a while now. The SEMANTICS[1] doc details what is and isn't supported from a POSIX filesystem API perspective, and this stands out: Write operations (write, writev, pwrite, pwritev) are not currently supported. In the future, Mountpoint for Amazon S3 will support sequential writes, but with some limitations: Writes will only be supported to new fil…

Forgive the question but I never quite understood the point of S3. It seems it’s a terrible protocol but it’s designed for bandwidth. Why couldn’t they have used something like, say, 9P or Ceph? Surely I’m missing something fundamental. EDIT: In my personal experience with S3 it’s always been super slow.

Simple api

Absurdly cheap storage

Extremely HA

Absurdly durable

Effectively unlimited bandwidth

Effectively unbounded storage without reservation or other management

Everything supports its api

It’s not a file system. It’s a blob store. It’s useful for spraying vast amounts of data into it and getting vast amounts of data out of it at any scale. It’s not low latency, it’s not a block store, but it is really cheap and the scaling of bandwidth and storage and concurrency make it possible to build stuff like snowflake that couldn’t be built on Ceph in any reasonable way.

Post reply on HN