Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A distributed Posix file system built on top of Redis and S3

github.com

Post reply on HN
Print view View original

111–120 of 120 posts

Re: A distributed Posix file system built on top of Redis and S3

#111

Post by daviesliu »

shaicoleman wrote: post #76
daviesliu wrote: post #62

Earlier quoted context omitted.

FUSE is not support by AWS Lambda, so we can't mount JuiceFS in Lambda. We can have a SDK to access JuiceFS from Lambda, similar to S3 SDK, when you need to use JuiceFS outside of Lambda. Same to Fargate, we can not mount JuiceFS in Farget because of lacking FUSE permission, people are asking for it[1]. https://github.com/aws/containers-roadmap/issues/412

What programming languages is the SDK available for?

Since JuiceFS written in Go, GO SDK will be the first, then we can have other languages using CGO. We already have Java SDK internally, Python will be the next.

But the way, A S3 gateway is on our roadmap, you can spin up an S3 gateway for JuiceFS, and talk to that using existing S3 SDK. This only make sense when you have other applications outside of Lambda using JuiceFS.

Re: A distributed Posix file system built on top of Redis and S3

#112

Post by daviesliu »

corford wrote: post #75
daviesliu wrote: post #34

Earlier quoted context omitted.

The underlying S3 bucket still have intelligent tiering, you can also put life cycle rules on it.

If a lifecycle rule deletes an object from the bucket, does the redis metadata server gracefully handle this (i.e. bucket state drifting)?

JuiceFS access objects in S3 using an unique id to generate key, so the result of lifecycle will not change the way JuiceFS accessing it, except the StorageClass Glacier or Deep Arching, which is not accessible instantly, will hurt the user experience for JuiceFS.

Re: A distributed Posix file system built on top of Redis and S3

#113

Post by 0xbadcafebee »

daviesliu wrote: post #97
0xbadcafebee wrote: post #74

I've still got PTSD from NFS so I would want to see some really abusive testing to know that this was more reliable than a vendored NFS SAN.

Understood, the annoying part of NFS is that some operation can not be interrupted when the network is not healthy, and we have no way to kill it. We have put lots of effort to make the operations interruptible when either meta or object store is slow or down. There are a few cases that the `close` can't be interrupted, we can still abort the FUSE connection or kill the JuiceFS process, then all the operation are can…

So operations-wise, applications would still require workarounds to be resilient to failure in production. From what I recall, all network filesystems have this problem, so I avoid them all.

The essential problem is trying to do something with an interface that it wasn't designed for. It's the same problem network protocols have. They can't communicate metadata about each layer across the layers, so your HTTP protocol has no idea that it's actually being tunneled in another HTTP protocol and that that protocol's TCP connection just got a PSH,FIN,RST packet. Your file i/o app also has no idea that you just lost a quorum, or that one section of the network just crapped out (and even if it did know, what would it do?)

Re: A distributed Posix file system built on top of Redis and S3

#114

Post by daviesliu »

fock wrote: post #47
toomuchtodo wrote: post #11

Earlier quoted context omitted.

NFS doesn’t scale to effectively infinity with an underlying object store. This is to give you a ton of storage without using a traditional volume target with your app that, for whatever reason, requires a posix filesystem. I’m sure someone from AWS can’t comment, but I imagine this is how AWS’ EFS service is built (NFS wire protocol to clients, but using S3 and metadata caching under the hood). Blobs or blocks doesn…

well, NFS may not scale to infinity, but easily beats this thing I guess... And for scaling to infinity: how about benchmarking vs. GPFS, BeeGFS or Gluster?

These are scalable but very expensive in the clouds.

They require a cluster of machines, the replicate the data across them, using either expensive EBS or local disk (a few larger instance to pick).

Maintaining them well is another burden. The cool idea of JuiceFS is to shift the maintenance to hosted Redis and S3.

Re: A distributed Posix file system built on top of Redis and S3

#116

Post by daviesliu »

Labo333 wrote: post #90
daviesliu wrote: post #51

Earlier quoted context omitted.

Redis can be persisted with RDB and AOF, can also be replicated to another machine. In the cloud, you don't need to worry about that, hosted Redis are ready to use. The is an ongoing effort [1] to improve the persistency and availability in general, which is expected to be GA in 2021. [1] https://github.com/RedisLabs/redisraft

Thanks! I really have trouble to trust in memory databases that require 100% uptime. What happens with a power outage?

As mentioned in other place, the perfect solution would be adding Raft to redis [1].

[1] https://github.com/RedisLabs/redisraft

Re: A distributed Posix file system built on top of Redis and S3

#117

Post by guenthert »

the8472 wrote: post #59

How POSIX-compatible is it exactly? There's a lot of niche features that tend to break on not fully compliant network filesystems. Do unlinked files remain accessible (the dreaded ESTALE on some NFS implementations)? mmap? atomic rename? atomic append? range locks? what's the consistency model? Some of those things don't appear to be covered by pjdfstest.

That and further POSIX requires that a read which can be proven to occur after a write returns the new data (I'd think that would be quite difficult to implement efficiently, if multiple clients are allowed). I couldn't find that being mentioned in either the documentation of juicefs nor the pjdfs test.

Re: A distributed Posix file system built on top of Redis and S3

#118

Post by daviesliu »

guenthert wrote: post #117
the8472 wrote: post #59

How POSIX-compatible is it exactly? There's a lot of niche features that tend to break on not fully compliant network filesystems. Do unlinked files remain accessible (the dreaded ESTALE on some NFS implementations)? mmap? atomic rename? atomic append? range locks? what's the consistency model? Some of those things don't appear to be covered by pjdfstest.

That and further POSIX requires that a read which can be proven to occur after a write returns the new data (I'd think that would be quite difficult to implement efficiently, if multiple clients are allowed). I couldn't find that being mentioned in either the documentation of juicefs nor the pjdfs test.

Most of network file system provide open-after-close consistency, same to JuiceFS, it's mentioned here [1]:

https://github.com/juicedata/juicefs#posix-compatibility

Re: A distributed Posix file system built on top of Redis and S3

#119

Post by corford »

daviesliu wrote: post #112
corford wrote: post #75

Earlier quoted context omitted.

If a lifecycle rule deletes an object from the bucket, does the redis metadata server gracefully handle this (i.e. bucket state drifting)?

JuiceFS access objects in S3 using an unique id to generate key, so the result of lifecycle will not change the way JuiceFS accessing it, except the StorageClass Glacier or Deep Arching, which is not accessible instantly, will hurt the user experience for JuiceFS.

Thanks. Sorry I wasn't being very clear, I meant: what happens if an object is permanently deleted from a bucket by an out-of-band process, invisible to JuuiceFS (e.g. by a user operating on the bucket directly or by a misconfigured life cycle rule that deletes old objects after X days/months/years)?

Does JuiceFS's metadata server handle this loss of synchronisation gracefully?

Re: A distributed Posix file system built on top of Redis and S3

#120

Post by shaicoleman »

daviesliu wrote: post #111
shaicoleman wrote: post #76

Earlier quoted context omitted.

What programming languages is the SDK available for?

Since JuiceFS written in Go, GO SDK will be the first, then we can have other languages using CGO. We already have Java SDK internally, Python will be the next. But the way, A S3 gateway is on our roadmap, you can spin up an S3 gateway for JuiceFS, and talk to that using existing S3 SDK. This only make sense when you have other applications outside of Lambda using JuiceFS.

After building the Go SDK, I think the next step would be to build a CLI.

That would help dogfood the SDK, and allow it to be used across all languages and environments.

Post reply on HN