A distributed Posix file system built on top of Redis and S3

FUSE is not support by AWS Lambda, so we can't mount JuiceFS in Lambda.

We can have a SDK to access JuiceFS from Lambda, similar to S3 SDK, when you need to use JuiceFS outside of Lambda.

Same to Fargate, we can not mount JuiceFS in Farget because of lacking FUSE permission, people are asking for it[1].

https://github.com/aws/containers-roadmap/issues/412

A zone transfer or grep really isn't so terrible, even for someone who doesn't know much about dns it's just a quick memorized dig away. HTTP or grpc aren't really better CLI UX and if this is your infra you can just write a cli to display your dns-driven information cleanly.

Storing key-value pairs in TXT records is easily parseable in any language if you use a simple delimiter. This isn't even an abuse of DNS, the protocol has been used to serve arbitrary data forever.

All the other modern services like consul, etcd, zk, etc require keeping a quorum of servers happy and have pretty heavy clients. By contrast, it's so hard to take down DNS and spinning up a new server is as easy as copying a zone file to a new server and the new server could even be running a totally different dns implementation because zone files are so standardized. Plus, your tooling can directly parse zone files with whatever dns library you were using and have a trivial way to dump the data without any server at all.

DNS can be replicated in arbitrary configurations and everything supports DNS caching for really high HA.

When you do want dynamic discovery and don't want to implement direct zone file generation, there's always CoreDNS which has plugins for so many datasources.

And if you don't want to host it, there are tons of DNS providers with great uptime.

At this point, I really can't think of any solution for service discovery that's better than DNS for most cases. Especially since the majority of service discovery solutions end up returning hostnames instead of IP addresses so you're already taking a dependency on DNS. Other solutions only really add benefit if you need to store tons of metadata or take advantage of things like leader election, etc

> If you’re writing new apps, I don’t see why you would want to add another abstraction layer

I can see the usefulness in basing your app on FS and other POSIXly primitives (as opposed to the "cloud-native" storage du jour) if you want your app to continue to be usable on the largest class of machines and scenarios including local deployment under traditional Unix site autonomy assumptions. The general purpose being portability, need for on-premise deployment, (very) long-term viability, developer experience, accountability, integration with legacy software and permission infrastructure, use of existing upload/download or VCS software, straightforward file or metadata exchange, forensic or academic transparency, and avoidance of lock-in.

We were doing this at Avere in 2015. The system built a POSIX filesystem out of S3 objects on the backend (including metadata) and then served it over NFS or SMB from a cluster of cache nodes. Keeping metadata and data in separate data stores with different consistency models is a disaster waiting to happen - ask anyone who has run Lustre. Having fast caches with SSDs was the key to getting any kind of decent performance out of S3. The fun part was mounting a filesystem and running df and seeing an exabyte of capacity. They were acquired by Microsoft in 2018 and integrated into Azure.

I saw Avere was recommended in GCP before, but never find the details, thanks for sharing that. It seems that Avere is close to ObjectiveFS, which also use S3 both for data and metadata.

My guest is that Avere could require a cluster of nodes as the fast layer for write and synchronization. In JuiceFS, Redis is used for synchronization and persisting metadata. Local SSD could used for data read-caching, not for writing.

JuiceFS is in production for 3+ years, we have not find much challenge that was difficult with this design, since it's borrowed from GFS, HDFS and MooseFS, have be proved in production for more than 10 years. I'd like to hear what's the challenge you were facing.