probably one of the few cases where agpl is not that bad.
A distributed Posix file system built on top of Redis and S3
71–80 of 120 posts
Re: A distributed Posix file system built on top of Redis and S3
#72Earlier quoted context omitted.
A low latency metadata server is good, but you still have the latency to the S3 server, e.g. Amazon S3 (from your README.md). Now you could say, that I could host my own S3 e.g. MinIO in the same DC, but then I could also simply deploy Ceph, which is battle tested for years up to the petabyte range and with iSCSI, S3 and FS interfaces. So I think this project might give the wrong impression that you can simply combin…
In AWS, it's yes. The latency of first byte from S3 is about 20-30ms, close to what you can expect from HDD. Ceph is great, if you can master the complexity under the hood, MinIO + Redis + JuiceFS could be the easier answer for beginners.
TTFB in S3 is 20-30ms around the 50th percentile. it can go much higher at p99 [1]. In any case, rotational latency for HDD drives is an order of magnitude lower (typically 2-5ms for a seek operation).
S3 is great for higher throughput workloads where TTFB is amortized across larger downloads (this is why it's very common to use S3 as a "data lake" where larger columnar files are stored, usually at the order of hundreds of MiB).
I think it's an interesting project but perhaps explaining the use cases where this solution is beneficial would go a long way here.
Re: A distributed Posix file system built on top of Redis and S3
#73Is metadata also replicated in S3? Else I don't understand how the metadata can be persistent after reboot as AFAIK redis cannot dump and reload its state.
Redis can be persisted with RDB and AOF, can also be replicated to another machine. In the cloud, you don't need to worry about that, hosted Redis are ready to use. The is an ongoing effort [1] to improve the persistency and availability in general, which is expected to be GA in 2021. [1] https://github.com/RedisLabs/redisraft
given a lost write to Redis would translate to corruption or missing data at the filesystem level, the only "safe" way to run this ATM is using Redis' extremely inefficient "always fsync" setting for its AOF log [1].
Keep in mind that Elasticache does not support it (in general, it doesn't really support running Redis in a durable way).
[1] https://redis.io/topics/persistence#:~:text=The%20suggested%....
Re: A distributed Posix file system built on top of Redis and S3
#74I've still got PTSD from NFS so I would want to see some really abusive testing to know that this was more reliable than a vendored NFS SAN.
Re: A distributed Posix file system built on top of Redis and S3
#75How much does it cost to use on S3? For example, how many GetObject and other misc non-free API calls does it use? And does it store data using intelligent tiering?
The underlying S3 bucket still have intelligent tiering, you can also put life cycle rules on it.
If a lifecycle rule deletes an object from the bucket, does the redis metadata server gracefully handle this (i.e. bucket state drifting)?
Re: A distributed Posix file system built on top of Redis and S3
#76Would there be any way to mount JuiceFS from AWS Lambda or Fargate?
FUSE is not support by AWS Lambda, so we can't mount JuiceFS in Lambda. We can have a SDK to access JuiceFS from Lambda, similar to S3 SDK, when you need to use JuiceFS outside of Lambda. Same to Fargate, we can not mount JuiceFS in Farget because of lacking FUSE permission, people are asking for it[1]. https://github.com/aws/containers-roadmap/issues/412
What programming languages is the SDK available for?
Re: A distributed Posix file system built on top of Redis and S3
#77Am I the only one who always feels uneasy when using NFS-like filesystems? In my experience way too much software has been built without any kind of fault tolerance regarding file system access, and no matter how good your network filesystem is, it still can cause havoc and every sort of data loss.
I've seen so many disasters related to software basically assuming a file can't just vanish into thin air, something that can very much happen when your FS is running on top of an arbitrary network connection. Hiding away such a fundamental detail in order to provide a file-like API tends to instill every sort of bad ideas into people (NFS via Wi-Fi? Why not!).
Re: A distributed Posix file system built on top of Redis and S3
#78Am I the only one who always feels uneasy when using NFS-like filesystems? In my experience way too much software has been built without any kind of fault tolerance regarding file system access, and no matter how good your network filesystem is, it still can cause havoc and every sort of data loss. I've seen so many disasters related to software basically assuming a file can't just vanish into thin air, something tha…
I learned not to use Dropbox for things like Git repos. I'd have a fresh copy on one computer, and then have an older copy on a different computer. Sometimes I'd accidentally overwrite the newer code w/ autosaving on the older code. Now I'm skeptical that I'll be able to keep everything properly in sync on any sort of network-ish file system that is wrestling for control with my local file system.
Re: A distributed Posix file system built on top of Redis and S3
#79We started to build JuiceFS since 2016, released it as a SaaS solution in 2017. After years of improvements, we released the core of JuiceFS recently, hopefully you will find it useful. I'm the founder of JuiceFS, would like to answer any questions here.
Hi. I'm pretty interested and excited about this project. Under "Credits" the project states:
>"The design of JuiceFS was inspired by Google File System, HDFS and MooseFS, thanks to their great work."
Would you consider writing up a design doc for JuiceFS. I would be interested to know more about what specific implementation ideas you used for each of those if any, design choices and tradeoffs made, learnings etc. It would make a great blog post. Cheers.
Re: A distributed Posix file system built on top of Redis and S3
#80We started to build JuiceFS since 2016, released it as a SaaS solution in 2017. After years of improvements, we released the core of JuiceFS recently, hopefully you will find it useful. I'm the founder of JuiceFS, would like to answer any questions here.
What's the recommended Redis configuration for JuiceFS?