81–90 of 120 posts
We started to build JuiceFS since 2016, released it as a SaaS solution in 2017. After years of improvements, we released the core of JuiceFS recently, hopefully you will find it useful. I'm the founder of JuiceFS, would like to answer any questions here.
2. Is there any access control and if there is who enforces it?
We were doing this at Avere in 2015. The system built a POSIX filesystem out of S3 objects on the backend (including metadata) and then served it over NFS or SMB from a cluster of cache nodes. Keeping metadata and data in separate data stores with different consistency models is a disaster waiting to happen - ask anyone who has run Lustre. Having fast caches with SSDs was the key to getting any kind of decent perform…
Can you elaborate - is the issue corruption or performance? I've never used Lustre.
Earlier quoted context omitted.
OOC, what are the downsides of storing service discovery information like this in TXT records? As opposed to using ZK, consul, etcd or a more standard solution?
There are better tools for the job now. This is basically a misuse of a service instead of using things that are purpose built. I mean, your inventory is either a grep or a zone transfer this way, for one.
Am I the only one who always feels uneasy when using NFS-like filesystems? In my experience way too much software has been built without any kind of fault tolerance regarding file system access, and no matter how good your network filesystem is, it still can cause havoc and every sort of data loss. I've seen so many disasters related to software basically assuming a file can't just vanish into thin air, something tha…
I learned not to use Dropbox for things like Git repos. I'd have a fresh copy on one computer, and then have an older copy on a different computer. Sometimes I'd accidentally overwrite the newer code w/ autosaving on the older code. Now I'm skeptical that I'll be able to keep everything properly in sync on any sort of network-ish file system that is wrestling for control with my local file system.
We started to build JuiceFS since 2016, released it as a SaaS solution in 2017. After years of improvements, we released the core of JuiceFS recently, hopefully you will find it useful. I'm the founder of JuiceFS, would like to answer any questions here.
1. Does JuiceFS support TLS encrypted Redis? 2. Is there any access control and if there is who enforces it?
Yes, please use `rediss://host:port/`
> 2. Is there any access control and if there is who enforces it?
You may specify port 0 to disable the non-TLS port completely. To enable only TLS on the default Redis port, use:
-port 0 -tls-port 6379
We started to build JuiceFS since 2016, released it as a SaaS solution in 2017. After years of improvements, we released the core of JuiceFS recently, hopefully you will find it useful. I'm the founder of JuiceFS, would like to answer any questions here.
What's the recommended Redis configuration for JuiceFS?
If you run it on your own, please pay attention on the persistency options and HA solution, there should be plenty of article on these.
We started to build JuiceFS since 2016, released it as a SaaS solution in 2017. After years of improvements, we released the core of JuiceFS recently, hopefully you will find it useful. I'm the founder of JuiceFS, would like to answer any questions here.
Hi. I'm pretty interested and excited about this project. Under "Credits" the project states: >"The design of JuiceFS was inspired by Google File System, HDFS and MooseFS, thanks to their great work." Would you consider writing up a design doc for JuiceFS. I would be interested to know more about what specific implementation ideas you used for each of those if any, design choices and tradeoffs made, learnings etc. It…
The POSIX and FUSE stuff was learned from MooseFS, but changed to use read-only chunk, and merge them together, and do compaction in background. Since most of object storage provide eventual consistency, the model work pretty well, also simplify the burden on cache eviction. In order to access object store in parallel, we divide the chunk into smaller blocks (4MB), which is also a good unit for caching.
The Hadoop SDK (not released yet) was learned from HDFS.
One key thing in the implementation is to use Redis transaction to guarantee atomicy on metadata operations, otherwise we will get into millions of random bugs.
Is metadata also replicated in S3? Else I don't understand how the metadata can be persistent after reboot as AFAIK redis cannot dump and reload its state.
Redis can be persisted with RDB and AOF, can also be replicated to another machine. In the cloud, you don't need to worry about that, hosted Redis are ready to use. The is an ongoing effort [1] to improve the persistency and availability in general, which is expected to be GA in 2021. [1] https://github.com/RedisLabs/redisraft