FS metadata stored in Redis? What will happen if Redis is down?
A distributed Posix file system built on top of Redis and S3
101–110 of 120 posts
Re: A distributed Posix file system built on top of Redis and S3
#102We were doing this at Avere in 2015. The system built a POSIX filesystem out of S3 objects on the backend (including metadata) and then served it over NFS or SMB from a cluster of cache nodes. Keeping metadata and data in separate data stores with different consistency models is a disaster waiting to happen - ask anyone who has run Lustre. Having fast caches with SSDs was the key to getting any kind of decent perform…
I saw Avere was recommended in GCP before, but never find the details, thanks for sharing that. It seems that Avere is close to ObjectiveFS, which also use S3 both for data and metadata. My guest is that Avere could require a cluster of nodes as the fast layer for write and synchronization. In JuiceFS, Redis is used for synchronization and persisting metadata. Local SSD could used for data read-caching, not for writi…
The nice thing about storing everything including metadata in S3 is that you have a consistent copy of your filesystem. How do you back up a filesystem that is split across Redis and S3 for example? I was able to blow away the cluster running it, spin up a new one, point it at the bucket and add the encryption key, and it would bring up a working filesystem. Just like plugging in a hard drive.
Re: A distributed Posix file system built on top of Redis and S3
#103We were doing this at Avere in 2015. The system built a POSIX filesystem out of S3 objects on the backend (including metadata) and then served it over NFS or SMB from a cluster of cache nodes. Keeping metadata and data in separate data stores with different consistency models is a disaster waiting to happen - ask anyone who has run Lustre. Having fast caches with SSDs was the key to getting any kind of decent perform…
>"Keeping metadata and data in separate data stores with different consistency models is a disaster waiting to happen - ask anyone who has run Lustre." Can you elaborate - is the issue corruption or performance? I've never used Lustre.
Lustre is all about performance, at the cost of being split across multiple distributed systems. So there are opportunities to have different components get out of sync - the metadata servers are usually the weak link. And the more moving parts the harder it is to debug when something goes wrong.
Re: A distributed Posix file system built on top of Redis and S3
#104We started to build JuiceFS since 2016, released it as a SaaS solution in 2017. After years of improvements, we released the core of JuiceFS recently, hopefully you will find it useful. I'm the founder of JuiceFS, would like to answer any questions here.
Congratulations for this work! Do you have some documentation describing how the metadata is modelled in Redis?
Re: A distributed Posix file system built on top of Redis and S3
#105Earlier quoted context omitted.
Hi. I'm pretty interested and excited about this project. Under "Credits" the project states: >"The design of JuiceFS was inspired by Google File System, HDFS and MooseFS, thanks to their great work." Would you consider writing up a design doc for JuiceFS. I would be interested to know more about what specific implementation ideas you used for each of those if any, design choices and tradeoffs made, learnings etc. It…
The whole idea was came from GFS: separate the metadata and data, load all the meta into memory, single meta server for simplicity, fixed-size chunk. The POSIX and FUSE stuff was learned from MooseFS, but changed to use read-only chunk, and merge them together, and do compaction in background. Since most of object storage provide eventual consistency, the model work pretty well, also simplify the burden on cache evic…
Thanks. Is the Hadoop SDK part of the official Hadoop project? I wasn't able to find any information about it.
Re: A distributed Posix file system built on top of Redis and S3
#106We started to build JuiceFS since 2016, released it as a SaaS solution in 2017. After years of improvements, we released the core of JuiceFS recently, hopefully you will find it useful. I'm the founder of JuiceFS, would like to answer any questions here.
Congratulations for this work! Do you have some documentation describing how the metadata is modelled in Redis?
They are summaried here: https://github.com/juicedata/juicefs/blob/main/pkg/redis/red...
Re: A distributed Posix file system built on top of Redis and S3
#107Earlier quoted context omitted.
The whole idea was came from GFS: separate the metadata and data, load all the meta into memory, single meta server for simplicity, fixed-size chunk. The POSIX and FUSE stuff was learned from MooseFS, but changed to use read-only chunk, and merge them together, and do compaction in background. Since most of object storage provide eventual consistency, the model work pretty well, also simplify the burden on cache evic…
Thanks. Is the Hadoop SDK part of the official Hadoop project? I wasn't able to find any information about it.
No, The Hadoop SDK is a connector to access JuiceFS, is built using JuiceFS client, will be open sourced later on, stay tuned.
Re: A distributed Posix file system built on top of Redis and S3
#108Earlier quoted context omitted.
> The architect of JuiceFS is very close to GFS, which use a single node master for many years, even now. GFS has since evolved into Colossus which doesn't have this architecture limitation.
Based on some information found on internet, Colossus is built on top of GFS, which use BigTable (or Spanner) as the meta store, and the BigTable still use GFS. The recent GFS may have multiple masters[1], but they are separate namespace, similar to HDFS federation. [1] https://queue.acm.org/detail.cfm?id=1594206
Based on experience - no, you are mistaken. Wish I could say more.
Re: A distributed Posix file system built on top of Redis and S3
#109Earlier quoted context omitted.
I saw Avere was recommended in GCP before, but never find the details, thanks for sharing that. It seems that Avere is close to ObjectiveFS, which also use S3 both for data and metadata. My guest is that Avere could require a cluster of nodes as the fast layer for write and synchronization. In JuiceFS, Redis is used for synchronization and persisting metadata. Local SSD could used for data read-caching, not for writi…
The nice thing about storing everything including metadata in S3 is that you have a consistent copy of your filesystem. How do you back up a filesystem that is split across Redis and S3 for example? I was able to blow away the cluster running it, spin up a new one, point it at the bucket and add the encryption key, and it would bring up a working filesystem. Just like plugging in a hard drive.
JuiceFS has a single node version working like this, it store in the metadata in SQLite and upload S3 in background, it's useful in some cases but not in general, you may lose recent metadata if the single node is broken.
The difficulty of using S3 as metadata is that they is not way to persistent metadata under 1 ms, for example, created a symlink, it will take more than 20ms, or you may lose it.
With en external persisted database, we could have the ACID for metadata operations. Also, the meta is the source of truth, whenever the object store is out of sync (losing or leaking a object), the whole file system is still consistent, rather than part of a file is corrupted, should be safer than put everything into S3.
The database is the key, redis is our first choice, we will add support to other databases in future.
Re: A distributed Posix file system built on top of Redis and S3
#110Earlier quoted context omitted.
Based on some information found on internet, Colossus is built on top of GFS, which use BigTable (or Spanner) as the meta store, and the BigTable still use GFS. The recent GFS may have multiple masters[1], but they are separate namespace, similar to HDFS federation. [1] https://queue.acm.org/detail.cfm?id=1594206
Based on experience - no, you are mistaken. Wish I could say more.
Good to know, I'm always curious on Colossus.