Can I install redis on it?
Do you mean use JuiceFS as the place to store RDB or AOF? Yes, for backup. The fsync() is much slower than SSD(20ms vs 100us), so it may hurt performance.
A distributed Posix file system built on top of Redis and S3
91–100 of 120 posts
Re: A distributed Posix file system built on top of Redis and S3
#92I've been using s3ql for a while, but this looks great. After testing it for a few hours, the performance seems solid. I'm glad it has a data cache feature like s3ql.
Thanks for verifying that.
I have saw people asking SQL database support for s3ql, to make s3ql accessible from multiple machines, right now, JuiceFS could be a choice.
Re: A distributed Posix file system built on top of Redis and S3
#93We started to build JuiceFS since 2016, released it as a SaaS solution in 2017. After years of improvements, we released the core of JuiceFS recently, hopefully you will find it useful. I'm the founder of JuiceFS, would like to answer any questions here.
What's the recommended Redis configuration for JuiceFS?
Redisraft [1] is a good thing to expect.
Re: A distributed Posix file system built on top of Redis and S3
#94Earlier quoted context omitted.
In AWS, it's yes. The latency of first byte from S3 is about 20-30ms, close to what you can expect from HDD. Ceph is great, if you can master the complexity under the hood, MinIO + Redis + JuiceFS could be the easier answer for beginners.
TTFB in S3 is 20-30ms around the 50th percentile. it can go much higher at p99 [1]. In any case, rotational latency for HDD drives is an order of magnitude lower (typically 2-5ms for a seek operation). S3 is great for higher throughput workloads where TTFB is amortized across larger downloads (this is why it's very common to use S3 as a "data lake" where larger columnar files are stored, usually at the order of hundr…
Agreed, thanks for putting these number together.
JuiceFS was initial designed for big data workload in Cloud, and we have tens of paying customer use it in this use case.
For NAS use case, the latency would be slower when case missed, but the overall IOPS could be higher. In the case of overwhelmed HDD, the latency could also go up to hundreds of milliseconds. We have pay lots of effort to improve the caching (in kernel and on disk) and prefetching. The overall performance is comparable to HDD over NFS.
Re: A distributed Posix file system built on top of Redis and S3
#95this is really cool! using a fast redis for metadata means that suddenly s3 style blob stores become feasible as real networked filesystems. nice!
Thanks, this is the goal we are targeting.
Re: A distributed Posix file system built on top of Redis and S3
#96Earlier quoted context omitted.
The architect of JuiceFS is very close to GFS, which use a single node master for many years, even now. Since Redis is only responsible for metadata, a single node can serve hundreds of millions of files, and tens of thousands of IOPS, that should be enough for many use cases. The term `distributed`, means that JuiceFS is not a `local` file system, or can only be used by single machine. JuiceFS should be qualified as…
> The architect of JuiceFS is very close to GFS, which use a single node master for many years, even now. GFS has since evolved into Colossus which doesn't have this architecture limitation.
Based on some information found on internet, Colossus is built on top of GFS, which use BigTable (or Spanner) as the meta store, and the BigTable still use GFS.
The recent GFS may have multiple masters[1], but they are separate namespace, similar to HDFS federation.
Re: A distributed Posix file system built on top of Redis and S3
#97I've still got PTSD from NFS so I would want to see some really abusive testing to know that this was more reliable than a vendored NFS SAN.
Understood, the annoying part of NFS is that some operation can not be interrupted when the network is not healthy, and we have no way to kill it.
We have put lots of effort to make the operations interruptible when either meta or object store is slow or down. There are a few cases that the `close` can't be interrupted, we can still abort the FUSE connection or kill the JuiceFS process, then all the operation are canceled.
In general, JuiceFS is built on FUSE, so we can have more control on it.
Re: A distributed Posix file system built on top of Redis and S3
#98Am I the only one who always feels uneasy when using NFS-like filesystems? In my experience way too much software has been built without any kind of fault tolerance regarding file system access, and no matter how good your network filesystem is, it still can cause havoc and every sort of data loss. I've seen so many disasters related to software basically assuming a file can't just vanish into thin air, something tha…
I learned not to use Dropbox for things like Git repos. I'd have a fresh copy on one computer, and then have an older copy on a different computer. Sometimes I'd accidentally overwrite the newer code w/ autosaving on the older code. Now I'm skeptical that I'll be able to keep everything properly in sync on any sort of network-ish file system that is wrestling for control with my local file system.
I'd have thought the very point of git would have negated the need for Dropbox or network file systems.
Re: A distributed Posix file system built on top of Redis and S3
#99We started to build JuiceFS since 2016, released it as a SaaS solution in 2017. After years of improvements, we released the core of JuiceFS recently, hopefully you will find it useful. I'm the founder of JuiceFS, would like to answer any questions here.
Congratulations on picking Go instead of being yet another C project.
Good luck with the project.
Re: A distributed Posix file system built on top of Redis and S3
#100Am I the only one who always feels uneasy when using NFS-like filesystems? In my experience way too much software has been built without any kind of fault tolerance regarding file system access, and no matter how good your network filesystem is, it still can cause havoc and every sort of data loss. I've seen so many disasters related to software basically assuming a file can't just vanish into thin air, something tha…
My instinct is with yours, but in some very simple use cases, it can make sense.
For example, you've got a legacy PHP web application you have to maintain, and it's got spaghetti code all over the place, but all uploaded user content is stored / served from a single directory. You can probably use an S3-backed file system to replace that directory.
Obviously if you try to do, say, an "ls -la" on that directory and there are a lot of files in there, you may be waiting a while, since that translates to a lot of API calls. Similarly if you have something like a virus scan running on the box, it's likely to be tripped up.
But if you know the only thing using that directory is doing simple CRUD operations, it might be easier than trying to retrofit the application itself to talk to S3. Especially now that S3 has strong consistency.