Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JuiceFS is a distributed POSIX file system built on top of Redis and S3

github.com

Post reply on HN
Print view View original

81–90 of 119 posts

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#81

Post by suavesu »

staticassertion wrote: post #50

Do people really trust Redis for something like this? I feel like it's sort of pointless to pair Redis with S3 like this, and it'd be better to see benchmarks with metadata stores that can provide actual guarantees for durability/availability. Unfortunately, the benchmarks use Redis. Why would I care about distributed storage on a system like S3, which is all about consistency/durability/availability guarantees, just…

JuiceFS metadata engine comparison -> https://juicefs.com/docs/community/metadata_engines_benchmar...

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#82

Post by suavesu »

ycombinatrix wrote: post #69
staticassertion wrote: post #50

Do people really trust Redis for something like this? I feel like it's sort of pointless to pair Redis with S3 like this, and it'd be better to see benchmarks with metadata stores that can provide actual guarantees for durability/availability. Unfortunately, the benchmarks use Redis. Why would I care about distributed storage on a system like S3, which is all about consistency/durability/availability guarantees, just…

It says MySQL can be used instead of Redis for the metadata

yes, support more 10 options, include redis, SQL-like DB, TiKV, FoundationDB, and more. see here -> https://juicefs.com/docs/community/databases_for_metadata

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#83

Post by suavesu »

c4pt0r wrote: post #74
willbeddow wrote: post #19

Juice is cool, but tradeoffs around which metadata store you choose end up being very important. It also writes files in it's own uninterpretable format to object storage, so if you lose the metadata store, you lose your data. When we tried it at Krea we ended up moving on because we couldn't get sufficient performance to train on, and having to choose which datacenter to deploy our metadata store on essentially forc…

In large-scale metadata scenarios, JFS recommends using a distributed key-value store to host metadata, such as TiKV or FoundationDB. Based on my experience with large JFS users, most of them choose TiKV. Disclaimer: I'm the co-founder of TiKV.

Truth. It supports very large volume, more than 100PiB and 10B files in a single volume.

Disclosure: I'm co-founder of JuiceFS

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#84

Post by IshKebab »

jabl wrote: post #78
IshKebab wrote: post #51

Earlier quoted context omitted.

> In general, file systems make for poor IPC implementations. I agree but also they do have advantages such as simplicity, not needing to explicitly declare which files are needed, lazy data transfer, etc. > you'll also want some mechanism for the writer to notify the reader that it's finished, be it with file locks, or some other entirely different protocol to send signals over the network. The writer is always fini…

> The writer is always finished before the reader starts in these scenarios. The issue is reads on one machine aren't guaranteed to be ordered after writes on a different machine due to write caching. In such a case it should be sufficient to rely on NFS close-to-open consistency as explained in the RFC I linked to in the previous message. Closing a file forces a flush of any dirty data to the server, and opening a f…

It wasn't my job so I didn't look into this fully, but the main issue we had was clients claiming that files didn't exist when they did. I just reread the NFS man page and I guess this is the issue:

> To detect when directory entries have been added or removed on the server, the Linux NFS client watches a directory's mtime. If the client detects a change in a directory's mtime, the client drops all cached LOOKUP results for that directory. Since the directory's mtime is a cached attribute, it may take some time before a client notices it has changed. See the descriptions of the acdirmin, acdirmax, and noac mount options for more information about how long a directory's mtime is cached.

> Caching directory entries improves the performance of applications that do not share files with applications on other clients. Using cached information about directories can interfere with applications that run concurrently on multiple clients and need to detect the creation or removal of files quickly, however. The lookupcache mount option allows some tuning of directory entry caching behavior.

People did talk about using Lustre or GPFS but apparently they are really complex to set up and maybe need fancier networking than ethernet, I don't remember.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#85

Post by suavesu »

ifoxhz wrote: post #60
mrkurt wrote: post #37

Earlier quoted context omitted.

It backs its metadata up to S3. You do need metadata to map inodes / slices / chunks to s3 objects, though. Tigris has a one-to-one FUSE that does what you want: https://github.com/tigrisdata/tigrisfs

FUSE generally has low overall performance because of an additional data transfer process between the kernel space and user space, which is less than ideal for AI training.

The MLPerf performance of JuiceFS last year, FUSE client, and TCP/IP network. There're some analysis about performance and bottleneck. -> https://juicefs.com/en/blog/engineering/mlperf-storage-v2-ai...

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#86

Post by suavesu »

cbarrick wrote: post #44
AdamJacobMuller wrote: post #34

Earlier quoted context omitted.

> It also writes files in it's own uninterpretable format to object storage, so if you lose the metadata store, you lose your data. That's so confusing to me I had to read it five times. Are you saying you lose the metadata, or that the underlying data is actually mangled or gone, or merely that you lose the metadata? One of the greatest features of something like this to me would be the ability to durable even beyon…

As I understand it, if the metadata is lost then the whole filesystem is lost. I think this is a common failure mode in filesystems. For example, in ZFS, if you store your metadata on a separate device and that device is destroyed, the whole pool is useless.

metadata backup is very important. don't forget.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#87

Post by suavesu »

weinzierl wrote: post #76

The consistency guarantees are what makes this interesting in my opinion. > * Close-to-open consistency. Once a file is written and closed, it is guaranteed to view the written data in the following opens and reads from any client. Within the same mount point, all the written data can be read immediately.* > Rename and all other metadata operations are atomic, which are guaranteed by supported metadata engine transac…

Can run MySQL and PG, but don't recommend, not good performance for production. but for temporary it's OK. Here's a case study -> https://juicefs.com/en/blog/user-stories/xiachufang-mysql-ba...

And here's a POSIX compatibility comparison with other cloud file system, like AWS EFS. https://juicefs.com/en/blog/engineering/posix-compatibility-...

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#88

Post by suavesu »

tuhgdetzhh wrote: post #54

If tested various Posix FS projects over the years and everyone has their shortcomings in one way or the other. Although the maintainers of these projects disagree, I mostly consider them as a workaround for smaller projects. For big data (PB range) and critical production workloads I recommend to bite the bullet and make your software nativley S3 compatible without going over a POSIX mounted S3 proxy.

Agree, don't recommend POSIX proxy on S3 for complex workload, like S3FS. In the design of JuiceFS, S3 is like raw disk, JuiceFS metadata engine is like partition table, compare with local file system.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#89

Post by suavesu »

jeffbee wrote: post #39

It is not clear that pjdfstest establishes full POSIX semantic compliance. After a short search of the repo I did not see anything that exercises multiple unrelated processes atomically writing with O_APPEND, for example. And the fact that their graphic shows applications interfacing with JuiceFS over NFS and SMB casts further doubt, since both of those lack many POSIX semantic properties. Over the decades I have wri…

pjdtest result of JuiceFS -> https://juicefs.com/en/blog/engineering/posix-compatibility-...

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#90

Post by ridruejo »

onionjake wrote: post #73
staticassertion wrote: post #50

Do people really trust Redis for something like this? I feel like it's sort of pointless to pair Redis with S3 like this, and it'd be better to see benchmarks with metadata stores that can provide actual guarantees for durability/availability. Unfortunately, the benchmarks use Redis. Why would I care about distributed storage on a system like S3, which is all about consistency/durability/availability guarantees, just…

We developed Object Mount (formerly cunoFS) ( https://www.storj.io/object-mount?hn=1 ) specifically to not rely on any metadata storage other than S3 AND preserve 1:1 mapping of objects to files AND support for POSIX. We have a direct mode that uses LD_PRELOAD to keep everything in userspace so no FUSE overhead. This approach isn't right for every use case and juice might be a better fit for this sort of 'direct bloc…

Pretty cool
Post reply on HN