Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JuiceFS is a distributed POSIX file system built on top of Redis and S3

github.com

Post reply on HN
Print view View original

71–80 of 119 posts

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#71

Post by [deleted] »

AdamJacobMuller wrote: post #34
willbeddow wrote: post #19

Juice is cool, but tradeoffs around which metadata store you choose end up being very important. It also writes files in it's own uninterpretable format to object storage, so if you lose the metadata store, you lose your data. When we tried it at Krea we ended up moving on because we couldn't get sufficient performance to train on, and having to choose which datacenter to deploy our metadata store on essentially forc…

> It also writes files in it's own uninterpretable format to object storage, so if you lose the metadata store, you lose your data. That's so confusing to me I had to read it five times. Are you saying you lose the metadata, or that the underlying data is actually mangled or gone, or merely that you lose the metadata? One of the greatest features of something like this to me would be the ability to durable even beyon…

[deleted]

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#72

Post by seany »

satoru42 wrote: post #68
hsn915 wrote: post #67

This is upside down. We need a kernel native distributed file system so that we can build distributed storage/databases on top of it. This is like building an operating system on top of a browser.

Show me an operating system built on top of a browser that can be used to solve real-world problems like JuiceFS.

https://github.com/tractordev/apptron ?

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#73

Post by onionjake »

staticassertion wrote: post #50

Do people really trust Redis for something like this? I feel like it's sort of pointless to pair Redis with S3 like this, and it'd be better to see benchmarks with metadata stores that can provide actual guarantees for durability/availability. Unfortunately, the benchmarks use Redis. Why would I care about distributed storage on a system like S3, which is all about consistency/durability/availability guarantees, just…

We developed Object Mount (formerly cunoFS) (https://www.storj.io/object-mount?hn=1) specifically to not rely on any metadata storage other than S3 AND preserve 1:1 mapping of objects to files AND support for POSIX. We have a direct mode that uses LD_PRELOAD to keep everything in userspace so no FUSE overhead.

This approach isn't right for every use case and juice might be a better fit for this sort of 'direct block store', but wanted to include it here for folks that might want something like Juice but without having to maintain a metadata store.

(Disclosure: I work at Storj that develops Object Mount)

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#74

Post by c4pt0r »

willbeddow wrote: post #19

Juice is cool, but tradeoffs around which metadata store you choose end up being very important. It also writes files in it's own uninterpretable format to object storage, so if you lose the metadata store, you lose your data. When we tried it at Krea we ended up moving on because we couldn't get sufficient performance to train on, and having to choose which datacenter to deploy our metadata store on essentially forc…

In large-scale metadata scenarios, JFS recommends using a distributed key-value store to host metadata, such as TiKV or FoundationDB. Based on my experience with large JFS users, most of them choose TiKV.

Disclaimer: I'm the co-founder of TiKV.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#75

Post by sroerick »

hsn915 wrote: post #67

This is upside down. We need a kernel native distributed file system so that we can build distributed storage/databases on top of it. This is like building an operating system on top of a browser.

I'm glad you noticed this, I thought this was a wildly insane thing to do. Its like the satanic inversion of 9P protocol

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#76

Post by weinzierl »

The consistency guarantees are what makes this interesting in my opinion.

> * Close-to-open consistency. Once a file is written and closed, it is guaranteed to view the written data in the following opens and reads from any client. Within the same mount point, all the written data can be read immediately.*

> Rename and all other metadata operations are atomic, which are guaranteed by supported metadata engine transaction.

This is a lot more than other "POSIX compatible" overlays claim, and I think similar to what NFSv4 promises. There are lots of subtitles there, though, and I doubt you could safely run a database on it.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#77

Post by hsn915 »

satoru42 wrote: post #68
hsn915 wrote: post #67

This is upside down. We need a kernel native distributed file system so that we can build distributed storage/databases on top of it. This is like building an operating system on top of a browser.

Show me an operating system built on top of a browser that can be used to solve real-world problems like JuiceFS.

My criticism is of the basic architecture, not usability or fitness for a particular purpose.

If a distributed file system is useful, then a properly architectured one is 100x more useful and more performant.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#78

Post by jabl »

IshKebab wrote: post #51
jabl wrote: post #35

Earlier quoted context omitted.

> poor locking support (this sounds like it works better) File locking on Unix is in general a clusterf*ck. (There was a thread a few days ago at https://news.ycombinator.com/item?id=46542247 ) > no manual fence support; a bad but common way of distributing workloads is e.g. to compile a test on one machine (on an NFS mount), and then use SLURM or SGE to run the test on other machines. You use NFS to let the other ma…

> In general, file systems make for poor IPC implementations. I agree but also they do have advantages such as simplicity, not needing to explicitly declare which files are needed, lazy data transfer, etc. > you'll also want some mechanism for the writer to notify the reader that it's finished, be it with file locks, or some other entirely different protocol to send signals over the network. The writer is always fini…

> The writer is always finished before the reader starts in these scenarios. The issue is reads on one machine aren't guaranteed to be ordered after writes on a different machine due to write caching.

In such a case it should be sufficient to rely on NFS close-to-open consistency as explained in the RFC I linked to in the previous message. Closing a file forces a flush of any dirty data to the server, and opening a file forces a revalidation of any cached content.

If that doesn't work, your NFS is broken. ;-)

And if you need 'proper' cache coherency, something like Lustre is an option.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#79

Post by fpoling »

tuhgdetzhh wrote: post #53
staticassertion wrote: post #50

Do people really trust Redis for something like this? I feel like it's sort of pointless to pair Redis with S3 like this, and it'd be better to see benchmarks with metadata stores that can provide actual guarantees for durability/availability. Unfortunately, the benchmarks use Redis. Why would I care about distributed storage on a system like S3, which is all about consistency/durability/availability guarantees, just…

I think they should replace Redis with Valkey or even better use rocksdb.

But how would RocksDB work with S3? It needs support for append that generic S3 buckets do not provide and for checkpoints and backup it assumes the support for hardlinks that S3 does not have at all.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#80

Post by __turbobrew__ »

ChocolateGod wrote: post #18
Eikon wrote: post #3

ZeroFS [0] outperforms JuiceFS on common small file workloads [1] while only requiring S3 and no 3rd party database. [0] https://github.com/Barre/ZeroFS [1] https://www.zerofs.net/zerofs-vs-juicefs

Let's remember that JuiceFS can be setup very easily to not have a single point of failure (by replicating the metadata engine), meanwhile ZeroFS seems to have exactly that. If I was a company I know which one I'd prefer.

Yea, that is a big caveat to ZeroFS. Single point of failure. It is like saying I can write a faster etcd by only having a single node. Sure, that is possible, but the hard part of distributed systems is the coordination, and coordination always makes performance worse.

I personally have went with Ceph for distributed storage. I personally have a lot more confidence in Ceph over JuiceFS and ZeroFS, but realize building and running a ceph cluster is more complex, but with that complexity you get much cheaper S3, block storage, and cephfs.

Post reply on HN