Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JuiceFS is a distributed POSIX file system built on top of Redis and S3

github.com

Post reply on HN
Print view View original

91–100 of 119 posts

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#91

Post by eerikkivistik »

I've had to test out various networked filesystems this year for a few use cases (satellite/geo) on a multi petabyte scale. Some of my thoughts:

* JuiceFS - Works well, for high performance it has limited use cases where privacy concerns matter. There is the open source version, which is slower. The metadata backend selection really matters if you are tuning for latency.

* Lustre - Heavily optimised for latency. Gets very expensive if you need more bandwidth, as it is tiered and tied to volume sizes. Managed solutions available pretty much everywhere.

* EFS - Surprisingly good these days, still insanely expensive. Useful for small amounts of data (few terabytes).

* FlexFS - An interesting beast. It murders on bandwidth/cost. But slightly loses on latency sensitive operations. Great if you have petabyte scale data and need to parallel process it. But struggles when you have tooling that does many small unbuffered writes.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#92

Post by suavesu »

__turbobrew__ wrote: post #80
ChocolateGod wrote: post #18

Earlier quoted context omitted.

Let's remember that JuiceFS can be setup very easily to not have a single point of failure (by replicating the metadata engine), meanwhile ZeroFS seems to have exactly that. If I was a company I know which one I'd prefer.

Yea, that is a big caveat to ZeroFS. Single point of failure. It is like saying I can write a faster etcd by only having a single node. Sure, that is possible, but the hard part of distributed systems is the coordination, and coordination always makes performance worse. I personally have went with Ceph for distributed storage. I personally have a lot more confidence in Ceph over JuiceFS and ZeroFS, but realize buildi…

Some users use JuiceFS with CephFS RADOS, as alternative with Ceph MDS.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#93

Post by mickael-kerjean »

eerikkivistik wrote: post #91

I've had to test out various networked filesystems this year for a few use cases (satellite/geo) on a multi petabyte scale. Some of my thoughts: * JuiceFS - Works well, for high performance it has limited use cases where privacy concerns matter. There is the open source version, which is slower. The metadata backend selection really matters if you are tuning for latency. * Lustre - Heavily optimised for latency. Gets…

Nothing around content addressable storage? Has anyone used something like IPFS / Kubo in production at that kind of scale?

(for those who don't know IPFS, I find the original paper fascinating: https://arxiv.org/pdf/1407.3561)

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#94

Post by ChocolateGod »

__turbobrew__ wrote: post #80
ChocolateGod wrote: post #18

Earlier quoted context omitted.

Let's remember that JuiceFS can be setup very easily to not have a single point of failure (by replicating the metadata engine), meanwhile ZeroFS seems to have exactly that. If I was a company I know which one I'd prefer.

Yea, that is a big caveat to ZeroFS. Single point of failure. It is like saying I can write a faster etcd by only having a single node. Sure, that is possible, but the hard part of distributed systems is the coordination, and coordination always makes performance worse. I personally have went with Ceph for distributed storage. I personally have a lot more confidence in Ceph over JuiceFS and ZeroFS, but realize buildi…

I replaced a GlusterFS cluster with JuiceFS some years ago and it's been a relief. Just much easier to manage.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#95

Post by lstodd »

IshKebab wrote: post #84
jabl wrote: post #78

Earlier quoted context omitted.

> The writer is always finished before the reader starts in these scenarios. The issue is reads on one machine aren't guaranteed to be ordered after writes on a different machine due to write caching. In such a case it should be sufficient to rely on NFS close-to-open consistency as explained in the RFC I linked to in the previous message. Closing a file forces a flush of any dirty data to the server, and opening a f…

It wasn't my job so I didn't look into this fully, but the main issue we had was clients claiming that files didn't exist when they did. I just reread the NFS man page and I guess this is the issue: > To detect when directory entries have been added or removed on the server, the Linux NFS client watches a directory's mtime. If the client detects a change in a directory's mtime, the client drops all cached LOOKUP resu…

I did set up GPFS tadam... almost exactly 20 years ago. I wouldn't say it absolutely required fancy networking (infiniband) or was extraordinary complex to set up, certainly on par with NFS when you hit its quirks (which was the reason we went off experimenting with gpfs and whatnot).

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#96

Post by romantomjak »

eerikkivistik wrote: post #91

I've had to test out various networked filesystems this year for a few use cases (satellite/geo) on a multi petabyte scale. Some of my thoughts: * JuiceFS - Works well, for high performance it has limited use cases where privacy concerns matter. There is the open source version, which is slower. The metadata backend selection really matters if you are tuning for latency. * Lustre - Heavily optimised for latency. Gets…

Did you happen to look into CephFS? CERN (folks that operate Large Hadron Collider) use it to store ~30PB of scientific data. Their analysis cluster is serving ~30GB/s reads

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#97

Post by alexhornby »

fpoling wrote: post #79
tuhgdetzhh wrote: post #53

Earlier quoted context omitted.

I think they should replace Redis with Valkey or even better use rocksdb.

But how would RocksDB work with S3? It needs support for append that generic S3 buckets do not provide and for checkpoints and backup it assumes the support for hardlinks that S3 does not have at all.

[dead]

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#98

Post by lifty »

onionjake wrote: post #73
staticassertion wrote: post #50

Do people really trust Redis for something like this? I feel like it's sort of pointless to pair Redis with S3 like this, and it'd be better to see benchmarks with metadata stores that can provide actual guarantees for durability/availability. Unfortunately, the benchmarks use Redis. Why would I care about distributed storage on a system like S3, which is all about consistency/durability/availability guarantees, just…

We developed Object Mount (formerly cunoFS) ( https://www.storj.io/object-mount?hn=1 ) specifically to not rely on any metadata storage other than S3 AND preserve 1:1 mapping of objects to files AND support for POSIX. We have a direct mode that uses LD_PRELOAD to keep everything in userspace so no FUSE overhead. This approach isn't right for every use case and juice might be a better fit for this sort of 'direct bloc…

Is this open source? I am a happy Storj customer, would love to use it if it's open source.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#99

Post by adamcharnock »

We've been using JuiceFS in production for a few months now and I'm a big fan. I've felt for a while that block-level filesystems do not adapt at all well to being implemented across a network (with my personal experience being of AWS EBS and OpenEBS Mayastor). So the fact that JuiceFS is interfaces at the POSIX layer felt intuitively better to me.

I also like that it can keep a local read-cache, rather than having to hit object storage for every read. This is because it can perform a freshness check with with the (relatively fast) metadata store to determine if its cached data is valid, prior to serving the request from cache.

We back it with a 3-node (Redis-compatible) HA Valkey cluster, and in-cluster MinIO object storage, all in bare-metal Kubernetes. We can saturate a 25g NIC with (IIRC) 16+ concurrent users.

It is also one of the few Kubernetes storage providers that provides read-write-many (RWX) access. Which can also be rather helpful in some situations.

In an early test we were running it against MinIO with zero redundancy. Which is not recommended in any case. There we did see some file corruption creep in. In which case some files in JuiceFS become unreadable, but the system as a whole kept working.

Another reason I think JuiceFS works well, is indeed because of its custom block-based storage format. It is disconcerting because you cannot see your files in object storage, but instead just a lot of chunks. But this does buy some real performance benefits, especially when doing partial file reads or updates.

Another test we're doing is running a small-to-medium sized Prometheus persisted to JuiceFS. It hasn't shown any issues so far.

And, if you've made it this, far: check us out if you want a hand installing and operating this kind of infra: https://lithus.eu . We deploy to bare-metal Hetzner.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#100

Post by eerikkivistik »

romantomjak wrote: post #96
eerikkivistik wrote: post #91

I've had to test out various networked filesystems this year for a few use cases (satellite/geo) on a multi petabyte scale. Some of my thoughts: * JuiceFS - Works well, for high performance it has limited use cases where privacy concerns matter. There is the open source version, which is slower. The metadata backend selection really matters if you are tuning for latency. * Lustre - Heavily optimised for latency. Gets…

Did you happen to look into CephFS? CERN (folks that operate Large Hadron Collider) use it to store ~30PB of scientific data. Their analysis cluster is serving ~30GB/s reads

Sure, so the use case I have requires elastic storage and elastic compute. So CephFS really isn't a good fit in the cloud environment for that case. It would get prohibitively expensive.
Post reply on HN