Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JuiceFS is a distributed POSIX file system built on top of Redis and S3

github.com

Post reply on HN
Print view View original

101–110 of 119 posts

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#101

Post by eerikkivistik »

mickael-kerjean wrote: post #93
eerikkivistik wrote: post #91

I've had to test out various networked filesystems this year for a few use cases (satellite/geo) on a multi petabyte scale. Some of my thoughts: * JuiceFS - Works well, for high performance it has limited use cases where privacy concerns matter. There is the open source version, which is slower. The metadata backend selection really matters if you are tuning for latency. * Lustre - Heavily optimised for latency. Gets…

Nothing around content addressable storage? Has anyone used something like IPFS / Kubo in production at that kind of scale? (for those who don't know IPFS, I find the original paper fascinating: https://arxiv.org/pdf/1407.3561 )

The latency and bandwidth really isn't there for HPC.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#102

Post by hn92726819 »

adamcharnock wrote: post #99

We've been using JuiceFS in production for a few months now and I'm a big fan. I've felt for a while that block-level filesystems do not adapt at all well to being implemented across a network (with my personal experience being of AWS EBS and OpenEBS Mayastor). So the fact that JuiceFS is interfaces at the POSIX layer felt intuitively better to me. I also like that it can keep a local read-cache, rather than having t…

> There we did see some file corruption creep in

Did you figure out what caused corruption? Was minio losing blocks or was juicefs corrupted even though minio was consistent?

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#103

Post by adamcharnock »

hn92726819 wrote: post #102
adamcharnock wrote: post #99

We've been using JuiceFS in production for a few months now and I'm a big fan. I've felt for a while that block-level filesystems do not adapt at all well to being implemented across a network (with my personal experience being of AWS EBS and OpenEBS Mayastor). So the fact that JuiceFS is interfaces at the POSIX layer felt intuitively better to me. I also like that it can keep a local read-cache, rather than having t…

> There we did see some file corruption creep in Did you figure out what caused corruption? Was minio losing blocks or was juicefs corrupted even though minio was consistent?

It was definitely MinIO related, I probably should have made that clearer. We noticed that with zero fault tolerance, MinIO objects would randomly become corrupted, which MinIO would present as "you're making too many requests, please slow down". We were certainly not making too many requests.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#104

Post by joshstrange »

onionjake wrote: post #73
staticassertion wrote: post #50

Do people really trust Redis for something like this? I feel like it's sort of pointless to pair Redis with S3 like this, and it'd be better to see benchmarks with metadata stores that can provide actual guarantees for durability/availability. Unfortunately, the benchmarks use Redis. Why would I care about distributed storage on a system like S3, which is all about consistency/durability/availability guarantees, just…

We developed Object Mount (formerly cunoFS) ( https://www.storj.io/object-mount?hn=1 ) specifically to not rely on any metadata storage other than S3 AND preserve 1:1 mapping of objects to files AND support for POSIX. We have a direct mode that uses LD_PRELOAD to keep everything in userspace so no FUSE overhead. This approach isn't right for every use case and juice might be a better fit for this sort of 'direct bloc…

I am currently looking for a way to take a legacy application that uses the filesystem as it's database and needs to support locking (flock) on FreeBSD and scale it horizontally (right now we only scale vertically and rebuilding from a corrupted FS and/or re-pulling our, backup, data from S3 takes too long if we lose a machine). We investigated NFS but the FreeBSD NFS performance was 1/10th or worse than on Linux and switching to Linux is not in the cards for us right now.

Does Object Mount support file locks (flock specifically) and FreeBSD? I see some mention of FreeBSD but I can't find anything on locking.

For context, we are working with a large number of small (<10KB if not <4KB) files normally.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#105

Post by sroerick »

satoru42 wrote: post #68
hsn915 wrote: post #67

This is upside down. We need a kernel native distributed file system so that we can build distributed storage/databases on top of it. This is like building an operating system on top of a browser.

Show me an operating system built on top of a browser that can be used to solve real-world problems like JuiceFS.

Why are you building an operating system on top of a browser?

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#106

Post by staticassertion »

suavesu wrote: post #81
staticassertion wrote: post #50

Do people really trust Redis for something like this? I feel like it's sort of pointless to pair Redis with S3 like this, and it'd be better to see benchmarks with metadata stores that can provide actual guarantees for durability/availability. Unfortunately, the benchmarks use Redis. Why would I care about distributed storage on a system like S3, which is all about consistency/durability/availability guarantees, just…

JuiceFS metadata engine comparison -> https://juicefs.com/docs/community/metadata_engines_benchmar...

Thanks, good to see these. Looks like these only show metadata operations though. But I guess that's probably enough to extrapolate that the system is going to be roughly 2-4x slower with other metadata stores.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#107

Post by maxloh »

adamcharnock wrote: post #103
hn92726819 wrote: post #102

Earlier quoted context omitted.

> There we did see some file corruption creep in Did you figure out what caused corruption? Was minio losing blocks or was juicefs corrupted even though minio was consistent?

It was definitely MinIO related, I probably should have made that clearer. We noticed that with zero fault tolerance, MinIO objects would randomly become corrupted, which MinIO would present as "you're making too many requests, please slow down". We were certainly not making too many requests.

What is your plan after MinIO enters maintenance mode?

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#108

Post by tuhgdetzhh »

fpoling wrote: post #79
tuhgdetzhh wrote: post #53

Earlier quoted context omitted.

I think they should replace Redis with Valkey or even better use rocksdb.

But how would RocksDB work with S3? It needs support for append that generic S3 buckets do not provide and for checkpoints and backup it assumes the support for hardlinks that S3 does not have at all.

SeaweedFS has a RocksDB metadata backend for instance.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#109

Post by adamcharnock »

maxloh wrote: post #107
adamcharnock wrote: post #103

Earlier quoted context omitted.

It was definitely MinIO related, I probably should have made that clearer. We noticed that with zero fault tolerance, MinIO objects would randomly become corrupted, which MinIO would present as "you're making too many requests, please slow down". We were certainly not making too many requests.

What is your plan after MinIO enters maintenance mode?

We're looking at alternatives, I've made some previous comments on that front. Sadly MinIO was the only option with sufficient performance for this particular situation. Thankfully we're not using any MinIO-specific features, so at least the migration path away is clear.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#110

Post by bastawhiz »

staticassertion wrote: post #64
bastawhiz wrote: post #56

Earlier quoted context omitted.

Redis is as reliable as the storage you persist it to. If you're running Redis right, it's very reliable. Not S3 reliable, though. But if you need S3 reliable, you would turn to something else. I expect that most folks looking at this are doing it because it means: 1. Effectively unbounded storage 2. It's fast 3. It's pretty darn cheap 4. You can scale it horizontally in a way that's challenging to scale other filesy…

> Redis is as reliable as the storage you persist it to. For a single node, if you tank performance by changing the configuration, sure. Otherwise, no, not really. I don't get why you'd want a file system that isn't durable, but to each their own.

With the wal redis runs with perfectly reasonable performance. Of course you're not going to have the performance of an in-memory only DB if you're flushing to disk on every write.

There's no vacuuming, there's no need for indexing. You can see the time complexity of most operations. Key-value operations are mostly O(1). You'll never get that kind of performance with other databases because they intentionally don't give you that granularity.

The metadata of the filesystem isn't the performance bottleneck in most cases.

Post reply on HN