Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JuiceFS is a distributed POSIX file system built on top of Redis and S3

github.com

Post reply on HN
Print view View original

41–50 of 119 posts

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#41

Post by selfhoster1312 »

huntaub wrote: post #5
Eikon wrote: post #3

ZeroFS [0] outperforms JuiceFS on common small file workloads [1] while only requiring S3 and no 3rd party database. [0] https://github.com/Barre/ZeroFS [1] https://www.zerofs.net/zerofs-vs-juicefs

Respect to your work on ZeroFS, but I find it kind of off-putting for you to come in and immediately put down JuiceFS, especially with benchmark results that don't make a ton of sense, and are likely making apples-to-oranges comparisons with how JuiceFS works or mount options. For example, it doesn't really make sense that "92% of data modification operations" would fail on JuiceFS, which makes me question a lot of t…

I have very limited experiences with object storage, but my humble benchmarks with juicefs + minio/garage [1] showed very bad performance (i.e. total collapse within a few hours) when running lots of small operations (torrents).

I wouldn't be surprised if there's a lot of tuning that can be achieved, but after days of reading docs and experimenting with different settings i just assumed JuiceFS was a very bad fit for archives shared through Bittorrent. I hope to be proven wrong, but in the meantime i'm very glad zerofs was mentioned as an alternative for small files/operations. I'll try to find the time to benchmark it too.

[1] https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#42

Post by selfhoster1312 »

dpacmittal wrote: post #27
Eikon wrote: post #3

ZeroFS [0] outperforms JuiceFS on common small file workloads [1] while only requiring S3 and no 3rd party database. [0] https://github.com/Barre/ZeroFS [1] https://www.zerofs.net/zerofs-vs-juicefs

The magnitude of performance difference alone immediately makes me skeptical of your benchmarking methodology.

I'm not an expert in any way, but i personally benchmarked [1] juiceFS performance totalling collapsing under very small files/operations (torrenting). It's good to be skeptical, but it might just be that the bar is very low for this specific usecase (IIRC juiceFS was configured and optimized for block sizes of several MBs).

https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#43

Post by huntaub »

IshKebab wrote: post #36
huntaub wrote: post #28

Earlier quoted context omitted.

It depends on what level of FUSE you're working with. If you're running a FUSE adapter provided by a third party (Mountpoint, GCS FUSE), odds are that you aren't going to get great performance because it's going to have to run across a network super far away to work with your data. To improve performance, these adapters need to be sure to set fiddly settings (like using Kernel-side writeback caching) to avoid the pen…

> you need to support the ability to unlink a file from the file system and keep its data around. Want to run "vim", you need the ability to do renames and hard links Those seem like pretty basic POSIX filesystem features to be fair. Awkward, sure... there's also awkwardness like symlinks, file locking, sticky bits and so on. But these are just things you have to implement. Are there gotchas that are inherent to FUSE…

These are basic POSIX features, but I think the high-level point that Kurt is trying to make is that building a FUSE file system signs you up for a nearly unlimited amount of compatibility work (if you want to support most applications) whereas their approach (just do a loopback ext4 fs into a large file) avoids a lot of those problems.

My expectations are that in 2026 we will see more and more developers attempt to build custom FUSE file systems and then run into the long tail of compatibility pain.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#44

Post by cbarrick »

AdamJacobMuller wrote: post #34
willbeddow wrote: post #19

Juice is cool, but tradeoffs around which metadata store you choose end up being very important. It also writes files in it's own uninterpretable format to object storage, so if you lose the metadata store, you lose your data. When we tried it at Krea we ended up moving on because we couldn't get sufficient performance to train on, and having to choose which datacenter to deploy our metadata store on essentially forc…

> It also writes files in it's own uninterpretable format to object storage, so if you lose the metadata store, you lose your data. That's so confusing to me I had to read it five times. Are you saying you lose the metadata, or that the underlying data is actually mangled or gone, or merely that you lose the metadata? One of the greatest features of something like this to me would be the ability to durable even beyon…

As I understand it, if the metadata is lost then the whole filesystem is lost.

I think this is a common failure mode in filesystems. For example, in ZFS, if you store your metadata on a separate device and that device is destroyed, the whole pool is useless.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#45

Post by victorbjorklund »

Eikon wrote: post #3

ZeroFS [0] outperforms JuiceFS on common small file workloads [1] while only requiring S3 and no 3rd party database. [0] https://github.com/Barre/ZeroFS [1] https://www.zerofs.net/zerofs-vs-juicefs

Can SQLite run on it?

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#46

Post by xorcist »

IshKebab wrote: post #13

Interesting. Would this be suitable as a replacement for NFS? In my experience literally everyone in the silicon design industry uses NFS on their compute grid and it sucks in numerous ways: * poor locking support (this sounds like it works better) * it's slow * no manual fence support; a bad but common way of distributing workloads is e.g. to compile a test on one machine (on an NFS mount), and then use SLURM or SGE…

> NFSv4 but it seems like nobody actually uses that

Hurry up and you might be able to adopt it before its 30th birthday!

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#47

Post by IshKebab »

huntaub wrote: post #43
IshKebab wrote: post #36

Earlier quoted context omitted.

> you need to support the ability to unlink a file from the file system and keep its data around. Want to run "vim", you need the ability to do renames and hard links Those seem like pretty basic POSIX filesystem features to be fair. Awkward, sure... there's also awkwardness like symlinks, file locking, sticky bits and so on. But these are just things you have to implement. Are there gotchas that are inherent to FUSE…

These are basic POSIX features, but I think the high-level point that Kurt is trying to make is that building a FUSE file system signs you up for a nearly unlimited amount of compatibility work (if you want to support most applications) whereas their approach (just do a loopback ext4 fs into a large file) avoids a lot of those problems. My expectations are that in 2026 we will see more and more developers attempt to…

> just do a loopback ext4 fs into a large file

How does that work with multiple clients though?

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#48

Post by huntaub »

IshKebab wrote: post #47
huntaub wrote: post #43

Earlier quoted context omitted.

These are basic POSIX features, but I think the high-level point that Kurt is trying to make is that building a FUSE file system signs you up for a nearly unlimited amount of compatibility work (if you want to support most applications) whereas their approach (just do a loopback ext4 fs into a large file) avoids a lot of those problems. My expectations are that in 2026 we will see more and more developers attempt to…

> just do a loopback ext4 fs into a large file How does that work with multiple clients though?

tl;dr it doesn't. I'm not sure what they're planning in this capacity (I haven't checked out sprites myself), but I would guess that it's going to be a function of "snapshots" as a mechanism to give multiple clients ephemeral write access to the same disk.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#49

Post by Dylan16807 »

Eikon wrote: post #3

ZeroFS [0] outperforms JuiceFS on common small file workloads [1] while only requiring S3 and no 3rd party database. [0] https://github.com/Barre/ZeroFS [1] https://www.zerofs.net/zerofs-vs-juicefs

> ZeroFS supports running multiple instances on the same storage backend: one read-write instance and multiple read-only instances.

Well that's a big limiting factor that needs to be at the front in any distributed filesystem comparison.

Though I'm confused, the page says things like "ZeroFS makes S3 behave like a regular block device", but in that case how do read-only instances mount it without constantly getting their state corrupted out from under them? Is that implicitly talking about the NBD access, and the other access modes have logic to handle that?

Edit: What I want to see is a ZeroFS versus s3backer comparison.

Edit 2: changed the question at the end

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#50

Post by staticassertion »

Do people really trust Redis for something like this? I feel like it's sort of pointless to pair Redis with S3 like this, and it'd be better to see benchmarks with metadata stores that can provide actual guarantees for durability/availability.

Unfortunately, the benchmarks use Redis. Why would I care about distributed storage on a system like S3, which is all about consistency/durability/availability guarantees, just to put my metadata into Redis?

It would be nice to see benchmarks with another metadata store.

Post reply on HN