Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JuiceFS is a distributed POSIX file system built on top of Redis and S3

github.com

Post reply on HN
Print view View original

21–30 of 119 posts

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#21

Post by tptacek »

willbeddow wrote: post #19

Juice is cool, but tradeoffs around which metadata store you choose end up being very important. It also writes files in it's own uninterpretable format to object storage, so if you lose the metadata store, you lose your data. When we tried it at Krea we ended up moving on because we couldn't get sufficient performance to train on, and having to choose which datacenter to deploy our metadata store on essentially forc…

I'm betting this is on the front page today (as opposed to any other day; Juice is very neat and doesn't need us to hype it) because of our Sprites post, which goes into some detail about how we use Juice (for the time being; I'm not sure if we'll keep it this way).

The TL;DR relevant to your comment is: we tore out a lot of the metadata stuff, and our metadata storage is SQLite + Litestream.io, which gives us fast local read/write, enough systemwide atomicity (all atomicity in our setting runs asymptotically against "someone could just cut the power at any moment"), and preserves "durably stored to object storage".

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#22

Post by mrkurt »

IshKebab wrote: post #13

Interesting. Would this be suitable as a replacement for NFS? In my experience literally everyone in the silicon design industry uses NFS on their compute grid and it sucks in numerous ways: * poor locking support (this sounds like it works better) * it's slow * no manual fence support; a bad but common way of distributing workloads is e.g. to compile a test on one machine (on an NFS mount), and then use SLURM or SGE…

FUSE is full of gotchas. I wouldn't replace NFS with JuiceFS for arbitrary workloads. Getting the full FUSE set implemented is not easy -- you can't use sqlite on JuiceFS, for example.

The meta store is a bottleneck too. For a shared mount, you've got a bunch of clients sharing a metadata store that lives in the cloud somewhere. They do a lot of aggressive metadata caching. It's still surprisingly slow at times.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#23

Post by wgjordan »

Eikon wrote: post #20
anonymousDan wrote: post #17

Earlier quoted context omitted.

Why would this matter if you're just using the database?

It doesn’t, you are free to use ZeroFS for commercial and closed source products.

This clarification is helpful, thanks! The README currently implies a slightly different take, perhaps it could be made more clear that it's suitable for use unmodified in closed source products:

> The AGPL license is suitable for open source projects, while commercial licenses are available for organizations requiring different terms.

I was a bit unclear on where the AGPL's network-interaction clause draws its boundaries- so the commercial license would only be needed for closed-source modifications/forks, or if statically linking ZeroFS crate into a larger proprietary Rust program, is that roughly it?

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#24

Post by huntaub »

mrkurt wrote: post #22
IshKebab wrote: post #13

Interesting. Would this be suitable as a replacement for NFS? In my experience literally everyone in the silicon design industry uses NFS on their compute grid and it sucks in numerous ways: * poor locking support (this sounds like it works better) * it's slow * no manual fence support; a bad but common way of distributing workloads is e.g. to compile a test on one machine (on an NFS mount), and then use SLURM or SGE…

FUSE is full of gotchas. I wouldn't replace NFS with JuiceFS for arbitrary workloads. Getting the full FUSE set implemented is not easy -- you can't use sqlite on JuiceFS, for example. The meta store is a bottleneck too. For a shared mount, you've got a bunch of clients sharing a metadata store that lives in the cloud somewhere. They do a lot of aggressive metadata caching. It's still surprisingly slow at times.

> FUSE is full of gotchas

I want to go ahead and nominate this for the understatement of the year. I expect that 2026 is going to be filled with people finding this out the hard way as they pivot towards FUSE for agents.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#25

Post by Eikon »

wgjordan wrote: post #23
Eikon wrote: post #20

Earlier quoted context omitted.

It doesn’t, you are free to use ZeroFS for commercial and closed source products.

This clarification is helpful, thanks! The README currently implies a slightly different take, perhaps it could be made more clear that it's suitable for use unmodified in closed source products: > The AGPL license is suitable for open source projects, while commercial licenses are available for organizations requiring different terms. I was a bit unclear on where the AGPL's network-interaction clause draws its bound…

> so the commercial license would only be needed for closed-source modifications/forks

Indeed.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#26

Post by dpe82 »

huntaub wrote: post #24
mrkurt wrote: post #22

Earlier quoted context omitted.

FUSE is full of gotchas. I wouldn't replace NFS with JuiceFS for arbitrary workloads. Getting the full FUSE set implemented is not easy -- you can't use sqlite on JuiceFS, for example. The meta store is a bottleneck too. For a shared mount, you've got a bunch of clients sharing a metadata store that lives in the cloud somewhere. They do a lot of aggressive metadata caching. It's still surprisingly slow at times.

> FUSE is full of gotchas I want to go ahead and nominate this for the understatement of the year. I expect that 2026 is going to be filled with people finding this out the hard way as they pivot towards FUSE for agents.

Mind helping us all out ahead of time by expanding on what kind of gotchas FUSE is full of?

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#27

Post by dpacmittal »

Eikon wrote: post #3

ZeroFS [0] outperforms JuiceFS on common small file workloads [1] while only requiring S3 and no 3rd party database. [0] https://github.com/Barre/ZeroFS [1] https://www.zerofs.net/zerofs-vs-juicefs

The magnitude of performance difference alone immediately makes me skeptical of your benchmarking methodology.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#28

Post by huntaub »

dpe82 wrote: post #26
huntaub wrote: post #24

Earlier quoted context omitted.

> FUSE is full of gotchas I want to go ahead and nominate this for the understatement of the year. I expect that 2026 is going to be filled with people finding this out the hard way as they pivot towards FUSE for agents.

Mind helping us all out ahead of time by expanding on what kind of gotchas FUSE is full of?

It depends on what level of FUSE you're working with.

If you're running a FUSE adapter provided by a third party (Mountpoint, GCS FUSE), odds are that you aren't going to get great performance because it's going to have to run across a network super far away to work with your data. To improve performance, these adapters need to be sure to set fiddly settings (like using Kernel-side writeback caching) to avoid the penalty of hitting the disk for operations like write.

If you're trying to write a FUSE adapter, it's up to you to implement as much of the POSIX spec that you need for the programs that you want to run. The requirements per-program are often surprising. Want to run "git clone", then you need to support the ability to unlink a file from the file system and keep its data around. Want to run "vim", you need the ability to do renames and hard links. All of this work needs to happen in-memory in order to get the performance that applications expect from their file system, which often isn't how these things are built.

Regarding agents in particular, I'm hopeful that someone (which is quite possibly us), builds a FUSE-as-a-service primitive that's simple enough to use that the vast majority of developers don't have to worry about these things.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#29

Post by andydang »

wgjordan wrote: post #23
Eikon wrote: post #20

Earlier quoted context omitted.

It doesn’t, you are free to use ZeroFS for commercial and closed source products.

This clarification is helpful, thanks! The README currently implies a slightly different take, perhaps it could be made more clear that it's suitable for use unmodified in closed source products: > The AGPL license is suitable for open source projects, while commercial licenses are available for organizations requiring different terms. I was a bit unclear on where the AGPL's network-interaction clause draws its bound…

[dead]

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#30

Post by zaphirplane »

huntaub wrote: post #15
Eikon wrote: post #9

Earlier quoted context omitted.

> but I find it kind of off-putting for you to come in and immediately put down JuiceFS, especially with benchmark results that don't make a ton of sense, and are likely making apples-to-oranges comparisons with how JuiceFS works or mount options. The benchmark suite is trivial and opensource [1]. Is performing benchmarks “putting down” these days? If you believe that the benchmarks are unfair to juicefs for a reason…

Yes, I'm working in the space too. I think it's fine to do benchmarks, I don't think it's necessary to immediately post them any time a competitor comes up on HN. I don't want to see the cloud storage sector turn as bitter as the cloud database sector. I've previously looked through the benchmarking code, and I still have some serious concerns about the way that you're presenting things on your page.

> presenting things

I don’t have a dog in this race, have to say thou the vagueness of the hand waving in multiple comments is losing you credibility

Post reply on HN