Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JuiceFS is a distributed POSIX file system built on top of Redis and S3

github.com

Post reply on HN
Print view View original

31–40 of 119 posts

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#31

Post by jabl »

huntaub wrote: post #14
IshKebab wrote: post #13

Interesting. Would this be suitable as a replacement for NFS? In my experience literally everyone in the silicon design industry uses NFS on their compute grid and it sucks in numerous ways: * poor locking support (this sounds like it works better) * it's slow * no manual fence support; a bad but common way of distributing workloads is e.g. to compile a test on one machine (on an NFS mount), and then use SLURM or SGE…

> * The bloody .nfs000000 files. I think this might be fixed by NFSv4 but it seems like nobody actually uses that. (Not helped by the fact that CentOS 7 is considered "modern" to EDA people.) Unfortunately, NFSv4 also has the silly rename semantics...

AFAIU the NFSv4 protocol in principle allows implementing unlinking an open file without silly rename, but the Linux client still does the silly rename dance.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#32

Post by maxmcd »

Eikon wrote: post #3

ZeroFS [0] outperforms JuiceFS on common small file workloads [1] while only requiring S3 and no 3rd party database. [0] https://github.com/Barre/ZeroFS [1] https://www.zerofs.net/zerofs-vs-juicefs

does having to maintain the slatedb as a consistent singleton (even with write fencing) make this as operationally tricky as a third party db?

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#33

Post by Eikon »

maxmcd wrote: post #32
Eikon wrote: post #3

ZeroFS [0] outperforms JuiceFS on common small file workloads [1] while only requiring S3 and no 3rd party database. [0] https://github.com/Barre/ZeroFS [1] https://www.zerofs.net/zerofs-vs-juicefs

does having to maintain the slatedb as a consistent singleton (even with write fencing) make this as operationally tricky as a third party db?

It’s not great UX on that angle. I am currently working on coordination (through s3, not node to node communication), so that you can just spawn instances without thinking about it.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#34

Post by AdamJacobMuller »

willbeddow wrote: post #19

Juice is cool, but tradeoffs around which metadata store you choose end up being very important. It also writes files in it's own uninterpretable format to object storage, so if you lose the metadata store, you lose your data. When we tried it at Krea we ended up moving on because we couldn't get sufficient performance to train on, and having to choose which datacenter to deploy our metadata store on essentially forc…

> It also writes files in it's own uninterpretable format to object storage, so if you lose the metadata store, you lose your data.

That's so confusing to me I had to read it five times. Are you saying you lose the metadata, or that the underlying data is actually mangled or gone, or merely that you lose the metadata?

One of the greatest features of something like this to me would be the ability to durable even beyond JuiceFS access to my data in a bad situation. Even if JuiceFS totally messes up, my data is still in S3 (and with versioning etc even if juicefs mangles or deletes my data, still). So odd to design this kind of software and lose this property.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#35

Post by jabl »

IshKebab wrote: post #13

Interesting. Would this be suitable as a replacement for NFS? In my experience literally everyone in the silicon design industry uses NFS on their compute grid and it sucks in numerous ways: * poor locking support (this sounds like it works better) * it's slow * no manual fence support; a bad but common way of distributing workloads is e.g. to compile a test on one machine (on an NFS mount), and then use SLURM or SGE…

> poor locking support (this sounds like it works better)

File locking on Unix is in general a clusterf*ck. (There was a thread a few days ago at https://news.ycombinator.com/item?id=46542247 )

> no manual fence support; a bad but common way of distributing workloads is e.g. to compile a test on one machine (on an NFS mount), and then use SLURM or SGE to run the test on other machines. You use NFS to let the other machines access the data... and this works... except that you either have to disable write caches or have horrible hacks to make the output of the first machine visible to the others. What you really want is a manual fence: "make all changes to this directory visible on the server"

In general, file systems make for poor IPC implementations. But if you need to do it with NFS, the key is to understand the close-to-open consistency model NFS uses, see section 10.3.1 in https://www.rfc-editor.org/rfc/rfc7530#section-10.3 . Of course, you'll also want some mechanism for the writer to notify the reader that it's finished, be it with file locks, or some other entirely different protocol to send signals over the network.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#36

Post by IshKebab »

huntaub wrote: post #28
dpe82 wrote: post #26

Earlier quoted context omitted.

Mind helping us all out ahead of time by expanding on what kind of gotchas FUSE is full of?

It depends on what level of FUSE you're working with. If you're running a FUSE adapter provided by a third party (Mountpoint, GCS FUSE), odds are that you aren't going to get great performance because it's going to have to run across a network super far away to work with your data. To improve performance, these adapters need to be sure to set fiddly settings (like using Kernel-side writeback caching) to avoid the pen…

> you need to support the ability to unlink a file from the file system and keep its data around. Want to run "vim", you need the ability to do renames and hard links

Those seem like pretty basic POSIX filesystem features to be fair. Awkward, sure... there's also awkwardness like symlinks, file locking, sticky bits and so on. But these are just things you have to implement. Are there gotchas that are inherent to FUSE itself rather than FUSE implementations?

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#37

Post by mrkurt »

AdamJacobMuller wrote: post #34
willbeddow wrote: post #19

Juice is cool, but tradeoffs around which metadata store you choose end up being very important. It also writes files in it's own uninterpretable format to object storage, so if you lose the metadata store, you lose your data. When we tried it at Krea we ended up moving on because we couldn't get sufficient performance to train on, and having to choose which datacenter to deploy our metadata store on essentially forc…

> It also writes files in it's own uninterpretable format to object storage, so if you lose the metadata store, you lose your data. That's so confusing to me I had to read it five times. Are you saying you lose the metadata, or that the underlying data is actually mangled or gone, or merely that you lose the metadata? One of the greatest features of something like this to me would be the ability to durable even beyon…

It backs its metadata up to S3. You do need metadata to map inodes / slices / chunks to s3 objects, though.

Tigris has a one-to-one FUSE that does what you want: https://github.com/tigrisdata/tigrisfs

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#38

Post by wgjordan »

wgjordan wrote: post #23
Eikon wrote: post #20

Earlier quoted context omitted.

It doesn’t, you are free to use ZeroFS for commercial and closed source products.

This clarification is helpful, thanks! The README currently implies a slightly different take, perhaps it could be made more clear that it's suitable for use unmodified in closed source products: > The AGPL license is suitable for open source projects, while commercial licenses are available for organizations requiring different terms. I was a bit unclear on where the AGPL's network-interaction clause draws its bound…

Also worth noting (as a sibling comment pointed out) that despite these assurances the untested legal risks of AGPL-licensed code may still cause difficulties for larger, risk-averse companies. Google notably has a blanket policy [1] banning all AGPL code entirely as "the risks outweigh the benefits", so large organizations are probably another area where the commercial license comes into play.

[1] https://opensource.google/documentation/reference/using/agpl...

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#39

Post by jeffbee »

It is not clear that pjdfstest establishes full POSIX semantic compliance. After a short search of the repo I did not see anything that exercises multiple unrelated processes atomically writing with O_APPEND, for example. And the fact that their graphic shows applications interfacing with JuiceFS over NFS and SMB casts further doubt, since both of those lack many POSIX semantic properties.

Over the decades I have written test harnesses for many distributed filesystems and the only one that seemed to actually offer POSIX semantics was LustreFS, which, for related reasons, is also an operability nightmare.

Post reply on HN