Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JuiceFS is a distributed POSIX file system built on top of Redis and S3

github.com

Post reply on HN
Print view View original

51–60 of 119 posts

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#51

Post by IshKebab »

jabl wrote: post #35
IshKebab wrote: post #13

Interesting. Would this be suitable as a replacement for NFS? In my experience literally everyone in the silicon design industry uses NFS on their compute grid and it sucks in numerous ways: * poor locking support (this sounds like it works better) * it's slow * no manual fence support; a bad but common way of distributing workloads is e.g. to compile a test on one machine (on an NFS mount), and then use SLURM or SGE…

> poor locking support (this sounds like it works better) File locking on Unix is in general a clusterf*ck. (There was a thread a few days ago at https://news.ycombinator.com/item?id=46542247 ) > no manual fence support; a bad but common way of distributing workloads is e.g. to compile a test on one machine (on an NFS mount), and then use SLURM or SGE to run the test on other machines. You use NFS to let the other ma…

> In general, file systems make for poor IPC implementations.

I agree but also they do have advantages such as simplicity, not needing to explicitly declare which files are needed, lazy data transfer, etc.

> you'll also want some mechanism for the writer to notify the reader that it's finished, be it with file locks, or some other entirely different protocol to send signals over the network.

The writer is always finished before the reader starts in these scenarios. The issue is reads on one machine aren't guaranteed to be ordered after writes on a different machine due to write caching.

It's exactly the same problem as trying to do multithreaded code. Thread A writes a value, thread B reads it. But even if they happen sequentially in real time thread B can still read an old value unless you have an explicit fence.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#52

Post by eYrKEC2 »

huntaub wrote: post #5
Eikon wrote: post #3

ZeroFS [0] outperforms JuiceFS on common small file workloads [1] while only requiring S3 and no 3rd party database. [0] https://github.com/Barre/ZeroFS [1] https://www.zerofs.net/zerofs-vs-juicefs

Respect to your work on ZeroFS, but I find it kind of off-putting for you to come in and immediately put down JuiceFS, especially with benchmark results that don't make a ton of sense, and are likely making apples-to-oranges comparisons with how JuiceFS works or mount options. For example, it doesn't really make sense that "92% of data modification operations" would fail on JuiceFS, which makes me question a lot of t…

I'm always curious about the of the option space. I appreciate folks talking about the alternative s. What's yours?

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#53

Post by tuhgdetzhh »

staticassertion wrote: post #50

Do people really trust Redis for something like this? I feel like it's sort of pointless to pair Redis with S3 like this, and it'd be better to see benchmarks with metadata stores that can provide actual guarantees for durability/availability. Unfortunately, the benchmarks use Redis. Why would I care about distributed storage on a system like S3, which is all about consistency/durability/availability guarantees, just…

I think they should replace Redis with Valkey or even better use rocksdb.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#54

Post by tuhgdetzhh »

If tested various Posix FS projects over the years and everyone has their shortcomings in one way or the other.

Although the maintainers of these projects disagree, I mostly consider them as a workaround for smaller projects. For big data (PB range) and critical production workloads I recommend to bite the bullet and make your software nativley S3 compatible without going over a POSIX mounted S3 proxy.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#55

Post by huntaub »

eYrKEC2 wrote: post #52
huntaub wrote: post #5

Earlier quoted context omitted.

Respect to your work on ZeroFS, but I find it kind of off-putting for you to come in and immediately put down JuiceFS, especially with benchmark results that don't make a ton of sense, and are likely making apples-to-oranges comparisons with how JuiceFS works or mount options. For example, it doesn't really make sense that "92% of data modification operations" would fail on JuiceFS, which makes me question a lot of t…

I'm always curious about the of the option space. I appreciate folks talking about the alternative s. What's yours?

Our product is Archil [1], and we are building our service on top of a durable, distributed SSD storage layer. As a result, we have the ability to: (a) store and use data in S3 in its native format [not a block based format like the other solutions in this thread], (b) durably commit writes to our storage layer with lower latency than products which operate as installable OSS libraries and communicate with S3 directly, and (c) handle multiple writers from different instances like NFS.

Our team spent years working on NFS+Lustre products at Amazon (EFS and FSx for Lustre), so we understand the performance problems that these storage products have traditionally had.

We've built a custom protocol that allows our users to achieve high-performance for small file operations (git -- perfect for coding agents) and highly-parallel HPC workloads (model training, inference).

Obviously, there are tons of storage products because everyone makes different tradeoffs around durability, file size optimizations, etc. We're excited to have an approach that we think can flex around these properties dynamically, while providing best-in-class performance when compared to "true" storage systems like VAST, Weka, and Pure.

[1] https://archil.com

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#56

Post by bastawhiz »

staticassertion wrote: post #50

Do people really trust Redis for something like this? I feel like it's sort of pointless to pair Redis with S3 like this, and it'd be better to see benchmarks with metadata stores that can provide actual guarantees for durability/availability. Unfortunately, the benchmarks use Redis. Why would I care about distributed storage on a system like S3, which is all about consistency/durability/availability guarantees, just…

Redis is as reliable as the storage you persist it to. If you're running Redis right, it's very reliable. Not S3 reliable, though. But if you need S3 reliable, you would turn to something else.

I expect that most folks looking at this are doing it because it means:

1. Effectively unbounded storage

2. It's fast

3. It's pretty darn cheap

4. You can scale it horizontally in a way that's challenging to scale other filesystems

5. All the components are pretty easy to set up. Many folks are probably already running S3 and Redis.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#57

Post by mattbillenstein »

The key I think with s3 is using it mostly as a blobstore. We put the important metadata we want into postgres so we can quickly select stuff that needs to be updated based on other things being newer. So, we don't need to touch s3 that often if we don't need the actual data.

When we actually need to manipulate or generate something in Python, we download/upload to S3 and wrap it all in a tempfile.TemporaryDirectory() to cleanup the local disk when we're done. If you don't do this, you end up with a bunch of garbage eventually in /tmp/ you need to deal with.

We also have some longer-lived disk caches and using the data in the db and a os.stat() on the file we can easily know if the cache is up to date without hitting s3. And this cache, we can just delete stuff that's old wrt os.stat() to manage the size of it since we can always get it from s3 again if needed in the future.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#58

Post by ghm2199 »

tptacek wrote: post #21
willbeddow wrote: post #19

Juice is cool, but tradeoffs around which metadata store you choose end up being very important. It also writes files in it's own uninterpretable format to object storage, so if you lose the metadata store, you lose your data. When we tried it at Krea we ended up moving on because we couldn't get sufficient performance to train on, and having to choose which datacenter to deploy our metadata store on essentially forc…

I'm betting this is on the front page today (as opposed to any other day; Juice is very neat and doesn't need us to hype it) because of our Sprites post, which goes into some detail about how we use Juice (for the time being; I'm not sure if we'll keep it this way). The TL;DR relevant to your comment is: we tore out a lot of the metadata stuff, and our metadata storage is SQLite + Litestream.io, which gives us fast l…

Litestream.io is amazing. Using sqlite as a DB in a typical relational data model where objects are related mean most read then write transactions would have to one node, but if the using it for blobs as first class objects(e.g. video uploads or sensor data) which are independent probably means you can shard and scale your set up the wazoo right?

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#59

Post by ekropotin »

IshKebab wrote: post #13

Interesting. Would this be suitable as a replacement for NFS? In my experience literally everyone in the silicon design industry uses NFS on their compute grid and it sucks in numerous ways: * poor locking support (this sounds like it works better) * it's slow * no manual fence support; a bad but common way of distributing workloads is e.g. to compile a test on one machine (on an NFS mount), and then use SLURM or SGE…

How about CephFs?

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#60

Post by ifoxhz »

mrkurt wrote: post #37
AdamJacobMuller wrote: post #34

Earlier quoted context omitted.

> It also writes files in it's own uninterpretable format to object storage, so if you lose the metadata store, you lose your data. That's so confusing to me I had to read it five times. Are you saying you lose the metadata, or that the underlying data is actually mangled or gone, or merely that you lose the metadata? One of the greatest features of something like this to me would be the ability to durable even beyon…

It backs its metadata up to S3. You do need metadata to map inodes / slices / chunks to s3 objects, though. Tigris has a one-to-one FUSE that does what you want: https://github.com/tigrisdata/tigrisfs

FUSE generally has low overall performance because of an additional data transfer process between the kernel space and user space, which is less than ideal for AI training.
Post reply on HN