Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JuiceFS is a distributed POSIX file system built on top of Redis and S3

github.com

Post reply on HN
Print view View original

111–119 of 119 posts

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#111

Post by bastawhiz »

danpalmer wrote: post #61
bastawhiz wrote: post #56

Earlier quoted context omitted.

Redis is as reliable as the storage you persist it to. If you're running Redis right, it's very reliable. Not S3 reliable, though. But if you need S3 reliable, you would turn to something else. I expect that most folks looking at this are doing it because it means: 1. Effectively unbounded storage 2. It's fast 3. It's pretty darn cheap 4. You can scale it horizontally in a way that's challenging to scale other filesy…

Redis isn't durable unless you drastically reduce the performance. Filesystems are pretty much by definition durable.

Enabling the WAL doesn't make Redis slow. It's slower than the default, but it's still exceptionally fast.

> Filesystems are pretty much by definition durable.

Where do you think Redis persists its data to

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#112

Post by fakebizprez »

adamcharnock wrote: post #109
maxloh wrote: post #107

Earlier quoted context omitted.

What is your plan after MinIO enters maintenance mode?

We're looking at alternatives, I've made some previous comments on that front. Sadly MinIO was the only option with sufficient performance for this particular situation. Thankfully we're not using any MinIO-specific features, so at least the migration path away is clear.

Ceph. The answer is always Ceph.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#113

Post by onionjake »

joshstrange wrote: post #104
onionjake wrote: post #73

Earlier quoted context omitted.

We developed Object Mount (formerly cunoFS) ( https://www.storj.io/object-mount?hn=1 ) specifically to not rely on any metadata storage other than S3 AND preserve 1:1 mapping of objects to files AND support for POSIX. We have a direct mode that uses LD_PRELOAD to keep everything in userspace so no FUSE overhead. This approach isn't right for every use case and juice might be a better fit for this sort of 'direct bloc…

I am currently looking for a way to take a legacy application that uses the filesystem as it's database and needs to support locking (flock) on FreeBSD and scale it horizontally (right now we only scale vertically and rebuilding from a corrupted FS and/or re-pulling our, backup, data from S3 takes too long if we lose a machine). We investigated NFS but the FreeBSD NFS performance was 1/10th or worse than on Linux and…

No BSD support unfortunately. Just Linux, Mac, windows.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#114

Post by onionjake »

lifty wrote: post #98
onionjake wrote: post #73

Earlier quoted context omitted.

We developed Object Mount (formerly cunoFS) ( https://www.storj.io/object-mount?hn=1 ) specifically to not rely on any metadata storage other than S3 AND preserve 1:1 mapping of objects to files AND support for POSIX. We have a direct mode that uses LD_PRELOAD to keep everything in userspace so no FUSE overhead. This approach isn't right for every use case and juice might be a better fit for this sort of 'direct bloc…

Is this open source? I am a happy Storj customer, would love to use it if it's open source.

Not open source. What would be your use case?

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#115

Post by hnarn »

joshstrange wrote: post #104
onionjake wrote: post #73

Earlier quoted context omitted.

We developed Object Mount (formerly cunoFS) ( https://www.storj.io/object-mount?hn=1 ) specifically to not rely on any metadata storage other than S3 AND preserve 1:1 mapping of objects to files AND support for POSIX. We have a direct mode that uses LD_PRELOAD to keep everything in userspace so no FUSE overhead. This approach isn't right for every use case and juice might be a better fit for this sort of 'direct bloc…

I am currently looking for a way to take a legacy application that uses the filesystem as it's database and needs to support locking (flock) on FreeBSD and scale it horizontally (right now we only scale vertically and rebuilding from a corrupted FS and/or re-pulling our, backup, data from S3 takes too long if we lose a machine). We investigated NFS but the FreeBSD NFS performance was 1/10th or worse than on Linux and…

According to this link:

https://vonng.com/en/pg/pgfs/

> The magic is: JuiceFS also supports using PostgreSQL as both metadata and object data storage backend! This means you only need to change JuiceFS’s backend to an existing PostgreSQL instance to get a database-based “filesystem.”

Sounds ideal for your kind of situation where a filesystem is being abused as a database :-)

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#116

Post by staticassertion »

bastawhiz wrote: post #110
staticassertion wrote: post #64

Earlier quoted context omitted.

> Redis is as reliable as the storage you persist it to. For a single node, if you tank performance by changing the configuration, sure. Otherwise, no, not really. I don't get why you'd want a file system that isn't durable, but to each their own.

With the wal redis runs with perfectly reasonable performance. Of course you're not going to have the performance of an in-memory only DB if you're flushing to disk on every write. There's no vacuuming, there's no need for indexing. You can see the time complexity of most operations. Key-value operations are mostly O(1). You'll never get that kind of performance with other databases because they intentionally don't g…

> With the wal redis runs with perfectly reasonable performance. Of course you're not going to have the performance of an in-memory only DB if you're flushing to disk on every write.

But that's not what gets benchmarked against.

> You'll never get that kind of performance with other databases because they intentionally don't give you that granularity.

Okay but you also won't get horizontal scaling while maintaining consistency in Redis.

> The metadata of the filesystem isn't the performance bottleneck in most cases.

Okay, but they do show that other metadata stores are 2-4x slower than Redis, so it would be great to see whole-system benchmarks that use those.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#117

Post by danpalmer »

bastawhiz wrote: post #111
danpalmer wrote: post #61

Earlier quoted context omitted.

Redis isn't durable unless you drastically reduce the performance. Filesystems are pretty much by definition durable.

Enabling the WAL doesn't make Redis slow. It's slower than the default, but it's still exceptionally fast. > Filesystems are pretty much by definition durable. Where do you think Redis persists its data to

Redis using the WAL for durable data can only be as fast as you can fsync on every write, just like every other database. The time to fsync is going to dominate the write path for Redis, and you'll sacrifice almost all of Redis's speed to achieve it. If you're running Redis in always fsync mode it I'd suggest there are better storage systems to use.

> Where do you think Redis persists its data to

This is sort of the point, Redis isn't for persistence. I mean it can, you can use the WAL, it's handy for not having to recompute when you relaunch Redis etc, but Redis is fundamentally an in-memory system and should be treated as such. It is designed for use-cases that necessitate in-memory performance, and that don't require durability.

Re: JuiceFS is a distributed POSIX file system built on top of Redis and S3

#119

Post by yencabulator »

eerikkivistik wrote: post #100
romantomjak wrote: post #96

Earlier quoted context omitted.

Did you happen to look into CephFS? CERN (folks that operate Large Hadron Collider) use it to store ~30PB of scientific data. Their analysis cluster is serving ~30GB/s reads

Sure, so the use case I have requires elastic storage and elastic compute. So CephFS really isn't a good fit in the cloud environment for that case. It would get prohibitively expensive.

Ceph is more something you build your own cloud with than something you run on someone else's cloud.
Post reply on HN