Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZeroFS vs. Amazon S3 Files

zerofs.net

Post reply on HN
Print view View original

21–30 of 34 posts

Re: ZeroFS vs. Amazon S3 Files

#21

Post by pierrebarre »

tandr wrote: post #14
pierrebarre wrote: post #4

Author here. Thanks for posting this! It’s been quite a ride building ZeroFS, and I’m happy to answer any questions.

I'm thinking about using this and have a few questions: 1. How are hardlinks and duplicate files (same content, different paths) handled? 2. Does deduplication work on a block/chunk level for partially matching files, or does it only look at whole files? 3. Is there any specific integration or handling for Copy-on-Write (CoW)? Thank you!

Hardlinks work as you’d expect: multiple paths point to the same inode and data, so a write through one path is visible through the others. Two separate files with the same contents are stored separately.

There’s no deduplication, either whole-file or block-level. That’s intentional, mostly because of the impact it would have on locality.

If by CoW you mean reflinks, those aren’t currently planned either. They avoid the content matching part of deduplication, but still require sharing extents between files and come with similar locality and complexity tradeoffs. Internally ZeroFS is copy-on-write, with immutable segments and checkpoints, but that isn’t exposed as reflinks.

Re: ZeroFS vs. Amazon S3 Files

#22

Post by pierrebarre »

the_duke wrote: post #15
pierrebarre wrote: post #4

Author here. Thanks for posting this! It’s been quite a ride building ZeroFS, and I’m happy to answer any questions.

I was investigating the design a little. Two big questions: A) You notably don't write a recovery log (WAL/journal) for things not yet flushed, so data can be lost. Do you have plans to add this? I think it would be pretty crucial. B) the system is single writer. Do you have plans for adding horizontal scalability so a writer can be dynamically selected and routed to, transparent to the client? (Or with client cooper…

[dead]

Re: ZeroFS vs. Amazon S3 Files

#24

Post by selfhoster1312 »

pierrebarre wrote: post #4

Author here. Thanks for posting this! It’s been quite a ride building ZeroFS, and I’m happy to answer any questions.

I've been curious about ZeroFS. My usecase is running a real POSIX filesystem on top of garage for integration with non-S3 services. I've had very bad (short) experiment with JuiceFS (1). Is it worth benchmarking `zerofs mount` with garage?

My usecase is NAS storage of many small files + some big files (think big shared SMB for a non-profit). I need:

- fsync durability (as promised on zerofs homepage), including sqlite durability (so NFS is out of the question)

- inotify support for external script integration (did not find an issue/docs about this)

- support for reading/writing small blocks without hammering the CPU/disks (benchmarked with JuiceFS using torrents and was catastrophic)

Do you think ZeroFS currently makes a good candidate? If not, is the described usecase part of the ZeroFS roadmap?

(1) https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021

Re: ZeroFS vs. Amazon S3 Files

#26

Post by snapplebobapple »

selfhoster1312 wrote: post #24
pierrebarre wrote: post #4

Author here. Thanks for posting this! It’s been quite a ride building ZeroFS, and I’m happy to answer any questions.

I've been curious about ZeroFS. My usecase is running a real POSIX filesystem on top of garage for integration with non-S3 services. I've had very bad (short) experiment with JuiceFS (1). Is it worth benchmarking `zerofs mount` with garage? My usecase is NAS storage of many small files + some big files (think big shared SMB for a non-profit). I need: - fsync durability (as promised on zerofs homepage), including sqli…

Did you get garagefs to work well? I ended up moving to rustfs for my plakar and restic backup station because the performance was abysmal on garage. i also dont really get the point of zerofs. Seems like you are building a posix fs on top of s3 that is built on top of a posix fs... seems like a lot of extra steps that probably degrade performance vs just backing up that first posix fs semi regular.

Re: ZeroFS vs. Amazon S3 Files

#27

Post by pierrebarre »

selfhoster1312 wrote: post #24
pierrebarre wrote: post #4

Author here. Thanks for posting this! It’s been quite a ride building ZeroFS, and I’m happy to answer any questions.

I've been curious about ZeroFS. My usecase is running a real POSIX filesystem on top of garage for integration with non-S3 services. I've had very bad (short) experiment with JuiceFS (1). Is it worth benchmarking `zerofs mount` with garage? My usecase is NAS storage of many small files + some big files (think big shared SMB for a non-profit). I need: - fsync durability (as promised on zerofs homepage), including sqli…

Yes, I think it’s worth benchmarking.

zerofs mount has durable fsync and fcntl byte-range locking.

Inotify works for changes made through the local mount. It won’t report changes made through another client, though.

Small writes operate on 32 KiB extents. Partial overwrites are read-modify-write, but the FUSE writeback cache helps combine them, and the resulting extents are packed into segments rather than producing one S3 PUT per write.

I’d be interested in the results if you try it.

Re: ZeroFS vs. Amazon S3 Files

#28

Post by selfhoster1312 »

pierrebarre wrote: post #27
selfhoster1312 wrote: post #24

Earlier quoted context omitted.

I've been curious about ZeroFS. My usecase is running a real POSIX filesystem on top of garage for integration with non-S3 services. I've had very bad (short) experiment with JuiceFS (1). Is it worth benchmarking `zerofs mount` with garage? My usecase is NAS storage of many small files + some big files (think big shared SMB for a non-profit). I need: - fsync durability (as promised on zerofs homepage), including sqli…

Yes, I think it’s worth benchmarking. zerofs mount has durable fsync and fcntl byte-range locking. Inotify works for changes made through the local mount. It won’t report changes made through another client, though. Small writes operate on 32 KiB extents. Partial overwrites are read-modify-write, but the FUSE writeback cache helps combine them, and the resulting extents are packed into segments rather than producing…

Thanks for the extra info, i will definitely come back to you when i have some results. It won't probably be until the end of the summer though, before i find the chance to put back online a test server.

Re: ZeroFS vs. Amazon S3 Files

#29

Post by selfhoster1312 »

snapplebobapple wrote: post #26
selfhoster1312 wrote: post #24

Earlier quoted context omitted.

I've been curious about ZeroFS. My usecase is running a real POSIX filesystem on top of garage for integration with non-S3 services. I've had very bad (short) experiment with JuiceFS (1). Is it worth benchmarking `zerofs mount` with garage? My usecase is NAS storage of many small files + some big files (think big shared SMB for a non-profit). I need: - fsync durability (as promised on zerofs homepage), including sqli…

Did you get garagefs to work well? I ended up moving to rustfs for my plakar and restic backup station because the performance was abysmal on garage. i also dont really get the point of zerofs. Seems like you are building a posix fs on top of s3 that is built on top of a posix fs... seems like a lot of extra steps that probably degrade performance vs just backing up that first posix fs semi regular.

garage was working fine in other scenarios but it really had pathological behavior with the small read/write with torrenting benchmarks. (see my previous link for detailed benchmarking)

The reason for me for trying this setup was to provide native S3 features for apps that support it, while still supporting POSIX semantics for other apps (we have a few), while replacing native RAID with a multi-server storage backend to extend available space, and potentially removing the need for backups for most of the non-critical data once it's replicated across different sites.

I never got NFS/SMB to saturate a Gbit/s LAN link, and other solutions like cephfs/glusterfs seemed more complex and did not necessarily have all the features (inotify/fsync).

I never tried rustfs but was particularly interested in garage for the "philosophy" of development: developed by academics with proper R&D, fully FLOSS, with low-end hardware and unstable links/sites in mind, aiming for technological degrowth. From the looks at the rustfs website, it looks like the opposite of that.

You are right i was trying to overengineer things. For now i still run RAID0 (+ backups for important data).

Re: ZeroFS vs. Amazon S3 Files

#30

Post by snapplebobapple »

selfhoster1312 wrote: post #29
snapplebobapple wrote: post #26

Earlier quoted context omitted.

Did you get garagefs to work well? I ended up moving to rustfs for my plakar and restic backup station because the performance was abysmal on garage. i also dont really get the point of zerofs. Seems like you are building a posix fs on top of s3 that is built on top of a posix fs... seems like a lot of extra steps that probably degrade performance vs just backing up that first posix fs semi regular.

garage was working fine in other scenarios but it really had pathological behavior with the small read/write with torrenting benchmarks. (see my previous link for detailed benchmarking) The reason for me for trying this setup was to provide native S3 features for apps that support it, while still supporting POSIX semantics for other apps (we have a few), while replacing native RAID with a multi-server storage backend…

I think it's those small writes that was making garagefs perform so badly and rustfs perform a lot better. My setup sounds like a different scale than yours though. It's my home lab running on questionable hardware. As truenas apps for rustfs/garagefs. I use cephfs on my proxmox cluster and it's honestly not that great if you have a lot of small files either (which I do because of things like screenshots in jellyfin/music players/etc). My rustfs setup is not clustered either. It's just single node
Post reply on HN