Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: ZeroFS – A log-structured filesystem for S3

zerofs.net

Post reply on HN
Print view View original

51–60 of 60 posts

Re: Show HN: ZeroFS – A log-structured filesystem for S3

#51

Post by nyc_pizzadev »

tmach32 wrote: post #12

See also: JuiceFS, S3FS, and quite a few others. We have done loads of research into using object storage wherever we can (given how cheap it is compared to SSDs), and so far it seems like making your application object store-aware is a far surer bet than abstracting S3 behind the file system. The behavior is just too different. I'm more interested in applications that cleverly use object storage, e.g. AutoMQ, which…

I agree that most usecases are best suited just using S3 directly, but I would check this out if you want an S3 based filesystem: https://fiberfs.io/

Re: Show HN: ZeroFS – A log-structured filesystem for S3

#53

Post by dangoodmanUT »

If you actually bench it locally (local S3, actually writes to disk for "staged" operations), ZeroFS performs horrifically. Ceph blows it out of the water. I don't have the exact numbers, but when I was building a toy CoW distributed block device and filesystem I did a perf matrix, and ZeroFS (even with an hour of codex tuning it) was never within the same 1 or 2 orders of magnitude perf-wise.

Sure, Ceph isn't "S3-backed", but when you're talking about an actual filesystem or block device (the thing that does lots of small-IO), you care more about io-perf than sequential.

Large sequential jobs that everyone is targeting now (ie AI workloads) can use s3 directly just fine, because they don't have decades of code built on top of the filesystem.

Re: Show HN: ZeroFS – A log-structured filesystem for S3

#54

Post by [deleted] »

dangoodmanUT wrote: post #53

If you actually bench it locally (local S3, actually writes to disk for "staged" operations), ZeroFS performs horrifically. Ceph blows it out of the water. I don't have the exact numbers, but when I was building a toy CoW distributed block device and filesystem I did a perf matrix, and ZeroFS (even with an hour of codex tuning it) was never within the same 1 or 2 orders of magnitude perf-wise. Sure, Ceph isn't "S3-ba…

[deleted]

Re: Show HN: ZeroFS – A log-structured filesystem for S3

#55

Post by [deleted] »

Eikon wrote: post #10
dan_sbl wrote: post #3

> The test suites run in public CI. > Each card links to the CI pipeline. Thanks for being explicit, AI written marketing site. Wouldn't have been able to figure that out! Every currently maintained and reasonably popular open source project either runs CI in public or makes the tests extremely easy to run.

Thank you for the feedback, the idea behind this was to say "We make claims that are backed by workflows you can verify". I'll improve the phrasing.

[deleted]

Re: Show HN: ZeroFS – A log-structured filesystem for S3

#56

Post by neverartful »

dangoodmanUT wrote: post #53

If you actually bench it locally (local S3, actually writes to disk for "staged" operations), ZeroFS performs horrifically. Ceph blows it out of the water. I don't have the exact numbers, but when I was building a toy CoW distributed block device and filesystem I did a perf matrix, and ZeroFS (even with an hour of codex tuning it) was never within the same 1 or 2 orders of magnitude perf-wise. Sure, Ceph isn't "S3-ba…

There are a couple of different ways of using Ceph with a filesystem: (1) CephFS, or (2) RBD (Ceph block device) volume mounted and then create filesystem on the mounted RBD volume. Historically, the RBD approach would likely have been the more common of the 2. Which of these 2 ways were you referring to?

Re: Show HN: ZeroFS – A log-structured filesystem for S3

#57

Post by zenoprax »

xyzzy_plugh wrote: post #32
rockwotj wrote: post #27

The sub-millisecond writes with data in S3 is false and impossible. If you look at the benchmark the fsync is not timed, so this is just the latency of either the network or in kernel file operations depending on the mount settings

I hate it when databases celebrate their performance without synchronous flushing. You should be clear about data loss window (which should be zero for committed transactions by default!) and the flushing interval to persistent storage. I'm okay if you batch writes, I'm okay if you offer a low-latency mode with less durability, but by being unclear about this it just feels like a scam.

Where is the "data loss window"? Between nodes or between the client and the infra?

The front page lists under Capabilities:

> 4.8 > Honest fsync > A successful fsync means every acknowledged write is durable in S3. If a failover may have lost unflushed writes, the next fsync returns an error instead of a false success.

Someone else mentioned: "write() is buffered (that's the batching) and "committed" maps to fsync(), which returns only once data is durable."

---

It sounds like all writes are written synchronously to at least one node but failovers/replicas are just eventually consistent. If so, latency between nodes is not within ZeroFS's control and including. Or are you saying that the latency is impossible for even a single node? If so, that would mean much more than just a footnote is needed.

---

I don't see an issue with the benchmark but I might not be looking in the right place.

The [sequential writes](https://github.com/Barre/ZeroFS/blob/ec32199d48d0409d4cccd44...) and [append-only writes](https://github.com/Barre/ZeroFS/blob/ec32199d48d0409d4cccd44...) start where I'd expect and `success: true,` should equate to `fsync()` as previously mentioned.

(Apologies if I got this wrong, still learning this)

Re: Show HN: ZeroFS – A log-structured filesystem for S3

#59

Post by gvkhna »

rockwotj wrote: post #27

The sub-millisecond writes with data in S3 is false and impossible. If you look at the benchmark the fsync is not timed, so this is just the latency of either the network or in kernel file operations depending on the mount settings

Incidentally the fact that you could read and write from essentially thousands of harddrives at once for even a tiny file makes you wonder what novel use cases you could get out of that.

Re: Show HN: ZeroFS – A log-structured filesystem for S3

#60

Post by chillfox »

Eikon wrote: post #40
chillfox wrote: post #38

I don’t get why they went for NFSv3, v4 is quite old and I can’t think of any reason why you would choose v3 over v4.

NFSv4 is a hard beast to implement correctly, with a lot of protocol surface (state, compound ops, delegations) for benefits ZeroFS mostly gets through 9P with extensions, over a much simpler protocol: https://www.zerofs.net/docs/9p-extensions . NFSv3 stayed in ZeroFS mostly for client compatibility.

Thanks for explaining it.
Post reply on HN