ZFS: Apple's new filesystem that wasn't (2016) • Hacker News Board

Re: ZFS: Apple's new filesystem that wasn't (2016)

#181

Post by DannyBee » Mon, Apr 28, 2025, 5:33 PM UTC

cryptonector wrote: post #144

throw0101b wrote: post #37

Apple and Sun couldn't agree on a 'support contract'. From Jeff Bonwick, one of the co-creators ZFS: >> Apple can currently just take the ZFS CDDL code and incorporate it (like they did with DTrace), but it may be that they wanted a "private license" from Sun (with appropriate technical support and indemnification), and the two entities couldn't come to mutually agreeable terms. > I cannot disclose details, but that…

> indemnification That was the sticking point. In the context of the NetApp lawsuit Apple wanted indemnification should Sun/Oracle lose the suit.

That would be surprising - indemnification is usually very cheap to provide even for this sort of thing - there is always an insurer willing to sell you the policy you need to provide it at some reasonable price :)

Re: ZFS: Apple's new filesystem that wasn't (2016)

#182

Post by DannyBee » Mon, Apr 28, 2025, 5:35 PM UTC

alwillis wrote: post #71

jitl wrote: post #2

Besides the licensing issue, I wonder if optimizing ZFS for low latency + low RAM + low power on iPhone was an uphill battle or if it’s easy. My experiencing running ZFS years ago was poor latency and large RAM use with my NAS, but that hardware and drive configuration was optimized for low $ per gb stored and used parity stuff.

Apple wanted one operating system that ran on everything from a Mac Pro to an Apple Watch and there’s no way ZFS could have done that.

ZFS is even used on my PLC to provide consistent snapshots. So uh, going to go with "it would probably work fine".

Re: ZFS: Apple's new filesystem that wasn't (2016)

#183

Post by kjkjadksj » Mon, Apr 28, 2025, 5:39 PM UTC

GeekyBear wrote: post #171

whartung wrote: post #63

As a desktop user, I am content with APFS. The only feature from ZFS that I would like, is the corruption detection. I honestly don't know how robust the image and video formats are to bit corruption. On the one hand, potentially, "very" robust. But on the other, I would think that there are some very special bits that if toggled can potentially "ruin" the entire file. But I don't know. However, I can say, every time…

Back in 2011, Apple paid $500 million for Anobit, a company producing enterprise grade SSD controllers. Since Apple was already integrating custom SSD controllers onto their A series SOCs, presumably the purchase was about Anobit patents. > Anobit appears to be applying a lot of signal processing techniques in addition to ECC to address the issue of NAND reliability and data retention... promising significant improve…

What is the devices lifetime in apples eyes? 5 years till support is dropped? Gulp.

Re: ZFS: Apple's new filesystem that wasn't (2016)

#184

Post by freedomben » Mon, Apr 28, 2025, 5:43 PM UTC

freedomben wrote: post #176

wkat4242 wrote: post #136

Earlier quoted context omitted.

I just mean that GPL is a bit of a religion. There are very strong opinions and principles behind it. Whereas the BSD license is more like "do whatever you want". It makes sense that the followers of the former care more deeply about it, right? Personally I don't care about or obey any software licenses, as a user. But this is kinda the vibe I get from other BSD users if a license discussion comes up. Maybe it's my b…

I have to strongly disagree with you that GPL is a religion. While there Are certainly people who sound religious in nature when they talk about the GPL, The real reason it is important is because of what it forces developers to do on behalf of the users. It balances somewhat the power between the developer and the user. Under other licensing, developers wield an extraordinary amount of power over the users. Yes, The…

Oops that last sentence (which is a sentence fragment) was supposed to be deleted, but slipped in some how and it's too late to edit. The point I was going to make was just that with the code being out there, the odds that some offensive thing the devs might do can be discovered by someone and have the issue raised. It also provides a powerful incentive to not stick something gross in there for risk of it being discovered and getting called out for it :-)

Re: ZFS: Apple's new filesystem that wasn't (2016)

#185

Post by DannyBee » Mon, Apr 28, 2025, 5:43 PM UTC

ein0p wrote: post #112

ZFS sort of moved inside the NVMe controller - it also checksums and scrubs things all the time, you just don't see it. This does not, however, support multi-device redundant storage, but that is not a concern for Apple - the vast majority of their devices have only one storage device.

Everyone seems to focus on the checksumming and scrubbing.

But the real win (at least for me) - every device i have - laptops, desktops, server, even PLC's (they now use freebsd under the covers + ZFS) all backup using zfs snapshots and replication.

I do not ever worry about finding an old file i accidentally deleted. Or restoring a backup to a new machine and "did it really include everything" or anything else.

The machine storing backups is itself replicated to another machine in my detached garage.

If i wanted even more security, i could trivially further replicate it to offsite storage in the same manner.

All of this takes ~0 time to set up, and require 0 maintenance to keep working.

Meanwhile, Apple has gone backwards - time machine can't even make actual full system backups anymore.

Re: ZFS: Apple's new filesystem that wasn't (2016)

#186

Post by zie » Mon, Apr 28, 2025, 5:54 PM UTC

krupan wrote: post #172

zie wrote: post #27

Earlier quoted context omitted.

Basically it's that memory changes out from under you. As we know, computers use Binary, so everything boils down to it being a 0 or a 1. A bit flip is changing what was say a 0 into a 1. Usually attributed to "cosmic rays", but really can happen for any number of less exciting sounding reasons. Basically, there is zero double checking in your computer for almost everything except stuff that goes across the network.…

"Basically, there is zero double checking in your computer for almost everything except stuff that goes across the network." This is so not true. All the high speed busses (QPI, UPI, DMI, PCIe, etc.) have "bit flip" protection in multiple layers: differential pair signaling, 8b/10b (or higher) encoding, and packet CRCs. Hard drives (the old spinning rust kind) store data along with a CRC. SSD/NVMe drives use strong E…

I agree that transfers(like the high speed buses) have some checks to ensure transfers happen properly, but that doesn't help much if the data is/was corrupted on either side.

> If most filesystems don't do integrity checks it's probably because there's not much need to.

I would disagree that disks alone are good enough for daily consumer use. I see corruption often enough to be annoying with consumer grade hardware without ECC & ZFS. Small images are where people usually notice. They tend to be heavily compressed and small in size means minor changes can be more noticeable. In larger files, corruption tends to not get noticed as much in my experience.

We have 10k+ consumer devices at work and corruption is not exactly common, but it's not rare either. A few cases a year are usually identified at the helpdesk level. It seems to be going down over time, since hardware is getting more reliable, we have a strong replacement program and most people don't store stuff locally anymore. Our shared network drives all live on machines with ECC & ZFS.

We had a cloud provider recently move some VM's to new hardware for us, the ones with ZFS filesystems noticed corruption, the ones with ext4/NTFS/etc filesystems didn't notice any corruption. We made the provider move them all again, the second time around ZFS came up clean. Without ZFS we would have never known, as none of the EXT4/NTFS FS's complained at all. Who knows if all the ext4/NTFS machines were corruption free, it's anyone's guess.

Re: ZFS: Apple's new filesystem that wasn't (2016)

#187

Post by Marsymars » Mon, Apr 28, 2025, 5:55 PM UTC

GeekyBear wrote: post #171

whartung wrote: post #63

As a desktop user, I am content with APFS. The only feature from ZFS that I would like, is the corruption detection. I honestly don't know how robust the image and video formats are to bit corruption. On the one hand, potentially, "very" robust. But on the other, I would think that there are some very special bits that if toggled can potentially "ruin" the entire file. But I don't know. However, I can say, every time…

Back in 2011, Apple paid $500 million for Anobit, a company producing enterprise grade SSD controllers. Since Apple was already integrating custom SSD controllers onto their A series SOCs, presumably the purchase was about Anobit patents. > Anobit appears to be applying a lot of signal processing techniques in addition to ECC to address the issue of NAND reliability and data retention... promising significant improve…

That's excellent if you're using Apple SSD storage, but seems unhelpful if you're running, say, a large array of HDDs connected to your Mac.

Re: ZFS: Apple's new filesystem that wasn't (2016)

#188

Post by zie » Mon, Apr 28, 2025, 5:55 PM UTC

ewuhic wrote: post #110

What's the current state of ZFS on Macos? As far as I'm aware there's a supported fork.

Exactly: https://openzfsonosx.org

Works just fine for me.

Re: ZFS: Apple's new filesystem that wasn't (2016)

#189

Post by Marsymars » Mon, Apr 28, 2025, 5:59 PM UTC

water9 wrote: post #147

TacticalCoder wrote: post #104

Earlier quoted context omitted.

> The only feature from ZFS that I would like, is the corruption detection. I run ZFS on my main server at home (Proxmox: a Linux hypervisor based on Debian and Proxmox ships with ZFS) but... No matter the FS, for "big" files that aren't supposed to change, I append a (partial) cryptographic checksum to the filename. For example: 20240238-familyTripBari.mp4 becomes 20240238-familyTripBari-b3-8d77e2419a36.mp4 where "-…

The checksum doesn’t help you fix the flipped bit nor does it tell you which bit flipped. You would have to re-create from a complete back up instead of using the efficiency of parity discs. Basically Raid 1 vs Raid 5

There are tools that give you hashing + parity, e.g. SnapRAID: https://www.snapraid.it/

Re: ZFS: Apple's new filesystem that wasn't (2016)

#190

Post by Marsymars » Mon, Apr 28, 2025, 6:06 PM UTC

badc0ffee wrote: post #99

wkat4242 wrote: post #48

Earlier quoted context omitted.

That's called marketing. Give it a snazzy name, like say "TimeMachine" and users will jump on it. Also, ZFS has a bad name within the Linux community due to some licensing stuff. I find that most BSD users don't really care about such legalese and most people I know that run FreeBSD are running ZFS on root. Which works amazingly well I might add. Especially with something like sanoid added to it, it basically does th…

Time Machine was released 17 years ago, and I wish Windows had anything that good. And they're on their 3rd backup system since then.

Time Machine is good for the technically savvy, but for the non-tech-savvy, without something like the AirPort Time Capsule there's no real easy configuration for it.