Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS in Detail

dtrace.org

Post reply on HN
Print view View original

31–40 of 98 posts

Re: APFS in Detail

#31

Post by veidr »

chongli wrote: post #30
isomorphic wrote: post #19

Earlier quoted context omitted.

I agree; no checksumming of user data is very disappointing. If there were performance issues, they could build checksumming into the filesystem, but make it a volume-specific option. No checksumming on the watch, strong integrity guarantees on the Mac Pro. Their filesystem goals are in some ways consistent with Apple's (marketing) vision: Users would never have terabyte libraries of anything, as the various iService…

I don't see why Apple couldn't just recommend that their pro users who have need of this sort of data integrity locally run their own server with FreeBSD + ZFS. Apple has really backed off on their attempts to market OS X Server to this crowd. Heck, they're probably using FreeBSD already if they need that much data integrity.

Here's the thing: everybody needs this sort of data integrity.

Literally nobody wants their files to be silently corrupted. ZFS made it much easier for (nerds like us) to attain very high levels of data integrity.

APFS was (and maybe still is?) a chance to make that the default for regular people.

Re: APFS in Detail

#32

Post by anexprogrammer »

chongli wrote: post #30
isomorphic wrote: post #19

Earlier quoted context omitted.

I agree; no checksumming of user data is very disappointing. If there were performance issues, they could build checksumming into the filesystem, but make it a volume-specific option. No checksumming on the watch, strong integrity guarantees on the Mac Pro. Their filesystem goals are in some ways consistent with Apple's (marketing) vision: Users would never have terabyte libraries of anything, as the various iService…

I don't see why Apple couldn't just recommend that their pro users who have need of this sort of data integrity locally run their own server with FreeBSD + ZFS. Apple has really backed off on their attempts to market OS X Server to this crowd. Heck, they're probably using FreeBSD already if they need that much data integrity.

It isn't just pro users.

With TB file systems, assuming you haven't outsourced everything to iCloud, data integrity matters. If you have, now you're trusting them not to screw up, ever.

From the movie or mp3 that mysteriously no longer plays, through to more important things - business data or family photos. I suspect many people have experienced bit rot, even if they don't recognise it as such. We've even reached a point where with quoted drive figures copying 2tb from one drive to another will likely result in a bit flip (source - Ars ZFS+btrfs article a couple of years back).

Heck, most people have some level of data loss from a HDD or flash drive fail. Sometimes even when they tried to do all the right things. Only question is whether it was backed up. In the case of personal users, unlikely. Self healing could have been quite some selling point!

Re: APFS in Detail

#33

Post by cyphar »

rdtsc wrote: post #26
jakobegger wrote: post #25

Earlier quoted context omitted.

If hardware fails silently, you won't buy more hardware. You'll just come across something odd and say, hmm, typical Apple-bugginess. It would be much nicer if your computer said, “I've detected a bit flip, please restore this file from backup”

Even more fun. Data gets corrupted, and backups pick it up, and start overwriting good backups with it eventually.

If your backup system involves overwriting old backups, it's not a backup system. It's a data loss system.

Re: APFS in Detail

#34

Post by nier »

veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

When all you have is Apple’s Disk Utility.app, all storage media is perfect. That was irony. Truth is hard drives can have more than 30 bad blocks and still have a verified S.M.A.R.T. status in their app.

I recommend sending every file system engineer on a year-long journey as a traveling system integrator.

Re: APFS in Detail

#35

Post by snarf »

matthewmcg wrote: post #5
idorosen wrote: post #2

In my opinion, APFS does not seem to improve upon ZFS in several key areas (compression, sending/receiving snapshots, dedup, etc.). Apple is reimplementing many features already implemented in OpenZFS, btrfs (which itself reimplemented a lot of ZFS features), BSD HAMMER, etc. Maybe extending one of these existing filesystems to add any functionality Apple needs on top of its existing features (and, hopefully, contrib…

Agreed. A few years ago Apple was on the verge of using ZFS but cancelled the project due to licensing issues. See http://arstechnica.com/apple/2009/10/apple-abandons-zfs-on-m...

A better/more up to date article on this from the same blog as this item: http://dtrace.org/blogs/ahl/2016/06/15/apple_and_zfs/

Re: APFS in Detail

#36

Post by tfar »

mindajar wrote: post #9
idorosen wrote: post #2

In my opinion, APFS does not seem to improve upon ZFS in several key areas (compression, sending/receiving snapshots, dedup, etc.). Apple is reimplementing many features already implemented in OpenZFS, btrfs (which itself reimplemented a lot of ZFS features), BSD HAMMER, etc. Maybe extending one of these existing filesystems to add any functionality Apple needs on top of its existing features (and, hopefully, contrib…

Classic comic, but I don't think it applies. APFS looks intended to solve Apple's product problems really well, and it doesn't even try to be a filesystem for everyone. Apple has said from time to time that they're all about owning and controlling the key technologies that go into their products. APFS makes a lot of sense from that perspective, and this seems one of those cases where going their own way is better tha…

I would not be surprised if one could write a ZFS implementation optimized for more constrained devices. If you already know you are going to a have flash storage you can probably ditch some of the N layers of cache you see in common ZFS implementations. Not that ZFS is a one size fits all, but the file systems specification could be implemented in more than one way.

Re: APFS in Detail

#37

Post by maureenash »

APFS makes no claims with regard to data redundancy. As Apple's Eric Tamura noted at WWDC, most Apple devices have a single storage. A file system is method used by a computer to store and retrieve data.Custom essay writing service (http://www.essayguardian.com/) helps to reduce your workload while writing dissertation proposal and meets all your requirements.

Re: APFS in Detail

#38

Post by chongli »

anexprogrammer wrote: post #32
chongli wrote: post #30

Earlier quoted context omitted.

I don't see why Apple couldn't just recommend that their pro users who have need of this sort of data integrity locally run their own server with FreeBSD + ZFS. Apple has really backed off on their attempts to market OS X Server to this crowd. Heck, they're probably using FreeBSD already if they need that much data integrity.

It isn't just pro users. With TB file systems, assuming you haven't outsourced everything to iCloud, data integrity matters. If you have, now you're trusting them not to screw up, ever. From the movie or mp3 that mysteriously no longer plays, through to more important things - business data or family photos. I suspect many people have experienced bit rot, even if they don't recognise it as such. We've even reached a…

I have experienced many bitrotted mp3s in my day. Thankfully I've been able to replace them online. As for other files? I can't recall any that are now unable to open for mysterious reasons.

I also happen to run a home file server on FreeBSD + ZFS, though I don't think that machine has ECC memory so it is still technically vulnerable to corruption.

Re: APFS in Detail

#39

Post by ot »

veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

If the storage is 100% reliable, why do they checksum the metadata?

Re: APFS in Detail

#40

Post by gmac »

Slightly worried by the vibe that comes off this. "I asked him about looking for inspiration in other modern file systems ... he was aware of them, but didn’t delve too deeply for fear, he said, of tainting himself". And (to paraphrase): 'bit-rot? What's that?'.

I would have hoped that a new filesystem with such wide future adoption would have come from a roomful of smart people with lots of experience of (for example) contributing to various modern filesystems, understanding their strengths and weaknesses, and dealing with data corruption issues in the field. This doesn't come across that way at all.

Post reply on HN