Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS in Detail

dtrace.org

Post reply on HN
Print view View original

41–50 of 98 posts

Re: APFS in Detail

#41

Post by ghshephard »

For example, my 1TB SSD includes 1TB (2^30 = 1024^3) bytes of flash but only reports 931GB of available space, sneakily matching the storage industry’s self-serving definition of 1TB (1000^3 = 1 trillion bytes).

Great article, but a couple nitpicking corrections (which seem appropriate for a storage article) Per: https://en.wikipedia.org/wiki/Terabyte - Terabyte is 1000^4, not 1000^3.

Also It's been 6+ years since we all agreed that TiB means 2^40 or 1024^4, and TB means 10^12. Indeed, only in the case of memory does "T" ever mean 2^40 anyways. It's always been the case that in both data rates, as well as storage, that T means 10^12. This convention is strong enough that we most of us just have thrown up our hands and agree when referring to DRAM memory, that Terabyte will mean 1024^4, and 1000^4 everywhere else.

Indeed, in the rare case where someone uses TiB to refer to a data rate, they are almost without exception incorrectly using it, and, they actually mean TB.

Re: APFS in Detail

#42

Post by kev009 »

gmac wrote: post #40

Slightly worried by the vibe that comes off this. "I asked him about looking for inspiration in other modern file systems ... he was aware of them, but didn’t delve too deeply for fear, he said, of tainting himself". And (to paraphrase): 'bit-rot? What's that?'. I would have hoped that a new filesystem with such wide future adoption would have come from a roomful of smart people with lots of experience of (for exampl…

Not looking at HAMMER/HAMMER2 is just NIH too, no 'taint' from BSD as Apple should well know on their UNIX side.

Re: APFS in Detail

#43

Post by XorNot »

cyphar wrote: post #33
rdtsc wrote: post #26

Earlier quoted context omitted.

Even more fun. Data gets corrupted, and backups pick it up, and start overwriting good backups with it eventually.

If your backup system involves overwriting old backups, it's not a backup system. It's a data loss system.

Storage is not that cheap yet.

Re: APFS in Detail

#44

Post by kev009 »

ahl wrote: post #21
veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

Talking to the Apple engineers it really didn't seem to be an issue of computation. They seemed genuine in their belief that they could solve data integrity with device qualification. While I asked them 100 questions they asked me 2: had I ever actually seen bit rot (yes), and what kind of drives did we ship with the ZFS Storage Appliance (mostly 7200 nearline drives).

That's dumbfounding. I know first hand a certain monthly-fee movie streaming service and the CDN I work for can tell anyone who wants to hear about handling silent corruption and bit rot and we have a relatively small fleets. At home ZFS saved me from a faulty power supply on my old workstation.

And.. the red herring here is, Apple users will want to plug in third party storage. There's just no way to contain what someone will plug in to USB and ThunderBolt, and it's insane to think APFS would not be ready to help there.

Re: APFS in Detail

#45

Post by ghshephard »

ahl wrote: post #21
veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

Talking to the Apple engineers it really didn't seem to be an issue of computation. They seemed genuine in their belief that they could solve data integrity with device qualification. While I asked them 100 questions they asked me 2: had I ever actually seen bit rot (yes), and what kind of drives did we ship with the ZFS Storage Appliance (mostly 7200 nearline drives).

That would suggest that APFS is only relevant for internal storage procured by Apple. Do they not intend for it to be used on external storage?

Re: APFS in Detail

#46

Post by ghshephard »

ahl wrote: post #15
TazeTSchnitzel wrote: post #6

> Also, APFS removes the most common way of a user achieving local data redundancy: copying files. A copied file in APFS actually creates a lightweight clone with no duplicated data. No, it doesn't. APFS supports copying files, if you want that. It's just that the default in Finder is to make a “clone” (copy-on-write).

Fair enough; and right now cp doesn't use the fast clone functionality, but it assuredly will. I'm not sure 'cat file.dup' is reasonable for most users.

Just teach everyone to use dd instead - dd if=file of=file.dup :-)

Re: APFS in Detail

#47

Post by cyphar »

XorNot wrote: post #43
cyphar wrote: post #33

Earlier quoted context omitted.

If your backup system involves overwriting old backups, it's not a backup system. It's a data loss system.

Storage is not that cheap yet.

Making backups more granular means you remove sets of backups (or you collapse incremental backups). If a new backup causes corruption to back-propogate then it's not a backup.

Re: APFS in Detail

#48

Post by leonroy »

veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

Agreed. What an arrogant attitude especially considering their Macbook Pro (2015) recently had a corruption bug which necessitated a firmware update: https://support.apple.com/kb/DL1830?locale=en_GB

Good checksumming to detect bit rot is exactly what is needed since as an owner of said laptop I have NO idea whether any of my data was affected.

If Apple want to say 'the majority of our devices are mobile and checksumming puts a large performance overhead' then that's one thing. But to claim it's not needed is just plain wrong and makes me worry that Apple's product managers sit in an echo chamber hearing only what they want to hear.

Re: APFS in Detail

#49

Post by therein »

chongli wrote: post #30
isomorphic wrote: post #19

Earlier quoted context omitted.

I agree; no checksumming of user data is very disappointing. If there were performance issues, they could build checksumming into the filesystem, but make it a volume-specific option. No checksumming on the watch, strong integrity guarantees on the Mac Pro. Their filesystem goals are in some ways consistent with Apple's (marketing) vision: Users would never have terabyte libraries of anything, as the various iService…

I don't see why Apple couldn't just recommend that their pro users who have need of this sort of data integrity locally run their own server with FreeBSD + ZFS. Apple has really backed off on their attempts to market OS X Server to this crowd. Heck, they're probably using FreeBSD already if they need that much data integrity.

I hear they use RHEL nowadays.

Re: APFS in Detail

#50

Post by amelius »

I think the value of this new proprietary filesystem is limited, since you can't run it on servers (Apple does not make servers anymore). Also, compatibility/porting issues may become a problem if you build your software for it.
Post reply on HN