Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS in Detail

dtrace.org

Post reply on HN
Print view View original

21–30 of 98 posts

Re: APFS in Detail

#21

Post by ahl »

veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

Talking to the Apple engineers it really didn't seem to be an issue of computation. They seemed genuine in their belief that they could solve data integrity with device qualification. While I asked them 100 questions they asked me 2: had I ever actually seen bit rot (yes), and what kind of drives did we ship with the ZFS Storage Appliance (mostly 7200 nearline drives).

Re: APFS in Detail

#22

Post by asveikau »

veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

> The engineers contend that Apple devices basically don’t return bogus data.

Holy shit.

I guess that explains why my Mac recently had a bunch of daemons burning all cpu crashing repeatedly in a tight loop when getting sqlite errors on a db in ~/Library. Cause disk corruption never happens.

Re: APFS in Detail

#23

Post by rdtsc »

veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

Hmm, I guess like there is the "sufficiently smart compiler" falacy, there is now the "sufficently reliable hardware" falacy.

But on another level, I guess if hardware fails, then well, you buy more hardware, which is good for Apple. Presumably people who bought in the past from Apple won't turn around and buy an Acer or HP laptop. They'll still buy Apple.

Re: APFS in Detail

#24

Post by amluto »

> I get that maybe a checksumming filesystem could conceivably be too computationally expensive for the little jewelry-computers Apple is into these days, but it's a terrible omission on something that is supposed to be the new filesystem for macOS.

Checksumming has another cost that isn't immediately obvious. Suppose you write to a file and the writes are cached. Then the filesystem starts to flush to disk. On a conventional filesystem, you can keep writing to the dirty page while the disk DMAs data out of it. On a checksumming filesystem, you can't: you have to compute the checksum and then write out data consistent with the checksum. This means you either have to delay user code that tries to write, or you have to copy the page, or you need hardware support for checksumming while writing.

On Linux, this type of delay is called "stable pages", and it destroys performance on some workloads on btrfs.

Re: APFS in Detail

#25

Post by jakobegger »

rdtsc wrote: post #23
veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

Hmm, I guess like there is the "sufficiently smart compiler" falacy, there is now the "sufficently reliable hardware" falacy. But on another level, I guess if hardware fails, then well, you buy more hardware, which is good for Apple. Presumably people who bought in the past from Apple won't turn around and buy an Acer or HP laptop. They'll still buy Apple.

If hardware fails silently, you won't buy more hardware. You'll just come across something odd and say, hmm, typical Apple-bugginess.

It would be much nicer if your computer said, “I've detected a bit flip, please restore this file from backup”

Re: APFS in Detail

#26

Post by rdtsc »

jakobegger wrote: post #25
rdtsc wrote: post #23

Earlier quoted context omitted.

Hmm, I guess like there is the "sufficiently smart compiler" falacy, there is now the "sufficently reliable hardware" falacy. But on another level, I guess if hardware fails, then well, you buy more hardware, which is good for Apple. Presumably people who bought in the past from Apple won't turn around and buy an Acer or HP laptop. They'll still buy Apple.

If hardware fails silently, you won't buy more hardware. You'll just come across something odd and say, hmm, typical Apple-bugginess. It would be much nicer if your computer said, “I've detected a bit flip, please restore this file from backup”

Even more fun. Data gets corrupted, and backups pick it up, and start overwriting good backups with it eventually.

Re: APFS in Detail

#27

Post by ahl »

rdtsc wrote: post #23
veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

Hmm, I guess like there is the "sufficiently smart compiler" falacy, there is now the "sufficently reliable hardware" falacy. But on another level, I guess if hardware fails, then well, you buy more hardware, which is good for Apple. Presumably people who bought in the past from Apple won't turn around and buy an Acer or HP laptop. They'll still buy Apple.

The failure mode of hardware is not all or nothing. A single sector in an HDD or a single page in an SSD can fail and the rest can be fine for years; both HDDs and SSDs have many spares for this expected condition.

Re: APFS in Detail

#28

Post by comex »

idorosen wrote: post #2

In my opinion, APFS does not seem to improve upon ZFS in several key areas (compression, sending/receiving snapshots, dedup, etc.). Apple is reimplementing many features already implemented in OpenZFS, btrfs (which itself reimplemented a lot of ZFS features), BSD HAMMER, etc. Maybe extending one of these existing filesystems to add any functionality Apple needs on top of its existing features (and, hopefully, contrib…

> Either way, we will now have yet another filesystem to contend with, implement in non-Darwin kernels (maybe), and this adds to the overall support overhead of all operating systems that want to be compatible with Apple devices.

On the other hand, if Apple decides to open source the APFS implementation (hard to tell what their plans are from current statements, but I'm holding out hope), it'll probably be under a permissive license that allows porting to Linux. The implementation is in C (not C++) so porting is probably generally feasible. Compare to ZFS, which, even if some distros have finally started shipping it, will never quite be free of licensing issues unless Oracle does a 180.

Re: APFS in Detail

#29

Post by vvhn »

idorosen wrote: post #2

In my opinion, APFS does not seem to improve upon ZFS in several key areas (compression, sending/receiving snapshots, dedup, etc.). Apple is reimplementing many features already implemented in OpenZFS, btrfs (which itself reimplemented a lot of ZFS features), BSD HAMMER, etc. Maybe extending one of these existing filesystems to add any functionality Apple needs on top of its existing features (and, hopefully, contrib…

>and this adds to the overall support overhead of all operating systems that want to be compatible with Apple devices.

What operating systems describe tremselves as being "compatible with Apple devices"

> Since the older versions of macOS (OSX) don't support APFS, only HFS+, this means Apple and others will also have to continue supporting HFS+.

Who else actually "supports" HFS+ ? Sure there are linux "ports" based on the spec but nobody claims them as being "supported". Apple would have had to continue supporting HFS+ whether they chose to implement ZFS, btrfs or HAMMER.

>It just seems wasteful of everyone's time to me.

I don't know how Apple writing their own filesystem is wasteful of anybody else's time ( except possibly Apple's and/or Disk utility software for vendors for OS X)

>Also: https://xkcd.com/927/

The standard is the interface ( POSIX / SUS ) and unless APFS breaks that how is this applicable ?

Re: APFS in Detail

#30

Post by chongli »

isomorphic wrote: post #19
veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

I agree; no checksumming of user data is very disappointing. If there were performance issues, they could build checksumming into the filesystem, but make it a volume-specific option. No checksumming on the watch, strong integrity guarantees on the Mac Pro. Their filesystem goals are in some ways consistent with Apple's (marketing) vision: Users would never have terabyte libraries of anything, as the various iService…

I don't see why Apple couldn't just recommend that their pro users who have need of this sort of data integrity locally run their own server with FreeBSD + ZFS. Apple has really backed off on their attempts to market OS X Server to this crowd. Heck, they're probably using FreeBSD already if they need that much data integrity.
Post reply on HN