Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS in Detail

dtrace.org

Post reply on HN
Print view View original

71–80 of 98 posts

Re: APFS in Detail

#71

Post by benmmurphy »

veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

i think it makes a lot of sense. turning on these kinds of checks can be scary. the current situation is mostly no-one is effected by bit-rot. this is probably because when it rarely happens it flips some bits that don't really matter anyway. but as soon as you turn on checksumming in software without any automatic error correction people are going to start freaking out when their files become inaccessible or they have to jump through some hoops to access the 'corrupted' file which looks entirely fine to them anyway.

same deal with some heap protections. say you are running a kernel which doesn't have byte patterns to detect heap overflows or reuse after free. maybe you have some heap overflows which because of their nature never cause any corruption but now you turn on heap protections and peoples kernels are getting more panics :/

Re: APFS in Detail

#72

Post by rdtsc »

cyphar wrote: post #33
rdtsc wrote: post #26

Earlier quoted context omitted.

Even more fun. Data gets corrupted, and backups pick it up, and start overwriting good backups with it eventually.

If your backup system involves overwriting old backups, it's not a backup system. It's a data loss system.

Unless you have infinite storage, you'll have to overwrite some backups at some point in the future.

Re: APFS in Detail

#73

Post by josho »

benmmurphy wrote: post #71
veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

i think it makes a lot of sense. turning on these kinds of checks can be scary. the current situation is mostly no-one is effected by bit-rot. this is probably because when it rarely happens it flips some bits that don't really matter anyway. but as soon as you turn on checksumming in software without any automatic error correction people are going to start freaking out when their files become inaccessible or they ha…

What is the user experience for when a checksumming filesystem detects an error?

If the fs detects a bit error does it flag the file as entirely unreadable? Move it to lost+found? Force me to restore the file from a backup? All these options seem more scary for an end user than blissful ignorance.

Don't misunderstand me, I've lost a few family photos over the years due to bit rot. So, I appreciate a fs that offers more protections. But, I honestly don't know offhand how an end user would recover from an error in /System or even an error in a family photo, or for that matter a word doc.

Re: APFS in Detail

#74

Post by deagle50 »

tveita wrote: post #70
veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

> The engineers contend that Apple devices basically don’t return bogus data. It's much easier to pretend that this is the case when the file system isn't verifying it. Checksumming would probably expose problems that would otherwise go unnoticed by users or be blamed on computer gremlins. It's hard to say if doing the "correct" thing here would improve the subjective user experience. Maybe putting on airs of infalli…

Good point. Going by the tone of Apple engineers' response, it sure does sound like they are going for plausible deniability.

Re: APFS in Detail

#75

Post by pwg »

ghshephard wrote: post #53
veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

A lot of the CoW patents, WAFL, snapshot patents that Network Appliance filed in the late 1990s have expired, or are expiring this year. For example, https://www.google.com/patents/US6289356 was filed in 1998, so I presume it's expiring fairly soon. Given that some of the original lawsuits were Network Appliance suing Sun/Oracle, I'm wondering how much of a role this played in the timing of the release of these featu…

6289356 also lists a "priority date" on the google page of 1993. If that is an actual "priority date" rather than a google metadata "add on" then this patent expired (absent any term extensions) on June 3, 2013.

Re: APFS in Detail

#76

Post by codemac »

cm3 wrote: post #63
codemac wrote: post #18

Earlier quoted context omitted.

Checksums are usually very fast to compute as the ARM CPUs of any modern phone have crypto engines, and their laptops do as well. I think trading data protection for performance reasons would be pretty irrational.

Except that with NVME drives, and the parallel operations you can run on these, the performance of checksums becomes important again. Recent experiments with HAMMER2: https://www.dragonflydigest.com/2016/06/15/18281.html

Yeah, the ever present march of storage --> memory has really put a strain on our current compute architectures. Thanks for the note, will be reading more about it.

Re: APFS in Detail

#77

Post by glhaynes »

josho wrote: post #73
benmmurphy wrote: post #71

Earlier quoted context omitted.

i think it makes a lot of sense. turning on these kinds of checks can be scary. the current situation is mostly no-one is effected by bit-rot. this is probably because when it rarely happens it flips some bits that don't really matter anyway. but as soon as you turn on checksumming in software without any automatic error correction people are going to start freaking out when their files become inaccessible or they ha…

What is the user experience for when a checksumming filesystem detects an error? If the fs detects a bit error does it flag the file as entirely unreadable? Move it to lost+found? Force me to restore the file from a backup? All these options seem more scary for an end user than blissful ignorance. Don't misunderstand me, I've lost a few family photos over the years due to bit rot. So, I appreciate a fs that offers mo…

If the fs detects a bit error does it flag the file as entirely unreadable? Move it to lost+found? Force me to restore the file from a backup?

For files stored in iCloud Drive, if that version of the file exists in the cloud, the OS could automatically re-fetch the file. But, yeah, for lots of circumstances there's not going to be a "good" option to give the user.

EDIT: Same applies to Time Machine (or whatever Apple's backup solution will be called in the APFS era).

Re: APFS in Detail

#78

Post by drvdevd »

cm3 wrote: post #66

I'm still looking for a widely supported (at least FreeBSD and Linux kernels) filesystem for external drives to carry around that doesn't have the FAT32 limitations. There's exFAT but no stable and supported implementation. Then there's NTFS, but that's also not 100% reliable in my experience when used through FUSE (NTFS-3G). I've considered UFS but that also was a no go. I'm hopeful for lklfuse[1] that also runs on…

Thanks for this! I didn't realize (or had forgotten) LUKS had been ported to Dragonfly. Also you touch upon my #1 frustration with APFS without really knowing anything about it: simple portability.

Re: APFS in Detail

#79

Post by idorosen »

vvhn wrote: post #29
idorosen wrote: post #2

In my opinion, APFS does not seem to improve upon ZFS in several key areas (compression, sending/receiving snapshots, dedup, etc.). Apple is reimplementing many features already implemented in OpenZFS, btrfs (which itself reimplemented a lot of ZFS features), BSD HAMMER, etc. Maybe extending one of these existing filesystems to add any functionality Apple needs on top of its existing features (and, hopefully, contrib…

>and this adds to the overall support overhead of all operating systems that want to be compatible with Apple devices. What operating systems describe tremselves as being "compatible with Apple devices" > Since the older versions of macOS (OSX) don't support APFS, only HFS+, this means Apple and others will also have to continue supporting HFS+. Who else actually "supports" HFS+ ? Sure there are linux "ports" based o…

> What operating systems describe tremselves as being "compatible with Apple devices"

I was referring to the Linux kernel modules implementing HFS+ and other Apple FSes.

> Who else actually "supports" HFS+ ? Sure there are linux "ports" based on the spec but nobody claims them as being "supported".

Yes, by support I meant other developers who want to be able to read and write to devices in APFS format.

> Apple would have had to continue supporting HFS+ whether they chose to implement ZFS, btrfs or HAMMER.

Yes, Apple would have to continue supporting HFS+, but other kernel developers would not have to port yet another filesystem (APFS) with all of its own quirks; and, who knows, maybe it would be less work for Apple to inherit ZFS/btrfs/HAMMER/some other filesystem's solutions to some of the same problems they're trying to solve from scratch here. My point was more that by reinventing the wheel to implement some of these features, they've created not just more work for themselves potentially, but more for the open source kernel development community as well in the long run.

> I don't know how Apple writing their own filesystem is wasteful of anybody else's time ( except possibly Apple's and/or Disk utility software for vendors for OS X)

APFS will find its way to external HDDs/SSDs/flash drives, etc., then in order to read those filesystems someone else will have to port it to any other devices/readers of that device/FS.

> The standard is the interface ( POSIX / SUS ) and unless APFS breaks that how is this applicable ?

I didn't mention POSIX, VFS, or filesystem _interfaces_. The analogy to the XKCD strip was that we already had N filesystems that have a large subset of (or in some cases superset of) the features of APFS as of right now, now we have N+1 complex filesystems to contend with and port and interoperate with in other kernels/OSes (mainly Linux + non-Darwin BSDs).

This may just be the price of progress, which is fine. I think it'll be fantastic if Apple makes progress in this area and improves upon the work of others. The developer seemed to be ignoring history so as not to "taint" himself (did he mean IP/legally tainted?), which is slightly worrying to me.

I hope Apple open sources their implementation under a BSD/GPL dual license to make it easier for others to port it directly into other kernels, rather than having to reimplement it themselves.

Re: APFS in Detail

#80

Post by idorosen »

mindajar wrote: post #9
idorosen wrote: post #2

In my opinion, APFS does not seem to improve upon ZFS in several key areas (compression, sending/receiving snapshots, dedup, etc.). Apple is reimplementing many features already implemented in OpenZFS, btrfs (which itself reimplemented a lot of ZFS features), BSD HAMMER, etc. Maybe extending one of these existing filesystems to add any functionality Apple needs on top of its existing features (and, hopefully, contrib…

Classic comic, but I don't think it applies. APFS looks intended to solve Apple's product problems really well, and it doesn't even try to be a filesystem for everyone. Apple has said from time to time that they're all about owning and controlling the key technologies that go into their products. APFS makes a lot of sense from that perspective, and this seems one of those cases where going their own way is better tha…

Should have said s/standards/filesystems/g... :-)

I was reacting to the idea of APFS for macOS, as well as having yet another filesystem to deal with on external media that interacts with multiple computers (HDDs/SSDs/USB flash drives/etc.).

Post reply on HN