Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS in Detail

dtrace.org

Post reply on HN
Print view View original

81–90 of 98 posts

Re: APFS in Detail

#81

Post by blumentopf »

ahl wrote: post #17
scarface74 wrote: post #10

Earlier quoted context omitted.

Is ZFS lightweight enough to run on the watch? Their ultimate goal was to have a file system efficient enough and flexible enough to run across all of their OS's - MacOS, iOS, tvOs, and watchOS

Unclear, but a member of the ZFS development team at Apple told me that Giampaolo complained that ZFS would never work on the phone. So the team demonstrated it working on the phone. I know of no obstacles to making it work on the watch other than engineering.

So they still have a team working on ZFS? Could the anonymous "ilovezfs" involved with OpenZFSonOSX and coming from a California IP address be part of this team?

Re: APFS in Detail

#82

Post by blumentopf »

ahl wrote: post #21
veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

Talking to the Apple engineers it really didn't seem to be an issue of computation. They seemed genuine in their belief that they could solve data integrity with device qualification. While I asked them 100 questions they asked me 2: had I ever actually seen bit rot (yes), and what kind of drives did we ship with the ZFS Storage Appliance (mostly 7200 nearline drives).

I've had an Intel S3500 brick within 4 weeks and a SanDisk Extreme Pro start to show occasional I/O errors after a few months. The latter doesn't just lead to bit rot, but unreadable files. With ZFS I was able to identify those with a quick zpool scrub. Which shows how valuable checksumming is even in the absence of ECC memory. At least according to my anecdotal experience, flash is much more flakey than conventional hard disks, so the assumption that stuff just doesn't happen seems ludicrous.

Re: APFS in Detail

#83

Post by cm3 »

drvdevd wrote: post #78
cm3 wrote: post #66

I'm still looking for a widely supported (at least FreeBSD and Linux kernels) filesystem for external drives to carry around that doesn't have the FAT32 limitations. There's exFAT but no stable and supported implementation. Then there's NTFS, but that's also not 100% reliable in my experience when used through FUSE (NTFS-3G). I've considered UFS but that also was a no go. I'm hopeful for lklfuse[1] that also runs on…

Thanks for this! I didn't realize (or had forgotten) LUKS had been ported to Dragonfly. Also you touch upon my #1 frustration with APFS without really knowing anything about it: simple portability.

Yeah, I believe it's by the same Dragonfly developer who also wrote tcplay[1] for TrueCrypt volumes.

[1] https://leaf.dragonflybsd.org/cgi/web-man?command=tcplay&sec...

Re: APFS in Detail

#84

Post by mindajar »

idorosen wrote: post #80
mindajar wrote: post #9

Earlier quoted context omitted.

Classic comic, but I don't think it applies. APFS looks intended to solve Apple's product problems really well, and it doesn't even try to be a filesystem for everyone. Apple has said from time to time that they're all about owning and controlling the key technologies that go into their products. APFS makes a lot of sense from that perspective, and this seems one of those cases where going their own way is better tha…

Should have said s/standards/filesystems/g... :-) I was reacting to the idea of APFS for macOS, as well as having yet another filesystem to deal with on external media that interacts with multiple computers (HDDs/SSDs/USB flash drives/etc.).

Is moving data between computers that way a thing that non-technical people do often? FAT-formatted USB sticks seem to be good enough for that, but e-mail/Dropbox/file sharing/cloud sharing/AirDrop have much better UX for the average person.

Re: APFS in Detail

#85

Post by KaiserPro »

veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

Either the engineer is young, or hasn't been doing systems programming for long enough.

Re: APFS in Detail

#86

Post by mindajar »

tfar wrote: post #36
mindajar wrote: post #9

Earlier quoted context omitted.

Classic comic, but I don't think it applies. APFS looks intended to solve Apple's product problems really well, and it doesn't even try to be a filesystem for everyone. Apple has said from time to time that they're all about owning and controlling the key technologies that go into their products. APFS makes a lot of sense from that perspective, and this seems one of those cases where going their own way is better tha…

I would not be surprised if one could write a ZFS implementation optimized for more constrained devices. If you already know you are going to a have flash storage you can probably ditch some of the N layers of cache you see in common ZFS implementations. Not that ZFS is a one size fits all, but the file systems specification could be implemented in more than one way.

If you assume Apple cares about having a disk format in common with other platforms, sure, I'd agree that's probably possible. But I don't think they do; they seem to care a lot more about things like a unified codebase across their platforms, the energy-efficiency initiatives they've been pushing for a few years, owning the key tech in the products, etc.

One slide in the WWDC talk deck showed a bunch of divergent Apple storage technologies across all their platforms that are being replaced by APFS. If ZFS has to fork into weird variants to run well on the phone or watch, that seems less appealing than a single codebase optimized for just the stuff Apple products do.

Re: APFS in Detail

#87

Post by mindajar »

veidr wrote: post #31
chongli wrote: post #30

Earlier quoted context omitted.

I don't see why Apple couldn't just recommend that their pro users who have need of this sort of data integrity locally run their own server with FreeBSD + ZFS. Apple has really backed off on their attempts to market OS X Server to this crowd. Heck, they're probably using FreeBSD already if they need that much data integrity.

Here's the thing: everybody needs this sort of data integrity. Literally nobody wants their files to be silently corrupted. ZFS made it much easier for (nerds like us) to attain very high levels of data integrity. APFS was (and maybe still is?) a chance to make that the default for regular people.

Do checksums actually need to be in the filesystem, though? It does seem like an important feature, but couldn't they be done at a higher level, like the way Spotlight indexing works on the Mac today?

Re: APFS in Detail

#88

Post by mindajar »

glhaynes wrote: post #77
josho wrote: post #73

Earlier quoted context omitted.

What is the user experience for when a checksumming filesystem detects an error? If the fs detects a bit error does it flag the file as entirely unreadable? Move it to lost+found? Force me to restore the file from a backup? All these options seem more scary for an end user than blissful ignorance. Don't misunderstand me, I've lost a few family photos over the years due to bit rot. So, I appreciate a fs that offers mo…

If the fs detects a bit error does it flag the file as entirely unreadable? Move it to lost+found? Force me to restore the file from a backup? For files stored in iCloud Drive, if that version of the file exists in the cloud, the OS could automatically re-fetch the file. But, yeah, for lots of circumstances there's not going to be a "good" option to give the user. EDIT: Same applies to Time Machine (or whatever Apple…

It was a stealthy feature addition that went totally unannounced, but as of 10.11, Time Machine stores file checksums in the backup. See 'tmutil verifyChecksums'.

Re: APFS in Detail

#89

Post by kevincox »

ghshephard wrote: post #45
ahl wrote: post #21

Earlier quoted context omitted.

Talking to the Apple engineers it really didn't seem to be an issue of computation. They seemed genuine in their belief that they could solve data integrity with device qualification. While I asked them 100 questions they asked me 2: had I ever actually seen bit rot (yes), and what kind of drives did we ship with the ZFS Storage Appliance (mostly 7200 nearline drives).

That would suggest that APFS is only relevant for internal storage procured by Apple. Do they not intend for it to be used on external storage?

They mentioned that it would be used on removable media as well.

Re: APFS in Detail

#90

Post by vvhn »

michaelmrose wrote: post #55
millstone wrote: post #11

Earlier quoted context omitted.

APFS is targeted to run on low-powered devices like the Apple Watch. It may be that the alternatives cannot be made suitable for such devices: ZFS is famously memory hungry, HAMMER says it's not designed for < 50GB devices.

ZFS requires a lot of ram to enable on line deduplication. From Freebsd Mastery: ZFS Pg 135 "For a rough-and-dirty approximation, you can assume that 1 TB of deduplicated data uses about 5 GB of RAM. You can more closely approximate memory needs for your particular data by looking at your data pool and doing some math. We recommend always doing the math and computing how much RAM your data needs, then using the most…

From the 2nd edition of "The Design and Implementation of the FreeBSD Operating System"

ZFS, Pg 549

" However it is not designed for or well suited to run on resource constrained systems using 32 bit CPUs with less than 8 Gbyte of memory and one small nearly full disk, which is typical of many embedded systems "

Post reply on HN