Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS in Detail

dtrace.org

Post reply on HN
Print view View original

11–20 of 98 posts

Re: APFS in Detail

#11

Post by millstone »

idorosen wrote: post #2

In my opinion, APFS does not seem to improve upon ZFS in several key areas (compression, sending/receiving snapshots, dedup, etc.). Apple is reimplementing many features already implemented in OpenZFS, btrfs (which itself reimplemented a lot of ZFS features), BSD HAMMER, etc. Maybe extending one of these existing filesystems to add any functionality Apple needs on top of its existing features (and, hopefully, contrib…

APFS is targeted to run on low-powered devices like the Apple Watch. It may be that the alternatives cannot be made suitable for such devices: ZFS is famously memory hungry, HAMMER says it's not designed for < 50GB devices.

Re: APFS in Detail

#12

Post by mahyarm »

veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

External hard drives / SSDs that are using APFS? I thought that would be a pretty obvious use case.

Re: APFS in Detail

#13

Post by toast0 »

scarface74 wrote: post #10
idorosen wrote: post #2

In my opinion, APFS does not seem to improve upon ZFS in several key areas (compression, sending/receiving snapshots, dedup, etc.). Apple is reimplementing many features already implemented in OpenZFS, btrfs (which itself reimplemented a lot of ZFS features), BSD HAMMER, etc. Maybe extending one of these existing filesystems to add any functionality Apple needs on top of its existing features (and, hopefully, contrib…

Is ZFS lightweight enough to run on the watch? Their ultimate goal was to have a file system efficient enough and flexible enough to run across all of their OS's - MacOS, iOS, tvOs, and watchOS

The Apple watch has 8 GB of storage and 512 Mb of ram; I don't think that's an unreasonable ratio. Most people discussing zfs memory use have large arrays or deduplication enabled; I'm not sure why dedup would be very useful on a watch. Maybe CPU for checksums is an issue? Apple could probably add acceleration for the checksum algorithm/use an algorithm that was faster/skip the checksums on read + do a sweep when plugged in and fully charged. An 8gb sad only takes minutes to do a complete read, as opposed to a large rotational drive, so you would likely be able to do a full scan in a reasonable amount of time.

Re: APFS in Detail

#14

Post by sho_hn »

I'm extremely confused by this:

> With APFS, if you copy a file within the same file system (or possibly the same container; more on this later), no data is actually duplicated. [...] I haven’t see this offered in other file systems [...]

To my knowledge, this is what cp --reflink does on GNU/Linux on a supporting filesystem, most notably btrfs, and has been doing by default in newer combinations of the kernel and GNU coreutils.

This guy seems too well-informed and experienced in the domain to miss something so obvious, though. So what am I missing?

Also interesting to me is the paragraph about prioritizing certain I/O requests to optimize interactive latency: On Linux this is done by the I/O scheduler, exchangable and agnostic to the filesystem. Perhaps greater insight into the filesystem could aid I/O scheduling (this has been the argument for moving RAID code into filesystems as well, though, which APFS opts against) -- hearing a well-informed opinion on this point would be interesting. Unless this post gets it wrong and I/O scheduling isn't technically implemented in APFS either.

It seems like this perspective might be one written from within a Solaris/ZFS bubble and further hamstrung by macOS' closed-source development model. Which is interesting in light of the Giampaolo quote about intentionally not looking closely at the competition, either.

Re: APFS in Detail

#15

Post by ahl »

TazeTSchnitzel wrote: post #6

> Also, APFS removes the most common way of a user achieving local data redundancy: copying files. A copied file in APFS actually creates a lightweight clone with no duplicated data. No, it doesn't. APFS supports copying files, if you want that. It's just that the default in Finder is to make a “clone” (copy-on-write).

Fair enough; and right now cp doesn't use the fast clone functionality, but it assuredly will. I'm not sure 'cat file.dup' is reasonable for most users.

Re: APFS in Detail

#16

Post by ahl »

sho_hn wrote: post #14

I'm extremely confused by this: > With APFS, if you copy a file within the same file system (or possibly the same container; more on this later), no data is actually duplicated. [...] I haven’t see this offered in other file systems [...] To my knowledge, this is what cp --reflink does on GNU/Linux on a supporting filesystem, most notably btrfs, and has been doing by default in newer combinations of the kernel and GN…

This guy (i.e. me) wasn't aware of this functionality in btrfs. Are reflinks commonly used? Yes, I know more about ZFS than the other filesystems mentioned.

Re: APFS in Detail

#17

Post by ahl »

scarface74 wrote: post #10
idorosen wrote: post #2

In my opinion, APFS does not seem to improve upon ZFS in several key areas (compression, sending/receiving snapshots, dedup, etc.). Apple is reimplementing many features already implemented in OpenZFS, btrfs (which itself reimplemented a lot of ZFS features), BSD HAMMER, etc. Maybe extending one of these existing filesystems to add any functionality Apple needs on top of its existing features (and, hopefully, contrib…

Is ZFS lightweight enough to run on the watch? Their ultimate goal was to have a file system efficient enough and flexible enough to run across all of their OS's - MacOS, iOS, tvOs, and watchOS

Unclear, but a member of the ZFS development team at Apple told me that Giampaolo complained that ZFS would never work on the phone. So the team demonstrated it working on the phone. I know of no obstacles to making it work on the watch other than engineering.

Re: APFS in Detail

#18

Post by codemac »

veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

Checksums are usually very fast to compute as the ARM CPUs of any modern phone have crypto engines, and their laptops do as well. I think trading data protection for performance reasons would be pretty irrational.

Re: APFS in Detail

#19

Post by isomorphic »

veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

I agree; no checksumming of user data is very disappointing. If there were performance issues, they could build checksumming into the filesystem, but make it a volume-specific option. No checksumming on the watch, strong integrity guarantees on the Mac Pro.

Their filesystem goals are in some ways consistent with Apple's (marketing) vision: Users would never have terabyte libraries of anything, as the various iServices would (should) be hosting that stuff in the cloud (where one presumes it is stored on a filesystem that actually includes data integrity). Since users won't be storing much of anything locally, Apple needn't care too much about data integrity. This is of course, nonsense.

The idea that Apple's storage devices are error-free is arrogant--but even assuming that were true, there can still be bit errors in the SATA/PCI bus, errors in memory, race conditions, gamma rays, etc. Apple uses ECC memory on their Mac Pro, so obviously someone still believes that sort of thing is possible.

Re: APFS in Detail

#20

Post by e12e »

sho_hn wrote: post #14

I'm extremely confused by this: > With APFS, if you copy a file within the same file system (or possibly the same container; more on this later), no data is actually duplicated. [...] I haven’t see this offered in other file systems [...] To my knowledge, this is what cp --reflink does on GNU/Linux on a supporting filesystem, most notably btrfs, and has been doing by default in newer combinations of the kernel and GN…

I wasn't aware of --reflink either, apparently support is coming to XFS as well (at some point): https://pkalever.wordpress.com/2016/01/22/xfs-reflinks-tutor...
Post reply on HN