Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS in Detail

dtrace.org

Post reply on HN
Print view View original

61–70 of 98 posts

Re: APFS in Detail

#61

Post by Freaky »

amelius wrote: post #51
veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

Perhaps they assume that you will sync everything to the iCloud anyway (?)

How are you expected to know you need to restore from the backup if the damage is silent?

How can you have confidence in your backup if damaged data can be silently written to it?

Re: APFS in Detail

#62

Post by TazeTSchnitzel »

ahl wrote: post #15
TazeTSchnitzel wrote: post #6

> Also, APFS removes the most common way of a user achieving local data redundancy: copying files. A copied file in APFS actually creates a lightweight clone with no duplicated data. No, it doesn't. APFS supports copying files, if you want that. It's just that the default in Finder is to make a “clone” (copy-on-write).

Fair enough; and right now cp doesn't use the fast clone functionality, but it assuredly will. I'm not sure 'cat file.dup' is reasonable for most users.

If they patched it to use a different syscall, maybe.

Re: APFS in Detail

#63

Post by cm3 »

codemac wrote: post #18
veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

Checksums are usually very fast to compute as the ARM CPUs of any modern phone have crypto engines, and their laptops do as well. I think trading data protection for performance reasons would be pretty irrational.

Except that with NVME drives, and the parallel operations you can run on these, the performance of checksums becomes important again. Recent experiments with HAMMER2: https://www.dragonflydigest.com/2016/06/15/18281.html

Re: APFS in Detail

#64

Post by speleding »

veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

Perhaps you (or Apple) would still be able to achieve the checksum feature by a smart choice of encryption algorithm?

APFS has file level encryption, so you would in theory be able to detect a flip by selecting an encryption algorithm that gives error upon decrypting modified data. I could see this being worked into apps_fsck at some point.

A similar case could be made for adding it into the compression algorithm, which the OP thinks will be coming to APFS later, popular algorithms such as deflate already have this built in.

Re: APFS in Detail

#65

Post by [deleted] »

sho_hn wrote: post #14

I'm extremely confused by this: > With APFS, if you copy a file within the same file system (or possibly the same container; more on this later), no data is actually duplicated. [...] I haven’t see this offered in other file systems [...] To my knowledge, this is what cp --reflink does on GNU/Linux on a supporting filesystem, most notably btrfs, and has been doing by default in newer combinations of the kernel and GN…

[deleted]

Re: APFS in Detail

#66

Post by cm3 »

I'm still looking for a widely supported (at least FreeBSD and Linux kernels) filesystem for external drives to carry around that doesn't have the FAT32 limitations. There's exFAT but no stable and supported implementation. Then there's NTFS, but that's also not 100% reliable in my experience when used through FUSE (NTFS-3G). I've considered UFS but that also was a no go. I'm hopeful for lklfuse[1] that also runs on FreeBSD and givess access to ext4, xfs, etc. in a way like Rump and allows you to use the same drivers on FreeBSD. I'm cautious though, given that I don't want corrupted data I might notice too late. Let's see if lklfuse provides LUKS as well, otherwise Dragonfly's LUKS implementation might need to be ported to FreeBSD or something like that. External drives one might lose need to be encrypted.

[1] https://www.freshports.org/sysutils/fusefs-lkl/

Re: APFS in Detail

#67

Post by cm3 »

gmac wrote: post #40

Slightly worried by the vibe that comes off this. "I asked him about looking for inspiration in other modern file systems ... he was aware of them, but didn’t delve too deeply for fear, he said, of tainting himself". And (to paraphrase): 'bit-rot? What's that?'. I would have hoped that a new filesystem with such wide future adoption would have come from a roomful of smart people with lots of experience of (for exampl…

Given Dominic's other output, I'm going to believe there's more to the story because he didn't strike me as someone who would actively ignore past innovations. I know it's a popular concept to NIH stuff when devs believe they know enough to build it, but so much stuff is just built poorly without consideration for existing designs and it shows in the poor software we have to live with.

Re: APFS in Detail

#68

Post by pixelbeat »

sho_hn wrote: post #14

I'm extremely confused by this: > With APFS, if you copy a file within the same file system (or possibly the same container; more on this later), no data is actually duplicated. [...] I haven’t see this offered in other file systems [...] To my knowledge, this is what cp --reflink does on GNU/Linux on a supporting filesystem, most notably btrfs, and has been doing by default in newer combinations of the kernel and GN…

Note mv tries reflink by default (when moving files across BTRFS subvols) since doesn't need a separate copy of the data. Now as storage systems evolve there are less guarantees that one does get multiple copies, with deduplicating at lower layers etc., and therefore cp may change at some stage to reflinking by default, especially as clone_file_range() moves to the VFS level. Actual data redundancy would then achieved at a higher level with separate file systems, devices, data centers, ... where arguably it needs to happen now anyway.

Re: APFS in Detail

#69

Post by ptx »

philjohn wrote: post #57
XorNot wrote: post #43

Earlier quoted context omitted.

Storage is not that cheap yet.

It is, if you outsource the storage. Backblaze is $5 a month per computer for (virtually) unlimited storage. They keep old copies of files for 30 days.

To paraphrase:

>>>> If your data gets silently corrupted and you keep backing up that corrupted data, eventually there won't be any backups left that have the original uncorrupted data.

>>> You shouldn't delete old backups!

>> Storage is too expensive to keep old backups forever.

> Backblaze! ... will delete backups after 30 days.

Yes. So 30 days after your file was corrupted, you will only have corrupted copies left.

Re: APFS in Detail

#70

Post by tveita »

veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

> The engineers contend that Apple devices basically don’t return bogus data.

It's much easier to pretend that this is the case when the file system isn't verifying it.

Checksumming would probably expose problems that would otherwise go unnoticed by users or be blamed on computer gremlins. It's hard to say if doing the "correct" thing here would improve the subjective user experience. Maybe putting on airs of infallibility is the more profitable route.

Post reply on HN