Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APFS in Detail

dtrace.org

Post reply on HN
Print view View original

91–98 of 98 posts

Re: APFS in Detail

#91

Post by michaelmrose »

vvhn wrote: post #90
michaelmrose wrote: post #55

Earlier quoted context omitted.

ZFS requires a lot of ram to enable on line deduplication. From Freebsd Mastery: ZFS Pg 135 "For a rough-and-dirty approximation, you can assume that 1 TB of deduplicated data uses about 5 GB of RAM. You can more closely approximate memory needs for your particular data by looking at your data pool and doing some math. We recommend always doing the math and computing how much RAM your data needs, then using the most…

From the 2nd edition of "The Design and Implementation of the FreeBSD Operating System" ZFS, Pg 549 " However it is not designed for or well suited to run on resource constrained systems using 32 bit CPUs with less than 8 Gbyte of memory and one small nearly full disk, which is typical of many embedded systems "

Even mobile cpus will be all 64 bit before long and it seemed to run great with 4gb others reported success on systems with 2gb. Meanwhile ram available in all sorts of devices is ever increasing.

What fs a laptop/workstation uses shouldn't be determined by whats suitable for a watch in 2016.

Re: APFS in Detail

#92

Post by e_proxus »

isomorphic wrote: post #19
veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

I agree; no checksumming of user data is very disappointing. If there were performance issues, they could build checksumming into the filesystem, but make it a volume-specific option. No checksumming on the watch, strong integrity guarantees on the Mac Pro. Their filesystem goals are in some ways consistent with Apple's (marketing) vision: Users would never have terabyte libraries of anything, as the various iService…

Does it not matter anyway though? If the file lives locally for a while, and it rots there, the corrupt version will be synced back into the cloud and the corruption will spread. I admit the window of corruption will be smaller, but it will still be there, no?

Re: APFS in Detail

#93

Post by ksec »

speleding wrote: post #64
veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

Perhaps you (or Apple) would still be able to achieve the checksum feature by a smart choice of encryption algorithm? APFS has file level encryption, so you would in theory be able to detect a flip by selecting an encryption algorithm that gives error upon decrypting modified data. I could see this being worked into apps_fsck at some point. A similar case could be made for adding it into the compression algorithm, wh…

Same thoughts here as well, but how does encryption correct data?

Re: APFS in Detail

#94

Post by idorosen »

mindajar wrote: post #84
idorosen wrote: post #80

Earlier quoted context omitted.

Should have said s/standards/filesystems/g... :-) I was reacting to the idea of APFS for macOS, as well as having yet another filesystem to deal with on external media that interacts with multiple computers (HDDs/SSDs/USB flash drives/etc.).

Is moving data between computers that way a thing that non-technical people do often? FAT-formatted USB sticks seem to be good enough for that, but e-mail/Dropbox/file sharing/cloud sharing/AirDrop have much better UX for the average person.

Yes, it is a thing people do. The problem is, those non-technical people do not understand on-disk formats. (Nowadays, most USB sticks come preformatted as ExFAT.) There are also offline and low bandwidth situations. In healthcare it's common too thanks to HIPAA and nervous hospitals: let's say a patient wants to transfer a set of MRI or CT scan images/videos (typically provided on CD or DVD, in which case it's ISO9660, but sometimes USB stick - hopefully ExFAT but sometimes worse).

Re: APFS in Detail

#95

Post by adiabatty »

ptx wrote: post #69
philjohn wrote: post #57

Earlier quoted context omitted.

It is, if you outsource the storage. Backblaze is $5 a month per computer for (virtually) unlimited storage. They keep old copies of files for 30 days.

To paraphrase: >>>> If your data gets silently corrupted and you keep backing up that corrupted data, eventually there won't be any backups left that have the original uncorrupted data. >>> You shouldn't delete old backups! >> Storage is too expensive to keep old backups forever. > Backblaze! ... will delete backups after 30 days. Yes. So 30 days after your file was corrupted, you will only have corrupted copies left…

…that's why I use both Backblaze and Time Machine.

Re: APFS in Detail

#96

Post by fanf2 »

ahl wrote: post #21
veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

Talking to the Apple engineers it really didn't seem to be an issue of computation. They seemed genuine in their belief that they could solve data integrity with device qualification. While I asked them 100 questions they asked me 2: had I ever actually seen bit rot (yes), and what kind of drives did we ship with the ZFS Storage Appliance (mostly 7200 nearline drives).

If the crypto layer has proper MACs then presumably checksums at lower layers aren't so important. Did they give you much indication that they thought disk encryption would become standard?

Re: APFS in Detail

#97

Post by fanf2 »

ahl wrote: post #21
veidr wrote: post #7

What a great and valuable post, especially since this info is the result of talking to the APFS team at WWDC, and has not been published anywhere else yet. Of particular interest (to me) was the "Checksums" section: Notably absent from the APFS intro talk was any mention of checksums....APFS checksums its own metadata but not user data. ...The APFS engineers I talked to cited strong ECC protection within Apple storag…

Talking to the Apple engineers it really didn't seem to be an issue of computation. They seemed genuine in their belief that they could solve data integrity with device qualification. While I asked them 100 questions they asked me 2: had I ever actually seen bit rot (yes), and what kind of drives did we ship with the ZFS Storage Appliance (mostly 7200 nearline drives).

If the crypto layer has proper MACs then presumably checksums at lower layers aren't so important. Did they give you much indication that they thought disk encryption would become standard?

Re: APFS in Detail

#98

Post by speleding »

ksec wrote: post #93
speleding wrote: post #64

Earlier quoted context omitted.

Perhaps you (or Apple) would still be able to achieve the checksum feature by a smart choice of encryption algorithm? APFS has file level encryption, so you would in theory be able to detect a flip by selecting an encryption algorithm that gives error upon decrypting modified data. I could see this being worked into apps_fsck at some point. A similar case could be made for adding it into the compression algorithm, wh…

Same thoughts here as well, but how does encryption correct data?

Correcting data is much harder, and would require a significant amount of additional storage to provide enough redundancy to be able to deduce the original data. But detecting is good enough for many uses, you would be able to restore the files from the Time Machine before those get silently corrupted as well.
Post reply on HN