Live data from Hacker News

MacOS may lose data on APFS-formatted disk images

bombich.com

81–90 of 193 posts

Re: MacOS may lose data on APFS-formatted disk images

#81

so why exactly does somebody backup into a image with variable size, when they could just dd together a fixed size image that makes these free space calculations unnecessary?

Variable size (sparse) disk images make it possible to backup e.g. two 1TB volumes that are on average <50% full to a single 1TB disk. Very useful to have for networked backups (Time Machine) of multiple machines.

Re: MacOS may lose data on APFS-formatted disk images

#82
post #71

Sure, it's a new filesystem; it's bound to have bugs. However, one pain point I've identified is that the existing tools often have no idea of how to deal with APFS. I'm currently typing this from a Mac with an APFS drive that is almost certainly experiencing filesystem corruption–I have folders that suddenly lose track of their contents and become impossible to delete ; however, existing tools such as fsck, diskutil…

There is a fsck_apfs, I’m pretty sure that’s what the repair button in the disk util runs.

`fsck_apfs -n -l -x /dev/rdisk1s1`

Re: MacOS may lose data on APFS-formatted disk images

#83

Yes, fairly limited impact, but still. Data loss is fucking important. We aren't talking about an occasional stutter when moving windows or a sound driver that sometimes needs a reboot to resume working. This is a major bug. I would really really like it if Apple would get their shit together so that people who actually rely on their computers to work correctly can upgrade at some point.

It's rather obnoxious that they prod us to upgrade every other day while all this is in the news.

I updated day one and had usual apple problems (finder broke so I had to use my terminal, sometimes it wouldn't wake up from sleep so I had to force it to reboot, my fans would spin up for seemingly no reason). Most of the issues were fixed a week later or so.

Anyway, I recently switched to Arch on a 2018 LG Gram and I'm not really missing anything. Battery life is great (8-12 hours of Firefox) and it has a quad core x64 processor for non-browser things.

Windows Ultrabooks are worth the purchase again.

Re: MacOS may lose data on APFS-formatted disk images

#84
post #74

Earlier quoted context omitted.

This is by far the best explanation of what’s happening at Apple and why we’re noticing these bugs and problems: https://www.imore.com/understanding-apple-scale

Not really. If you built a new filesystem, competent software engineers will heavily test the corner cases. What happens when the fs runs out of space? What happens when the metadata store runs out of space? etc. The original article mentions bugs that are pretty obvious cases to test. What precisely happens when you have a sparsebundle that exceeds the storage capacity of the containing volume? A PM needs to define…

Agreed. Many of the issues with Apple's software quality lately are fundamentals, not issues of scale. Stuff like this, the root password bug, the many issues with reliability in High Sierra, etc.

These things speak to organizational issues.

Re: MacOS may lose data on APFS-formatted disk images

#85

Earlier quoted context omitted.

>How do you know that? Can you point to any other APFS issues that were reported before this one?

My wife's laptop suddenly decided that the boot drive was corrupt a few weeks after she updated to APFS. None of the recovery tools were of any use. We had to reinstall the OS and pull the files from a backup. This story did not make it to Hacker News.

Anecdata != filesystem issue.

If you have more information about the problem you encountered and how it implicates/interacts with APFS, please do link to it. Otherwise, bug reports via circumstantial evidence are, while not inherently false, certainly suspect.

Re: MacOS may lose data on APFS-formatted disk images

#86

Earlier quoted context omitted.

As a counterpoint, APFS was deployed to almost a billion devices over the span of several months when it was release and, IIRC, this is the first major issue.

How do you know that? Filesystem corruption is frequently silent, and every-time it happens customers don't get on the phone and send the disks to apple so that they can root cause the problem. Its quite possible this bug has happened an untold number of times before it happened to someone who went through the effort to reproduce and isolate it.

We know that because it was deployed to almost a billion devices without a problem. If there was a problem that affected even a fraction of a percent of people, it would have been all over the news given how many devices that is.

Edit: Thanks for the downvotes. If you disagree, please tell me why. Apple's deployment of APFS to iPhones was so flawless, most people probably still don't even know they did it.

Re: MacOS may lose data on APFS-formatted disk images

#87

Can you even test a filesystem properly internally? Seems to me we're all involved in a massive public beta.

Apple authored a tool called fsx (file system exerciser), but I doubt it checks for corner cases for loopback mounts. Maybe they should add such tests.

That tool has been around for ages, I think it might originally be from SGI.

Re: MacOS may lose data on APFS-formatted disk images

#88
post #15

Is the incident with the matching checksum that he mentions because APFS only checksum metadata (which are on preallocated space on the image), or is it his own checksum (say sha1sum), I wonder? It seems strange that the filesystem driver would cache 500 GB of sequentially written data in RAM.

> It seems strange that the filesystem driver would cache 500 GB of sequentially written data in RAM.

That was the most interesting/worrying part of TFA, and I would love to see how the checksum tests were conducted clarified in the text.

Presumably, the "md5" commandline tool has no special fallback to the filesystem checksum cache (if it does, rather a lot of my life has been a lie, I'm afraid). Since that's the case, could we assume that, if the "lost" writes totalled $X GB of data, that any evil memory-caching of the file will only work in the presence of at least $X GB of free system memory (RAM plus swap).

I'd also be interested in learning what happens if there's less than that amount of memory available. Will the checksum fail? Will an error occur elsewhere? Will the system have some sort of memory (and swap) exhaustion failure/panic?

Re: MacOS may lose data on APFS-formatted disk images

#89

Not to downplay the importance of this, but it reads as clickbait that you wait until the second paragraph to say "oh yeah, it's only sparsebundles. Just those things that almost nobody uses."

Eh, it's a minority but not an "almost nobody" minority. There are plenty of pieces of software even besides CCC that rely on/give prominent options for the use of sparsebundles.

It's more like "the latest OS update leaves you vulnerable to malicious javascript everywhere, but only if you use Opera". It's a minority of users, sure, but it's still really important information to note.

Re: MacOS may lose data on APFS-formatted disk images

#90
post #43

I upgraded my hackintosh to High Sierra with APFS. The next day, I accidentally switched off the machine while it was in the _process_ of shutting down (the screen had gone blank, but casing still emitting lights). Next time I turned it on, I couldn't get past login screen (giving me forever beach ball). I put the ssd inside my old MBP as slave to recover data. The ssd was corrupted, most data gone, as in shown in Fi…

Hypothesis: since it does _get to_ the login screen, the OS does think that the disk is in a consistent state. Maybe your unclean shutdown forced the async conversion from HFS+ to APFS to become forced-synchronous? Try leaving the drive in the Hackintosh machine, spinning at the login screen, for a few hours. Maybe it’ll “finish.”

If it has Filevault, the "login" screen is the firmware's disk unlock screen. In my experience that will often load OK despite the filesystem being completely corrupted.
Post reply on HN