Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple File System

developer.apple.com

Post reply on HN
Print view View original

371–380 of 406 posts

Re: Apple File System

#371

Post by djrogers »

zingplex wrote: post #33
jasonjei wrote: post #8

Earlier quoted context omitted.

Apple says they intend to open-source the file system in the docs. Open Source : "An open source implementation is not available at this time. Apple plans to document and publish the APFS volume format when Apple File System is released in 2017."

Just like they promised to open source FaceTime and iMessages? Apple has a history of making false promises in regards to open sourcing products.

> Apple has a history of making false promises in regards to open sourcing products

No, not true at all. iMessage was never promised, mentioned, or even hinted at being open source. Also, on the FaceTime from they said that the underlying protocols would be submitted to standards bodies as an open standard - that's a completely different things from open source. And the reasons that Apple didn't wind up doing that are well known (lawsuit from patent trolls).

Sounds like you have an axe to grind or only listen to people who do...

Re: Apple File System

#372

Post by tedmiston »

newscracker wrote: post #364
tedmiston wrote: post #241

Earlier quoted context omitted.

CCC was what I used back in the day (IIRC it used to be free) to make the most reliable bit-for-bit backups of OS X machines. Time Machine skipping over so many files is kind of a nuisance to me.

Unless you set the privacy in Time Machine to ignore certain drives/volumes/folders, you shouldn't really miss anything with a Time Machine backup. It usually ignores those files that aren't necessary and can be created by the OS or applications (like caches or temporary files, for example). What kinds of personal files did you lose with a Time Machine backup?

Of course you can see the full list at /System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist.

One of the biggest is iPhone backups from iTunes. If I were a normal user and my MacBook didn't boot up tomorrow, I'd be surprised to find my backup didn't include critical files like this. For example, apps that are no longer in the App Store but are in your backup can be restored. Once you don't have that backup anymore, they can't.

It's just one of the reasons Apple users are forced to have multiple backup services if they want reliable and complete backups.

Re: Apple File System

#373

Post by PhantomGremlin »

bemmu wrote: post #341
laumars wrote: post #325

Earlier quoted context omitted.

I appreciate this was a long time ago and you've learned a lot since, but your story perfectly illustrates the importance of not just performing backups, but periodically testing them as well. That way you eliminate your blind faith in a black box by ensuring you actually have a working solution to fallback on.

I keep hearing this, but as a Mac user who uses Time Machine and CCC, what's a good way to test the backups? After all deleting my only system to attempt a restore seems even more dangerous.

what's a good way to test the backups?

Here's a way to do it. You need to feel comfortable around the OS X terminal command line. You only rely on OS X's built in programs, so it's an independent way to check if your backup program is doing the right thing.

As root, I used to do something like this:

   cd /
   find -x \
      . \
      -type f -print0 \
      | xargs -0 -n 100 -x md5 \
      | sort > /tmp/src.md5
Then I would run the same type of script in the destination directory and diff the results.

There are some annoyances. E.g. (from memory) the ~/Library/Caches files aren't backed up, so they will be missing at the destination. I wrote some sed commands to first remove some of these from the result checksums to keep my diff's more manageable.

Instead of the above, I currently use a Python script that I wrote, that let me fine tune things. At the heart of it is (as root, of course) using Python's os.walk to traverse a directory tree. For each file I use hashlib.sha256 to generate a checksum. I also don't descend into certain directories. Etc.

Using a few Python scripts to generate and process the source and destination checksums allows me to, at the end, use a simple

   vimdiff source.checksums destination.checksums
to quickly see the few differences that remain.

Keep in mind that the perfect is the enemy of the "good enough". Just using the basic 'find' (and not a custom Python script) is plenty good. I used that method for many years.

Many errors stick out right away. E.g. if you have 1,000,000 files checksummed in your source but only 250,000 in your destination, then you quickly know that you screwed up.

Re: Apple File System

#374

Post by PhantomGremlin »

laumars wrote: post #336
Twisell wrote: post #332

Earlier quoted context omitted.

I my case (and I can't emphasis more on my teenager status back then) it was a permission issue, everything looked fine from my perspective, but CCC somewhat didn't have the right permissions to sync folder of others users. So their home folder where actually empty. From my perspective the backup was a success... This traumatizing error is when I first learned about superuser and permissions. I guess CCC has gone a l…

Those points don't change the fact that you (and everyone else) should test back ups periodically. If it's not file system permissions nor other configuration problems then it will be a failing storage medium or just dumb user error. So the only way to be sure you have a working back up is to test that back up - ideally before you actually need to use it.

Yeah, once upon a time at a startup, I wanted a file restored. We were small; the CEO was actually the one doing the backups (to QIC tape cartridges).

Of course, the recovery failed. In fact, the backups hadn't been working properly for months. I lost a few hours of work; if our file server had actually failed, not having a good backup could have literally been catastrophic. As in would the company have survived?!

I recall an ancient quip, more or less: "if you don't test your backups, you don't have backups, you have dreams".

Re: Apple File System

#375

Post by masklinn »

snaky wrote: post #227
lighttower wrote: post #224

Earlier quoted context omitted.

I've had F2FS on an Android tablet for many years. Resurrected it. However I'm running Debian on my laptop and I'm scared to try f2fs on / Because i get warnings about it being not fully supported "yet" i would love to have an SSD optimized FS on Linux. Since AAPL will open source the release version, is it conceivable that AFS could replace ext4 as the default Linux FS?

Do you think Apple will release it with GPL-compatible license?

Most Apple OSS stuff is released under Apache (Swift 2.2 is Apache 2), so probably?

Re: Apple File System

#377

Post by tukelully »

dalore wrote: post #315
sametmax wrote: post #309

Earlier quoted context omitted.

- Some people don't want to use the cloud to store their stuff for privacy / ethic / commercial / legal reasons. - Some others have been burnt by the cloud failing them (corruption, data loss, copyright abuse, etc) and want an additional layer of security. - There are many activities that are fully not covered by the cloud, such as 3D / video / music edition, art in general, programming (all those dev envs)... - You…

What's wrong with sex tapes in the cloud?

I'm glad I had cloudtobutt enabled for your comment.

Re: Apple File System

#378

Post by KMag »

spullara wrote: post #297
1amzave wrote: post #152

Earlier quoted context omitted.

According to their documented "Current Limitations" ( https://developer.apple.com/library/prerelease/content/docum... ): > Case Sensitivity: Filenames are currently case-sensitive only. First thought: they have seen the light! A moment later: wait...they consider this a "limitation", and it's only "currently" the case. So maybe they're going to perpetuate the brain-damage anyway. Sigh.

Ok, what is the argument against case preserving but insensitive file systems?

It pushes a localization and UI problem down into the filesystem layer. Case-insensitivity is pretty easy for US-ASCII, but in release 2 of your filesystem, you realized you didn't properly handle LATIN WIDE characters, the Cyrillic alphabet, etc. In release 7 of your FS, you get case sensitivity correct for Klingon, but some popular video game relied on everything except Klingon being case-insensitive on your FS, and now all of the users are complaining.

How do you handle the case where the only difference between two file names is that one uses Latin wide characters and the other uses Latin characters? This one bit me when writing a CAPTCHA system back in 2004. (Long story, but existing systems wouldn't work between a credit card processing server that had to validate in Perl, and a web form that had to be written in PHP, where the two systems couldn't share a file system. It's simple enough to do using HMAC and a shared key between the two servers, but for some reason, none of the available solutions did it.) I noticed that Japanese users had a disturbingly high CAPTCHA failure rate. It turns out that many East Asian languages have characters that are roughly square, and most Latin characters are roughly half as wide as they are tall, so mixing the two looks odd. So, Unicode has a whole set of Latin wide characters that are the same as the Latin characters we use in English, except they're roughly square, so they look better when mixed with Unified Han and other characters. Apparently most Japanese web browsers (or maybe it's an OS level keyboard layout setting) will by default emit Latin wide unicode code points when the user types Latin characters. Whether or not to normalize wide Latin characters to Latin characters is a highly context-dependent choice. In my case, it was definitely necessary, but in other cases it will throw out necessary information and make documents look ugly/odd. Good arguments can be made both ways about how a case-insensitive filesystem should handle Latin wide characters, and that's a relatively simple case.

Most users don't type names of existing files, exclusively accessing files through menus, file pickers, and the OS's graphical command shell (Finder/Explorer). So, if you want to avoid users getting confused over similar file names, that can be handled at file creation time (as well as more subtle issues that are actually more likely to confuse users, such as file names that have two consecutive spaces, etc., etc.) via UI improvements.

Re: Apple File System

#379

Post by dhagz »

spott wrote: post #231
entee wrote: post #60

Earlier quoted context omitted.

Presumably as with today, you'll have the option. I don't have a strong opinion on case sensitivity of file names, but I suspect they'll keep it case insensitive by default. I think for the average non-technical user that two files, "MyFile.txt" and "myfile.txt", being different could lead to some confusion, and Apple historically has apparently considered that confusion unacceptable.

I believe the problem is also present for a large amount of third party software, making the move to case sensitive drives pretty hard to do: [0] https://helpx.adobe.com/creative-suite/kb/error-case-sensiti... [1] http://apple.stackexchange.com/questions/192185/os-x-case-se... [2] http://dcatteeu.github.io/article/2015/12/31/case-sensitive-...

I've noticed some bugs with case-sensitivity recently in Ruby, of all things.

Re: Apple File System

#380

Post by cmurf »

zrm wrote: post #127
joelg236 wrote: post #107

Earlier quoted context omitted.

Apple has (relatively, you can replace some harddrives) the most control on hardware, so at least from that perspective it's easier for them.

Control over hardware doesn't really buy you anything here. Just about any hardware can use any filesystem with, in the worst case, the requirement that you have a small boot partition using the legacy filesystem.

Apple's EFI firmware has an HFS driver built into it. The way today's macOS boots is the firmware reads the bootloader off the boot partition created these days on Core Storage installations, and the bootloader (more correctly OSLoader) is what enables the firmware pre-boot environment to read core storage (encrypted or not) and thus find and load the kernel and kext cache and then boot.
Post reply on HN