Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple File System Reference [pdf]

developer.apple.com

Post reply on HN
Print view View original

51–60 of 67 posts

Re: Apple File System Reference [pdf]

#51

Post by arthurfm »

mrpippy wrote: post #16

The 'Fusion' section at the end is interesting. macOS 10.13 didn't convert Fusion drives to APFS, but 10.14 will. Presumably this specific support for Fusion drives is new in 10.14. HFS+ had no knowledge about Fusion drives, the caching was handled entirely at block-level by the lower CoreStorage layer (although later versions did add some flags so CoreStorage could pin metadata/swap blocks to the SSD). Now what I'm…

The only bad thing about Apple supporting APFS on Fusion Drives is that it gives them an incentive to continue selling future iMacs and Mac minis with Fusion Drives.

Having had to replace failed HDDs in Fusion Drive iMacs at work, it's certainly no fun. For all new Mac purchases I ensure they are SSD only now.

Re: Apple File System Reference [pdf]

#52

Post by tonyedgecombe »

aasasd wrote: post #47
nicky0 wrote: post #12

Earlier quoted context omitted.

Care to enlighten ignorant me why we would want that?

I've recently listened through an old-but-good episode of the Hypercritical podcast with John Siracusa's informative rant about this very topic: http://5by5.tv/hypercritical/56

I've just been going through Siracusa's old OS/X reviews, he does talk about this a lot right back to the earliest days.

Re: Apple File System Reference [pdf]

#54

Post by israrkhan »

tambourine_man wrote: post #38
israrkhan wrote: post #8

In past, I implemented HFS+ on an embedded rtos platform,. Technical Note TN1150 [1] proved to be extremely useful asset. From a cursory look, I feel TN1150 was much more detailed, and perhaps can be treated as a pre-req to this document. At least it should have been mentioned in this document. [1] https://developer.apple.com/library/archive/technotes/tn/tn1...

You got me curious, can you explain why you needed to implement HFS+? Was it read only?

I was working on a video recorder product that could record videos directly to iPods, iPhones and other media players. iPods were formatted to HFS+, if you connected them to a Mac out-of-box, and were formatted to FAT32, if you connected them to a PC. The platform i was working on was an RTOS, and we had to develop both FAT32, and HFS+ from scratch. It was not read-only, it supported both read/write.

Re: Apple File System Reference [pdf]

#55

Post by tambourine_man »

israrkhan wrote: post #54
tambourine_man wrote: post #38

Earlier quoted context omitted.

You got me curious, can you explain why you needed to implement HFS+? Was it read only?

I was working on a video recorder product that could record videos directly to iPods, iPhones and other media players. iPods were formatted to HFS+, if you connected them to a Mac out-of-box, and were formatted to FAT32, if you connected them to a PC. The platform i was working on was an RTOS, and we had to develop both FAT32, and HFS+ from scratch. It was not read-only, it supported both read/write.

Nice, that sounds fun, thanks.

Re: Apple File System Reference [pdf]

#56

Post by josephg »

giobox wrote: post #30
nicky0 wrote: post #12

Earlier quoted context omitted.

Care to enlighten ignorant me why we would want that?

Simple, to help prevent “bit rot”. The problem is exacerbated further in that many of us treat cloud sync services as backup, which they arguably aren’t - they can inconveniently just spread the decay. I’d also hoped that a next generation file system from Apple would have had more to say on this topic, but it seems like features that promote their iOS device agenda took front seat over less “sexy” features like data…

Word on the street is that Apple's ZFS integration was mostly finished and it was going to be announced at WWDC. Sun opensourced ZFS under the CDDL. But then Oracle bought Sun, and Apple's lawyers wanted to make sure Oracle wouldn't try to sue them over ZFS somehow anyway. Negotiations between Apple and Oracle for a clear ZFS license fell through. Without legal go-ahead the feature was pulled from macos at the 11th hour and buried.

When ZFS was opensourced under the CDDL, lots of people complained that they should have chosen a clearer, more permissive opensource license. Other people said it was fine, because the license was good enough and Sun is full of good people. The way everything played out, its clear the first group's concerns were valid.

Its a huge shame. ZFS is a fantastic piece of engineering. It was ahead of its time in lots of ways. It would take years for btrfs to become usable and for apfs to appear on the scene. If not for the weird licensing decision, zfs would almost certainly have landed in the linux and macos kernels. We almost had an ubiquitous, standard, cross platform filesystem.

For more history about Sun and Oracle, this talk by Bryan Cantrill is a great watch: https://www.youtube.com/watch?v=-zRN7XLCRhc

Re: Apple File System Reference [pdf]

#57

Post by barrkel »

djrogers wrote: post #28
petecooper wrote: post #15

One of my long-time favourite macOS applications -- iDefrag -- had support withdrawn shortly after APFS appeared. Reasons cited were lack of an APFS spec and increased System Integrity Protection. I fear this is too little, too late to have iDefrag make a comeback. I understand defragmenting an SSD typically does more harm than good [edit: and I only defrag spinning drives), but nothing touched it for effectiveness o…

> I fear this is too little, too late to have iDefrag make a comeback Defragging has been snake oil for more than a decade, anything that hastens it’s demise is a good thing.

You have not used a file system exceeding 90% use with large files and lots of little files coming and going.

ZFS in particular completely falls off a cliff somewhere between 80% and 90%, due to the copy on write nature of ZFS always allocating and freeing small bits of space. That creates the little gaps all over the FS which murder performance when the big gaps run out.

Re: Apple File System Reference [pdf]

#58

Post by lathiat »

barrkel wrote: post #57
djrogers wrote: post #28

Earlier quoted context omitted.

> I fear this is too little, too late to have iDefrag make a comeback Defragging has been snake oil for more than a decade, anything that hastens it’s demise is a good thing.

You have not used a file system exceeding 90% use with large files and lots of little files coming and going. ZFS in particular completely falls off a cliff somewhere between 80% and 90%, due to the copy on write nature of ZFS always allocating and freeing small bits of space. That creates the little gaps all over the FS which murder performance when the big gaps run out.

The old default for this was lower but most modern versions of the ZFS defaults don’t hit this until the vicinity of 96%

It’s not technically the little holes or even CoW that is the problem. It simply switched from a “first fit” to a “best fit” algorithm as it got full which was quite expensive to do the search for.

Re: Apple File System Reference [pdf]

#59

Post by petecooper »

tambourine_man wrote: post #39
petecooper wrote: post #15

One of my long-time favourite macOS applications -- iDefrag -- had support withdrawn shortly after APFS appeared. Reasons cited were lack of an APFS spec and increased System Integrity Protection. I fear this is too little, too late to have iDefrag make a comeback. I understand defragmenting an SSD typically does more harm than good [edit: and I only defrag spinning drives), but nothing touched it for effectiveness o…

HFS+ has had on-the-fly defrag for ages now. Did you have issues with it? What kind of files do you work with?

A moderate amount of my work involves video training and the component parts to build, so raw video and audio. It's subjective, but there are times that video and audio stutters a bit and it's not down to the available CPU or RAM (plenty of headroom there).

Re: Apple File System Reference [pdf]

#60

Post by petecooper »

pfranz wrote: post #42
petecooper wrote: post #31

Earlier quoted context omitted.

My computer (macOS Sierra, HFS (Journaled)) has 2x spinning drives of 3TB, and operations are noticeably quicker after an overnight defragmentation run, which I do twice a year or so. Anecdata, granted - but it works for my situation.

If it works for you, then great, but here's an article from almost 10 years ago where Apple discourages defragging[1]. There's a bunch of other articles with more specifics showing that advanced defrag features from the 90s were built into the OS years ago[2]. [1] https://support.apple.com/en-us/HT1375 (last updated 2010) [2] http://osxbook.com/software/hfsdebug/fragmentation.html (from 2004)

I just took a snapshot of one of my hard drives (2TB) from iDefrag:

https://imgur.com/bdGcXXC

The red slivers towards the lower middle are where the reported fragmented files are located, and the lower right is the fragmented files listed by number of fragments descending.

The anonymised .mkv files are training videos with multiple language and subtitle streams. They're exported to a scratch drive and then copied to the the drive they currently reside on.

After a full, offline defrag (including a b-tree rebuild) the legend for defragmentation is a neatly arranged list of that blue/grey colour, no red to be seen.

Post reply on HN