Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple File System Reference [pdf]

developer.apple.com

Post reply on HN
Print view View original

61–67 of 67 posts

Re: Apple File System Reference [pdf]

#61

Post by petecooper »

sjwright wrote: post #46
petecooper wrote: post #31

Earlier quoted context omitted.

My computer (macOS Sierra, HFS (Journaled)) has 2x spinning drives of 3TB, and operations are noticeably quicker after an overnight defragmentation run, which I do twice a year or so. Anecdata, granted - but it works for my situation.

Given that macOS automatically defragments sub-20MB files on the fly (which covers 99% of the files which affect perceived system performance) I'd wager money that your experience is 100% placebo. I know from personal experience how powerful this placebo effect can be.

>Given that macOS automatically defragments sub-20MB files on the fly (which covers 99% of the files which affect perceived system performance) I'd wager money that your experience is 100% placebo.

Perhaps. I can certainly tell the difference performance when an overnight defrag run has finished, and I have no beef with you to prove or disprove a point.

If you're ever in North Cornwall, UK: drop me a line and I'll show you a before and after over a mug of coffee/tea/etc.

Re: Apple File System Reference [pdf]

#62

Post by Someone »

There are several cases where the file system has monotonically increasing integers that, when overflowing, are unrecoverable errors.

Those counters always are 64 bits, and won’t overflow in normal use (for example, the text says: ”if you created 1,000,000 transactions per second, it would take more than 5,000 centuries to exhaust the available transaction identifiers.”), but I can see people making ‘interesting’ disk images, for example ones where writing to a specific directory is impossible or, depending on how the implementation handles it, even panics the OS.

Re: Apple File System Reference [pdf]

#63

Post by giobox »

hollerith wrote: post #49
giobox wrote: post #30

Earlier quoted context omitted.

Simple, to help prevent “bit rot”. The problem is exacerbated further in that many of us treat cloud sync services as backup, which they arguably aren’t - they can inconveniently just spread the decay. I’d also hoped that a next generation file system from Apple would have had more to say on this topic, but it seems like features that promote their iOS device agenda took front seat over less “sexy” features like data…

Might it not be the case that combating bit rot is best done at higher layers of the stack similar to how it is best done at levels higher than the IP layer in a networking stack? For example, data painstakingly entered by the user a character at a time with a keyboard might deserve more redundancy than for example a movie downloaded by iTunes.

I’m no expert, but my understanding is that bits can “flip” and introduce errors due to things as unpredictable as background radiation etc, even on files the system has had no interaction with, which is why it’s kind of desirable to implement this kind of integrity check at the file system level. A higher level check may be completely unaware of this kind of passive background error.

Also, if I pull the drive and move it to another machine, again it’s kind of nice if the data integrity features are tied to the drive format rather than higher level software. I don’t think it’s too unreasonable to expect the file system to make sensible guarantees that the sequence of bytes I record today will remain the same until I next interact with them.

I’m not sure how appropriate comparisons with IP error correction is either; it’s a markedly different class of problem really (you are not dealing with long term storage issues at all).

Re: Apple File System Reference [pdf]

#64

Post by forapurpose »

akvadrako wrote: post #34
forapurpose wrote: post #26

Earlier quoted context omitted.

> Probably often Defragging an SSD is often a good thing? Why? It would seem to greatly increase wear for no benefit.

If your fragments are really small like 16kb then you could see a significant performance improvement, due to better predictive loading and packet overhead. This is clear from SSD benchmarks. But I don’t see how this would ever realistically happen.

What about the massive effect on wear?

Re: Apple File System Reference [pdf]

#65

Post by cerberusss »

sneak wrote: post #37
aaaaaaaaaab wrote: post #36

Earlier quoted context omitted.

It takes bad luck for sure, but I once ruined a bunch (a big bunch) of my photos by syncing them to a NAS with a faulty RAM. It was a Synology Ds212 I think, back in 2012. Mind you, the device didn’t produce symptoms other than messing up regularly spaced bytes in the transferred files.

I am super paranoid about this kind of stuff and don’t consider a copy finished until it is first done copying then also passes an independent rsync -c.

For my family photo's, I create par2 files. The rest, I don't care so much.

Re: Apple File System Reference [pdf]

#66

Post by tinus_hn »

forapurpose wrote: post #64
akvadrako wrote: post #34

Earlier quoted context omitted.

If your fragments are really small like 16kb then you could see a significant performance improvement, due to better predictive loading and packet overhead. This is clear from SSD benchmarks. But I don’t see how this would ever realistically happen.

What about the massive effect on wear?

You don’t know if there is wear. The controller is a separate computer. It may be clever and just remap blocks if you copy or move them around. It could be that defragmentation allows the system to work in a way that is better for the controller or allows that controller to place the blocks in a better way.

Re: Apple File System Reference [pdf]

#67

Post by mrpippy »

arthurfm wrote: post #51
mrpippy wrote: post #16

The 'Fusion' section at the end is interesting. macOS 10.13 didn't convert Fusion drives to APFS, but 10.14 will. Presumably this specific support for Fusion drives is new in 10.14. HFS+ had no knowledge about Fusion drives, the caching was handled entirely at block-level by the lower CoreStorage layer (although later versions did add some flags so CoreStorage could pin metadata/swap blocks to the SSD). Now what I'm…

The only bad thing about Apple supporting APFS on Fusion Drives is that it gives them an incentive to continue selling future iMacs and Mac minis with Fusion Drives. Having had to replace failed HDDs in Fusion Drive iMacs at work, it's certainly no fun. For all new Mac purchases I ensure they are SSD only now.

On that note, I am surprised that they added Fusion-awareness to APFS, rather than just putting APFS on top of CoreStorage.

It certainly is better to have the filesystem aware of the Fusion situation, but...measurably, significantly better? Would the experience have been significantly worse without it? 10.13 betas allowed APFS use on Fusion drives, presumably without any Fusion-awareness in the FS.

I'm surprised, but happy to see they did it.

Post reply on HN