Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZFS: Apple's new filesystem that wasn't (2016)

ahl.dtrace.org

Post reply on HN
Print view View original

101–110 of 209 posts

Re: ZFS: Apple's new filesystem that wasn't (2016)

#101

Post by bcantrill »

ryao wrote: post #68
ahl wrote: post #31

Earlier quoted context omitted.

ZFS was developed in Solaris, and at the time we were mostly selling SPARC systems. That changed rapidly and the biggest commercial push was in the form of the ZFS Storage Appliance that our team (known as Fishworks) built at Sun. Those systems were based on AMD servers that Sun was making at the time such as Thumper [1]. Also in 2016, Ubuntu leaned in to use of ZFS for containers [2]. There was nothing that specific…

> There was nothing that specific about Solaris that made sense for ZFS, and even less of a connection to the SPARC architecture. Although it does not change the answer to the original question, I have long been under the impression that part of the design of ZFS had been influenced by the Niagara processor. The heavily threaded ZIO pipeline had been so forward thinking that it is difficult to imagine anyone devising…

Niagara did not shape design decisions at all -- remember that Niagara was really only doing on a single socket what we had already done on large SMP machines (e.g., Starfire/Starcat). What did shape design decisions -- or at least informed thinking -- was a belief that all main memory would be non-volatile within the lifespan of ZFS. (Still possible, of course!) I don't know that there are any true artifacts of that within ZFS, but I would say that it affected thinking much more than Niagara.

As for Thumper using Opteron over Niagara: that was due to many reasons, both technological (Niagara was interesting but not world-beating) and organizational (Thumper was a result of the acquisition of Kealia, which was independently developing on AMD).

Re: ZFS: Apple's new filesystem that wasn't (2016)

#102

Post by Modified3019 »

tmoertel wrote: post #80
Modified3019 wrote: post #50

Earlier quoted context omitted.

ZFS detects corruption. A very long ago someone named cyberjock was a prolific and opinionated proponent of ZFS, who wrote many things about ZFS during a time when the hobbyist community was tiny and not very familiar with how to use it and how it worked. Unfortunately, some of their most misguided and/or outdated thoughts still haunt modern consciousness like an egregore. What you are probably thinking of is the pro…

> Hell, I’ve never even had bad ram. To what degree can you separate this claim from "I've never noticed RAM failures"?

You can take that as meaning “I’ve never had a noticed issue that was detected by extensive ram testing, or solved by replacing ram”.

I got into overclocking both regular and ECC DDR4 ram for a while when AMD’s 1st gen ryzen stuff came out, thanks to asrock’s x399 motherboard which unofficially supporting ECC, allowing both it’s function and reporting of errors (produced when overlocking)

Based on my own testing and issues seen from others, regular memory has quite a bit of leeway before it becomes unstable, and memory that’s generating errors tends to constantly crash the system, or do so under certain workloads.

Of course, without ECC you can’t prove every single operation has been fault free, but as some point you call it close enough.

I am of the opinion that ECC memory is the best memory to overclock, precisely because you can prove stability simply by using the system.

All that said, as things become smaller with tighter specifications to squeeze out faster performance, I do grow more leery of intermittent single errors that occur on the order of weeks or months in newer generations of hardware. I was once able to overclock my memory to the edge of what I thought was stability as it passed all tests for days, but about every month or two there’d be a few corrected errors show up in my logs. Typically, any sort of stability is caught by manual tests within minutes or the hour.

Re: ZFS: Apple's new filesystem that wasn't (2016)

#103

Post by nikhizzle »

whitepoplar wrote: post #85
nikhizzle wrote: post #18

Earlier quoted context omitted.

Ex-Apple File System engineer here who shared an office with the other ZFS lead at the time. Can confirm they link to the wrong profile for Don Brady. This is the correct person: https://github.com/don-brady Also can confirm Don is one of the kindest, nicest principal engineer level people I’ve worked with in my career. Always had time to mentor and assist.

Given your expertise, any chance you can comment on the risk of data corruption on APFS given that it only checksums metadata?

I moved out of the kernel in 2008 and never went back, so don’t have a wise opinion here which would be current.

Re: ZFS: Apple's new filesystem that wasn't (2016)

#104

Post by TacticalCoder »

whartung wrote: post #63

As a desktop user, I am content with APFS. The only feature from ZFS that I would like, is the corruption detection. I honestly don't know how robust the image and video formats are to bit corruption. On the one hand, potentially, "very" robust. But on the other, I would think that there are some very special bits that if toggled can potentially "ruin" the entire file. But I don't know. However, I can say, every time…

> The only feature from ZFS that I would like, is the corruption detection.

I run ZFS on my main server at home (Proxmox: a Linux hypervisor based on Debian and Proxmox ships with ZFS) but...

No matter the FS, for "big" files that aren't supposed to change, I append a (partial) cryptographic checksum to the filename. For example:

20240238-familyTripBari.mp4 becomes 20240238-familyTripBari-b3-8d77e2419a36.mp4 where "-b3-" indicates the type of cryptographic hash ("b3" for Blake3 in my case for it's very fast) and 8d77e2419a36 is the first x hexdigits of the cryptographic hash.

I play the video file (or whatever file it is) after I added the checksum: I know it's good.

I do that for movies, pictures, rips of my audio CDs (although these ones are matched with a "perfect rips" online database too), etc. Basically with everything that isn't supposed to change and that I want to keep.

I then have a shell script (which I run on several machines) that uses random sampling where I pick the percentage of files that have such a cryptographic checksum in their filenames that I want to check and that verifies that each still has its checksum matching. I don't verify 100% of the files all the time. Typically I'll verify, say, 3% of my files, randomly, daily.

Does it help? Well sure yup. For whatever reason one file was corrupt on one of my system: it's not too clear why for the file had the correct size but somehow a bit had flipped. During some sync probably. And my script caught it.

The nice thing is I can copy such files on actual backups: DVDs or BluRays or cloud or whatever. The checksum is part of the filename, so I know if my file changed or not no matter the OS / backup medium / cloud or local storage / etc.

If you have "bit flip anxiety", it helps ; )

Re: ZFS: Apple's new filesystem that wasn't (2016)

#105

Post by zie »

crazygringo wrote: post #38
zie wrote: post #27

Earlier quoted context omitted.

Basically it's that memory changes out from under you. As we know, computers use Binary, so everything boils down to it being a 0 or a 1. A bit flip is changing what was say a 0 into a 1. Usually attributed to "cosmic rays", but really can happen for any number of less exciting sounding reasons. Basically, there is zero double checking in your computer for almost everything except stuff that goes across the network.…

> disks are not checked for correctness, basically ever on any machine anywhere. I'm not sure that's really accurate -- all modern hard drives and SSD's use error-correcting codes, as far as I know. That's different from implementing additional integrity checking at the filesystem level. But it's definitely there to begin with.

Yes, the disk mostly promises what you write there will be read back correctly, but that's at the disk level only. The OS, Filesystem and Memory generally do no checking, so any errors at those levels will propagate. We know it happens, we just mostly choose to not do anything about it.

My point was, on most consumer compute, there is no promises or guarantees that what you see on day 1 will be there on day 2. It mostly works, and the chances are better than even that your data will be mostly safe on day 2, but there is zero promises or guarantees, even though we know how to do it. Some systems do, those with ECC memory and ZFS for example. Other filesystems also support checksumming, like BTRFS being the most common counter-example to ZFS. Even though parts of BTRFS are still completely broken(see their status page for details).

Re: ZFS: Apple's new filesystem that wasn't (2016)

#106

Post by pseudalopex »

ryao wrote: post #62
hs86 wrote: post #9

Earlier quoted context omitted.

While its deduplication feature clearly demands more memory, my understanding is that the ZFS ARC is treated by the kernel as a driver with a massive, persistent memory allocation that cannot be swapped out ("wired" pages). Unlike the regular file system cache, ARC's eviction is not directly managed by the kernel. Instead, ZFS itself is responsible for deciding when and how to shrink the ARC. This can lead to problem…

The ZFS driver will release memory if the kernel requests it. The only integration level issue is that the free command does not show ARC as a buffer/cache, so it misrepresents reality, but as far as I know, this is an issue with caches used by various filesystems (e.g. extent caches). It is only obvious in the case of ZFS because the ARC can be so large. That is a feature, not a bug, since unused memory is wasted me…

> The ZFS driver will release memory if the kernel requests it.

Not fast enough always.

Re: ZFS: Apple's new filesystem that wasn't (2016)

#107

Post by mixmastamyk »

GeekyBear wrote: post #75
thyristan wrote: post #35

Earlier quoted context omitted.

Both are imho linked to "offline desktop use cases are not important anymore". Both companies saw their future gains elsewhere, in internet-related functions and what became known as "cloud". No need to have a fancy, featurefull and expensive filesystem when it is only to be used as a cache for remote cloud stuff.

Internet connections of the day didn't yet offer enough speed for cloud storage. Apple was already working to integrate ZFS when Oracle bought Sun. From TFA: > ZFS was featured in the keynotes, it was on the developer disc handed out to attendees, and it was even mentioned on the Mac OS X Server website. Apple had been working on its port since 2006 and now it was functional enough to be put on full display. However,…

Was it not open source at that point?

Re: ZFS: Apple's new filesystem that wasn't (2016)

#108

Post by zie »

lgg wrote: post #65
tpetry wrote: post #45

Earlier quoted context omitted.

But SSDs (to my knowledge) only implement checksum for the data transfer. Its a requirement of the protocol. So you can be sure that the Stuff in memory and checksum computed by the CPU arrives exactly like that in the SSD driver. In the past this was a common error source with hardware raid which was faulty. But there is ABSOLUTELY NO checksum for the bits stored on a SSD. So bit rot at the cells of the SSDs are und…

That is ABSOLUTELY incorrect. SSDs have enormous amounts of error detection and correction builtin explicitly because errors on the raw medium are so common that without it you would never be able to read correct data from the device. It has been years since I was familiar enough with the insides of SSDs to tell you exactly what they are doing now, but even ~10-15 years ago it was normal for each raw 2k block to actu…

> I can say with assurance that there is no chance your OS sees uncorrected bits from your flash dies.

While true, there is zero promises that what you meant to save and what gets saved are the same things. All the drive mostly promises is that if the drive safely wrote XYZ to the disk and you come back later, you should expect to get XYZ back.

There are lots of weasel words there on purpose. There is generally zero guarantee in reality and drives lie all the time about data being safely written to disk, even if it wasn't actually safely written to disk yet. This means on power failure/interruption the outcome of being able to read XYZ back is 100% unknown. Drive Manufacturers make zero promises here.

On most consumer compute, there is no promises or guarantees that what you wrote on day 1 will be there on day 2+. It mostly works, and the chances are better than even that your data will be mostly safe on day 2+, but there is zero promises or guarantees. We know how to guarantee it, we just don't bother(usually).

You can buy laptops and desktops with ECC RAM and use ZFS(or other checksumming FS), but basically nobody does. I'm not aware of any mobile phones that offer either option.

Re: ZFS: Apple's new filesystem that wasn't (2016)

#109

Post by zie »

amarshall wrote: post #44
zie wrote: post #27

Earlier quoted context omitted.

Basically it's that memory changes out from under you. As we know, computers use Binary, so everything boils down to it being a 0 or a 1. A bit flip is changing what was say a 0 into a 1. Usually attributed to "cosmic rays", but really can happen for any number of less exciting sounding reasons. Basically, there is zero double checking in your computer for almost everything except stuff that goes across the network.…

Btrfs and bcachefs both have data checksumming. I think ReFS does as well.

Yes, ZFS is not the only filesystem with data checksumming and guarantees, but it's one of the very rare exceptions that do.

ZFS has been in productions work loads since 2005, 20 years now. It's proven to be very safe.

BTRFS has known fundamental issues past one disk. It is however improving. I will say BTRFS is fine for a single drive. Even the developers last I checked(a few years ago) don't really recommend it past a single drive, though hopefully that's changing over time.

I'm not familiar enough with bcachefs to comment.

Post reply on HN