Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In Praise of ZFS on Linux's ZED 'ZFS Event Daemon'

utcc.utoronto.ca

Post reply on HN
Print view View original

101–110 of 124 posts

Re: In Praise of ZFS on Linux's ZED 'ZFS Event Daemon'

#101

Post by ghaff »

rleigh wrote: post #100
ghaff wrote: post #38

Earlier quoted context omitted.

Because Danese Cooper said so. (See video link from the Wikipedia article: https://en.wikipedia.org/wiki/Common_Development_and_Distrib... ) I think there's also a general suspicion that Sun could have just chosen the GPL if they cared about compatibility. Although, for various reasons, it's probably at least somewhat more complicated than that because of patent protection, etc.

"Sun could have just chosen the GPL if they cared about compatibility." That's a very loaded statement. I've seen it said quite a lot over the years. But, have you thought about its implications? The implicit assumption here is the primacy of the GPL over all other open source licences. Why should other companies and organisations treat it as "more special" than any other free/open source licence when it comes down t…

I should have written compatibility with the GPL (or really the Linux kernel which was what was most relevant from the perspective of Solaris). And, of course, Sun could have chosen a fully permissive license but AFAIK nothing like that was seriously considered.

Nit: Apache 2.0 is compatible with GPLv3 (but not v2).

Re: In Praise of ZFS on Linux's ZED 'ZFS Event Daemon'

#102

Post by accelbred »

tw04 wrote: post #71
briffle wrote: post #56

Earlier quoted context omitted.

or install BTRFS and be able to detect bitrot as well, but also be able to grow your storage by adding more disks. (even different sizes)

RAID-Z expansion is being worked on: https://github.com/openzfs/zfs/pull/8853 Last I checked BTRFS RAID5/6 was a dumpster fire and unusable in production. Have they actually open sourced the ability to fix bitrot detection with mdraid? If not, it's kind of irrelevant. So... once again down votes without response - BTRFS raid still isn't recommended and the file healing isn't compatible with MDRAID I assume and you ju…

If you're using RAID-Z on zfs, your comparison isn't fair. Rather than use RAID56 with btrfs, the equivalent would be to get 1 or 2 disk redundancy with raid1 or raid1c3.

Re: In Praise of ZFS on Linux's ZED 'ZFS Event Daemon'

#103

Post by theamk »

louwrentius wrote: post #84
theamk wrote: post #73

Earlier quoted context omitted.

Silent data corruption is a real thing, I have a few thousands files for the evidence. If you are building a home NAS from quality rackmounted server parts, then maybe you are fine. But this was not an option for me, as I did not have dedicated server space and needed something quiet. And once you have to start with to mess with desktop cases full of hard drives, it is very easy to get corrupted data. I run ZFS on my…

The question is always: what did exactly happen and how would other solutions have fared. A plain statement like this doesn't prove anything.

It provides another example that silent data corruption is a thing, and that it can happen. While SATA protocol has error detection, it is pretty weak (32 bit CRC) and it does not always help, especially since there is no way to tell how often the packets are retried.

I had a few cases of data damage. One of the worst ones was when I moved to a different place, and had to leave much of my stuff behind. I had half a dozen or so smaller drives in my PC (SATA + IDE) which were working just fine. I got about three new drives (I believe SATA 1TB?), installed them into PC, and copied all the files to the new drives. I then left the old PC, and only took the new drives with me.

This was Linux, ext3 and JBOD (no RAID or anything). I did not have a good filing system, so some data got copied multiple times.

Once I got into new place, I bought a new PC and installed the hard drives I had. I have noticed that some files are damaged. I had some checksumming scripts, and I was recording checksums, and found out that some checksums would not match - and each copy had a different set of damaged files. I ended up cherry-picking files from multiple copies to assemble a good set.

I don't know the exact reason, but I am fairly sure they were transfer errors. The original PC was working fine for a long time, so source data was likely clean. The new PC did not show any more data corruptions, and it was reading the same data every time. So my theory is either transfer errors while copying files, or silent data corruption on disk.

I don't know of any solutions that would have helped here except custom data checksum tools or ZFS (I suppose btrfs might have helped too, but I heard too many horror stories about it).

I actually had this come up the second time: when I moved again, and built another NAS box (desktop motherboard, 5x 4TB drives with ZFS), I started copying files off the old SATA drives (ext3) and saw the data transfer mismatch. It was pretty freaky: "rsync" the file, flush caches, md5sum source and destinations -- and they are different. kernel log was quiet, memtest was not showing any errors, so I got a beefier power supply and replaced all the SATA cables. This helped.

Re: In Praise of ZFS on Linux's ZED 'ZFS Event Daemon'

#104

Post by throw0101a »

ghaff wrote: post #38
Wowfunhappy wrote: post #31

Earlier quoted context omitted.

Do you happen to know where the claim that it was intended came from?

Because Danese Cooper said so. (See video link from the Wikipedia article: https://en.wikipedia.org/wiki/Common_Development_and_Distrib... ) I think there's also a general suspicion that Sun could have just chosen the GPL if they cared about compatibility. Although, for various reasons, it's probably at least somewhat more complicated than that because of patent protection, etc.

> I think there's also a general suspicion that Sun could have just chosen the GPL if they cared about compatibility.

There were 'technical' reasons why they did go not with GPL, and specifically GPLv2 (GPLv3 was not out yet). IIRC, they did consider waiting for GPLv3, but it was unknown when it would be out, and one thing they desired was a patent grant, which v2 does not have.

Another condition was that they wanted a file-based copyright rather than a work-based copyright (i.e., applies to any individual files of ZFS as opposed to "ZFS" in aggregate).

* https://nawilson.com/2007/12/02/why-the-dislike-for-cddl/

Re: In Praise of ZFS on Linux's ZED 'ZFS Event Daemon'

#105

Post by ghaff »

throw0101a wrote: post #104
ghaff wrote: post #38

Earlier quoted context omitted.

Because Danese Cooper said so. (See video link from the Wikipedia article: https://en.wikipedia.org/wiki/Common_Development_and_Distrib... ) I think there's also a general suspicion that Sun could have just chosen the GPL if they cared about compatibility. Although, for various reasons, it's probably at least somewhat more complicated than that because of patent protection, etc.

> I think there's also a general suspicion that Sun could have just chosen the GPL if they cared about compatibility. There were 'technical' reasons why they did go not with GPL, and specifically GPLv2 (GPLv3 was not out yet). IIRC, they did consider waiting for GPLv3, but it was unknown when it would be out, and one thing they desired was a patent grant, which v2 does not have. Another condition was that they wanted…

I had forgotten about some of the reasons they specifically wanted file-based copyright. Sun were clients at the time and I spoke fairly frequently with the open source folks there. But I didn't remember all the details and was certainly not privy to all the internal discussions.

Re: In Praise of ZFS on Linux's ZED 'ZFS Event Daemon'

#106

Post by throw0101a »

giantrobot wrote: post #79
e40 wrote: post #74

Earlier quoted context omitted.

UFS on what OS?

NetFlix is a heavy FreeBSD shop, so I'd assume FreeBSD.

> NetFlix is a heavy FreeBSD shop, so I'd assume FreeBSD.

Kind of: their edge-cache appliances run FreeBSD. IIRC they run Linux on their Amazon cloud for all their 'internal' stuff.

If you do some searches there's some good presentations on their work on getting encrypted streaming to go Very Fast:

* https://www.phoronix.com/scan.php?page=news_item&px=Netflix-...

* https://2019.eurobsdcon.org/talk-speakers/#numa

* https://netflixtechblog.com/serving-100-gbps-from-an-open-co...

Re: In Praise of ZFS on Linux's ZED 'ZFS Event Daemon'

#107

Post by throw0101a »

curt15 wrote: post #24
diffeomorphism wrote: post #13

Earlier quoted context omitted.

Can you comment on btrfs by comparison? Also, what is the status of the license incompatibility/integration of ZFS with distros? Canonical seems to think that shipping it with ubuntu is legal, but other distros seem less sure.

Fedora is proposing to default to btrfs for its next release: https://fedoraproject.org/wiki/Changes/BtrfsByDefault

Contrast with Debian's warnings on the topic:

* https://wiki.debian.org/Btrfs#Warnings

Re: In Praise of ZFS on Linux's ZED 'ZFS Event Daemon'

#108

Post by StavrosK »

washadjeffmad wrote: post #88
StavrosK wrote: post #57

Earlier quoted context omitted.

Probably not, since it's a home NAS the reads won't be very repeatable. I wanted to prevent the hard disks from waking up, but it's mostly writes, so it doesn't help there either.

Add the SSD to its own pool, set up as a working share, then use a utility or script to sync contents a few times a day?

That would probably be easier, yep, thanks. It's more straightforward to just make a partition on it and rsync twice a day.

Re: In Praise of ZFS on Linux's ZED 'ZFS Event Daemon'

#109

Post by Geezus_42 »

briffle wrote: post #56
m0zg wrote: post #29

Earlier quoted context omitted.

> ZFS is fine, but it is overkill for most home applications Yeah, not having silent data corruption is "overkill", sure. /s Why not use ZFS? It takes 15 seconds to install, and its CLI is fairly intuitive. Works fine. Costs $0. Why not, even for "home" applications? I could see how it could be unsuitable for "entreprise" applications where there are strict performance requirements etc, but for home, I wish I could u…

or install BTRFS and be able to detect bitrot as well, but also be able to grow your storage by adding more disks. (even different sizes)

Just avoid RAID 5/6

Re: In Praise of ZFS on Linux's ZED 'ZFS Event Daemon'

#110

Post by drewg123 »

throw0101a wrote: post #106
giantrobot wrote: post #79

Earlier quoted context omitted.

NetFlix is a heavy FreeBSD shop, so I'd assume FreeBSD.

> NetFlix is a heavy FreeBSD shop, so I'd assume FreeBSD. Kind of: their edge-cache appliances run FreeBSD. IIRC they run Linux on their Amazon cloud for all their 'internal' stuff. If you do some searches there's some good presentations on their work on getting encrypted streaming to go Very Fast: * https://www.phoronix.com/scan.php?page=news_item&px=Netflix-... * https://2019.eurobsdcon.org/talk-speakers/#numa * ht…

UFS on FreeBSD. And that's me :)
Post reply on HN