Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A data corruption bug in OpenZFS?

despairlabs.com

Post reply on HN
Print view View original

31–40 of 115 posts

Re: A data corruption bug in OpenZFS?

#31

Post by cesarb »

IMO, part of the issue is that something which used to be just a low-level optimization (don't store large sequences of zeros) became visible to userspace (SEEK_HOLE and friends). Quoting from this article:

"This is allowed; its always safe to say there’s data where there’s a hole, because reading a hole area will always find “zeroes”, which is valid data."

But I recall reading elsewhere a discussion about some userspace program which did depend on holes being present in the filesystem as actual holes (visible to SEEK_HOLE and so on) and not as runs of zeros.

Combined with the holes being restricted to specific alignments and sizes, this means that the underlying "sequence of fixed-size blocks" implementation is leaking too much over the abstract "stream of bytes" representation we're more used to. Perhaps it might be time to rethink our filesystem abstractions?

Re: A data corruption bug in OpenZFS?

#32

Post by ericbarrett »

tetha wrote: post #12
MenhirMike wrote: post #3

Periodic reminder to check if your backups are working, and if you can also restore them. It doesn't matter which file system or operating system you use, make sure to backup your stuff. In a way that's immune to ransomware as well, so not just a RAID-1/5/Z or another form of hot/warm storage (RAID is not a backup, it's an uptime/availability mechanism) but cold storage. (I snapshot and tar that snapshot every night,…

It is also a good idea to test the restore procedures and documentation as well. Don't have the grizzled old storage admin / DBA test the backup. They know a million and one weird necessary workarounds and just execute them. However, if you need a restore and they are currently exploring caves or something, things turn dire. Have a chipper junior restore something based off of the documentation (and prepare to spend…

As a grizzled old storage admin who somehow made a career out of database backups, I wholeheartedly agree with all of this. Especially having someone else do a test restore. They don't have to be junior, just not intimately familiar with the systems involved.

Re: A data corruption bug in OpenZFS?

#33

Post by feanaro »

samastur wrote: post #18
dewey wrote: post #14

Earlier quoted context omitted.

“Average home NAS user” doesn’t have 40TB of data. With a subset of data that’s important like photos it’s not that expensive and with Backblaze and other services that are directly integrated in operating systems like Synology also not that hard to do.

I agree with the advice which is what we do. Average home user (with emphasis on average) doesn't have 40TB, but a "normal" non-professional one might. We have about 9TB of photos. I can easily imagine someone like us, who is into video, of having more than 40TB of videos.

When will you ever be able to appreciate and look at 9T of photos?

Re: A data corruption bug in OpenZFS?

#34

Post by grepfru_it »

stevenAthompson wrote: post #13
MenhirMike wrote: post #3

Periodic reminder to check if your backups are working, and if you can also restore them. It doesn't matter which file system or operating system you use, make sure to backup your stuff. In a way that's immune to ransomware as well, so not just a RAID-1/5/Z or another form of hot/warm storage (RAID is not a backup, it's an uptime/availability mechanism) but cold storage. (I snapshot and tar that snapshot every night,…

I see this advice repeated frequently, but it's always very general. Do you have any advice as to HOW the average home NAS user can affordably backup modern NAS devices? The last time I looked it could easily cost hundreds of dollars per month to back up as little as 40TB to the cloud.

LTO. I bought an LTO-5 system to backup 6TB of critical data and 12TB of nice-to-have data. LTO-6 is better if you can afford it.

Downside to tape backup is you need throughput, or the ability to do disk-disk backups

Re: A data corruption bug in OpenZFS?

#35

Post by grepfru_it »

andyjohnson0 wrote: post #22
MenhirMike wrote: post #3

Periodic reminder to check if your backups are working, and if you can also restore them. It doesn't matter which file system or operating system you use, make sure to backup your stuff. In a way that's immune to ransomware as well, so not just a RAID-1/5/Z or another form of hot/warm storage (RAID is not a backup, it's an uptime/availability mechanism) but cold storage. (I snapshot and tar that snapshot every night,…

I'd be interested to know what tape setup you use? I occasionally look into using LTO tapes for home backup, but the media and hardware always seems a bit too expensive compared to something like Backblaze (which I currently use). Also afaik tapes need a stable storage environment: how do you manage that?

LTO5. The cost of my LTO5 system is the cost of downloading all of my data once from a remote cloud provider. It's a nobrainer

Re: A data corruption bug in OpenZFS?

#36

Post by ajross »

cesarb wrote: post #31

IMO, part of the issue is that something which used to be just a low-level optimization (don't store large sequences of zeros) became visible to userspace (SEEK_HOLE and friends). Quoting from this article: "This is allowed; its always safe to say there’s data where there’s a hole, because reading a hole area will always find “zeroes”, which is valid data." But I recall reading elsewhere a discussion about some users…

Indeed, sparse files are simply a mistake to have included in Unix in the first place (I think we blame this on early SunOS? Not sure, though almost certain that 3BSD and v7 didn't have them). Yes, they have been used productively for various tricks, but they create a bunch of complexity that every filesystem needs to carry along with it. It's a bad trade.

Re: A data corruption bug in OpenZFS?

#37

Post by toast0 »

tetha wrote: post #12
MenhirMike wrote: post #3

Periodic reminder to check if your backups are working, and if you can also restore them. It doesn't matter which file system or operating system you use, make sure to backup your stuff. In a way that's immune to ransomware as well, so not just a RAID-1/5/Z or another form of hot/warm storage (RAID is not a backup, it's an uptime/availability mechanism) but cold storage. (I snapshot and tar that snapshot every night,…

It is also a good idea to test the restore procedures and documentation as well. Don't have the grizzled old storage admin / DBA test the backup. They know a million and one weird necessary workarounds and just execute them. However, if you need a restore and they are currently exploring caves or something, things turn dire. Have a chipper junior restore something based off of the documentation (and prepare to spend…

One nice thing about a circa 2010 MySQL setup is setting up a new replica is easiest by restoring a backup. If you have to do that from time to time, your backups get tested by regular process.

Re: A data corruption bug in OpenZFS?

#38

Post by doublepg23 »

feanaro wrote: post #33
samastur wrote: post #18

Earlier quoted context omitted.

I agree with the advice which is what we do. Average home user (with emphasis on average) doesn't have 40TB, but a "normal" non-professional one might. We have about 9TB of photos. I can easily imagine someone like us, who is into video, of having more than 40TB of videos.

When will you ever be able to appreciate and look at 9T of photos?

AI tools analyze photos pretty well now. It’s very common they bubble up old photos I had forgotten about.

Re: A data corruption bug in OpenZFS?

#39

Post by dist-epoch »

grepfru_it wrote: post #34
stevenAthompson wrote: post #13

Earlier quoted context omitted.

I see this advice repeated frequently, but it's always very general. Do you have any advice as to HOW the average home NAS user can affordably backup modern NAS devices? The last time I looked it could easily cost hundreds of dollars per month to back up as little as 40TB to the cloud.

LTO. I bought an LTO-5 system to backup 6TB of critical data and 12TB of nice-to-have data. LTO-6 is better if you can afford it. Downside to tape backup is you need throughput, or the ability to do disk-disk backups

For 20 TB LTO seems too expensive.

20 TB of SSD costs about $1000.

Or you could get a 20 TB hard drive for $300.

Re: A data corruption bug in OpenZFS?

#40

Post by pferde »

feanaro wrote: post #33
samastur wrote: post #18

Earlier quoted context omitted.

I agree with the advice which is what we do. Average home user (with emphasis on average) doesn't have 40TB, but a "normal" non-professional one might. We have about 9TB of photos. I can easily imagine someone like us, who is into video, of having more than 40TB of videos.

When will you ever be able to appreciate and look at 9T of photos?

When you're old and retired, and are reminiscing about your kids or grandkids back when they were small, or about past vacations.

My parents tend to take a lot of photos whenever the family is together, and it used to bother me. Only in recent years I started to understand them.

Post reply on HN