41–50 of 115 posts
Fascinating write up. As someone with a ZFS system, how can I check if I’m affected?
> zpool get all tank | grep bclone
> kc3000 bcloneused 442M
> kc3000 bclonesaved 1.42G
> kc3000 bcloneratio 4.30x
> My understanding is this: If the result is 0 for both bcloneused and bclonesaved then it's safe to say that you don't have silent corruption.
Periodic reminder to check if your backups are working, and if you can also restore them. It doesn't matter which file system or operating system you use, make sure to backup your stuff. In a way that's immune to ransomware as well, so not just a RAID-1/5/Z or another form of hot/warm storage (RAID is not a backup, it's an uptime/availability mechanism) but cold storage. (I snapshot and tar that snapshot every night,…
I see this advice repeated frequently, but it's always very general. Do you have any advice as to HOW the average home NAS user can affordably backup modern NAS devices? The last time I looked it could easily cost hundreds of dollars per month to back up as little as 40TB to the cloud.
Backup what you value the most, ignore what you don't and apply tiers depending on what needs to be kept but you can deal with transferring it back home slowly and what you need immediately in case of a failure.
My rules of thumb are:
- always invest 3x the price of your hot live NAS storage in backups. If you can't afford buying 40TB of storage, you can't afford having 10TB of live storage. Period. Goal is to have at least one copy locally and one externally and have more space to on the backup storages to account for retentions, changes and help with migrations.
- if you can't afford 3 redundant storages(RAID), favor having 3 times non redundant storage (no RAID) over having less copies of redundant one.
Additional tip to reduce cost and avoid expensive cloud offering is to find a reliable and trustable relative or friend that can host your external copy of your backup. Nebula or Tailscale now makes it very easy without having to configure routers and stuff. In exchange you can offer that person to host his/her backup storage.
Also digitalizing material stuff is nice, but printing digital photos is also a great way to preserve copies. I'd rather save the photos I cherish the most than having 3 backup copies of 10TB of blurry or non outstanding photos. After years of having them all digitally, I am inveting back in printing photos and making albums. You can also print photobook multiple times and have some stored at a relative's place.
Periodic reminder to check if your backups are working, and if you can also restore them. It doesn't matter which file system or operating system you use, make sure to backup your stuff. In a way that's immune to ransomware as well, so not just a RAID-1/5/Z or another form of hot/warm storage (RAID is not a backup, it's an uptime/availability mechanism) but cold storage. (I snapshot and tar that snapshot every night,…
I see this advice repeated frequently, but it's always very general. Do you have any advice as to HOW the average home NAS user can affordably backup modern NAS devices? The last time I looked it could easily cost hundreds of dollars per month to back up as little as 40TB to the cloud.
I know you can use ZFS snapshots but I'm not experienced enough to trust that I could make a 20-40tb snapshot without screwing something up. Plus it's all video files so I can roughly keep track of what's what and I can ignore the stupid LTO compression.
It takes days, its noisy, and very tedious. But thats #hoarderLyfe lol
Earlier quoted context omitted.
When will you ever be able to appreciate and look at 9T of photos?
When you're old and retired, and are reminiscing about your kids or grandkids back when they were small, or about past vacations. My parents tend to take a lot of photos whenever the family is together, and it used to bother me. Only in recent years I started to understand them.
Fascinating write up. As someone with a ZFS system, how can I check if I’m affected?
https://github.com/openzfs/zfs/issues/15526#issuecomment-181... > zpool get all tank | grep bclone > kc3000 bcloneused 442M > kc3000 bclonesaved 1.42G > kc3000 bcloneratio 4.30x > My understanding is this: If the result is 0 for both bcloneused and bclonesaved then it's safe to say that you don't have silent corruption.
It's also not worth checking for because this bug has existed for many years. Your data probably wasn't affected. None of the massive ZFS storage companies out there ran into it by now either.
Your data is fine. Sleep easy.
IMO, part of the issue is that something which used to be just a low-level optimization (don't store large sequences of zeros) became visible to userspace (SEEK_HOLE and friends). Quoting from this article: "This is allowed; its always safe to say there’s data where there’s a hole, because reading a hole area will always find “zeroes”, which is valid data." But I recall reading elsewhere a discussion about some users…
Indeed, sparse files are simply a mistake to have included in Unix in the first place (I think we blame this on early SunOS? Not sure, though almost certain that 3BSD and v7 didn't have them). Yes, they have been used productively for various tricks, but they create a bunch of complexity that every filesystem needs to carry along with it. It's a bad trade.
IMO, part of the issue is that something which used to be just a low-level optimization (don't store large sequences of zeros) became visible to userspace (SEEK_HOLE and friends). Quoting from this article: "This is allowed; its always safe to say there’s data where there’s a hole, because reading a hole area will always find “zeroes”, which is valid data." But I recall reading elsewhere a discussion about some users…
Indeed, sparse files are simply a mistake to have included in Unix in the first place (I think we blame this on early SunOS? Not sure, though almost certain that 3BSD and v7 didn't have them). Yes, they have been used productively for various tricks, but they create a bunch of complexity that every filesystem needs to carry along with it. It's a bad trade.
Earlier quoted context omitted.
I agree with the advice which is what we do. Average home user (with emphasis on average) doesn't have 40TB, but a "normal" non-professional one might. We have about 9TB of photos. I can easily imagine someone like us, who is into video, of having more than 40TB of videos.
When will you ever be able to appreciate and look at 9T of photos?
Today you might take 10 photos of your family and keep the best one where everyone is smiling.
But 10-20 years from now you will probably appreciate having kept the other 9 where the baby is crying, the kid is making a face, and grandma has started to wander off.
Earlier quoted context omitted.
Indeed, sparse files are simply a mistake to have included in Unix in the first place (I think we blame this on early SunOS? Not sure, though almost certain that 3BSD and v7 didn't have them). Yes, they have been used productively for various tricks, but they create a bunch of complexity that every filesystem needs to carry along with it. It's a bad trade.
This a feature I was completely unaware of. Why would you choose to use a sparse file instead of multiple files?
Earlier quoted context omitted.
Indeed, sparse files are simply a mistake to have included in Unix in the first place (I think we blame this on early SunOS? Not sure, though almost certain that 3BSD and v7 didn't have them). Yes, they have been used productively for various tricks, but they create a bunch of complexity that every filesystem needs to carry along with it. It's a bad trade.
This a feature I was completely unaware of. Why would you choose to use a sparse file instead of multiple files?