Triple-Parity Raid and Beyond (2009)
queue.acm.org
Triple-Parity Raid and Beyond (2009)
1–10 of 13 posts
Re: Triple-Parity Raid and Beyond (2009)
#2I wasn’t building RAID arrays when this paper was written, but I have always heard RAID6 is to be avoided and to just use RAID10 (or RAID5 if you don’t have enough disks). Lately I’ve heard (and agree) the conventional wisdom is to avoid hardware RAID altogether in favor of redundancy solutions at the OS level such as MDADM on Linux or Storage Spaces on Windows (macOS supports software RAID as well, created for their now-dead Fusion Drive system).
I feel like the need for a triple-redundancy option in RAID is superseded by more “advanced” software “RAID” at the file system level such as ZFS or ButterFS (to an extent). Further, the increased availability and affordability of ECC RAM in non-enterprise hardware makes the call for additional redundancy even less-urgent.
There is a nicety to having that backup battery on a RAID card for write-through operations to finish in the event of a power outage, however this is easily-solved by a UPS. In the event of a power outage, not losing any data I might have been transferring to the array is nice, but I’m still losing my OS state and any unsaved things I may have been working on.
Re: Triple-Parity Raid and Beyond (2009)
#3I wasn’t building RAID arrays when this paper was written, but I have always heard RAID6 is to be avoided and to just use RAID10 (or RAID5 if you don’t have enough disks). Lately I’ve heard (and agree) the conventional wisdom is to avoid hardware RAID altogether in favor of redundancy solutions at the OS level such as MDADM on Linux or Storage Spaces on Windows (macOS supports software RAID as well, created for their…
there’s always use cases but I think the interfaces are fast enough
NVMe SSDs, direct or over PCIe solve lots of the issue that people made RAID arrays for to begin with
sure in theory you can go even faster and redundant with RAID concepts on these nvme ssds, but tad overkill
Re: Triple-Parity Raid and Beyond (2009)
#4I wasn’t building RAID arrays when this paper was written, but I have always heard RAID6 is to be avoided and to just use RAID10 (or RAID5 if you don’t have enough disks). Lately I’ve heard (and agree) the conventional wisdom is to avoid hardware RAID altogether in favor of redundancy solutions at the OS level such as MDADM on Linux or Storage Spaces on Windows (macOS supports software RAID as well, created for their…
there’s always use cases but I think the interfaces are fast enough NVMe SSDs, direct or over PCIe solve lots of the issue that people made RAID arrays for to begin with sure in theory you can go even faster and redundant with RAID concepts on these nvme ssds, but tad overkill
There will always be a need for platter drives (or other magnetic storage) because flash memory loses data over time if it's powered-off. Theoretically you can store a tape or other magnetic drive for decades in a safety deposit box or the like and your data will still be there if/when you need it.
I don't like the trend of SSDs (NVMe or otherwise) getting cheaper and cheaper because it's coming at the cost of reliability and endurance. Sure I can get 2TB for ~$100 but at this point I'm not convinced it will outlast spinning rust as has been the colloquial assumption since 2.5" SSDs first hit the scene circa 2008.
I've quickly destroyed (consumer-grade) SSDs before by running stuff that is constantly reading and writing to them. Microsoft's Azure Stack Development Kit (ASDK) is one example.
Therefore, I'm actually very receptive to RAID'ing SSDs, be it MDADM, ZFS or some other means. I do agree that to RAID (RAID0) NVMe drives is a bit ridiculous, but RAID1 definitely adds value.
Re: Triple-Parity Raid and Beyond (2009)
#5Earlier quoted context omitted.
there’s always use cases but I think the interfaces are fast enough NVMe SSDs, direct or over PCIe solve lots of the issue that people made RAID arrays for to begin with sure in theory you can go even faster and redundant with RAID concepts on these nvme ssds, but tad overkill
There will always be a need for platter drives (or other magnetic storage) because flash memory loses data over time if it's powered-off. Theoretically you can store a tape or other magnetic drive for decades in a safety deposit box or the like and your data will still be there if/when you need it. I don't like the trend of SSDs (NVMe or otherwise) getting cheaper and cheaper because it's coming at the cost of reliab…
> because flash memory loses data over time if it's powered-off.
That's a red herring; the magnitude of this effect is usually greatly exaggerated, and in reality never comes close to being as important as the fact that flash memory is simply too expensive to use for cold storage.
And of course, the very idea of cold storage is dangerous: if you really want to ensure your data lasts for decades, you should be verifying your backups at least annually and making plans to migrate your data off any media that is obsolete and at risk of becoming hard to read using commodity hardware. This also entirely eliminates the above flash memory data retention concerns.
Re: Triple-Parity Raid and Beyond (2009)
#6Earlier quoted context omitted.
There will always be a need for platter drives (or other magnetic storage) because flash memory loses data over time if it's powered-off. Theoretically you can store a tape or other magnetic drive for decades in a safety deposit box or the like and your data will still be there if/when you need it. I don't like the trend of SSDs (NVMe or otherwise) getting cheaper and cheaper because it's coming at the cost of reliab…
> because flash memory loses data over time if it's powered-off. That's a red herring; the magnitude of this effect is usually greatly exaggerated, and in reality never comes close to being as important as the fact that flash memory is simply too expensive to use for cold storage. And of course, the very idea of cold storage is dangerous: if you really want to ensure your data lasts for decades, you should be verifyi…
I remember reading something somewhere about a guy doing tests on "3D" NAND and other TLC/MLC NAND SSDs and finding that data was lost well before the "10 years or so" that we have come to assume unpowered SSDs can retain data for. I can't locate that article at the moment but it isn't unreasonable to expect faster degradation on denser NAND.
Re: Triple-Parity Raid and Beyond (2009)
#7I wasn’t building RAID arrays when this paper was written, but I have always heard RAID6 is to be avoided and to just use RAID10 (or RAID5 if you don’t have enough disks). Lately I’ve heard (and agree) the conventional wisdom is to avoid hardware RAID altogether in favor of redundancy solutions at the OS level such as MDADM on Linux or Storage Spaces on Windows (macOS supports software RAID as well, created for their…
> I feel like the need for a triple-redundancy option in RAID is superseded by more “advanced” software “RAID” at the file system level such as ZFS or ButterFS (to an extent).
ZFS has triple-parity with RAID-Z3:
> The need for RAID-Z3 arose in the early 2000s as multi-terabyte capacity drives became more common. This increase in capacity—without a corresponding increase in throughput speeds—meant that rebuilding an array due to a failed drive could "easily take weeks or months" to complete.[38] During this time, the older disks in the array will be stressed by the additional workload, which could result in data corruption or drive failure. By increasing parity, RAID-Z3 reduces the chance of data loss by simply increasing redundancy.[40]
* https://en.wikipedia.org/wiki/ZFS#ZFS's_approach:_RAID-Z_and...
The use of software versus "hardware" (firmware) does not remove the need for extra copies of data.
Re: Triple-Parity Raid and Beyond (2009)
#8Earlier quoted context omitted.
There will always be a need for platter drives (or other magnetic storage) because flash memory loses data over time if it's powered-off. Theoretically you can store a tape or other magnetic drive for decades in a safety deposit box or the like and your data will still be there if/when you need it. I don't like the trend of SSDs (NVMe or otherwise) getting cheaper and cheaper because it's coming at the cost of reliab…
> because flash memory loses data over time if it's powered-off. That's a red herring; the magnitude of this effect is usually greatly exaggerated, and in reality never comes close to being as important as the fact that flash memory is simply too expensive to use for cold storage. And of course, the very idea of cold storage is dangerous: if you really want to ensure your data lasts for decades, you should be verifyi…
> That's a red herring; the magnitude of this effect is usually greatly exaggerated […]
Per JEDEC, Client SSDs have to retain data for 1 year at 30C, and Enterprise SSDs have to retain data for 3 months at 40C:
* https://www.jedec.org/sites/default/files/Alvin_Cox%20[Compa...
I'm sure most last longer, but as a CYA I wouldn't want to rely on that.
Re: Triple-Parity Raid and Beyond (2009)
#9Earlier quoted context omitted.
there’s always use cases but I think the interfaces are fast enough NVMe SSDs, direct or over PCIe solve lots of the issue that people made RAID arrays for to begin with sure in theory you can go even faster and redundant with RAID concepts on these nvme ssds, but tad overkill
There will always be a need for platter drives (or other magnetic storage) because flash memory loses data over time if it's powered-off. Theoretically you can store a tape or other magnetic drive for decades in a safety deposit box or the like and your data will still be there if/when you need it. I don't like the trend of SSDs (NVMe or otherwise) getting cheaper and cheaper because it's coming at the cost of reliab…
> I do agree that to RAID (RAID0) NVMe drives is a bit ridiculous, but RAID1 definitely adds value.
RAID0 is handy in HPC for local scratch space.
Re: Triple-Parity Raid and Beyond (2009)
#10Earlier quoted context omitted.
> because flash memory loses data over time if it's powered-off. That's a red herring; the magnitude of this effect is usually greatly exaggerated, and in reality never comes close to being as important as the fact that flash memory is simply too expensive to use for cold storage. And of course, the very idea of cold storage is dangerous: if you really want to ensure your data lasts for decades, you should be verifyi…
> That's a red herring; the magnitude of this effect is usually greatly exaggerated […] Per JEDEC, Client SSDs have to retain data for 1 year at 30C, and Enterprise SSDs have to retain data for 3 months at 40C: * https://www.jedec.org/sites/default/files/Alvin_Cox%20[Compa... I'm sure most last longer, but as a CYA I wouldn't want to rely on that.
That's the standard for drives that are at end of life, having exhausted their rated write endurance. Those aren't the drives anyone would use for a cold storage backup system. Drives that have only been written to a few times will retain data much longer.