51–60 of 123 posts
Earlier quoted context omitted.
This is one reason why RAID 0+1 is a best practice, and RAID 5 & 6 are no longer recommended. It takes too long to rebuild the array, leading to a multi-failed disk situation.
Raid 01 has its own risk. If the wrong two disks fail your entire array is toast
Earlier quoted context omitted.
Raid 01 has its own risk. If the wrong two disks fail your entire array is toast
as opposed to raid 5 where if any two disks fail your array is toast, raid 6 increases this to 3. However both raid 5 and 6 have 2 huge problems: Data inflight at write time (power/hardware failures are more likely to corrupt the array, especially silently, which is the worst outcome). Parity calculations require you to spin up the whole raid5/6 array during a rebuild, massively increasing the chance of a multi drive…
How is that? RAID doesn't affect data persistence behavior in any meaningful way. FUA/SyncCache/etc are supported by RAID controllers same as the underlying disks in writeback enviroments, parity updates included. Put another way, if you FUA or flush the writeback cache, those operations won't complete in a properly implemented RAID environment until the data is persisted somewhere, even if that means passing FUA down to the underlying storage. Granted there are a number of ways to mess this up, RMW cycles in a controller that doesn't have some kind of persistent memory and flush on power restore. Anyway, none of this is any worse than what happens in any other WB cached storage technology.
Finally, all this fearmongering about loss on rebuild is also something that should be more fully explored in the context of the fact that decent RAID systems run background scrub operations on a regular basis. Those operations by themselves are going to "stress test" the array on a regular basis when its consistent and not degraded. I've actually got a fair amount of experience in this area, and I'm here to tell you that if you think this is a risk consider what happens to non-raided unscrubbed drives that have a lot of data silently bitrotting on the platters. That latter effect is nearly always the problem in RAID environments when someone starts a rebuild on drives/sectors that have been unread for extended periods of time. But, in the case of RAID, a properly implemented system won't fail a drive for a single read failure during a rebuild, instead reconstructing from the other drives and leaving the drive online long enough to complete the rebuild and then taking it offline.
Basically raid 1 setups don't actually fix any of these problems, except through the use of massive additional parity disks overhead. Overhead that can also be applied to other RAID algorithsm to much better effect. AKA a mirrored RAID 6 provides far more protection than a mirrored raid 0. Similar levels can be had with 6+6 in environments where that is possible, with trivial capacity overhead.
Earlier quoted context omitted.
Investment in new spinning drive technologies is going away though. Nobody wants to spend R&D money on coming up with patents and ideas which will be worthless in 5 years when SSD's overtake. Science investment requires a new technology to have a prospect of a return for most of the ~20 year patent lifespan for it to look like a good investment, and spinning bits of metal aren't that right now.
In the same way that hard drives didn't kill off tape, SSD won't kill off hard drives. The price differential is too great for many applications and they have different operational strengths and weaknesses.
SSDs are higher performance than HDDs and have none of the packaging constraints. Flash storage is going to be put into everything and the economies of scale look quite good.
Storage is scaling but the r/w speeds of hdds aren't keeping up. Following the trend line and we see huge hdds that are functionally useless due to how long it takes to do disk operations.
HDDs only exist above tapes because of their performance. And only exist below SSDs due to cost. Tapes are the floor and SSDs are the quickly lowering ceiling. HDDs are likely to be crushed between.
I was about to buy 3x 12TB Toshiba for Deep Learning datasets; now I need to reconsider... Does anyone know what are the current reliability stats for >10TB drives? My old 6x 4TB HGST in NAS are running without a single problem for the past 3 years...
How do people use such large HDDs when internet download speeds are still low as compared to downloading directly on cloud services (like aws)?
I have 18 TB: 5 TB Seagate (x2), 4 TB WesternDigital, 2 TB WesternDigital, and 2 TB internal.
Backups take the most space - I fix laptops for friends from church, and they don't back up but still want their files to be safe. I had to shuffle some files around to free up 650 GB for a recent repair, mostly photos & videos.
Virtual machines use a lot of space too. I made VMWare Fusion images of every Mac OS version 10.5-10.13, Windows 95, 98, 2000, XP, 7, and 10, in several languages ( https://peterburk.github.com/i2018n ), and some Linux distros.
Another 1 TB is a dataset of Chinese characters from a machine learning project of mine ( https://blog.usejournal.com/making-of-a-chinese-characters-d... ).
Music, mostly from repaired iPods back in high school, accounts for a lot as well. There's some movies too, though I missed a chance to get 2 TB from a friend because I didn't have enough space at the time. If I upload those, even those that I legally ripped from CDs & DVDs, I'm worried that it'll trigger content filters.
For these, local disks are more useful than cloud services in my opinion.
Earlier quoted context omitted.
>We are clearly close to the end of spinning rust In every aspect, except price. Samsung 1 TB SSD for 150€, Seagate 8 TB for 220€.
Agree. And for a NAS the performance of SSD are unlikely to be required. In fact at one point I made the mistake of enabling SSD caching on a NAS. The SSD became the bottleneck because of the limitation of SATA, ie one SSD on SATA is slower than 8 or 10 HD in RAID5. So unless you really need very high iops, HD are likely to be good enough.
Earlier quoted context omitted.
This is one reason why RAID 0+1 is a best practice, and RAID 5 & 6 are no longer recommended. It takes too long to rebuild the array, leading to a multi-failed disk situation.
Interesting. What do you think the advantage of raid01 instead of raid10? The latter looks safer at first sight.
Earlier quoted context omitted.
Copy on write filesystems can probably be optimized for SMR by using TRIM commands to punch holes and rewrite the content sequentially in a new zone. Afaik both zfs and btrfs have plans to do this. That way they can be useful for more than archival.
I assume that the drive firmware remaps all your writes to make them sequential anyway for increased write performance.
Earlier quoted context omitted.
In the same way that hard drives didn't kill off tape, SSD won't kill off hard drives. The price differential is too great for many applications and they have different operational strengths and weaknesses.
Tapes have a use case that hard drives do not. Tapes are the lowest cost/GB stored and are more shelf stable than hard drives. SSDs are higher performance than HDDs and have none of the packaging constraints. Flash storage is going to be put into everything and the economies of scale look quite good. Storage is scaling but the r/w speeds of hdds aren't keeping up. Following the trend line and we see huge hdds that ar…
Earlier quoted context omitted.
Agree. And for a NAS the performance of SSD are unlikely to be required. In fact at one point I made the mistake of enabling SSD caching on a NAS. The SSD became the bottleneck because of the limitation of SATA, ie one SSD on SATA is slower than 8 or 10 HD in RAID5. So unless you really need very high iops, HD are likely to be good enough.
I'm curious who sold you a NAS with SSD caching that didn't support bypassing the cache for sequential I/O. That's a pretty basic and obvious feature, and it seems like the manufacturer must not have been taking their SSD caching feature seriously if they didn't implement bypassing.
It's an SMR drive. So better for archiving than regular IO
Copy on write filesystems can probably be optimized for SMR by using TRIM commands to punch holes and rewrite the content sequentially in a new zone. Afaik both zfs and btrfs have plans to do this. That way they can be useful for more than archival.
However since I initially saw the news of these drives a few days ago Samsung also axed some Linux devs, which gives me pause and makes me reconsider the long term viability of this filesystem...