RAID6 should be fine rebuilding online (in RAID5 mode) even under a moderate write load. Of course one should source RAID disks form 3 different vendors, to ensure that they are from different batches, and are not going to fail at approximately the same time.
Do other manufacturers produce this size of drive? It's difficult to source from 3 vendors if there's only one making the product.
Get one from amazon, one from newegg, one from the manufacturer directly or some such.
Copy on write filesystems can probably be optimized for SMR by using TRIM commands to punch holes and rewrite the content sequentially in a new zone. Afaik both zfs and btrfs have plans to do this. That way they can be useful for more than archival.
You probably mean log structured, not copy on write. CoW doesn't help make writes sequential, unlike log structured filesystems.
log-structured is a special-case of CoW; specifically it's CoW where the allocation strategy is sequential blocks.
It's an SMR drive. So better for archiving than regular IO
Copy on write filesystems can probably be optimized for SMR by using TRIM commands to punch holes and rewrite the content sequentially in a new zone. Afaik both zfs and btrfs have plans to do this. That way they can be useful for more than archival.
I assume that the drive firmware remaps all your writes to make them sequential anyway for increased write performance.
I was about to buy 3x 12TB Toshiba for Deep Learning datasets; now I need to reconsider... Does anyone know what are the current reliability stats for >10TB drives? My old 6x 4TB HGST in NAS are running without a single problem for the past 3 years...
How do people use such large HDDs when internet download speeds are still low as compared to downloading directly on cloud services (like aws)?
My dad and I created backups locally, mailed each other hard-drives, and we just do a weekly rsync. Storage is large, but network traffic is relatively low.
I am going for the 15tb instead of that 14 so I have extra space for backups. Says no body. We are clearly close to the end of spinning rust, absent some new breakthrough.
For archival spinning rust is still massively more cost effective than SSDs while offering better performance than tape, especially for random access. I'm sure eventually SSDs will be cheap enough that HDD will go the way of the floppy but we're not there yet.
Is tape even cheaper than spinning rust? Last time I priced it, the per-GB costs were similar and the tape drives themselves are quite expensive. Tape is a more reliable backup, since the moving parts are not part of the storage, but it's not a cheaper backup.
I am going for the 15tb instead of that 14 so I have extra space for backups. Says no body. We are clearly close to the end of spinning rust, absent some new breakthrough.
HAMR and then HDMR are expected to allow data densities to increase by 5 to 10 times what is currently achievable. HAMR will probably start showing up in a year or two. Spinning drives are definitely not going away anytime soon unless there is a much more significant drop in the cost of SSDs.
Investment in new spinning drive technologies is going away though. Nobody wants to spend R&D money on coming up with patents and ideas which will be worthless in 5 years when SSD's overtake.
Science investment requires a new technology to have a prospect of a return for most of the ~20 year patent lifespan for it to look like a good investment, and spinning bits of metal aren't that right now.
Will we hit a point where the size of the drive is simply too big to get the data off in any decent amount of time?
This is one reason why RAID 0+1 is a best practice, and RAID 5 & 6 are no longer recommended. It takes too long to rebuild the array, leading to a multi-failed disk situation.
HAMR and then HDMR are expected to allow data densities to increase by 5 to 10 times what is currently achievable. HAMR will probably start showing up in a year or two. Spinning drives are definitely not going away anytime soon unless there is a much more significant drop in the cost of SSDs.
Investment in new spinning drive technologies is going away though. Nobody wants to spend R&D money on coming up with patents and ideas which will be worthless in 5 years when SSD's overtake. Science investment requires a new technology to have a prospect of a return for most of the ~20 year patent lifespan for it to look like a good investment, and spinning bits of metal aren't that right now.
In the same way that hard drives didn't kill off tape, SSD won't kill off hard drives. The price differential is too great for many applications and they have different operational strengths and weaknesses.
I was about to buy 3x 12TB Toshiba for Deep Learning datasets; now I need to reconsider... Does anyone know what are the current reliability stats for >10TB drives? My old 6x 4TB HGST in NAS are running without a single problem for the past 3 years...
How do people use such large HDDs when internet download speeds are still low as compared to downloading directly on cloud services (like aws)?
If you are doing serious Deep/Machine Learning you often get massive datasets, you might need to pre-process substantial portions of them for different models you want to try, you might want to do custom models that are trained e.g. on 4k/8k video footage (think DLSS, self-driving car or drone footage); space gets exhausted pretty quickly. You might also want multiple independent drives, as you might do this all in parallel and it would be too slow to do it on a single drive at the same time, and even RAID with two drives would have a large performance penalty for seeks.