101–110 of 276 posts
Earlier quoted context omitted.
Just FYI for anyone for whom this didn't work by default: I needed to use the --all flag with smartctl (and install smartmontools if you don't have it).
The -a flag from my example should be an alias for --all (man smartctl | grep -A1 ' -a' | head -2). Is that not the case in your version?
Earlier quoted context omitted.
Wow, thanks for sharing. I didn't realize how closely related they were. (TLDR For anyone wondering, "recent HN issues" means HN very likely went down yesterday because of this same bug, when two (edit: two pairs, four total) enterprise SSDs with old firmware died after 40,000 hours close together. An admin of HN and its host both like this theory. See details in that thread.) Edit: If you want to discuss that theory…
the chance of two SSD's failing at the same time under normal circumstances is extremely slim. So this might actually be a good cause of this incident.
Cisco is not a SSD manufacturer. They write industry-wide bug. Does that mean that more than one SSD manufacturer is affected (because they use partially the same firmware)? Further down they mention only Sandisk. Or is the industry-wide just their newspeak for saying any Sandisk of affected model, regardless whether installed in a Cisco box or somewhere else?
Cisco is not a SSD manufacturer. They write industry-wide bug. Does that mean that more than one SSD manufacturer is affected (because they use partially the same firmware)? Further down they mention only Sandisk. Or is the industry-wide just their newspeak for saying any Sandisk of affected model, regardless whether installed in a Cisco box or somewhere else?
The problem seems to be widely experienced. The Cisco report turned up in response to a post I'd made of the HN issue on the Fediverse: https://mastodon.infra.de/@galaxis/108622795822100862
None of which gained traction at the time:
Earlier quoted context omitted.
It seems more likely it was four drives (though dang and Mike both refer to "two" in the earlier thread). Both primary and failover servers had RAID arrays. I suspect RAID 10 (striped mirror), which would mean two drives would have to fail to take down a single server. Four drives of the same manufacturer spec and batch would do that.
This is what always worries me about our home server. It's running ZFS with multiple redundant drives but the supplier refused (when I explicitly asked) to supply it with disks known to be from different batches claiming that the odds of multiple failures close together were negligible. Obviously we have backups as well but the time and cost to restore a full server from online backups can be significant.
https://news.ycombinator.com/item?id=32031655
I've increasingly come to view systems operations / SRE as a risk management exercise, where the goal is to reduce the odds of a catastrophic failure. Total system outage is one level, unrecoverable total system outage is even worse.
Having multiple redundant backups / storage systems, in different locations, with different vendor hardware / stacks, all helps reduce risk of a single-factor outage. Though complexity risk is its own issue.
I’m moving all my storage to vellum with papyrus backups.
Earlier quoted context omitted.
It seems more likely it was four drives (though dang and Mike both refer to "two" in the earlier thread). Both primary and failover servers had RAID arrays. I suspect RAID 10 (striped mirror), which would mean two drives would have to fail to take down a single server. Four drives of the same manufacturer spec and batch would do that.
This is what always worries me about our home server. It's running ZFS with multiple redundant drives but the supplier refused (when I explicitly asked) to supply it with disks known to be from different batches claiming that the odds of multiple failures close together were negligible. Obviously we have backups as well but the time and cost to restore a full server from online backups can be significant.
Would companies be willing to contribute to the OpenSSD project? OCP (Open Compute Project) has shown that customer-operators can cooperate on open hardware designs, successfully influencing enterprise hardware supply chains. Commercial DPUs and SmartNICs were preceded by a decade of open hardware and research by the NetFPGA project ( https://netfpga.org ). Why not DiskFPGA? 2017 OpenSSD overview, based on Xilinx: ht…
Edit: this obviously ignores any troubles one would have sourcing the ICs (such as possible NDAs)
Would companies be willing to contribute to the OpenSSD project? OCP (Open Compute Project) has shown that customer-operators can cooperate on open hardware designs, successfully influencing enterprise hardware supply chains. Commercial DPUs and SmartNICs were preceded by a decade of open hardware and research by the NetFPGA project ( https://netfpga.org ). Why not DiskFPGA? 2017 OpenSSD overview, based on Xilinx: ht…
An open source SSD is also a lot more feasible than an open source hard drive. Even if you managed to get an open source HDD controller, you still need the precision mechanical parts that are impossible for the average person to make. With SSDs, however, it’s just a PCB with ICs. Edit: this obviously ignores any troubles one would have sourcing the ICs (such as possible NDAs)