Live data from Hacker News

HN is up again

news.ycombinator.com

151–160 of 390 posts

Re: HN is up again

#152

Earlier quoted context omitted.

Yep: if you buy a pair disks together, there's a fair chance they'll both be from the same manufacturing batch, which correlates with disk defects.

This makes total sense but I've never heard of it. Is there any literature or writing about this phenomenon? I guess proper redundancy is having different brands of equipment also in some cases.

I hadn't heard of it either until disks in our storage cluster at work started failing faster than the cluster could rebuild in an event our ops team named SATApocalypse. It was a perfect storm of cascading failures.

https://web.archive.org/web/20220330032426/https://ops.faith...

Re: HN is up again

#155

I don't think I had ever fully internalized how often I open this site throughout the day. Finish a task? HN. Got frustrated/stuck on a problem? HN break. Waiting for something to install/upload/compile/etc? HN. Needless to say I opened a new tab, typed "n", and hit enter countless times today before my brain caught up with my muscle memory.

[deleted]

Re: HN is up again

#157

Earlier quoted context omitted.

> Double disk failure is improbable but not impossible. It's not even improbable if the disks are the same kind purchased at the same time.

Yep: if you buy a pair disks together, there's a fair chance they'll both be from the same manufacturing batch, which correlates with disk defects.

[deleted]

Re: HN is up again

#158

Earlier quoted context omitted.

Yep. Same. It says a lot about the quality of HN, I think. Also, I can't remember the last time it was down. For a while, I thought there must be something wrong with my internet connection or DNS config or something.

Might have tried to curl it from a production system just to make sure it wasn't my internet.

I understood this reference.

Re: HN is up again

#159

HN was down because the failover server also failed: https://twitter.com/HNStatus/status/1545409429113229312 Double disk failure is improbable but not impossible. The most impressive thing is that there seems to be no dataloss, almost whatsoever. Whatever the backup system is, it seems rock solid.

I'm extremely curious about the makes & models of the failed hardware...

Re: HN is up again

#160
post #143

Earlier quoted context omitted.

Yep: if you buy a pair disks together, there's a fair chance they'll both be from the same manufacturing batch, which correlates with disk defects.

Yeah just coming here to say this. Multiple disk failures are pretty probable. I've had batches of both disks and SSDs with sequential serial numbers, subjected to the same workloads, all fail within the same ~24 hour periods.

Seems like it was only a few days ago that there was a comment from a former Dropbox engineer here pointing out that a lot of disk drives they bought when they stood up their own datacenter had been found to all have a common flaw involving tiny metal slivers.
Post reply on HN