Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HN is up again

news.ycombinator.com

Post reply on HN
Print view View original

331–340 of 390 posts

Re: HN is up again

#331

Post by mkl »

kabdib wrote: post #309
mikiem wrote: post #307

Earlier quoted context omitted.

You are never going to guess how long the HN SSDs were in the servers... never ever... OK... I'll tell you: 4.5years. I am not even kidding.

Let me narrow my guess: They hit 4 years, 206 days and 16 hours . . . or 40,000 hours. And that they were sold by HP or Dell, and manufactured by SanDisk. Do I win a prize? (None of us win prizes on this one).

I wonder if it might be closer to 40,032 hours. The official Dell wording [1] is "after approximately 40,000 hours of usage". 2^57 nanoseconds is 40031.996687737745 hours. Not sure what's special about 57, but a power of 2 limit for a counter makes sense. That time might include some manufacturer testing too.

[1] https://www.reddit.com/r/sysadmin/comments/f5k95v/dell_emc_u...

Re: HN is up again

#332

Post by colinmhayes »

cheesewhizemacs wrote: post #213
joshstrange wrote: post #34

I don't think I had ever fully internalized how often I open this site throughout the day. Finish a task? HN. Got frustrated/stuck on a problem? HN break. Waiting for something to install/upload/compile/etc? HN. Needless to say I opened a new tab, typed "n", and hit enter countless times today before my brain caught up with my muscle memory.

> Needless to say I opened a new tab, typed "n", and hit enter countless times today before my brain caught up with my muscle memory. I do this too, and it's because this site is an addictive slot machine just like every other social networking site. I actually really hate this website, but I'm here almost every day, because I can't seem to break the habit. Neat. It's probably because I have a common impulse control…

I ended up just DNS blocking it on my work computer. Now I just spend a lot more time on my phone though :/

Re: HN is up again

#333

Post by boulos »

mkl wrote: post #331
kabdib wrote: post #309

Earlier quoted context omitted.

Let me narrow my guess: They hit 4 years, 206 days and 16 hours . . . or 40,000 hours. And that they were sold by HP or Dell, and manufactured by SanDisk. Do I win a prize? (None of us win prizes on this one).

I wonder if it might be closer to 40,032 hours. The official Dell wording [1] is "after approximately 40,000 hours of usage". 2^57 nanoseconds is 40031.996687737745 hours. Not sure what's special about 57, but a power of 2 limit for a counter makes sense. That time might include some manufacturer testing too. [1] https://www.reddit.com/r/sysadmin/comments/f5k95v/dell_emc_u...

See! People should register via mail for those important notifications! (Or alternatively do quarterly checks that your firmware is up to date).

Re: HN is up again

#334

Post by boulos »

mikiem wrote: post #323
dang wrote: post #322

Earlier quoted context omitted.

Wow. It's possible that you have nailed this. Edit: here's why I like this theory. I don't believe that the two disks had similar levels of wear, because the primary server would get more writes than the standby, and we switched between the two so rarely. The idea that they would have failed within hours of each other because of wear doesn't seem plausible. But the two servers were set up at the same time , and it's…

This morning, I googled for issues with the firmware and the model of SSD, I got nothing. But now I am searching for "40000 hours SSD" and a million relevant results. Of course, why would I search for 40000 hours. This thread is making me feel a lot less crazy.

I'm hoping that deep in your spam folder is a critical firmware update notice from Dell/EMC/HP/SanDisk from 2 years ago :).

Re: HN is up again

#335

Post by dang »

Johnny555 wrote: post #82

Will you be posting a postmortem? Not that I deserve or expect one from a free service, but because I enjoy reading postmortems from failures where both the primary and backup systems failed, I like to see what holes I might have in my own failover setup.

The postmortem is sort of dissolved into the bloodstream of these threads:

Tell HN: HN Moved from M5 to AWS - https://news.ycombinator.com/item?id=32030400 - July 2022 (116 comments)

Ask HN: What'd you do while HN was down? - https://news.ycombinator.com/item?id=32026639 - July 2022 (218 comments)

HN is up again - https://news.ycombinator.com/item?id=32026571 - July 2022 (314 comments)

I'd particularly look here: https://news.ycombinator.com/item?id=32026606 and here: https://news.ycombinator.com/item?id=32031025.

If you scroll through my comments from today via https://news.ycombinator.com/comments?id=dang&next=32039936, there are additional details. (Sorry for recommending my own comments.)

If you (or anyone) skim through that stuff and have a question that isn't answered there, I'd be happy to take a crack at it.

Re: HN is up again

#337

Post by chinathrow »

mikiem wrote: post #307
kabdib wrote: post #231

Earlier quoted context omitted.

I once had a small fleet of SSDs fail because they had some uptime counters that overflowed after 4.5 years, and that somehow persistently wrecked some internal data structures. It turned them into little, unrecoverable bricks. It was not awesome seeing a bunch of servers go dark in just about the order we had originally powered them on. Not a fun day at all.

You are never going to guess how long the HN SSDs were in the servers... never ever... OK... I'll tell you: 4.5years. I am not even kidding.

How many other customers will/have hit this?

Re: HN is up again

#338

Post by perilunar »

davedunkin wrote: post #66
sillysaurusx wrote: post #20

HN was down because the failover server also failed: https://twitter.com/HNStatus/status/1545409429113229312 Double disk failure is improbable but not impossible. The most impressive thing is that there seems to be no dataloss, almost whatsoever. Whatever the backup system is, it seems rock solid.

> Double disk failure is improbable but not impossible. It's not even improbable if the disks are the same kind purchased at the same time.

There's a principle in aviation of staggering engine maintenance on multiple-engined airplanes to avoid maintenance-induced errors leading to complete power loss.

e.g. Simultaneous Engine Maintenance Increases Operating Risks, Aviation Mechanics Bulletin, September–October 1999 https://flightsafety.org/amb/amb_sept_oct99.pdf

Re: HN is up again

#339

Post by tempestn »

dang wrote: post #322
kabdib wrote: post #309

Earlier quoted context omitted.

Let me narrow my guess: They hit 4 years, 206 days and 16 hours . . . or 40,000 hours. And that they were sold by HP or Dell, and manufactured by SanDisk. Do I win a prize? (None of us win prizes on this one).

Wow. It's possible that you have nailed this. Edit: here's why I like this theory. I don't believe that the two disks had similar levels of wear, because the primary server would get more writes than the standby, and we switched between the two so rarely. The idea that they would have failed within hours of each other because of wear doesn't seem plausible. But the two servers were set up at the same time , and it's…

This kind of thing is why I love Hacker News. Someone runs into a strange technical situation, and someone else happens to share their own obscure, related anecdote, which just happens to precisely solve the mystery. Really cool to see it benefit HN itself this time.

Re: HN is up again

#340

Post by sofixa »

spiffytech wrote: post #119
davedunkin wrote: post #66

Earlier quoted context omitted.

> Double disk failure is improbable but not impossible. It's not even improbable if the disks are the same kind purchased at the same time.

Yep: if you buy a pair disks together, there's a fair chance they'll both be from the same manufacturing batch, which correlates with disk defects.

That's why serious SAN vendors take care to provide you a mix of disks (e.g. on a brand new NetApp you can see that disks are of 2-3 different types, and with quite different serial numbers).
Post reply on HN