Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HN is up again

news.ycombinator.com

Post reply on HN
Print view View original

321–330 of 390 posts

Re: HN is up again

#321

Post by digitallyfree »

dang wrote: post #313
digitallyfree wrote: post #110

Earlier quoted context omitted.

By second disk failure do they mean that the disks on both the primary and fallback servers failed? Or do they mean that two disks (of a RAID1 or similar setup) in the fallback server failed? The latter is understandable, the former would be quite a surprise for such a popular site. That means that the machines have no disk redundancy and the server is going down immediately on disk failure. The fallback server would…

The disks on both the primary and fallback servers definitely failed. Each was in a RAID setup, but those failed too in both cases.

Ouch! I'm assuming the disks were from the same batch and installed at the same time, but having at least four fail like that is just crazy unlucky.

Re: HN is up again

#322

Post by dang »

kabdib wrote: post #309
mikiem wrote: post #307

Earlier quoted context omitted.

You are never going to guess how long the HN SSDs were in the servers... never ever... OK... I'll tell you: 4.5years. I am not even kidding.

Let me narrow my guess: They hit 4 years, 206 days and 16 hours . . . or 40,000 hours. And that they were sold by HP or Dell, and manufactured by SanDisk. Do I win a prize? (None of us win prizes on this one).

Wow. It's possible that you have nailed this.

Edit: here's why I like this theory. I don't believe that the two disks had similar levels of wear, because the primary server would get more writes than the standby, and we switched between the two so rarely. The idea that they would have failed within hours of each other because of wear doesn't seem plausible.

But the two servers were set up at the same time, and it's possible that the two SSDs had been manufactured around the same time (same make and model). The idea that they hit the 40,000 hour mark within a few hours of each other seems entirely plausible.

Mike of M5 (mikiem in this thread) told us today that it "smelled like a timing issue" to him, and that is squarely in this territory.

Re: HN is up again

#323

Post by mikiem »

dang wrote: post #322
kabdib wrote: post #309

Earlier quoted context omitted.

Let me narrow my guess: They hit 4 years, 206 days and 16 hours . . . or 40,000 hours. And that they were sold by HP or Dell, and manufactured by SanDisk. Do I win a prize? (None of us win prizes on this one).

Wow. It's possible that you have nailed this. Edit: here's why I like this theory. I don't believe that the two disks had similar levels of wear, because the primary server would get more writes than the standby, and we switched between the two so rarely. The idea that they would have failed within hours of each other because of wear doesn't seem plausible. But the two servers were set up at the same time , and it's…

This morning, I googled for issues with the firmware and the model of SSD, I got nothing. But now I am searching for "40000 hours SSD" and a million relevant results. Of course, why would I search for 40000 hours.

This thread is making me feel a lot less crazy.

Re: HN is up again

#324

Post by Amfy »

kabdib wrote: post #309
mikiem wrote: post #307

Earlier quoted context omitted.

You are never going to guess how long the HN SSDs were in the servers... never ever... OK... I'll tell you: 4.5years. I am not even kidding.

Let me narrow my guess: They hit 4 years, 206 days and 16 hours . . . or 40,000 hours. And that they were sold by HP or Dell, and manufactured by SanDisk. Do I win a prize? (None of us win prizes on this one).

is this leased to HN as dedicated/baremetal servers or colocation aka HN owns the hardware?

Re: HN is up again

#325

Post by iratewizard »

butterNaN wrote: post #211
christophilus wrote: post #122

Earlier quoted context omitted.

Yep. Same. It says a lot about the quality of HN, I think. Also, I can't remember the last time it was down. For a while, I thought there must be something wrong with my internet connection or DNS config or something.

Habitual usage doesn't necessarily correlate to quality, I'd say. People who use Facebook/Twitter also have this sort of muscle reflex developed over time. That said, HN does have quality content and the signal/noise is way better than sites designed specifically to keep you addicted.

I think they meant quality from a hardware and ops perspective.

Re: HN is up again

#326

Post by dang »

digitallyfree wrote: post #321
dang wrote: post #313

Earlier quoted context omitted.

The disks on both the primary and fallback servers definitely failed. Each was in a RAID setup, but those failed too in both cases.

Ouch! I'm assuming the disks were from the same batch and installed at the same time, but having at least four fail like that is just crazy unlucky.

Veteran programmer and HN user kabdib has proposed a striking theory that could explain everything: https://news.ycombinator.com/item?id=32028511.

Re: HN is up again

#327

Post by dang »

Amfy wrote: post #324
kabdib wrote: post #309

Earlier quoted context omitted.

Let me narrow my guess: They hit 4 years, 206 days and 16 hours . . . or 40,000 hours. And that they were sold by HP or Dell, and manufactured by SanDisk. Do I win a prize? (None of us win prizes on this one).

is this leased to HN as dedicated/baremetal servers or colocation aka HN owns the hardware?

The former.

Re: HN is up again

#330

Post by haswell »

rco8786 wrote: post #54
haswell wrote: post #35

This downtime made me realize (again) how much I appreciate the kind of interesting topics that show up here, the depth of discussion, and a general attitude of good faith that (most) engage with here. I realized how little of this I find elsewhere in my life - whether through Reddit or even my IRL friend circles. This realization saddens me - I feel like I shouldn’t have to rely on HN so much to scratch this particu…

> This realization saddens me - I feel like I shouldn’t have to rely on HN so much to scratch this particular itch. > Perhaps I need to get out more. Another way to look at it is that you have a particular set of interests and HN is the online outlet that serves those interests. There's nothing wrong with that, at all and you don't need to have multiple sources for it. No different than someone who likes to ride bike…

I think that’s fair. Aside from the topics though I think the key is “meaningful discourse”.

I think I’m just realizing I need to find that in more places, regardless of topic focus.

Post reply on HN