Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HN is up again

news.ycombinator.com

Post reply on HN
Print view View original

371–380 of 390 posts

Re: HN is up again

#371

Post by rbanffy »

kabdib wrote: post #231
davedunkin wrote: post #66

Earlier quoted context omitted.

> Double disk failure is improbable but not impossible. It's not even improbable if the disks are the same kind purchased at the same time.

I once had a small fleet of SSDs fail because they had some uptime counters that overflowed after 4.5 years, and that somehow persistently wrecked some internal data structures. It turned them into little, unrecoverable bricks. It was not awesome seeing a bunch of servers go dark in just about the order we had originally powered them on. Not a fun day at all.

I had a similar issue, but it was a single RAID-5 array and wear of some other manufacture defect. They were the same brand, model, and batch. When the first failed and the array got in recovery mode I ordered 3 replacements and upped the backup frequency. It was good that I did that because the two remaining drives died shortly after.

The lesson I learned is that the three replacements went to different arrays and we never again let drives from the same batch be part of the same array.

Re: HN is up again

#372

Post by exikyut »

qu1j0t3 wrote: post #361
chinathrow wrote: post #337

Earlier quoted context omitted.

How many other customers will/have hit this?

Every large DC will have hit it (Amazon, Facebook, Google, etc). But it's a shame that all their operational knowledge is kept secret.

I understand BackBlaze is more HDD rather than SSD, but perhaps they might have some level of awareness.

Re: HN is up again

#373

Post by gpshead »

dredmorbius wrote: post #366
dang wrote: post #353

Earlier quoted context omitted.

It's also an example of the dharma of /newest – the rising and falling away of stories that get no attention: HPE releases urgent fix to stop enterprise SSDs conking out at 40K hours - https://news.ycombinator.com/item?id=22706968 - March 2020 (0 comments) HPE SSD flaw will brick hardware after 40k hours - https://news.ycombinator.com/item?id=22697758 - March 2020 (0 comments) Some HP Enterprise SSD will brick after…

Popularity is a very poor relevance / truth heuristic.

I wanted to upvote this comment but that just feels wrong.

Re: HN is up again

#374

Post by dang »

yahelc wrote: post #369
dang wrote: post #353

Earlier quoted context omitted.

It's also an example of the dharma of /newest – the rising and falling away of stories that get no attention: HPE releases urgent fix to stop enterprise SSDs conking out at 40K hours - https://news.ycombinator.com/item?id=22706968 - March 2020 (0 comments) HPE SSD flaw will brick hardware after 40k hours - https://news.ycombinator.com/item?id=22697758 - March 2020 (0 comments) Some HP Enterprise SSD will brick after…

Easy to imagine why this didn’t capture peoples’ attention in late March 2020…

Yes, an enterprisey firmware update - all very boring until BLAM!

Re: HN is up again

#375

Post by mcsee »

kabdib wrote: post #231
davedunkin wrote: post #66

Earlier quoted context omitted.

> Double disk failure is improbable but not impossible. It's not even improbable if the disks are the same kind purchased at the same time.

I once had a small fleet of SSDs fail because they had some uptime counters that overflowed after 4.5 years, and that somehow persistently wrecked some internal data structures. It turned them into little, unrecoverable bricks. It was not awesome seeing a bunch of servers go dark in just about the order we had originally powered them on. Not a fun day at all.

No post body was provided.

Re: HN is up again

#377

Post by lupire »

yahelc wrote: post #369
dang wrote: post #353

Earlier quoted context omitted.

It's also an example of the dharma of /newest – the rising and falling away of stories that get no attention: HPE releases urgent fix to stop enterprise SSDs conking out at 40K hours - https://news.ycombinator.com/item?id=22706968 - March 2020 (0 comments) HPE SSD flaw will brick hardware after 40k hours - https://news.ycombinator.com/item?id=22697758 - March 2020 (0 comments) Some HP Enterprise SSD will brick after…

Easy to imagine why this didn’t capture peoples’ attention in late March 2020…

[dead]

Re: HN is up again

#378

Post by winternett »

dang wrote: post #353
tempestn wrote: post #339

Earlier quoted context omitted.

This kind of thing is why I love Hacker News. Someone runs into a strange technical situation, and someone else happens to share their own obscure, related anecdote, which just happens to precisely solve the mystery. Really cool to see it benefit HN itself this time.

It's also an example of the dharma of /newest – the rising and falling away of stories that get no attention: HPE releases urgent fix to stop enterprise SSDs conking out at 40K hours - https://news.ycombinator.com/item?id=22706968 - March 2020 (0 comments) HPE SSD flaw will brick hardware after 40k hours - https://news.ycombinator.com/item?id=22697758 - March 2020 (0 comments) Some HP Enterprise SSD will brick after…

Interesting how something that is so specifically and unexpectedly devastating, yet known for such a long time without any serious public awareness from companies involved, is referred to as a "bug".

It makes you lose data and need to purchase new hardware, where I come from, that's usually referred to as "planned" or "convenient" obsolescence.

Re: HN is up again

#379

Post by dreamcompiler »

gomijacogeo wrote: post #367
mkl wrote: post #331

Earlier quoted context omitted.

I wonder if it might be closer to 40,032 hours. The official Dell wording [1] is "after approximately 40,000 hours of usage". 2^57 nanoseconds is 40031.996687737745 hours. Not sure what's special about 57, but a power of 2 limit for a counter makes sense. That time might include some manufacturer testing too. [1] https://www.reddit.com/r/sysadmin/comments/f5k95v/dell_emc_u...

It might not be nanoseconds, but something that's a power of 2 number of nanoseconds going into an appropriately small container seems likely. For example, a 62.5MHz counter going into 53 bits breaks at the same limit. Why 53 bits? That's where things start to get weird with IEEE doubles - adding 1 no longer fits into the mantissa and the number doesn't change. So maybe someone was doing a bit of fp math to figure ou…

53 is indeed a magic value for IEEE doubles, but why would anybody count an inherently integer value with floating-point? That's a serious rookie mistake.

Of course there's no law that says SSD firmware writers can't be rookies.

Re: HN is up again

#380

Post by mulmen »

winternett wrote: post #378
dang wrote: post #353

Earlier quoted context omitted.

It's also an example of the dharma of /newest – the rising and falling away of stories that get no attention: HPE releases urgent fix to stop enterprise SSDs conking out at 40K hours - https://news.ycombinator.com/item?id=22706968 - March 2020 (0 comments) HPE SSD flaw will brick hardware after 40k hours - https://news.ycombinator.com/item?id=22697758 - March 2020 (0 comments) Some HP Enterprise SSD will brick after…

Interesting how something that is so specifically and unexpectedly devastating, yet known for such a long time without any serious public awareness from companies involved, is referred to as a "bug". It makes you lose data and need to purchase new hardware, where I come from, that's usually referred to as "planned" or "convenient" obsolescence.

The difference between planned and convenient seems to be intent. And in this context that difference very much matters. I wouldn’t conflate the two.
Post reply on HN