Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

EPYC 7002 CPUs may hang after 1042 days of uptime

old.reddit.com

Post reply on HN
Print view View original

81–90 of 109 posts

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#81

Post by bee_rider »

__alexs wrote: post #78
bee_rider wrote: post #38

Earlier quoted context omitted.

It probably depends on your workload, which is a bigger deal. The fdiv bug was pretty bad, but at least fixable in software (at some cost). Anyway, recall is the right decision in either case (unless there’s a good enough workaround).

If every CPU with an errata that needed software workarounds was recalled there would be no CPUs to use.

That’s why I said “unless there are good enough workarounds.” You buy a part with some performance/power consumption expectations.

It sounds like a workaround here could be to disable C6 sleep, so I guess we’ll see how much that violates those expectations. I guess they didn’t add the feature for no reason, though.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#82

Post by lanstin »

xp84 wrote: post #74
PragmaticPulp wrote: post #61

Earlier quoted context omitted.

> Back then intel were pressured into a recall, today we seem too willing to put up with being sold broken stuff. This bug only applies to servers that haven’t been rebooted for 3 years and have the CC6 sleep state enabled. It can be worked around by disabling CC6 sleep state or rebooting once every 3 years. If you think operators of these servers can’t be bothered to update and reboot their machines once in 3 years…

I’m picturing a long 50’ aisle filled with racks and a guy with a huge box marked “replacement CPUs” and a screwdriver. Good lord, can you imagine how long just a few of those would take in a data center?

The old every CPU is sacred idea lives on.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#83

Post by 0xr0kk3r »

Waterluvian wrote: post #22
RobotToaster wrote: post #18

I feel like some of the comments here are missing the point. Yes it's only likely to effect a small number of users, so did the intel fdiv bug, both are defective products. Back then intel were pressured into a recall, today we seem too willing to put up with being sold broken stuff.

It feels a little bit different. One creates uncertainty in all floating point results, given you don’t know when it happens. The other requires you to reboot maybe every ~3 years and you know exactly when it happens. I’m not saying we should tolerate a defect, but it doesn’t feel nearly as problematic.

To be fair, it was possible to tell what operations would be off in the FDIV bug and by how much. It was 100% deterministic. Problem was, checking all the operands in SW before performing the computation to make adjustments completely defeated the purpose of having an FPU.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#85

Post by ly3xqhl8g9 »

dale_glass wrote: post #79
ly3xqhl8g9 wrote: post #64

Earlier quoted context omitted.

3 years is irresponsible? To quote Logan Roy, you, software developers, "are not serious people" [1]. Just out of curiosity looked for a list of longest running electrical devices [2]: 1840 - The Oxford Electric Bell 1871 – Souter Lighthouse in South Shields, UK 1896 – The Isle of Man’s Manx Electric Railway 1902 – The Centennial Bulb Apparently, "The Centennial Bulb has seen just two interruptions: for a week in 193…

Those are completely trivial complexity-wise compared to a modern server, and many don't have a real function, and mostly are artificially maintained as a curiosity. I mean, the centennial bulb barely glows, that's why it still works. The hotter the filament gets the faster it evaporates, so a light bulb that barely makes any light can stay working forever.

Sure, was looking for electrical devices, a better example of what great engineering can achieve I suppose it's the Pons Fabricius [1], bridge built 2,085 years ago, still in use.

The problem is, if we can't expect software to run essentially forever, to update without 'restarts', and so forth, how are we ever going to achieve neural chip implants, artificial organs, synthetic agents mining ore in outer space, and so on? Software is not a gear mechanism, a rack and pinion, there is absolutely no reason to restart an 'operating system' or to ever lose state, however we became accustomed and we commit these sort of crimes daily, restarts and refreshes.

[1] https://en.wikipedia.org/wiki/Pons_Fabricius

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#86

Post by dathinab »

ly3xqhl8g9 wrote: post #64
dale_glass wrote: post #2

A machine staying up for almost 3 years is irresponsible in this day and age. Yeah, I remember people having uptime competitions on Slashdot and the like some decades back, but you only need to look at the ssh logs of a 5 minutes old machine to realize this is a terrible idea in modern times.

3 years is irresponsible? To quote Logan Roy, you, software developers, "are not serious people" [1]. Just out of curiosity looked for a list of longest running electrical devices [2]: 1840 - The Oxford Electric Bell 1871 – Souter Lighthouse in South Shields, UK 1896 – The Isle of Man’s Manx Electric Railway 1902 – The Centennial Bulb Apparently, "The Centennial Bulb has seen just two interruptions: for a week in 193…

yes 3 years without hardware reset of a component not designed for long term high reliably use is irresponsible (the are servers fir very high reliability, they are just WAY more expensive)

BUT this doesn't mean you need to have downtime, in the same way a train unit in a railway system going through maintenance doesn't mean your railway system has downtime.

Redundancy is a must have feature for reliable systems and that means you system must be able to cope with random hardware failure or rebooting a server unit.

And both planned and unplanned maintenance of components are important normal business which in a well desingned reliable system should not lead to downtime.

Similar testing failure cases is important and should be done.

so either you don't run a high reliably system (and likely don't run into this bug ever), or you run a proper reliable system (and it's not a big deal), or you run a badly desingned or operated system pretending to be high reliably but but really being that... which is irresponsible (if you are aware)

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#88

Post by kllrnohj »

bee_rider wrote: post #81
__alexs wrote: post #78

Earlier quoted context omitted.

If every CPU with an errata that needed software workarounds was recalled there would be no CPUs to use.

That’s why I said “unless there are good enough workarounds.” You buy a part with some performance/power consumption expectations. It sounds like a workaround here could be to disable C6 sleep, so I guess we’ll see how much that violates those expectations. I guess they didn’t add the feature for no reason, though.

The other workaround is to reboot at least once every 3 years, which surely most users are doing anyway to pick up on security patches & similar.

Exceptions definitely exist, but the workarounds are both pretty straightforward and you can pick whichever is less impactful.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#89

Post by imtringued »

bee_rider wrote: post #81
__alexs wrote: post #78

Earlier quoted context omitted.

If every CPU with an errata that needed software workarounds was recalled there would be no CPUs to use.

That’s why I said “unless there are good enough workarounds.” You buy a part with some performance/power consumption expectations. It sounds like a workaround here could be to disable C6 sleep, so I guess we’ll see how much that violates those expectations. I guess they didn’t add the feature for no reason, though.

It is a non-issue. If you need 3 years of permanent uptime, then what exactly do you need a deep sleep state for that is basically the same as turning the CPU off?

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#90

Post by dale_glass »

ly3xqhl8g9 wrote: post #85
dale_glass wrote: post #79

Earlier quoted context omitted.

Those are completely trivial complexity-wise compared to a modern server, and many don't have a real function, and mostly are artificially maintained as a curiosity. I mean, the centennial bulb barely glows, that's why it still works. The hotter the filament gets the faster it evaporates, so a light bulb that barely makes any light can stay working forever.

Sure, was looking for electrical devices, a better example of what great engineering can achieve I suppose it's the Pons Fabricius [1], bridge built 2,085 years ago, still in use. The problem is, if we can't expect software to run essentially forever, to update without 'restarts', and so forth, how are we ever going to achieve neural chip implants, artificial organs, synthetic agents mining ore in outer space, and so…

Don't get me wrong, I'm not saying it's not a problem. It should be fixed.

But if you need a single system to stay up for 3 years straight that's probably not good. There's too much going on in a modern high tech server for that to be a good idea. Everything has a CPU in it (including disks, video cards, network cards, etc). And any of that could make your system unusable by hitting some rare condition.

> The problem is, if we can't expect software to run essentially forever, to update without 'restarts', and so forth, how are we ever going to achieve neural chip implants, artificial organs, synthetic agents mining ore in outer space, and so on?

I would hope such things to be purpose-made and to be made in a way that the user can survive a reboot/firmware update. Eg, your neural implant should be built in such a way that it's not going to be life threatening if the battery runs out. The system has to be designed with that accounted for.

Maybe there's a secondary, minimal implementation acting as a backup and keeping critical functions working while the fully featured one is being updated. Hopefully everything is implemented in a failsafe way so that if it completely stops working you're not in a worse state than before you got it.

Any plan where there's a crucial component that must not stop even for a second isn't a very good plan.

Post reply on HN