Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

EPYC 7002 CPUs may hang after 1042 days of uptime

old.reddit.com

Post reply on HN
Print view View original

61–70 of 109 posts

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#61

Post by PragmaticPulp »

RobotToaster wrote: post #18

I feel like some of the comments here are missing the point. Yes it's only likely to effect a small number of users, so did the intel fdiv bug, both are defective products. Back then intel were pressured into a recall, today we seem too willing to put up with being sold broken stuff.

> Back then intel were pressured into a recall, today we seem too willing to put up with being sold broken stuff.

This bug only applies to servers that haven’t been rebooted for 3 years and have the CC6 sleep state enabled. It can be worked around by disabling CC6 sleep state or rebooting once every 3 years.

If you think operators of these servers can’t be bothered to update and reboot their machines once in 3 years or change a single BIOS setting, what makes you think they’d be interested in tearing down their servers, physically replacing the CPU, and reassembling all of them with the associated downtime and inevitable accidental damage to some units? Nothing about that makes sense from a business perspective.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#62

Post by gchamonlive »

PragmaticPulp wrote: post #56
gchamonlive wrote: post #28

Earlier quoted context omitted.

It means anyone launching amd powered virtual machines on cloud providers can experience this now, at any point, and you don't know when it will happen, given this type of CPU could have been bought, booted or rebooted anytime in the past three years. Seems comparably problematic to me.

This depends on the C6 sleep state being enabled and the server not having been restarted for 3 years. It’s extremely unlikely that your cloud provider servers are going to meet this criteria and ignore this errata for a part they’ve bought thousands of. So no, it’s not going to start randomly hitting people. > Seems comparably problematic to me. Not even close. The FDIV bug hit common operations that could be issued…

Thanks for explaining the issue more! I must say I wasn't too familiar with Intel's issue when I wrote the comment.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#63

Post by numpad0 »

callamdelaney wrote: post #51
gchamonlive wrote: post #28

Earlier quoted context omitted.

It means anyone launching amd powered virtual machines on cloud providers can experience this now, at any point, and you don't know when it will happen, given this type of CPU could have been bought, booted or rebooted anytime in the past three years. Seems comparably problematic to me.

I think 30 seconds of downtime over 3 years probably isn't that much of an issue for anybody. Floating point calculations being wrong though.. that's a bigger problem.

Server hardware routinely take longer than 30 seconds to boot up, sometimes just to wait for power to stabilize just in case it matters, sometimes to do a staggered spinup of HDD to avoid current spike overloading something(it sounds cool!)

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#64

Post by ly3xqhl8g9 »

dale_glass wrote: post #2

A machine staying up for almost 3 years is irresponsible in this day and age. Yeah, I remember people having uptime competitions on Slashdot and the like some decades back, but you only need to look at the ssh logs of a 5 minutes old machine to realize this is a terrible idea in modern times.

3 years is irresponsible? To quote Logan Roy, you, software developers, "are not serious people" [1]. Just out of curiosity looked for a list of longest running electrical devices [2]:

    1840 - The Oxford Electric Bell
    1871 – Souter Lighthouse in South Shields, UK
    1896 – The Isle of Man’s Manx Electric Railway
    1902 – The Centennial Bulb
Apparently, "The Centennial Bulb has seen just two interruptions: for a week in 1937 when the Firehouse was refurbished, and in May 2013 when it was off for nine and a half hours due to a failed power supply."

[1] https://www.youtube.com/watch?v=LZTaXjt2Ggk

[2] https://www.drax.com/electrification/4-of-the-longest-runnin...

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#65

Post by gsich »

kuratkull wrote: post #12
dale_glass wrote: post #2

A machine staying up for almost 3 years is irresponsible in this day and age. Yeah, I remember people having uptime competitions on Slashdot and the like some decades back, but you only need to look at the ssh logs of a 5 minutes old machine to realize this is a terrible idea in modern times.

Are you perhaps a Windows user? In the Linux world updates don't necessarily require reboots.

They do. Kernel and libs require it, unless you want to be unsure if your system is still reboot-safe

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#67

Post by edude03 »

dale_glass wrote: post #10
hardware2win wrote: post #7

Earlier quoted context omitted.

I dont understand opinons like this Just because it would be dangerous for your nodejs web_app.exe running on ubuntu behind apache fully exposed on the internet then there are billion other ways to use computers, like even air gapped systems. So, dont try to justify obvious flaw

I mean, hardware is cheap enough that any server of importance should be individually disposable. Yeah, you can do stuff to maximize uptime but if it needs to stay up that badly you have to consider the case of the hardware needing to be turned off at some point. > So, dont try to justify obvious flaw I'm not, it's a bug and should be fixed. But I think if anything is powered for 3 years straight it's a bit concernin…

As an additional data point -

I have ~1000 7002 cores in my home DC (8 dual socket R7525s with 48-64 cores each) that run kubernetes but are connected to a battery backup and use kexec to perform upgrades. So, while I am very bought into the cattle not pets philosophy, it's rare that any of these machines need to be turned off and I could see them being on for three years continuously without problem otherwise.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#68

Post by atmavatar »

RobotToaster wrote: post #18

I feel like some of the comments here are missing the point. Yes it's only likely to effect a small number of users, so did the intel fdiv bug, both are defective products. Back then intel were pressured into a recall, today we seem too willing to put up with being sold broken stuff.

Don't divide, Intel inside!

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#69

Post by jeroenhd »

RobotToaster wrote: post #18

I feel like some of the comments here are missing the point. Yes it's only likely to effect a small number of users, so did the intel fdiv bug, both are defective products. Back then intel were pressured into a recall, today we seem too willing to put up with being sold broken stuff.

These types of bugs have been in hardware forever. Nobody is going to replace hundreds of EPYC servers even if they could get a free replacement from AMD.

There are definitely cases where hardware should be exchanged with fixed chips, particularly the small business/consumer/hobbyist range where exchanging CPUs is worth the time and effort. The RDRAND problem with Ryzen chips was much worse because it actually happened all the time and there is still no microcode fix available for some motherboards (though AMD already makes the fix available so it's more of an issue about a lack of motherboard support than broken hardware).

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#70

Post by cesarb »

dale_glass wrote: post #2

A machine staying up for almost 3 years is irresponsible in this day and age. Yeah, I remember people having uptime competitions on Slashdot and the like some decades back, but you only need to look at the ssh logs of a 5 minutes old machine to realize this is a terrible idea in modern times.

> A machine staying up for almost 3 years is irresponsible in this day and age. [...] but you only need to look at the ssh logs of a 5 minutes old machine to realize this is a terrible idea in modern times.

You don't need to reboot a machine to update ssh.

You only need to reboot the machine to update the kernel; for everything else, you just have to restart the corresponding user-space processes (and even PID1 can re-exec itself). Most kernel vulnerabilities are not remotely exploitable, so as long as you can trust your user-space processes (and keep them updated), it should be safe enough.

Post reply on HN