EPYC 7002 CPUs may hang after 1042 days of uptime • Hacker News Board

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#101

Post by magicalhippo » Thu, Jun 01, 2023, 10:20 PM UTC

justinclift wrote: post #72

dale_glass wrote: post #16

Earlier quoted context omitted.

Actually as of late, Linux has been moving towards rebooting for update. Yeah, you technically can replace on-disk files while services are running. In practice this can cause trouble if an application wants to read an updated file at the wrong time, and library dependencies can require restarting a lot of stuff. For ages people would install an update containing a security fix in glibc or libz or something, and keep…

> Actually as of late, Linux has been moving ... That's a pretty broad generalisation. Which distro's are you meaning?

KDE Neon has done this. Before I had to reboot anyway because usually the desktop was full of random crashes if I updated without rebooting.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#102

Post by touisteur » Fri, Jun 02, 2023, 1:14 AM UTC

paulryanrogers wrote: post #96

touisteur wrote: post #94

Earlier quoted context omitted.

I can already see the pain of the myriads of compliance (to all energy reduction directives, at least in EU) people getting strangely obtuse notes from their sw/hw/platform teams, saying in essence, errrrr we need to amend our already thick justification folder, to disable a specific sleep state. I feel a migraine (or a kind of sketch) coming. 'oh and BTW we're field upgrading the whole fleet'. I guess fighting tooth…

Would this qualify as more CPU errata?

There's errata and errata...

As a systems seller you get most of the markup but also most of the responsibility, so handwaving 'sorry AMD fucked up' won't do it. You know have an installed base that might crash every 1024 days, which for unattended systems is long but not that long. Worse if you have hardware redundancy, there's still a chance they all booted around the same time so will crash around the same time.

Customers will be proactive and follow the intelligent periodic reboot schedule you propose for a time (see the 787 overflow bugs stories), while asking for a fix. The fix needs to still be OK with all the specs you sold. If one of these specs depends on sleep states, you'll have to find a solution around it and deploy it fleetwide. If a microcode update fixes it, yay. If the problem can't be winked away with a software patch, now the blast radius is bigger and you're still supposed to do as much as possible to use the least energy possible in most idle states...

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#103

Post by justinclift » Fri, Jun 02, 2023, 5:33 AM UTC

magicalhippo wrote: post #101

justinclift wrote: post #72

Earlier quoted context omitted.

> Actually as of late, Linux has been moving ... That's a pretty broad generalisation. Which distro's are you meaning?

KDE Neon has done this. Before I had to reboot anyway because usually the desktop was full of random crashes if I updated without rebooting.

Ahhh. As a first thought, that sounds like you could have restarted your desktop (eg logout -> login) without needing a reboot.

On a related topic, Ubuntu has an optional package that can be enabled to automatically restart the various systemd components that need it after their dependencies have been upgraded. From memory, that's specifically so people don't have to reboot unless it's really needed.

I don't remember the name of the package off hand though, but someone else here might... :)

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#104

Post by eqvinox » Fri, Jun 02, 2023, 11:50 AM UTC

MSR-poking Tool for Zen1 Ryzen CPUs to disable C6: https://github.com/r4m0n/ZenStates-Linux/blob/master/zenstat...

Not sure if this is applicable to EPYC CPUs, probably not. But I would expect that it's possible to disable C6 in some similar way on EPYC CPUs without rebooting the system. (If you are actually at risk of running into this issue, you likely don't want to reboot the system…)

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#105

Post by SpaghettiCthulu » Fri, Jun 02, 2023, 2:55 PM UTC

tedunangst wrote: post #95

The good news is now I know why my server crashed last month, and it wasn't some other defect.

You've had a Ryzen 7000 series CPU running for nearly 3 years already?

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#106

Post by nh2 » Fri, Jun 02, 2023, 3:26 PM UTC

I filed a kernel bug 'System thrashes with "AMD-Vi: Completion-Wait loop timed out" after 247 days of uptime'

for AMD Ryzen 7 3700X

https://bugzilla.kernel.org/show_bug.cgi?id=217257

Might this be potentially related?

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#107

Post by magicalhippo » Sat, Jun 03, 2023, 1:32 AM UTC

justinclift wrote: post #103

magicalhippo wrote: post #101

Earlier quoted context omitted.

KDE Neon has done this. Before I had to reboot anyway because usually the desktop was full of random crashes if I updated without rebooting.

Ahhh. As a first thought, that sounds like you could have restarted your desktop (eg logout -> login) without needing a reboot. On a related topic, Ubuntu has an optional package that can be enabled to automatically restart the various systemd components that need it after their dependencies have been upgraded. From memory, that's specifically so people don't have to reboot unless it's really needed. I don't remember…

I tried the logout/login dance several times, but it didn't always work. A reboot doesn't take much longer anyway so...

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#108

Post by justinclift » Sat, Jun 03, 2023, 9:55 AM UTC

magicalhippo wrote: post #107

justinclift wrote: post #103

Earlier quoted context omitted.

Ahhh. As a first thought, that sounds like you could have restarted your desktop (eg logout -> login) without needing a reboot. On a related topic, Ubuntu has an optional package that can be enabled to automatically restart the various systemd components that need it after their dependencies have been upgraded. From memory, that's specifically so people don't have to reboot unless it's really needed. I don't remember…

I tried the logout/login dance several times, but it didn't always work. A reboot doesn't take much longer anyway so...

Yeah, makes sense for that situation. :)

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#109

Post by lUserAMD » Tue, Jun 06, 2023, 6:13 PM UTC

Neil44 wrote: post #5

It seems a C6 state is an individual core sleeping. The intersection of people who don't reboot for 3 years and people who have sleep states enabled must be pretty small. It's an interesting bug though!

EPYCs have many cores, and most applications (including those with long uptime requirements) use only a subset of the cores continuously. So it is totally normal for some of the cores to go to deep sleep C6 during phases of lower load. It will cause server operators headaches, when those cores don't come back eventually. Reboots help, disabling C6 in the (already running) OS also helps.

Please note, that we are not talking about a core sleeping for three years. We are talking about a core going to deep sleep, when the system has been up for three years or longer.