Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

EPYC 7002 CPUs may hang after 1042 days of uptime

old.reddit.com

Post reply on HN
Print view View original

51–60 of 109 posts

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#51

Post by callamdelaney »

gchamonlive wrote: post #28
Waterluvian wrote: post #22

Earlier quoted context omitted.

It feels a little bit different. One creates uncertainty in all floating point results, given you don’t know when it happens. The other requires you to reboot maybe every ~3 years and you know exactly when it happens. I’m not saying we should tolerate a defect, but it doesn’t feel nearly as problematic.

It means anyone launching amd powered virtual machines on cloud providers can experience this now, at any point, and you don't know when it will happen, given this type of CPU could have been bought, booted or rebooted anytime in the past three years. Seems comparably problematic to me.

I think 30 seconds of downtime over 3 years probably isn't that much of an issue for anybody. Floating point calculations being wrong though.. that's a bigger problem.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#52

Post by gchamonlive »

Waterluvian wrote: post #32
gchamonlive wrote: post #28

Earlier quoted context omitted.

It means anyone launching amd powered virtual machines on cloud providers can experience this now, at any point, and you don't know when it will happen, given this type of CPU could have been bought, booted or rebooted anytime in the past three years. Seems comparably problematic to me.

I’d prefer a bug that crashes a program than one that quietly inserts wrong data and keeps going.

Me too, that is why I said the problems are comparable, not the same.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#53

Post by gchamonlive »

Sakos wrote: post #42
gchamonlive wrote: post #28

Earlier quoted context omitted.

It means anyone launching amd powered virtual machines on cloud providers can experience this now, at any point, and you don't know when it will happen, given this type of CPU could have been bought, booted or rebooted anytime in the past three years. Seems comparably problematic to me.

Google already does preemptive VMs where an instance can go down if it's needed elsewhere. It's something you can design your services to handle easily, if you aren't already doing so. Why wouldn't cloud providers be aware of how long a specific CPU has been up and plan around it? Also, do cloud providers generally never reboot their systems?

In AWS, if you keep a long running VM, it will keep running in the same server, even if degraded afaik. Even a reboot won't migrate to a new server. You have to shut it down and power it back up. This I learned back in 2019, so it could have changed but I doubt it.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#54

Post by bioemerl »

arp242 wrote: post #34
Waterluvian wrote: post #22

Earlier quoted context omitted.

It feels a little bit different. One creates uncertainty in all floating point results, given you don’t know when it happens. The other requires you to reboot maybe every ~3 years and you know exactly when it happens. I’m not saying we should tolerate a defect, but it doesn’t feel nearly as problematic.

It also has a fairly easy solution: disable the CC6 sleep state. The practical effects from that will most likely be minimal or non-existent for most users of these CPUs.

> disable the CC6 sleep state.

This is now the second time AMD has screwed up the C6 state. Ryzen first gen would hang daily for me when due to a similar bug.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#55

Post by gchamonlive »

viraptor wrote: post #40
gchamonlive wrote: post #28

Earlier quoted context omitted.

It means anyone launching amd powered virtual machines on cloud providers can experience this now, at any point, and you don't know when it will happen, given this type of CPU could have been bought, booted or rebooted anytime in the past three years. Seems comparably problematic to me.

> amd powered virtual machines on cloud providers Cloud providers are very unlikely to use sleep states. I mean, is possible... but I'd bet against it.

I have to admit I wasn't aware of sleep states being necessary for the problem do arise when I wrote the comment.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#56

Post by PragmaticPulp »

gchamonlive wrote: post #28
Waterluvian wrote: post #22

Earlier quoted context omitted.

It feels a little bit different. One creates uncertainty in all floating point results, given you don’t know when it happens. The other requires you to reboot maybe every ~3 years and you know exactly when it happens. I’m not saying we should tolerate a defect, but it doesn’t feel nearly as problematic.

It means anyone launching amd powered virtual machines on cloud providers can experience this now, at any point, and you don't know when it will happen, given this type of CPU could have been bought, booted or rebooted anytime in the past three years. Seems comparably problematic to me.

This depends on the C6 sleep state being enabled and the server not having been restarted for 3 years. It’s extremely unlikely that your cloud provider servers are going to meet this criteria and ignore this errata for a part they’ve bought thousands of.

So no, it’s not going to start randomly hitting people.

> Seems comparably problematic to me.

Not even close. The FDIV bug hit common operations that could be issued millions of times per second. This bug only applies to specific configurations that haven’t been rebooted for 3 years and has a clear workaround.

They’re not even close to comparable in impact and ability to work around. Literally many orders of magnitude different.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#57

Post by gchamonlive »

tpetry wrote: post #35
gchamonlive wrote: post #28

Earlier quoted context omitted.

It means anyone launching amd powered virtual machines on cloud providers can experience this now, at any point, and you don't know when it will happen, given this type of CPU could have been bought, booted or rebooted anytime in the past three years. Seems comparably problematic to me.

The cloud providers now know of this bug. They will live migrate you to a different machine or shutdown reboot. Only on-premise will have this issue.

They won't though. An EC2 stays in the same server even if its service is degraded, afaik.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#58

Post by sidewndr46 »

RobotToaster wrote: post #18

I feel like some of the comments here are missing the point. Yes it's only likely to effect a small number of users, so did the intel fdiv bug, both are defective products. Back then intel were pressured into a recall, today we seem too willing to put up with being sold broken stuff.

I'm way too young to remember it clearly but from what I was told it was nothing of the sort. Intel announced that they had identified a bug and would review on a case by case basis to see who was affected and would determine if you were worthy of getting a CPU that was fixed.

Again, this is secondhand but from people who worked directly in the industry at the time.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#59

Post by sidewndr46 »

bioemerl wrote: post #54
arp242 wrote: post #34

Earlier quoted context omitted.

It also has a fairly easy solution: disable the CC6 sleep state. The practical effects from that will most likely be minimal or non-existent for most users of these CPUs.

> disable the CC6 sleep state. This is now the second time AMD has screwed up the C6 state. Ryzen first gen would hang daily for me when due to a similar bug.

I don't understand the nature of the relationship between a motherboard manufacturer and AMD but when I got my MSI Tomahawk board for my Ryzen I really thought I was losing my mind. I would have USB devices stop working at the most random of times and some of them would continually cycle between connected and not connected.

A motherboard update from MSI applied something from AMD and that fixed the issue.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#60

Post by arbitrandomuser »

kuratkull wrote: post #12
dale_glass wrote: post #2

A machine staying up for almost 3 years is irresponsible in this day and age. Yeah, I remember people having uptime competitions on Slashdot and the like some decades back, but you only need to look at the ssh logs of a 5 minutes old machine to realize this is a terrible idea in modern times.

Are you perhaps a Windows user? In the Linux world updates don't necessarily require reboots.

On Arch Linux atleast any external hardware device not already loaded by the kernel will fail to load after a kernel update
Post reply on HN