Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

EPYC 7002 CPUs may hang after 1042 days of uptime

old.reddit.com

Post reply on HN
Print view View original

41–50 of 109 posts

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#41

Post by bee_rider »

dale_glass wrote: post #25
PedroBatista wrote: post #21

Earlier quoted context omitted.

You live in your own World with other people. Please just keep in mind there are many other Worlds with other people and laws of the Universe. I don't know if you're young or don't know much about history but what you describe is a fairly recent way of looking at things, it's not the only one and I guarantee you it will become "out of fashion".

Yeah, the "cattle not pets" philosophy is fairly recent, but I don't see it changing any time soon. If anything we're going even more in that direction. And it makes a lot of sense because if uptime is that important, then no matter how fancy the hardware it can't do anything about disasters or losing internet connectivity.

The 7002 seems like it could be used in a workstation, where the “cattle vs pets” thing is less of a distinction, right? (I guess a workstation is sort of like a work dog in this analogy).

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#42

Post by Sakos »

gchamonlive wrote: post #28
Waterluvian wrote: post #22

Earlier quoted context omitted.

It feels a little bit different. One creates uncertainty in all floating point results, given you don’t know when it happens. The other requires you to reboot maybe every ~3 years and you know exactly when it happens. I’m not saying we should tolerate a defect, but it doesn’t feel nearly as problematic.

It means anyone launching amd powered virtual machines on cloud providers can experience this now, at any point, and you don't know when it will happen, given this type of CPU could have been bought, booted or rebooted anytime in the past three years. Seems comparably problematic to me.

Google already does preemptive VMs where an instance can go down if it's needed elsewhere. It's something you can design your services to handle easily, if you aren't already doing so.

Why wouldn't cloud providers be aware of how long a specific CPU has been up and plan around it? Also, do cloud providers generally never reboot their systems?

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#43

Post by nicolaslem »

icybox wrote: post #23
vegardx wrote: post #8

Earlier quoted context omitted.

I had a very similar issue with some AMD-based servers (bulldozer, I think) about ten years ago. There was a bug where Xen-based virtual machines could set a C-state on cores it was assigned, but for whatever reason it wasn't able to wake them up. It was fun trying to figure out what the heck was going on.

I have C-states already disabled because of old linux kernel bug where the kernel hang on Zen3 architecture. So not much to see here :)

Do you mean a bug in an old version of linux that is now fixed? Because I have been using Zen3 and Zen3+ on linux since their release and never had to mess with C-states.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#44

Post by ant6n »

shpx wrote: post #9
dale_glass wrote: post #2

A machine staying up for almost 3 years is irresponsible in this day and age. Yeah, I remember people having uptime competitions on Slashdot and the like some decades back, but you only need to look at the ssh logs of a 5 minutes old machine to realize this is a terrible idea in modern times.

1042 days ought to be enough for anybody

> 1042 days ought to be enough for anybody

"640K ought to be enough for anybody."

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#45

Post by viraptor »

dale_glass wrote: post #16
kuratkull wrote: post #12

Earlier quoted context omitted.

Are you perhaps a Windows user? In the Linux world updates don't necessarily require reboots.

Actually as of late, Linux has been moving towards rebooting for update. Yeah, you technically can replace on-disk files while services are running. In practice this can cause trouble if an application wants to read an updated file at the wrong time, and library dependencies can require restarting a lot of stuff. For ages people would install an update containing a security fix in glibc or libz or something, and keep…

While Fedora did move towards that, it's not the only way. A lot of systems which require high reliability are built to reload correctly.

At a generic system level, for example upgrading Nixos will pull new packages and put them next to the current ones, then reexec where possible. Nginx can replace its master process (SIGUSR2). Telephony software can often reexec and keep connecting open. Etc.

Outside of desktops it's not that uncommon to do seamless live reloads of the whole system.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#46

Post by dale_glass »

bee_rider wrote: post #41
dale_glass wrote: post #25

Earlier quoted context omitted.

Yeah, the "cattle not pets" philosophy is fairly recent, but I don't see it changing any time soon. If anything we're going even more in that direction. And it makes a lot of sense because if uptime is that important, then no matter how fancy the hardware it can't do anything about disasters or losing internet connectivity.

The 7002 seems like it could be used in a workstation, where the “cattle vs pets” thing is less of a distinction, right? (I guess a workstation is sort of like a work dog in this analogy).

That's the EPYC lineup, which is the server model. Support for terabytes of RAM, 128 PCIe lanes, that sort of thing.

I mean you could use it in a workstation, but unless you need 4 video cards locally it's probably overkill for most uses.

And a workstation should have no problem rebooting once in a while.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#47

Post by foobiekr »

tpetry wrote: post #35
gchamonlive wrote: post #28

Earlier quoted context omitted.

It means anyone launching amd powered virtual machines on cloud providers can experience this now, at any point, and you don't know when it will happen, given this type of CPU could have been bought, booted or rebooted anytime in the past three years. Seems comparably problematic to me.

The cloud providers now know of this bug. They will live migrate you to a different machine or shutdown reboot. Only on-premise will have this issue.

Which CSPs do live migration?

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#48

Post by nielsole »

foobiekr wrote: post #47
tpetry wrote: post #35

Earlier quoted context omitted.

The cloud providers now know of this bug. They will live migrate you to a different machine or shutdown reboot. Only on-premise will have this issue.

Which CSPs do live migration?

https://cloud.google.com/compute/docs/instances/live-migrati...

Not sure about others

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#49

Post by dspillett »

RobotToaster wrote: post #18

I feel like some of the comments here are missing the point. Yes it's only likely to effect a small number of users, so did the intel fdiv bug, both are defective products. Back then intel were pressured into a recall, today we seem too willing to put up with being sold broken stuff.

A key difference between then and now is how much easier it is to distribute software/firmware workarounds or fixes. From an end users perspective replacing the CPU might be seen as far easier than updating their software. A software fix would affect performance, so of course it isn't as simple as that, but this difference is part of the dynamic.

Also, as a direct user of the CPU, if the fdiv bug would impact you it would affect you often rather than once every three years which is the impact frequency of this fault.

Another matter that affected the fdiv bug is that the Pentium line was the first time a CPU had been aggressively marketed directly at the general public in quite the way it was. Prior to that only manufacturers and techies would have known about it and they were used to errata for hardware components. The public more generally had an impression that hardware (at least undamaged hardware) was reliable and only software had bugs, and the fdiv bug invalidated that view of reality causing a bit of a panic.

Re: EPYC 7002 CPUs may hang after 1042 days of uptime

#50

Post by p_l »

Sakos wrote: post #42
gchamonlive wrote: post #28

Earlier quoted context omitted.

It means anyone launching amd powered virtual machines on cloud providers can experience this now, at any point, and you don't know when it will happen, given this type of CPU could have been bought, booted or rebooted anytime in the past three years. Seems comparably problematic to me.

Google already does preemptive VMs where an instance can go down if it's needed elsewhere. It's something you can design your services to handle easily, if you aren't already doing so. Why wouldn't cloud providers be aware of how long a specific CPU has been up and plan around it? Also, do cloud providers generally never reboot their systems?

GCP also does live migration for standard instances.
Post reply on HN