Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bug 1950764: Work Around Crash on Intel Raptor Lake CPU

phabricator.services.mozilla.com

Post reply on HN
Print view View original

51–60 of 62 posts

Re: Bug 1950764: Work Around Crash on Intel Raptor Lake CPU

#51

Post by Polizeiposaune »

cesarb wrote: post #49
userbinator wrote: post #10

Earlier quoted context omitted.

Use of the "h" register slices (bits 8..15) by compilers is thankfully pretty rare That's unfortunate, because it's precisely why things like this will keep happening. Agner Fog's optimization guide says "Any use of the high 8-bit registers AH, BH, CH, DH should be avoided because it can cause false dependences and less efficient code." The sad vicious cycle of compilers not exercising the hardware, and then the hard…

> That's unfortunate, because it's precisely why things like this will keep happening. I have the opposite opinion. Its use being rare means CPU designers have less need to optimize for that rare case, and hardware optimizations are precisely where these kinds of issues tend to pop up. And high 8-bit registers are a x86-specific feature, other CPU families don't have it. So that special case being less optimized (or…

Yep. The "high" registers as an alias for bits 8-15 of certain registers are one of many warts in the architecture; they should have been purged from 32-bit and 64-bit code, and left to rot in 16-bit mode only.

Intel blew it when they let them continue to work in to 32-bit code on the 386, and then AMD blew it when they repeated the mistake when defining the 64-bit ISA.

Re: Bug 1950764: Work Around Crash on Intel Raptor Lake CPU

#52

Post by Numerlor »

anonymars wrote: post #45
Numerlor wrote: post #38

Earlier quoted context omitted.

Undervolting would definitely help, and is the actual fix. The current Intel fixes were mostly just for the symptoms, as the main issue is high voltage+power when pushing high clocks, but they can't actually fix that as it'd downgrade the advertised clocks the cpus were sold with

Sorry, but that understanding is dangerously incomplete. You're describing the first set of issues they uncovered, but there's also: "Microcode and BIOS code requesting elevated core voltages which can cause Vmin shift especially during periods of idle and/or light activity " (emphasis mine) https://community.intel.com/t5/Blogs/Tech-Innovation/Client/... Recall also that "Vmin shift" means "the minimum voltage the pr…

Not sure what's supposed to be wrong with that? The clock tree degrades at high voltage. Some theories I've seen were on the CPU requesting significantly higher voltages during alternating clocks when there's a short lull in load from e.g. a pipeline stall. Then there doesn't seem to be a good enough of a sensor net in the correct places for the CPU to react to this, so it just "burns" itself down gradually. Assuming these are true, actual fixes from intel would be relaxing boost clocks to ones that are universally safe and open themselves to a lawsuit from everyone that bought the high end SKUs, or do a new stepping which is extremely expensive for a done design.

When you degrade the CPU naturally needs higher voltages to be stable, until the point where it just breaks completely and no amount of voltage it help it. But if your CPU doesn't degrade because it hasn't been overdoing it on voltages then there'll be no issues for Vmin to shift.

As an anecdotal experience from someone I know that runs these in prod for game servers, limiting the CPU to 80°C and 1.4V-1.45V, 400A has been keeping them alive for years doing 24/7 loads. Maybe a bit lower on the voltage if one wants to be sure longer term, as they are fine with just mass RMAing these. There's also large amount of differences in the silicon quality between samples that can make one run cool and completely fine even at the old stock settings, and an another sample that'll have to pull say 1.5x the power for the same load and clocks having it degrade.

Re: Bug 1950764: Work Around Crash on Intel Raptor Lake CPU

#53

Post by db48x »

varispeed wrote: post #29
throwaway2037 wrote: post #28

Earlier quoted context omitted.

Why is this downvoted? (At the time of writing, the text is grey, so it has at least a few downvotes.) This is a good question. As others have noted below, yes, and sometimes you can see kernel logging on start-up when the microcode is loaded.

[flagged]

It’s not elitism, it’s just self defense. Every forum starts out with high–value high–signal low–noise conversations and gradually decays towards low–signal high–noise conversations as new people are brought in. The new people are, by definition, new. They don’t know much if anything yet so they cannot participate meaningfully in advanced topics. Naturally they ask questions in order to fill in the gaps in their knowledge. It is simply unfortunate that the effect is to increase the amount of noise in the forum as each new member asks the same questions over and over again. This leads to the most knowledgeable members of the forum dropping out, as the quality of the discussion drops below the point where it is worth their time.

See also “Eternal September”.

Re: Bug 1950764: Work Around Crash on Intel Raptor Lake CPU

#54

Post by anonymars »

Numerlor wrote: post #52
anonymars wrote: post #45

Earlier quoted context omitted.

Sorry, but that understanding is dangerously incomplete. You're describing the first set of issues they uncovered, but there's also: "Microcode and BIOS code requesting elevated core voltages which can cause Vmin shift especially during periods of idle and/or light activity " (emphasis mine) https://community.intel.com/t5/Blogs/Tech-Innovation/Client/... Recall also that "Vmin shift" means "the minimum voltage the pr…

Not sure what's supposed to be wrong with that? The clock tree degrades at high voltage. Some theories I've seen were on the CPU requesting significantly higher voltages during alternating clocks when there's a short lull in load from e.g. a pipeline stall. Then there doesn't seem to be a good enough of a sensor net in the correct places for the CPU to react to this, so it just "burns" itself down gradually. Assuming…

You're implying that if you don't run the CPU at high power and high heat it won't have problems, and that undervolting or underclocking will prevent damage. This is not correct: while that is helpful, Vmin degradation occurs during idle or light activity as well

Vmin will creep up, and the headroom for undervolting will degrade. It will affect the high clocks first (they demand the highest voltage), which is why dropping the max boost multiplier a step or two can also work around it (at the cost of basically downgrading it to a cheaper processor)

Re: Bug 1950764: Work Around Crash on Intel Raptor Lake CPU

#55

Post by anarazel »

Polizeiposaune wrote: post #4
mike_hock wrote: post #3

Uh ... working around this in each and every piece of software sounds like a non-starter? Intel should be on the hook to fix this.

Use of the "h" register slices (bits 8..15) by compilers is thankfully pretty rare -- otherwise this would have been noticed much sooner! Agner Fog's optimization guide says "Any use of the high 8-bit registers AH, BH, CH, DH should be avoided because it can cause false dependences and less efficient code."

> Use of the "h" register slices (bits 8..15) by compilers is thankfully pretty rare -- otherwise this would have been noticed much sooner!

It's actually pretty easy to get compilers to use those, you mainly need a bunch of narrow accesses to neighboring memory. The oodle post contains a godbolt link to pretty ordinary c code triggering this.

I'd guess that you also need some other conditions (multiple in flight stores, high boost speeds) to trigger this.

Re: Bug 1950764: Work Around Crash on Intel Raptor Lake CPU

#56

Post by Numerlor »

anonymars wrote: post #54
Numerlor wrote: post #52

Earlier quoted context omitted.

Not sure what's supposed to be wrong with that? The clock tree degrades at high voltage. Some theories I've seen were on the CPU requesting significantly higher voltages during alternating clocks when there's a short lull in load from e.g. a pipeline stall. Then there doesn't seem to be a good enough of a sensor net in the correct places for the CPU to react to this, so it just "burns" itself down gradually. Assuming…

You're implying that if you don't run the CPU at high power and high heat it won't have problems, and that undervolting or underclocking will prevent damage. This is not correct: while that is helpful, Vmin degradation occurs during idle or light activity as well Vmin will creep up, and the headroom for undervolting will degrade. It will affect the high clocks first (they demand the highest voltage), which is why dro…

Idle and light load is bad for degradation only because that's the most common scenario where the boosting algorith will actually go to the highest clocks. More loaded cores will have the CPU target lower clocks on all cores so that it actually can get the power for it and have the CPU be coolable, but if you're idle and then some task loads just a single core for a bit the CPU will boost it the highest it can. The voltage spikes from those boosts will cause local hotspots even if the CPU is cool overall

Re: Bug 1950764: Work Around Crash on Intel Raptor Lake CPU

#57

Post by eigenform »

whadawha wrote: post #7
Polizeiposaune wrote: post #2

Details of the errata from a comment in the diff: "Write both dist bytes as a single 2-byte store. This avoids the `movb %ch, [mem]` instruction pattern (store from high-byte register alias) that LLVM otherwise emits when dist arrives as a wide register. That pattern triggers the Intel Raptor Lake CPU errata, causing silent 2-byte stores that corrupt the adjacent `len` byte."

How did this get past validation at Intel? This is worse than https://en.wikipedia.org/wiki/Pentium_FDIV_bug

Well, the FDIV bug is simpler. AFAICT this whole thing is tied up in the relationship between DVFS, silicon aging, and being able to have guarantees about clock reliability. That seems hard to reason about when you give BIOS vendors and users the ability to do over-volting/over-clocking (and of course, it's worse when you are maybe relying on all this in certain situations to try and approach parity with other hardware vendors who are taping out machines on CMOS processes that are more mature than yours).

Re: Bug 1950764: Work Around Crash on Intel Raptor Lake CPU

#58

Post by [deleted] »

Numerlor wrote: post #56
anonymars wrote: post #54

Earlier quoted context omitted.

You're implying that if you don't run the CPU at high power and high heat it won't have problems, and that undervolting or underclocking will prevent damage. This is not correct: while that is helpful, Vmin degradation occurs during idle or light activity as well Vmin will creep up, and the headroom for undervolting will degrade. It will affect the high clocks first (they demand the highest voltage), which is why dro…

Idle and light load is bad for degradation only because that's the most common scenario where the boosting algorith will actually go to the highest clocks. More loaded cores will have the CPU target lower clocks on all cores so that it actually can get the power for it and have the CPU be coolable, but if you're idle and then some task loads just a single core for a bit the CPU will boost it the highest it can. The v…

[deleted]

Re: Bug 1950764: Work Around Crash on Intel Raptor Lake CPU

#59

Post by altairprime »

xxs wrote: post #34
altairprime wrote: post #18

Earlier quoted context omitted.

https://github.com/intel/intel-linux-processor-microcode-dat... $ echo 1 > /sys/devices/system/cpu/microcode/reload Hot-swappable, even. TIL!

of course, it's hot-swap material, the microcode is 1st deployed by the bios, then the OS can apply changes as well. Just that it's writable by $ (not #) feels awkward.

> Of course it’s hot-swap

Microcode updates like this didn’t exist when I first learned sysadmin, so I guess I don’t take such things for granted, sorry!

Re: Bug 1950764: Work Around Crash on Intel Raptor Lake CPU

#60

Post by [deleted] »

altairprime wrote: post #59
xxs wrote: post #34

Earlier quoted context omitted.

of course, it's hot-swap material, the microcode is 1st deployed by the bios, then the OS can apply changes as well. Just that it's writable by $ (not #) feels awkward.

> Of course it’s hot-swap Microcode updates like this didn’t exist when I first learned sysadmin, so I guess I don’t take such things for granted, sorry!

[deleted]
Post reply on HN