Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ECC matters

realworldtech.com

Post reply on HN
Print view View original

531–540 of 567 posts

Re: ECC matters

#531

Post by exikyut »

rfoo wrote: post #458
saagarjha wrote: post #422

Earlier quoted context omitted.

There was an interesting challenge at DEF CON CTF a while back that tested this, actually. It turns out that it is possible to write x86 code that is 1-bit-flip tolerant–that is, a bit flip anywhere in its code can be detected and recovered from with the same output. Of course, finding the sequence took (or so I hear) something like 3600 cores running for a day to discover it ;)

Nit: not for a day, more like 8 hours, and that's because we were lazy and somebody said he "just happened" to have a cluster with unbalanced resources (mainly used for deep learning, but all GPUs occupied with quite a lot CPUs / RAMs left), so we decided to brute force the last 16 bits :) Also, the challenge host left useful state (which bit was flipped) in registers before running teams' code, without this I'm not…

This sounds really cool and interesting.

Was any code dumped anywhere?

I found this which corroborates everything you're saying but provides no further details: https://www.cspensky.info/slides/defcon_27_shortman.pdf

Re: ECC matters

#532

Post by MarkusWandel »

salmon wrote: post #512
MarkusWandel wrote: post #194

This is one justified Linus rant! My personal history includes data loss twice because of defective RAM, and many more RAMs discarded after the now obligatory overnight run of MemTest86+ (these were all secondhand RAMs - I would never buy a new one without a refund guarantee). My very first "PC" still had the ECC capability and I used it. My own now very dated rant on the subject: http://wandel.ca/homepage/memory_ran…

You bought used RAM DIMMs and were surprised that they failed?

Used computers that have RAM in them. But as I wrote, two of those computers were brand new with new RAMs in them.

Re: ECC matters

#533

Post by exikyut »

axiolite wrote: post #448
Jedd wrote: post #439

Earlier quoted context omitted.

At some point, whatever's watching the watchers is going to be vulnerable to bitflip and similar problems. Even with a triple-redundant quorum mechanism, slightly further up that stack you're going to have some bit of code running that processes the three returned results - if the memory that's sitting on gets corrupted, you're back where you started.

> At some point, whatever's watching the watchers is going to be vulnerable to bitflip One advantage of microkernels is that the "watcher" is so small that it could be run directly from ROM, instead of loaded into RAM. QNX has advocated that route for robotics and such in the past. Minix may not be the best example of the type. While it is a microkernel, it's real world reliability has been poor in the past. More mat…

> While it is a microkernel, it's real world reliability has been poor in the past.

Nitpick/clarification: it currently supervises the security posture, attestation state and overall health of several billion(?) Intel CPUs as the kernel used by the latest version of the Management Engine.

If ME is shut down completely apparently the CPU switches off within 20 minutes. Presumably this applies across the full uptime of the processor, and not just immediately after boot, and iff this is the case... percentage of Intel CPUs that randomly switch off === instability/unreliability of Minix in a tightly controlled industrial setting.

Re: ECC matters

#534

Post by buran77 »

behringer wrote: post #530
buran77 wrote: post #528

Earlier quoted context omitted.

Indeed, users don't know enough to care but more than that, for most users this is simply not a problem because they haven't been practically affected. Most may never experience a bit flip, and most of the flips won't be obvious anyway (e.g. no visible effect). So your hypothetical dialogue will sound more like a scam to a regular user. You're charging money to tackle a problem they don't have or see, only addressing…

My point is, they have been affected. I think most people have seen bitflips. >2009 Google's paper "DRAM Errors in the Wild: A Large-Scale Field Study" says that there can be up to 25000-75000 one-bit FIT per Mbit (failures in time per billion hours), which is equal to 1 - 5 bit errors per hour for 8GB of RAM after my calculations. Paper says the same: "mean correctable error rates of 2000–6000 per GB per year". http…

But behringer, regular users don't see bitflips, they see "a random error", a random reboot, a random green pixel in a photo. That's if they even consciously observed anything at all. You yourself said those could have a dozen other root causes. I for one have not actually observed a bitflip (so something that I can actually consciously detect as an issue) in a decade. No unexpected reboots, no unexpected file corruption, etc., nothing that made me say "weird, must be some memory corruption".

Don't get me wrong, I'm all for ECC memory, I practically showed the value of ECC after intentionally running the same multi-day Matlab job several times on a non-ECC machine with different results just to prove this very point. But you use your deeper knowledge to assume what someone with shallow knowledge and interest wants or needs, and that will almost always be off mark. Your car does not have a roll cage. Normal computers do not have ECC memory.

Re: ECC matters

#535

Post by projct »

buran77 wrote: post #534
behringer wrote: post #530

Earlier quoted context omitted.

My point is, they have been affected. I think most people have seen bitflips. >2009 Google's paper "DRAM Errors in the Wild: A Large-Scale Field Study" says that there can be up to 25000-75000 one-bit FIT per Mbit (failures in time per billion hours), which is equal to 1 - 5 bit errors per hour for 8GB of RAM after my calculations. Paper says the same: "mean correctable error rates of 2000–6000 per GB per year". http…

But behringer , regular users don't see bitflips , they see "a random error", a random reboot, a random green pixel in a photo. That's if they even consciously observed anything at all. You yourself said those could have a dozen other root causes. I for one have not actually observed a bitflip (so something that I can actually consciously detect as an issue) in a decade. No unexpected reboots, no unexpected file corr…

"Your car does not have a roll cage" No but the crash structures are one of the areas where we are absolutely trying to recreate the effect of a rollcage without the weight. The weight putting constraints on fuel efficiency is the big issue there.

For ECC, there's no reason to really expect that adding ECC to everything would really make things too expensive or slow long term. For a long time the reason why desktops did not have ECC was pretty much because intel wanted people who really need ECC to buy Xeons.

Re: ECC matters

#536

Post by supernovae »

ajnin wrote: post #309
supernovae wrote: post #79

Earlier quoted context omitted.

I've got nearly 30 years of experience and not once has non ECC memory lead to corruption. Maybe a crash, maybe a panic, maybe a kernel dump... But.. in all my time operating servers over 3 decades, it's always been bad drivers, bad code and problematic hardware that's caused most of my headaches. Have i seen ECC error correction in logs? yeah.. I don't advocate against it but, i've found for most people you design a…

> I've got nearly 30 years of experience and not once has non ECC memory lead to corruption How do you know?

Complex systems require constant probing. You always probe. If you stop probing to look for errors because you assumed ECC fixed it for you, then you're probably at more risk of corruption vs accepting the fact there will be an infinite risk of error and being defensive against all types.

Re: ECC matters

#537

Post by supernovae »

louwrentius wrote: post #279
supernovae wrote: post #79

Earlier quoted context omitted.

I've got nearly 30 years of experience and not once has non ECC memory lead to corruption. Maybe a crash, maybe a panic, maybe a kernel dump... But.. in all my time operating servers over 3 decades, it's always been bad drivers, bad code and problematic hardware that's caused most of my headaches. Have i seen ECC error correction in logs? yeah.. I don't advocate against it but, i've found for most people you design a…

The problem with memory corruption is not just crashes, those are the more benign outcomes. The real killer is data corruption. Houw would you even begin to know that data is corrupted until it is too late?

Corruption can happen for any number of reasons. ECC doesn't correct for human error, protocol error, coding error, logic error, type error so on and so forth.

Again, I don't advocate NOT using ECC, but i'd say in complex systems, never assume ECC alone is enough... and if ECC becomes your champion cause, how could you enforce it through every device that touches data, provides data, consumes data or injects data?

Re: ECC matters

#538

Post by saagarjha »

rfoo wrote: post #529
saagarjha wrote: post #490

Earlier quoted context omitted.

Sure, all's fair in a CTF. That story came to me through the mouths of at least a handful of people, who might have a bit of an incentive to exaggerate given that they hadn't quite been able to get to zero and might be a just a little sour :P The state was quite helpful, yes–for x86 it seems like a "clean slate" shellcode would be quite difficult, if impossible, to achieve as we saw. However, I am left wondering how…

Maybe being notoriously dense is not a bad thing? While those ModRM bytes popping up everywhere is annoying as f* (too easy to flip an instruction into a form with almost-guaranteed-to-be-invalid memory access), at least due to the density there won't be reserved bits. For example, in AArch64 if bit 28 and bit 27 is both zero the instruction will almost certainly be an invalid one (hitting unallocated area), and with…

Right, I was saying that the other ISAs would do wore because they aren't as dense and will hit something undefined much more readily. But the RISCs in general are much less likely to touch memory (only if you do a load/store from a register that isn't clean, maybe). From a glance, MIPS looks like it might work, since the opcode field seems to use all the bits and the remaining bits just encode reg/func/imm in various ways. The one caveat I see is that I think the top bit of opcode seems to encode memory accesses, so you may be forced to deal with at least one.

Re: ECC matters

#539

Post by saagarjha »

exikyut wrote: post #531
rfoo wrote: post #458

Earlier quoted context omitted.

Nit: not for a day, more like 8 hours, and that's because we were lazy and somebody said he "just happened" to have a cluster with unbalanced resources (mainly used for deep learning, but all GPUs occupied with quite a lot CPUs / RAMs left), so we decided to brute force the last 16 bits :) Also, the challenge host left useful state (which bit was flipped) in registers before running teams' code, without this I'm not…

This sounds really cool and interesting. Was any code dumped anywhere? I found this which corroborates everything you're saying but provides no further details: https://www.cspensky.info/slides/defcon_27_shortman.pdf

Oh, hey, it's Chad's slides!

Coverage of the finals is usually much less detailed, unfortunately, since the number of teams is much smaller and the challenges don't necessarily go up. However, https://oooverflow.io/dc-ctf-2020-quals/ has a couple more writeups linked from it; https://dttw.tech/posts/SJ40_7MNS#proof-by-exhaustion from PPP and http://www.secmem.org/blog/2019/08/19/Shellcoding-and-Bitfli... from SeoulPlusBadass.

Re: ECC matters

#540

Post by rcoveson »

shrewduser wrote: post #500
klodolph wrote: post #485

Earlier quoted context omitted.

Just because capitalism fails to provide something does not mean that communism would solve the problem, or that criticism of capitalism as a system is uncalled for. This is not some kind of binary capitalism/communism world we live in.

i'm just struggling to find the link between capitalism and ECC memory.

There is no link. Segmentation of ECC support was an Intel decision which other chip manufactures have copied to a greater or lesser extent. For what it's worth, ECC support is spotty in Chinese-designed chips as well.

Intel's decision could just as well have been made by a socialist government's regulatory body (in the form of "minimum requirements for consumer chip" legislation), or by a committee of decision makers in an employee-owned company. We may never know, because we only have 2-4 data points and they're all basically capitalist.

Honestly, I'm not sure it was intended to be a serious comment. It could just as easily have been "...failure of human cognition and Obama."

Post reply on HN