Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ECC matters

realworldtech.com

Post reply on HN
Print view View original

361–370 of 567 posts

Re: ECC matters

#361

Post by Nebasuke »

jjeaff wrote: post #337
otterley wrote: post #86

About 1/3 of Google's machines and 8% of Google's DIMMs in their fleet suffer at least one correctible memory error per year: http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf

Which means, assuming google is running very large machines with lots of memory that one might expect a single correctable error once every 6-10 years on your average workstation of small server. That's generously assuming your workstation has 1/3 as much memory as the average google server.

Google does not use very large or even large machines for most of their fleet. You can quickly see in the paper this is for 1, 2, and 4 GB RAM machines (in 2006-2008).

Re: ECC matters

#362

Post by avianes »

wagslane wrote: post #354

It really does. I did a write-up recently on it as I was diving in and understanding the benefits: https://qvault.io/2020/09/17/very-basic-intro-to-elliptic-cu...

Be careful not to confuse ECC memory with ECC encryption.

ECC memory = memory with Error-Correcting Code

ECC encryption = Elliptic Curve Cryptography

Re: ECC matters

#363

Post by incrudible »

SideQuark wrote: post #233
jjeaff wrote: post #193

Earlier quoted context omitted.

Ya, I'm not buying that biyflips are a problem. Or maybe modern software can correct better for this? Because I use my desktop all day everyday running tons of software on 64 gb of ram and I don't get errors or crashes often enough to remember ever having one.

> I'm not buying that biyflips are a problem. Google and read up - it is a problem, has killed people, has thrown election results, and much more. It's such a common problem than bitsquatting is a real thing :) Want to do an experiment? Pick a bitsquatted domain for a common site, and see how often you get hits. https://en.wikipedia.org/wiki/Bitsquatting

Nobody denies that bitflips happen. On the whole, you fail to make a case that preventing bitflips is the solution to a problem. Bitsquatting is not a real problem, it's a curiosity.

As for the case of bitflips killing someone: Bitflips are not the root cause here. The root cause is that somebody engineered something life-critical that mistakenly assumed hardware can not fail. Bitflips are just one of many reasons for hardware failure.

Re: ECC matters

#364

Post by a1369209993 »

musingsole wrote: post #130
JoeAltmaier wrote: post #92

ECC works if done right. Accessing a memory location can fix bit-flips (ECC is a 'correcting' code). But systems that don't regularly visit every memory location, can accumulate risk. Those dark corners of RAM can eventually get double-bit errors and be uncorrectable. So an OS might 'wash' RAM during idle moments, reading every location in a round-robin manner to get ECC to kick in and auto-correct. Doesn't matter ho…

A double-bit error in many cases is fine. If the error is at least detectable at the time of a read, your protection worked. What's scary is a triple-flip event. Most of those will still look like corrupted data, but if it happens to flip into looking like a fixable, single-bit error, you're out of luck and won't even know it.

> Most of those will still look like corrupted data,

Not if you're using a typical 72-bit SECDED code[0].

You have two error indicators: a summary parity bit (even number of errors: 0,2,etc vs odd number of errors: 1,etc), and a error index: 0 for no errors, or the bitwise xor of the locations each bit error.

For a triple error at bits a,b, and c, you'll have summary parity of 1 (odd number of errors, assumed to be 1), and a error index of a^b^c, in the range 0..127, of which 0..71[1] (56.25%, a clear albeit not overwhelming majority) will correspond to legitimate single-bit errors.

0: https://en.wikipedia.org/wiki/Hamming_code#Hamming_codes_wit...

1: or 72 out of 128 anyway; the active bits might not all be assigned contiguous indexes starting from zero, but it doesn't change the probability and it's simpler to analyse if summary is bit 0 and index bit i is substrate bit 2^i.

Re: ECC matters

#365

Post by ksec »

nostrademons wrote: post #237

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

>I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory."

Did they ( Google ) or He ( Craig Silverstein ) ever officially admit it on record? I did a Google search and results that came up were all on HN. Did they at least make a few PR pieces saying that they are using ECC memory now because I dont see any with searching. Admitting they made a mistake without officially saying it?

I mean the whole world of Server or computer might not need ECC insanity was started entirely because of Google [1] [2] with news and articles published even in the early 00s [3]. And after that it has spread like wildfire and became a common accepted fact that even Google doesn't need ECC. Just like Apple were using custom ARM instruction to achieve their fast JS VM performance became a "fact". ( For the last time, no they didn't ). And proponents of ECC memory has been fighting this misinformation like mad for decades. To the point giving up and only rant about every now and then. [3]

[1] https://blog.codinghorror.com/building-a-computer-the-google...

[2] https://blog.codinghorror.com/to-ecc-or-not-to-ecc/

[3] https://danluu.com/why-ecc/

Re: ECC matters

#366

Post by lostlogin »

koolba wrote: post #127
phkahler wrote: post #117

Earlier quoted context omitted.

>> I've never seen anyone shop a desktop CPU by TDP, rather than by performance and price. Oh oh, me! Back in the day I bought a 65W CPU for a system that could handle a 90W. I wanted quiet and figured that would keep fan noise down at a modest performance penalty. It should also last longer, being the same design but running cooler. I ran that from 2005 until a few years ago (it still run fine but is in storage). Pl…

SFF?

The Intel Nuc and Mac mini are good examples of this - however the Nuc doesn’t have its psu inside, it’s a brick. Great for fixing failures, horrible in general as a built in psu is so much tidier.

Re: ECC matters

#367

Post by ksec »

colejohnson66 wrote: post #317
marcosdumay wrote: post #72

Earlier quoted context omitted.

Keep in mind that Intel lies about its TDP.

They don’t. They just measure it differently than AMD. Intel measures at base clock, but AMD measures at sustained max clock IIRC. It’s definitely deceptive, but it’s not a lie as long as Intel tells you (which they do).

It is a lie when they change the definition of TDP without telling you first and later redefined the word to different thing once they got caught.

May be we should use a new term for it, something like iTDP.

Re: ECC matters

#368

Post by bollu »

slumdev wrote: post #339
eevilspock wrote: post #327

Earlier quoted context omitted.

I'm surprised that the other replies don't grasp this. This is the proper level to do the quorum. Doing quorum at the computer level would require synchronizing parallel computers, and unless that synchronization were to happen for each low level instruction, then it would have to be written into the software to take a vote at critical points. This is going to be greatly detrimental both to throughput and software co…

I would highly recommend a graduate-level course in computer architecture for anyone who thinks ECC is a 1980s solution to a modern problem. There are a lot of seemingly high-level problems that are solved (ingeniously) in hardware with very simple, very low-level solutions.

Could you please link me to such a course that displays the hardware level solutions? I'm super interested!

Re: ECC matters

#369

Post by andrewaylett »

eevilspock wrote: post #327
slumdev wrote: post #291

Earlier quoted context omitted.

Error-correcting code (the "ECC" in ECC) is just a quorum at the bit level.

I'm surprised that the other replies don't grasp this. This is the proper level to do the quorum. Doing quorum at the computer level would require synchronizing parallel computers, and unless that synchronization were to happen for each low level instruction, then it would have to be written into the software to take a vote at critical points. This is going to be greatly detrimental both to throughput and software co…

https://en.wikipedia.org/wiki/NonStop_(server_computers)

My first employer out of Uni had an option for their primary product to use a NonStop for storage -- I think HP funded development, and I'm not sure we ever sold any licenses for it.

Re: ECC matters

#370

Post by slumdev »

bollu wrote: post #368
slumdev wrote: post #339

Earlier quoted context omitted.

I would highly recommend a graduate-level course in computer architecture for anyone who thinks ECC is a 1980s solution to a modern problem. There are a lot of seemingly high-level problems that are solved (ingeniously) in hardware with very simple, very low-level solutions.

Could you please link me to such a course that displays the hardware level solutions? I'm super interested!

https://www.udacity.com/course/high-performance-computer-arc...
Post reply on HN