Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ECC matters

realworldtech.com

Post reply on HN
Print view View original

351–360 of 567 posts

Re: ECC matters

#351

Post by [deleted] »

tyoma wrote: post #338
nostrademons wrote: post #237

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

Figure this is as good of a time as any to ask this: There are many various DRAMs in a server (say, for disk cache). Has Google or anyone who operates at a similar scale seen single bit errors in these components?

[deleted]

Re: ECC matters

#352

Post by ikiris »

adrian_b wrote: post #331
incrudible wrote: post #149

Earlier quoted context omitted.

When you say bitflips were "common" on thousands of physical machines, does that mean you observed thousands of bitflips? Otherwise, I would think that an unlikely event becoming 1000x more likely by sheer numbers would have warped your perception. I believe that hardware reliability is mostly irrelevant, because software reliability is already far worse. It doesn't matter whether a bitflip (unlikely) or some bug (li…

On a single computer with a large memory, e.g. 32 GB or more, the time between errors can be of a few months, if you are lucky to have good modules. Moreover, some of the errors will have no effect, if they happened to affect free memory. Nevertheless, anyone who uses the computer for anything else besides games or movie watching, will greatly benefit from having ECC memory, because that is the only way to learn when…

The good modules bit is important. I'm told by some colleagues that most of the bit flips are from alpha particles from the ram casings surprisingly enough.

Re: ECC matters

#353

Post by gh02t »

tyoma wrote: post #338
nostrademons wrote: post #237

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

Figure this is as good of a time as any to ask this: There are many various DRAMs in a server (say, for disk cache). Has Google or anyone who operates at a similar scale seen single bit errors in these components?

The supercomputing community has looked at some of the effect on different parts of the GPU.

https://ieeexplore.ieee.org/abstract/document/7056044

Re: ECC matters

#355

Post by marcosdumay »

colejohnson66 wrote: post #317
marcosdumay wrote: post #72

Earlier quoted context omitted.

Keep in mind that Intel lies about its TDP.

They don’t. They just measure it differently than AMD. Intel measures at base clock, but AMD measures at sustained max clock IIRC. It’s definitely deceptive, but it’s not a lie as long as Intel tells you (which they do).

Well, it's a power measurement that isn't total and can't be used for design... So, it's a lie.

If they gave it some other name, it would be only misleading. Calling it TDP is a lie.

Re: ECC matters

#356

Post by rovr138 »

sobriquet9 wrote: post #343
maria_weber23 wrote: post #274

Earlier quoted context omitted.

ECC memory can't eliminate the chances of these failures entirely. They can still happen. Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. So in theory the behavior of software under random bit flips is well... Random. You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. I…

If you use multiple computers doing the same calculation and then take the answer from the quorum, how do you ensure the computer that does the comparison is not affected by memory failures? Remember that all queries have to through it, so it has to be comparable in scale and power.

> how do you ensure the computer that does the comparison is not affected by memory failures?

You do the comparison on multiple nodes too. Get the calculations. Pass them to multiple nodes, validate again and if it all matches, you use it.

Re: ECC matters

#357

Post by linsomniac »

This reminds me of last year we ordered a new $14K server, it arrived and we ran it through our burn-in process which included running memtest86 on it, and it would, after around 7 hours, generate errors.

Support was only interested if their built-in memory tester, which even on it's most thorough, would only run for ~3 hours, would show errors, which it wouldn't. IIRC, the BMC was logging "correctable memory errors", but I may be misremembering that.

"We've run this test on every server we've gotten from you, including several others that were exactly the same config as this, this is the only one that's ever thrown errors". Usually support is really great, but they really didn't care in this case.

We finally contacted sales. "Uh, how long do we have to return this server for a refund?" All of a sudden support was willing to ship us out a replacement memory module (memtest86 identified which slot was having the problem), which resolved the problem.

They were all too willing to have us go to production relying on ECC to handle the memory error.

Re: ECC matters

#358

Post by incrudible »

dkersten wrote: post #235
jhasse wrote: post #210

Earlier quoted context omitted.

So can a bug.

Yes. And? That doesn't suddenly make bitflips benign.

The point is that you can't prevent failure by just buying something. You have to deal with the fact that failure can not be prevented.

In other words, if a single defective DIMM somewhere in your deployment is causing catastraphic failure, your mistake was not buying the wrong RAM modules. Your mistake was relying on a single point of failure for mission critical data.

Re: ECC matters

#359

Post by sobriquet9 »

rovr138 wrote: post #356
sobriquet9 wrote: post #343

Earlier quoted context omitted.

If you use multiple computers doing the same calculation and then take the answer from the quorum, how do you ensure the computer that does the comparison is not affected by memory failures? Remember that all queries have to through it, so it has to be comparable in scale and power.

> how do you ensure the computer that does the comparison is not affected by memory failures? You do the comparison on multiple nodes too. Get the calculations. Pass them to multiple nodes, validate again and if it all matches, you use it.

> validate again

Recursion, see recursion.

Re: ECC matters

#360

Post by sroussey »

buildbuildbuild wrote: post #341
colejohnson66 wrote: post #311

Earlier quoted context omitted.

> You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. The Apollo missions (or was it the Space Shuttle?) did this. They had redundant computers that would work with each other to determine the “true” answer.

This remains common in aerospace, each voting computer is referred to as a "string". https://space.stackexchange.com/questions/45076/what-is-a-fl...

In aerospace where this is common, you often had multiple implementations, as you wanted to avoid software bugs made by humans. Problem was, different teams often created the same error at the same place, so it wasn’t as effective as it would have seemed.
Post reply on HN