Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ECC matters

realworldtech.com

Post reply on HN
Print view View original

331–340 of 567 posts

Re: ECC matters

#331

Post by adrian_b »

incrudible wrote: post #149
dijit wrote: post #16

I beg this, every time this conversation comes up it’s the same answer “I don’t see a problem”. It’s so easy to chalk these kind of errors to other issues, a little corruption here, a running program goes bezerk there- could be a buggy program or a little accidental memory overwrite. Reboot will fix it. But I ran many thousands of physical machines, petabytes of RAM, I tracked memory flip errors and they were _common…

When you say bitflips were "common" on thousands of physical machines, does that mean you observed thousands of bitflips? Otherwise, I would think that an unlikely event becoming 1000x more likely by sheer numbers would have warped your perception. I believe that hardware reliability is mostly irrelevant, because software reliability is already far worse. It doesn't matter whether a bitflip (unlikely) or some bug (li…

On a single computer with a large memory, e.g. 32 GB or more, the time between errors can be of a few months, if you are lucky to have good modules. Moreover, some of the errors will have no effect, if they happened to affect free memory.

Nevertheless, anyone who uses the computer for anything else besides games or movie watching, will greatly benefit from having ECC memory, because that is the only way to learn when the memory modules become defective.

Modern memories have a shorter lifetime than old memories and very frequently they begin to have bit errors from time to time long before breaking down completely.

Without ECC, you will become aware that a memory module is defective only when the computer crashes or no longer boots and severe data corruption in your files could have happened some months before that.

For myself, this was the most obvious reason why ECC was useful, because I was able in several cases to replace memory modules that began to have frequent correctable errors, after many years with little or no errors, without losing any precious data and without downtime.

Re: ECC matters

#332

Post by jjeaff »

jgalentine007 wrote: post #206
cbanek wrote: post #152

Earlier quoted context omitted.

Well, I've had to actually worry about radiation bitflips as well. It does happen. But usually not so much on Earth!

One of the tire pressure sensors in my car tires had a bit flip a couple years ago and I had to reprogram it's ID. Luckily it was a subaru, so only a light came on in the dash. My old Honda crv however would turn traction control on if your pressure was low - which worked by applying brakes to wheels that were slipping. If you were going up a slippery hill you would soon have no power, sliding backwards nearly off th…

How in the world would you ever know that problem was caused by a bit flip and not just one of the countless other reasons that a sensor could fail?

Re: ECC matters

#333

Post by jjeaff »

josefx wrote: post #121
ekianjo wrote: post #29

Earlier quoted context omitted.

> no good reason not to switch to ECC everywhere. Not all CPUs support ECC however.

Just Intel fucking over security by making ECC a non feature on consumer grade hardware - wouldn't be surprised if it was just a single bit flipped in a feature mask.

Well, with as common as a bunch of people in this thread seem to think bit flips are, it should just be a matter of time until that bit gets flipped on your cpu and activates the ecc feature.

Re: ECC matters

#334

Post by EvanAnderson »

colejohnson66 wrote: post #311
maria_weber23 wrote: post #274

Earlier quoted context omitted.

ECC memory can't eliminate the chances of these failures entirely. They can still happen. Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. So in theory the behavior of software under random bit flips is well... Random. You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. I…

> You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. The Apollo missions (or was it the Space Shuttle?) did this. They had redundant computers that would work with each other to determine the “true” answer.

The Space Shuttle had redundant computers. The Apollo Guidance Computer was not redundant (though there were two AGCs onboard-- one in the CM and one in the LEM). The aerospace industry has a history of using redundant dissimilar computers (different CPU architectures, multiple implementations of the control software developed by separate teams in different languages, etc) in voting-based architectures to hedge against various failure modes.

Re: ECC matters

#335

Post by Laforet »

colejohnson66 wrote: post #313
theevilsharpie wrote: post #186

Earlier quoted context omitted.

> And though ECC is not disabled in Ryzen CPUs, AFAIK it's not tested in (or advertised for) those ECC isn't validated by AMD for AM4 Ryzen models, but it's present and supported if the motherboard also supports it. Many motherboards have ECC support (the manual will say for sure), and a handful of models even explicitly advertise it as a feature. I have a Ryzen 9 3900X on an ASRock B450M Pro4 and 64 GB of ECC DRAM,…

What do you mean by “validated”? There’s the silicon, but they don’t test it?

More like "The feature is present in silicon but motherboard makers are not required to turn it on". At the end of the day, ECC support does require extra copper traces in the PCB and some low end models may deliberately choose to skip them, thus the expectation has to be managed.

Re: ECC matters

#336

Post by toast0 »

bitcharmer wrote: post #257
avian wrote: post #238

Earlier quoted context omitted.

As someone who also ran a similar test myself and haven't seen a bit flip, I'm also skeptical of the 96% figure. I'm too lazy to run the exact numbers right now, but with "4 GB, 96% percent chance, three days" as the hypothesis, I think you'll find that an experimental result of "8 GB, 0% chance, 14 days" is highly statistically significant. Edit: rough back of napkin estimate - you're not seeing an event in roughly…

The 96% figure comes from Google and was obtained in a large scale experiment over many months. I've been in this business long enough to have witnessed adverse effects of cosmic rays an non-ECC memory multiple times myself. I don't think you're sample gets anywhere near statistical significance. Not mentioning testing methodology.

My anecdotal evidence is far from rigorous, but the Google data from ten years ago doesn't match up with my experience running thousands of ECC enabled servers up to a few years ago. Their rates seem a lot higher than what my servers experienced; we would page on any ram errors, correctable or not (uncorrectable would halt the machine, so we would have to inspect the console to confirm; when we knowingly tried machines with uncorrectable errors after a halt, they nearly all failed again within 24 hours, so those we didn't inspect the console of probably were counted on their second failure), and while there were pages from time to time, it felt like a lot less than 8% of the machine having a

There's a lot of variables that go into RAM errors, including manufacturing quality and condition of the ram, the dimm, the dimm slot, the motherboard generally, the power supply, the wiring, and the temperature of all of those. Google was known for cost cutting in their servers, especially early on; so I wouldn't be surprised if some of that resulted in higher bitflip rate than running in commercially available servers. Things like running bare motherboards, supported only on the edges cause excess strain and can impact resistance and capacitance of traces on the board (and in extreme cases, break the traces).

Re: ECC matters

#337

Post by jjeaff »

otterley wrote: post #86

About 1/3 of Google's machines and 8% of Google's DIMMs in their fleet suffer at least one correctible memory error per year: http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf

Which means, assuming google is running very large machines with lots of memory that one might expect a single correctable error once every 6-10 years on your average workstation of small server. That's generously assuming your workstation has 1/3 as much memory as the average google server.

Re: ECC matters

#338

Post by tyoma »

nostrademons wrote: post #237

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

Figure this is as good of a time as any to ask this:

There are many various DRAMs in a server (say, for disk cache). Has Google or anyone who operates at a similar scale seen single bit errors in these components?

Re: ECC matters

#339

Post by slumdev »

eevilspock wrote: post #327
slumdev wrote: post #291

Earlier quoted context omitted.

Error-correcting code (the "ECC" in ECC) is just a quorum at the bit level.

I'm surprised that the other replies don't grasp this. This is the proper level to do the quorum. Doing quorum at the computer level would require synchronizing parallel computers, and unless that synchronization were to happen for each low level instruction, then it would have to be written into the software to take a vote at critical points. This is going to be greatly detrimental both to throughput and software co…

I would highly recommend a graduate-level course in computer architecture for anyone who thinks ECC is a 1980s solution to a modern problem.

There are a lot of seemingly high-level problems that are solved (ingeniously) in hardware with very simple, very low-level solutions.

Re: ECC matters

#340

Post by toast0 »

trevyn wrote: post #150
formerly_proven wrote: post #73

Earlier quoted context omitted.

RAM is not as reliable as you think. Some ECC memory hardly ever finds an error, some machines see them at a very consistent rate, e.g. 50 errors per TB-day. That would translate to 1-2 errors per day in a 32 GB PC. Without ECC you cannot know in which bucket you are.

If true, that seems like... a very straightforward bucket to test if you’re in.

The bucket can change over time though. If you want to be sure, you need to test often, which gets in the way of using the computer.
Post reply on HN