Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ECC matters

realworldtech.com

Post reply on HN
Print view View original

541–550 of 567 posts

Re: ECC matters

#541

Post by greyhair »

ECC is required on mission critical hardware.

I have spent 36 years fielding embedded devices in core network (D1/E1, SONET, ROADM/MPLS, Cellular basestation) and I will tell you that large ECC covered memory arrays always show small numbers of correctable error events over the course of a year. I have seen, over the course of my career, exactly one controller card replaced early in the field, because it started throwing excessive recoverable ECC events over time, until it hit a threshold of 10x the average of a typical board. On the order of ten recoverable ECC events per month instead of one event per month. I have never observed a logged non-correctable ECC event in the field. In the lab, yes, but never in fielded equipment.

If you are fine with your PC experiencing one or two bits flipped in memory every month, then you really don't need ECC. That is the question you need to answer.

For mission critical systems? ECC is a requirement.

Re: ECC matters

#543

Post by Jedd »

tomxor wrote: post #503
Jedd wrote: post #439

Earlier quoted context omitted.

At some point, whatever's watching the watchers is going to be vulnerable to bitflip and similar problems. Even with a triple-redundant quorum mechanism, slightly further up that stack you're going to have some bit of code running that processes the three returned results - if the memory that's sitting on gets corrupted, you're back where you started.

Yes, but this is the entire principle around which microkernels are designed: making the the last critical piece of code as small and reliable as possible. Minix3's kernel is As far as bitflips are concerned, having the critical kernel code occupy fewer bits reduces the probability of a bitflip causing an irrecoverable error.

Yes, I understand this -- basic risk mitigation by reducing the size of your vulnerability.

(I'll archaic brag a bit by mentioning I used to be a heavy user of Minix - my floppy images came in over an X25 network - and saw Andy Tanenbaum give his Minix 3 keynote at FOSDEM about a decade ago. I'm a big fan.)

Anyway, while reducing risk this way is laudable, and will improve your fleet's health, as per TFA it's a poor substitute, with bad economics and worse politics behind it, than simply stumping up for ECC.

I'll also note that, for example, Google's sitting on ~3 million servers so that ~4k LoC just blew out to 12,000,000,000 LoC -- and that's for the hypervisors only.

Multiply that out by ~50 to include VM's microkernels, and the amount of memory you've now got that is highly susceptible to undetected bit-flips is well into the mind-blowing range.

Re: ECC matters

#544

Post by Dylan16807 »

behringer wrote: post #530
buran77 wrote: post #528

Earlier quoted context omitted.

Indeed, users don't know enough to care but more than that, for most users this is simply not a problem because they haven't been practically affected. Most may never experience a bit flip, and most of the flips won't be obvious anyway (e.g. no visible effect). So your hypothetical dialogue will sound more like a scam to a regular user. You're charging money to tackle a problem they don't have or see, only addressing…

My point is, they have been affected. I think most people have seen bitflips. >2009 Google's paper "DRAM Errors in the Wild: A Large-Scale Field Study" says that there can be up to 25000-75000 one-bit FIT per Mbit (failures in time per billion hours), which is equal to 1 - 5 bit errors per hour for 8GB of RAM after my calculations. Paper says the same: "mean correctable error rates of 2000–6000 per GB per year". http…

Bit flips are reasonably common, but mean is of very little use when one stick can throw no errors and another stick can throw thousands. A more useful quote is "About a third of machines and over 8% of DIMMs in our fleet saw at least one correctable error per year."

Re: ECC matters

#545

Post by Dylan16807 »

DaiPlusPlus wrote: post #516
eptcyka wrote: post #505

Earlier quoted context omitted.

It's the act of maximizing profits by segmenting the market that creates artificially bad products that end up being the mainstream because of the price difference. If the companies responsible for making these decisions were optimizing just a tad bit more for practicality and usability rather than just profit, ECC would be the standard and that would be the end of that. Similar to how a lot of luxury and non-luxury…

> double-the-price ECC memory I was buying DDR4 RAM last week and the cost for a 4x8GB ECC+Registered DDR4 3200 RAM kit for a Xeon W build I’ll be doing soon was about 5% more expensive than for a kit of otherwise identical RAM kit for a non-ECC Core i7 rig - which also came with tacky RGB LED heatspreader - about $230 in total for each of them. The fact the pricing was so similar does make me wonder about the claims…

> The fact the pricing was so similar does make me wonder about the claims that non-ECC RAM is really ECC RAM just without the CPU/MMU being made aware of it - I think that’s possible if the RAM is already Buffered/Registered.

Who claims that? And can't you just count the chips?

Re: ECC matters

#546

Post by Dylan16807 »

buran77 wrote: post #494
Animats wrote: post #423

Earlier quoted context omitted.

The CPU chip can do it. Some motherboards bring out the pins to do it, but they're often called "workstation" boards and cost 2x the price of a standard desktop motherboard. ECC memory itself is overpriced. $60 for 16 GB DDR4 without ECC, $130 for 16 GB DDR4 with ECC. This is the legacy of Intel's policies.

Because if you give consumers a choice between having ECC or LEDs on otherwise identical boards with identical price, most will go for the LEDs. In reality the price isn't even the same because ECC realistically adds to the BOM (board, modules) more than LEDs do. So the price goes up with seemingly no benefit for the user. As such features that are unattractive to the regular consumer go into workstation/enterprise o…

> Because if you give consumers a choice between having ECC or LEDs on otherwise identical boards with identical price, most will go for the LEDs.

Citation needed.

I would bet that your typical ram-purchasing consumer is not seeing or even considering the existence of the ECC model.

> ECC realistically adds to the BOM (board, modules) more than LEDs do. So the price goes up with seemingly no benefit for the user.

LEDs are a great opportunity to increase profit margin, so I'm not sure about your price conclusions.

Re: ECC matters

#547

Post by Dylan16807 »

bitcharmer wrote: post #257
avian wrote: post #238

Earlier quoted context omitted.

As someone who also ran a similar test myself and haven't seen a bit flip, I'm also skeptical of the 96% figure. I'm too lazy to run the exact numbers right now, but with "4 GB, 96% percent chance, three days" as the hypothesis, I think you'll find that an experimental result of "8 GB, 0% chance, 14 days" is highly statistically significant. Edit: rough back of napkin estimate - you're not seeing an event in roughly…

The 96% figure comes from Google and was obtained in a large scale experiment over many months. I've been in this business long enough to have witnessed adverse effects of cosmic rays an non-ECC memory multiple times myself. I don't think you're sample gets anywhere near statistical significance. Not mentioning testing methodology.

> The 96% figure comes from Google

No it doesn't. You're assuming an even distribution of errors, which is very much not the case.

Google found that the average number of errors is around that range, but they also found that only one third of their servers had any errors in a year.

Re: ECC matters

#548

Post by Dylan16807 »

sally1620 wrote: post #461

Linux is accusing Intel of killing ECC intentionally. But that is not really the case, they just wanted people to pay up. If you care about ECC, you pay for Xeon. Majority of consumers don't run critical applications on their devices, so they are happy with a cheap device that may crash once in a while. AMD is only changing the game because they are trying to undercut Intel. They have been putting pro features into a…

Intel had to kill consumer ECC as part of making it a feature that people can "pay up" for. That's very intentional.

> AMD is only changing the game because they are trying to undercut Intel. They have been putting pro features into all of their CPUs including over-clocking, extra PCIE lanes and ECC.

You are correct to call them a corporation. AMD is not your friend, but they are the good actor in this fight.

Re: ECC matters

#549

Post by Dylan16807 »

mark-r wrote: post #218
KingMachiavelli wrote: post #160

Earlier quoted context omitted.

Well maybe. Rather than having to trust memory completely, it would just be better to use a binary format where each bit is verifiable so then at least a single bit flip would be immediately obvious. For example, a bit flip in a TLS session causes the whole session to fail rather than a random page element to change.

That's the principle behind Gray Code counting: https://en.wikipedia.org/wiki/Gray_code

Not really. Gray codes are designed so that if you're counting correctly, only one bit flips at a time. But if you flip the wrong bit by accident, you'll end up with a completely different number, no way to tell a problem happened.

If you want to detect a bit flip, use parity.

Re: ECC matters

#550

Post by buran77 »

Dylan16807 wrote: post #546
buran77 wrote: post #494

Earlier quoted context omitted.

Because if you give consumers a choice between having ECC or LEDs on otherwise identical boards with identical price, most will go for the LEDs. In reality the price isn't even the same because ECC realistically adds to the BOM (board, modules) more than LEDs do. So the price goes up with seemingly no benefit for the user. As such features that are unattractive to the regular consumer go into workstation/enterprise o…

> Because if you give consumers a choice between having ECC or LEDs on otherwise identical boards with identical price, most will go for the LEDs. Citation needed. I would bet that your typical ram-purchasing consumer is not seeing or even considering the existence of the ECC model. > ECC realistically adds to the BOM (board, modules) more than LEDs do. So the price goes up with seemingly no benefit for the user. LED…

That's pretty much exactly my conclusion

> Citation needed.

It really isn't. It was a hypothetical choice between 2 models, with ECC or LEDs, at the same price. Hypothetical because most boards don't offer the ECC support at all, and certainly not at the same price.

> LEDs are a great opportunity to increase profit margin, so I'm not sure about your price conclusions

You confused manufacturing costs, price of the product, and profit margins. LEDs cost far less to integrate than ECC but command a higher price premium (thus better profit margins) from the regular consumer. Again supporting my statement that even if presented with 2 absolutely identical parts save for ECC vs. LEDs the vast majority of consumers will go for LEDs because they don't care or know about ECC.

Post reply on HN