Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ECC matters

realworldtech.com

Post reply on HN
Print view View original

521–530 of 567 posts

Re: ECC matters

#521

Post by eptcyka »

DaiPlusPlus wrote: post #516
eptcyka wrote: post #505

Earlier quoted context omitted.

It's the act of maximizing profits by segmenting the market that creates artificially bad products that end up being the mainstream because of the price difference. If the companies responsible for making these decisions were optimizing just a tad bit more for practicality and usability rather than just profit, ECC would be the standard and that would be the end of that. Similar to how a lot of luxury and non-luxury…

> double-the-price ECC memory I was buying DDR4 RAM last week and the cost for a 4x8GB ECC+Registered DDR4 3200 RAM kit for a Xeon W build I’ll be doing soon was about 5% more expensive than for a kit of otherwise identical RAM kit for a non-ECC Core i7 rig - which also came with tacky RGB LED heatspreader - about $230 in total for each of them. The fact the pricing was so similar does make me wonder about the claims…

Oh wow, the prices have normalized a bit - about 18 months ago I had to pay about twice the money to get 64 gigabytes of ECC goodness.

Re: ECC matters

#522

Post by phkahler »

vvanders wrote: post #198
mlyle wrote: post #182

Earlier quoted context omitted.

On AMD, with Ryzen Master, you can set the TDP-envelope of the processor to what you want. Then the boost/frequency/voltage envelope it chooses to operate in under sustained load is different. IMO, shopping by performance/watt makes sense. Shopping by TDP doesn't. (Especially since there is no comparing the AMD and Intel TDP numbers as they're defined differently; neither is the maximum the processor can draw, and In…

TDP matters a fair bit in SFF(Small Form Factor) PCs. For instance the 3700x is a fantastic little CPU since it has a 65W TDP but pretty solid performance. In a sandwich style case you're usually limited to low profile coolers like Noctua L9i/L9a since vertical height is pretty limited.

I'm running a 2400G in a Mellori-ITX. Another issue is sizing the power supply.

Re: ECC matters

#523

Post by tpetry »

jjeaff wrote: post #337
otterley wrote: post #86

About 1/3 of Google's machines and 8% of Google's DIMMs in their fleet suffer at least one correctible memory error per year: http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf

Which means, assuming google is running very large machines with lots of memory that one might expect a single correctable error once every 6-10 years on your average workstation of small server. That's generously assuming your workstation has 1/3 as much memory as the average google server.

With a single bit flip on 8% of the dimms you only need 12.5 dimms in your workstation to have one bit flip every year. Not everyone has that much dimms, but at least 4 is pretty normal. So in average every 3 years for every workstation.

But i don‘t know how relevant these metrics from 2009 are. Did memory got better or worse compared to 2009 for bit flips?

Re: ECC matters

#524

Post by masklinn »

apankrat wrote: post #379
cbanek wrote: post #77

As someone who has had to read thousands of random game crash reports from all over the interwebs (you know when Windows says you might want to send that crash log? like that), I totally agree. Of all the things to be worried about, like OS bugs, bad hardware configuration, etc. bad memory is one of those really troubling things. You look at the code and say "it's can't make it here, because this was set" but when yo…

Aye. I have an assert in the code that fronts a very pedantic test of the context. In all cases when this assert was tripped (and reported) an overnight memtest86 test surfaced RAM issues. - Edit - Also, bit flips in the non-ECC memory are _the_ cause of the "bitrot" phenomenon. That is when you write out X to a storage device, but you get Y when you read it back. A common explanation is that the corruption happens _…

> Also, bit flips in the non-ECC memory are _the_ cause of the "bitrot" phenomenon. That is when you write out X to a storage device, but you get Y when you read it back. A common explanation is that the corruption happens _at rest_. However all drives from the last 30+ years have FEC support, so in reality the only way a bit rot can happen is if the data is damaged _in transit_, while in RAM, on the way to/from the storage media.

Problems can definitely happen in the IO controller, RAID controller, cable, and disk controller. AFAIK all of these were seen and motivations for the existence of ZFS. One of their biggest insistence was that drives are universally lying bastards and should not be trusted any further than they can be thrown.

Re: ECC matters

#525

Post by incrudible »

SideQuark wrote: post #509
incrudible wrote: post #424

Earlier quoted context omitted.

> So those systems didn't fail when a bitflip happened? I didn't say that. I'm saying that the root cause (as in "root cause analysis") is not the bitflip. Designating the bitflip as the root cause is like analyzing your drunk driving accident and concluding that the root cause must be ethanol, rather than your drinking habits. > The systems I am aware of were designed with bitflips in mind. NO software can handle ar…

>I don't care how often it happens. Yes, that is clear. > If you have solved the problem of data integrity... As above, this is not a binary, black and white thing, but you keep presenting it as such. It's probabilistic, and higher protection is not free - the tradeoff is engineering. > Redundancy alone reduces the probability of spurious failure by several magnitudes ECC "alone reduces the probability of spurious fa…

> As above, this is not a binary, black and white thing, but you keep presenting it as such. It's probabilistic, and higher protection is not free - the tradeoff is engineering.

The actual problem is binary. You either solved it, or you didn't. ECC is "free", but it doesn't actually solve the problem. Actually solving the problem requires engineering.

Of course there's a probabilistic element to it, but the problem is to drive the probability of failure to "vanishingly small". The utility of adding or removing a vanishingly small constant to another vanishingly small constant is vanishingly small. This is what ECC does for you.

> ECC "alone reduces the probability of spurious failure by several magnitudes". That's why it is used.

ECC reduces the probability of spurious failure due to bitflips in DRAM by several magnitudes. However, spurious failure can occur for so many more reasons that the bitflip issue becomes a vanishingly small part.

> I have a feeling your idea of redundancy is having multiple exact copies of a system or data and having them vote, which is a terribly expensive way to do data protection when there are vastly better methods.

As you know, having worked for NASA, this is the right choice under certain circumstances. If there are lives on the line and you have a choice between "not solving a problem" and "a terribly expensive solution", you should go with the latter.

> If you don't think ECC helps a real problem then I see why you don't understand bitflip causing problems.

ECC does not solve the problem of data integrity. If you actually solve the problem of data integrity, you will find that ECC becomes effectively redundant. Do we not fundamentally agree on this? If so, why not?

That's not to say ECC is entirely useless from an administrative standpoint. It makes DRAM bitflips one less thing to worry about. One less thing out of thousands of things. Commensurately, the cost of ECC in a given deployment, like its utility, is vanishingly small.

Re: ECC matters

#526

Post by behringer »

buran77 wrote: post #493
jandrese wrote: post #452

Earlier quoted context omitted.

I have always been angry that ECC was treated as an "enterprise" feature and increased the price way more than it should have.

It's treated as enterprise because most regular users had no realistic reason to care about it. The practical consequences of not having this on your phone or home PC are completely invisible for the vast majority of people. Because of this and the fact that in the consumer space prices matter most the situation is quite expected.

Users most definitely want it. They just aren't technically literate enough to know about it.

Hello user, for an extra 10 dollars, would you like to guarantee that cosmic radiation never affects your computing experience, including having to reformat, reinstall, or otherwise reboot your phone/computer randomly when one day something just doesn't work anymore?

Was it an update? Was it cosmic radiation? Was it a bad capacitor? Who knows!

Why do you think so many problems are solved by rebooting? Sure 99 out of 100 might be software bugs, but the other 1 out of 100 is straight up cosmic radiation, and that 1 percent is growing more and more every year as software becomes more robust and bug free with better tooling.

Re: ECC matters

#527

Post by [deleted] »

klodolph wrote: post #489
DSingularity wrote: post #456

Earlier quoted context omitted.

You need two alpha particles hitting the same rank of memory for failure to happen. Although super rare, even then it is still correctable. You need three before it is silent data corruption. Silent corruption is what you get with non ECC with even a single flip.

Where are you getting this from? My understanding is that these errors are predominantly caused by secondary particles from cosmic rays hitting individual memory cells, and I've never heard something so precise as "you need two alpha particles". Aren't the capacitances in modern DRAM chips extremely small?

[deleted]

Re: ECC matters

#528

Post by buran77 »

behringer wrote: post #526
buran77 wrote: post #493

Earlier quoted context omitted.

It's treated as enterprise because most regular users had no realistic reason to care about it. The practical consequences of not having this on your phone or home PC are completely invisible for the vast majority of people. Because of this and the fact that in the consumer space prices matter most the situation is quite expected.

Users most definitely want it. They just aren't technically literate enough to know about it. Hello user, for an extra 10 dollars, would you like to guarantee that cosmic radiation never affects your computing experience, including having to reformat, reinstall, or otherwise reboot your phone/computer randomly when one day something just doesn't work anymore? Was it an update? Was it cosmic radiation? Was it a bad ca…

Indeed, users don't know enough to care but more than that, for most users this is simply not a problem because they haven't been practically affected. Most may never experience a bit flip, and most of the flips won't be obvious anyway (e.g. no visible effect).

So your hypothetical dialogue will sound more like a scam to a regular user. You're charging money to tackle a problem they don't have or see, only addressing a single one of the root causes that trigger that same result, and in the end you're not even completely fixing it, just reducing the already infinitesimal odds it happens.

It will become mainstream when manufacturers just decides to include it everywhere and not really give the user a choice. Apple is a prime candidate for a company with enough clout to afford this.

Re: ECC matters

#529

Post by rfoo »

saagarjha wrote: post #490
rfoo wrote: post #458

Earlier quoted context omitted.

Nit: not for a day, more like 8 hours, and that's because we were lazy and somebody said he "just happened" to have a cluster with unbalanced resources (mainly used for deep learning, but all GPUs occupied with quite a lot CPUs / RAMs left), so we decided to brute force the last 16 bits :) Also, the challenge host left useful state (which bit was flipped) in registers before running teams' code, without this I'm not…

Sure, all's fair in a CTF. That story came to me through the mouths of at least a handful of people, who might have a bit of an incentive to exaggerate given that they hadn't quite been able to get to zero and might be a just a little sour :P The state was quite helpful, yes–for x86 it seems like a "clean slate" shellcode would be quite difficult, if impossible, to achieve as we saw. However, I am left wondering how…

Maybe being notoriously dense is not a bad thing? While those ModRM bytes popping up everywhere is annoying as f* (too easy to flip an instruction into a form with almost-guaranteed-to-be-invalid memory access), at least due to the density there won't be reserved bits. For example, in AArch64 if bit 28 and bit 27 is both zero the instruction will almost certainly be an invalid one (hitting unallocated area), and with a single bit flip all branch instructions will have [28:27] = b'00...

[1] https://developer.arm.com/docs/ddi0596/h/top-level-encodings...

Re: ECC matters

#530

Post by behringer »

buran77 wrote: post #528
behringer wrote: post #526

Earlier quoted context omitted.

Users most definitely want it. They just aren't technically literate enough to know about it. Hello user, for an extra 10 dollars, would you like to guarantee that cosmic radiation never affects your computing experience, including having to reformat, reinstall, or otherwise reboot your phone/computer randomly when one day something just doesn't work anymore? Was it an update? Was it cosmic radiation? Was it a bad ca…

Indeed, users don't know enough to care but more than that, for most users this is simply not a problem because they haven't been practically affected. Most may never experience a bit flip, and most of the flips won't be obvious anyway (e.g. no visible effect). So your hypothetical dialogue will sound more like a scam to a regular user. You're charging money to tackle a problem they don't have or see, only addressing…

My point is, they have been affected. I think most people have seen bitflips.

>2009 Google's paper "DRAM Errors in the Wild: A Large-Scale Field Study" says that there can be up to 25000-75000 one-bit FIT per Mbit (failures in time per billion hours), which is equal to 1 - 5 bit errors per hour for 8GB of RAM after my calculations. Paper says the same: "mean correctable error rates of 2000–6000 per GB per year".

https://stackoverflow.com/questions/2580933/cosmic-rays-what...

I believe bitflips are far more common than people realize. I see weird shit all the time from my users that is only explainable due to bizarre software bugs or bitflips.

Post reply on HN