ECC matters • Hacker News Board

Re: ECC matters

#551

Post by buran77 » Tue, Jan 05, 2021, 8:35 AM UTC

projct wrote: post #535

buran77 wrote: post #534

Earlier quoted context omitted.

But behringer , regular users don't see bitflips , they see "a random error", a random reboot, a random green pixel in a photo. That's if they even consciously observed anything at all. You yourself said those could have a dozen other root causes. I for one have not actually observed a bitflip (so something that I can actually consciously detect as an issue) in a decade. No unexpected reboots, no unexpected file corr…

"Your car does not have a roll cage" No but the crash structures are one of the areas where we are absolutely trying to recreate the effect of a rollcage without the weight. The weight putting constraints on fuel efficiency is the big issue there. For ECC, there's no reason to really expect that adding ECC to everything would really make things too expensive or slow long term. For a long time the reason why desktops…

> No but the crash structures are one of the areas where we are absolutely trying to recreate the effect of a rollcage without the weight

Exactly like how we try to recreate the effect of ECC hardware but without its "weight" (cost, complexity). For a problem that's not nearly as visible or life and death.

> too expensive or slow long term

Not "too" expensive long term still means more expensive especially short term, like when the person buys it. For an issue no average consumer is actually frustrated about. Why are we still debating why those consumers don't care about ECC? How is it different from hot swappable RAM for the average Joe? They'll upgrade or expand their RAM more often than they'll get frustrated about the effect of bitflips. So why not have hot swappable memory? Because as much as I'd love hot swappable anything, regular consumers neither care about it nor want to pay for it.

Re: ECC matters

#552

Post by alkonaut » Tue, Jan 05, 2021, 8:51 AM UTC

xxs wrote: post #498

alkonaut wrote: post #300

Earlier quoted context omitted.

I’m aware. But the risk is extremely small (and 99.9% of important data is not created on the machine but goes directly from e.g iOS camera to backup). My desktop machine is basically a gaming rig with disposable data. Hence the “performance over integrity”. I also never rotate anything out. Every version of everything is in the backups. Storage is that cheap these days.

>My desktop machine is basically a gaming rig ...unlike all the accountant machines which are just standard desktop grade ones that have invoices entered from their keyboards. >Hence the “performance over integrity”. If anything ECC would be better for performance, as it'd allow to clock memory higher. It's a mystery how Intel has managed to convince people ECC = enterprise/server market. The real cost of ECC is 1/8…

That's more a hypothetical "if ECC was common enough that the products would exist" scenario.

The current status is that it mostly doesn't exist outside OEM parts. Searching an online retailer for ECC brings me hundreds of results for "non-ECC" RAM. When they occasionally have 1-2 products with ECC, they aren't as aggressively binned as the non-ECC sticks. They'll have higher latencies and lower clock frequencies. Basically: you'll need to accept gambling on being able to overclock your ECC RAM.

Re: ECC matters

#553

Post by exikyut » Tue, Jan 05, 2021, 10:39 AM UTC

saagarjha wrote: post #539

exikyut wrote: post #531

Earlier quoted context omitted.

This sounds really cool and interesting. Was any code dumped anywhere? I found this which corroborates everything you're saying but provides no further details: https://www.cspensky.info/slides/defcon_27_shortman.pdf

Oh, hey, it's Chad's slides! Coverage of the finals is usually much less detailed, unfortunately, since the number of teams is much smaller and the challenges don't necessarily go up. However, https://oooverflow.io/dc-ctf-2020-quals/ has a couple more writeups linked from it; https://dttw.tech/posts/SJ40_7MNS#proof-by-exhaustion from PPP and http://www.secmem.org/blog/2019/08/19/Shellcoding-and-Bitfli... from SeoulPl…

I see. Thanks very much for this info.

Binary bitflip resilience is really cool. The radiation-hardened-quine idea (https://codegolf.stackexchange.com/questions/57257/radiation..., https://github.com/mame/radiation-hardened-quine) is cool, but these source-based approaches depend on a perfectly functioning and rather large (Ruby, V8, whole browser) binary stack. A bitflip-protected hex monitor or kernel, on the other hand...

Re: ECC matters

#554

Post by ksec » Tue, Jan 05, 2021, 10:54 AM UTC

djur wrote: post #441

ksec wrote: post #365

Earlier quoted context omitted.

>I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Did they ( Google ) or He ( Craig Silverstein ) ever officially admit it on record? I did a Google search and results that came up were all on HN. Did they at least make a few PR pieces saying that they are using ECC memory now because I dont see any with searching. Admitting they mad…

Your [3] has a footnote quoting a Google book that reads "Modern DRAM systems are a good example of a case in which powerful error correction can be provided at a very low additional cost... The following machine generation at Google did include memory parity detection, and once the price of memory with ECC dropped to competitive levels, all subsequent generations have used ECC DRAM"

Well They made a big deal out of it and lots of PR out of not having use ECC, and only quietly adding back ECC and listed it out in a book that barely any one reads.

Re: ECC matters

#555

Post by Dylan16807 » Tue, Jan 05, 2021, 10:59 AM UTC

buran77 wrote: post #550

Dylan16807 wrote: post #546

Earlier quoted context omitted.

> Because if you give consumers a choice between having ECC or LEDs on otherwise identical boards with identical price, most will go for the LEDs. Citation needed. I would bet that your typical ram-purchasing consumer is not seeing or even considering the existence of the ECC model. > ECC realistically adds to the BOM (board, modules) more than LEDs do. So the price goes up with seemingly no benefit for the user. LED…

That's pretty much exactly my conclusion > Citation needed. It really isn't. It was a hypothetical choice between 2 models, with ECC or LEDs, at the same price. Hypothetical because most boards don't offer the ECC support at all, and certainly not at the same price. > LEDs are a great opportunity to increase profit margin, so I'm not sure about your price conclusions You confused manufacturing costs, price of the pro…

> It really isn't. It was a hypothetical choice between 2 models, with ECC or LEDs, at the same price. Hypothetical because most boards don't offer the ECC support at all, and certainly not at the same price.

You're making a claim about what people would choose. If you have no related data, and logic could support multiple outcomes, then a claim like that is basically useless.

> You confused manufacturing costs, price of the product, and profit margins.

I'm not sure why you think this.

> Again supporting my statement that even if presented with 2 absolutely identical parts save for ECC vs. LEDs the vast majority of consumers will go for LEDs because they don't care or know about ECC.

Sure, if you don't tell them that it's ECC they won't pick the ECC part.

If you actually do a fair test, and put them side by side while explaining that one protects them from memory errors and the other looks cooler, you can't assume they'll all pick the LED.

When people never even think of ECC, that is not evidence that they wouldn't care or know about it in a head-to-head competition.

Re: ECC matters

#556

Post by tomxor » Tue, Jan 05, 2021, 12:01 PM UTC

Jedd wrote: post #543

tomxor wrote: post #503

Earlier quoted context omitted.

Yes, but this is the entire principle around which microkernels are designed: making the the last critical piece of code as small and reliable as possible. Minix3's kernel is As far as bitflips are concerned, having the critical kernel code occupy fewer bits reduces the probability of a bitflip causing an irrecoverable error.

Yes, I understand this -- basic risk mitigation by reducing the size of your vulnerability. (I'll archaic brag a bit by mentioning I used to be a heavy user of Minix - my floppy images came in over an X25 network - and saw Andy Tanenbaum give his Minix 3 keynote at FOSDEM about a decade ago. I'm a big fan.) Anyway, while reducing risk this way is laudable, and will improve your fleet's health, as per TFA it's a poor…

Oh i'm not saying it's the single best solution, I guess I got carried away in argument - It's simply a scenario where the concept shines, yet it's entirely artificial scenario and I agree ECC is the correct way.

Re: ECC matters

#557

Post by buran77 » Tue, Jan 05, 2021, 12:03 PM UTC

Dylan16807 wrote: post #555

buran77 wrote: post #550

Earlier quoted context omitted.

That's pretty much exactly my conclusion > Citation needed. It really isn't. It was a hypothetical choice between 2 models, with ECC or LEDs, at the same price. Hypothetical because most boards don't offer the ECC support at all, and certainly not at the same price. > LEDs are a great opportunity to increase profit margin, so I'm not sure about your price conclusions You confused manufacturing costs, price of the pro…

> It really isn't. It was a hypothetical choice between 2 models, with ECC or LEDs, at the same price. Hypothetical because most boards don't offer the ECC support at all, and certainly not at the same price. You're making a claim about what people would choose. If you have no related data, and logic could support multiple outcomes, then a claim like that is basically useless. > You confused manufacturing costs, pric…

> You're making a claim

My claims are common sense and supported by the real life: regular people don't know what ECC is, and those who do find the problem's impact is too minor to get palpable benefits from fixing it. Why are you being pedantic if you aren't actually going to bring arguments at the same level you expect from me?

> If you actually do a fair test, and put them side by side while explaining that one protects them from memory errors and the other looks cooler, you can't assume they'll all pick the LED.

Isn't this exactly the kind of claim you yourself characterize one paragraph above as "useless" because "you have no related data, and logic could support multiple outcomes"? Sure, if people were more tech-educated then my assumption might be wrong. But people aren't more educated so...

The benefits of LEDs are hard to miss (light) all the time. The benefits of ECC are hard to observe even in that fraction of a percent of the time. Human cellular "bitflips" happen every hour but they don't visibly affect you so you also consider it's not an issue that demands more attention, like constant solar protection. People aren't keen on paying to solve problems they never suffered from, or even noticed, especially when you tell them they happen so often still with no obvious impact. Unless they have no choice, like OEMs selling ECC RAM only devices.

Sell me ECC memory when my (actual real life) 10 year old desktop or 5 year old phone never glitched. Sell me ECC RAM when my Matlab calculations come back different every time. See the difference?

> When people never even think of ECC, that is not evidence that they wouldn't care or know about it in a head-to-head competition.

Well then, I guess none of us has any evidence except today people buy LEDs not ECC RAM. Educate people or wait until manufacturing process and design are so susceptible to bitflips that people notice and it will be a different conversation.

Re: ECC matters

#558

Post by Dylan16807 » Tue, Jan 05, 2021, 5:58 PM UTC

buran77 wrote: post #557

Dylan16807 wrote: post #555

Earlier quoted context omitted.

> It really isn't. It was a hypothetical choice between 2 models, with ECC or LEDs, at the same price. Hypothetical because most boards don't offer the ECC support at all, and certainly not at the same price. You're making a claim about what people would choose. If you have no related data, and logic could support multiple outcomes, then a claim like that is basically useless. > You confused manufacturing costs, pric…

> You're making a claim My claims are common sense and supported by the real life: regular people don't know what ECC is, and those who do find the problem's impact is too minor to get palpable benefits from fixing it. Why are you being pedantic if you aren't actually going to bring arguments at the same level you expect from me? > If you actually do a fair test, and put them side by side while explaining that one pr…

> My claims are common sense and supported by the real life: regular people don't know what ECC is, and those who do find the problem's impact is too minor to get palpable benefits from fixing it. Why are you being pedantic if you aren't actually going to bring arguments at the same level you expect from me?

Regular people aren't given the choice! The things you're quoting about the real world to support your argument are incompatible with a scenario where someone is actually looking at ECC and LED next to each other. And I'm not being "pedantic" to say that, it's a really core point.

> Isn't this exactly the kind of claim you yourself characterize one paragraph above as "useless" because "you have no related data, and logic could support multiple outcomes"?

A claim of a specific outcome is useless. "you can't assume" is another way of phrasing the lack of knowledge of specific outcomes.

> Sure, if people were more tech-educated then my assumption might be wrong. But people aren't more educated so...

It's the kind of thing that can go on a product page. But first someone has to actually make a consumer-focused sales page for ECC memory, and the ECC has to be plug-and-play without strong compatibility worries.

And just like when LEDs spread over everything, it's something that you can teach people about and create demand for with a bit of advertising.

> Sell me ECC memory when my (actual real life) 10 year old desktop or 5 year old phone never glitched. Sell me ECC RAM when my Matlab calculations come back different every time. See the difference?

That's a clear picture of one person. But "never glitched" is a very dubious claim, and you can't blindly extrapolate that to how everyone would act.

Re: ECC matters

#559

Post by xxs » Wed, Jan 06, 2021, 10:46 AM UTC

alkonaut wrote: post #552

xxs wrote: post #498

Earlier quoted context omitted.

>My desktop machine is basically a gaming rig ...unlike all the accountant machines which are just standard desktop grade ones that have invoices entered from their keyboards. >Hence the “performance over integrity”. If anything ECC would be better for performance, as it'd allow to clock memory higher. It's a mystery how Intel has managed to convince people ECC = enterprise/server market. The real cost of ECC is 1/8…

That's more a hypothetical " if ECC was common enough that the products would exist" scenario. The current status is that it mostly doesn't exist outside OEM parts. Searching an online retailer for ECC brings me hundreds of results for "non-ECC" RAM. When they occasionally have 1-2 products with ECC, they aren't as aggressively binned as the non-ECC sticks. They'll have higher latencies and lower clock frequencies. B…

ECC could have been common, if there was a meaningful CPU support for. Intel basically did - "you need Xeon to do exactly what an i7 does + ECC, it just costs 5 times more... We just fused the ECC part of memory controller of i7, so buy Xeon."

The entire discussion is about why ECC is not common and why ECC matters. There is no technical reason for ECC to be uncommon aside purposed market segmentation.

Of course, there is no intrinsic availability of ECC udimms for the retail market currently, however that does mean ECC has no use or benefits for a small extra production cost.

Re: ECC matters

#560

Post by _0ffh » Wed, Jan 06, 2021, 11:47 AM UTC

wmf wrote: post #328

_0ffh wrote: post #289

Please someone correct me if I'm wrong, but as far as I can remember memory with extra capacity for error detection used to be a rather common thing on early PCs. That really only changed a couple of decades in, in order to be able to offer lower prices to home users who didn't know or care about the difference. Probably about the time, or earlier, when with some hard disk manufacturers megabytes suddenly shrunk to 1…

Yes, PCs used to use parity memory.

That's the name I couldn't quite recover from my memory when I asked, exactly!