Live data from Hacker News

ECC matters

realworldtech.com

251–260 of 567 posts

Re: ECC matters

#251

Is there such a thing as 'software' ECC where a segment in memory also has a checksum stored in memory and the CPU just verifies it when the memory segment is accessed? It would be a lot slower than real ECC but it could just be used for operations that would be especially vulnerable to bit flips. It would also not know for certain if the memory segment of data or the memory segment holding the checksum was corrupted…

Intel has some recent patents on that.

Re: ECC matters

#252

Earlier quoted context omitted.

That's an interesting point. In an extreme case, an order or money transfer might be placed for an incorrect quantity, or to an incorrect recipient.

Well maybe. Rather than having to trust memory completely, it would just be better to use a binary format where each bit is verifiable so then at least a single bit flip would be immediately obvious. For example, a bit flip in a TLS session causes the whole session to fail rather than a random page element to change.

That doesn't help if the memory is corrupted before the verification code is applied. (the code will simply put a signature on incorrect data)

Or after it's been checked. (time-of-check vs time-of-use)

Re: ECC matters

#253
post #234

Earlier quoted context omitted.

> it's not true that ECC is supported well on AMD ECC is supported on most Ryzen models[1], as long as the motherboard supports it. In fact, ASUS and ASRock (possibly others) have Ryzen motherboards designed for workstation/server use where ECC support is specifically advertised. [1] The only exception is the Ryzen CPUs with integrated graphics.

Depends what you mean by supported. Semi-offically: ECC is not disabled. It works, but not validated for our consumer client platform. Validated means run it through server/workstation grade testing. For the first Ryzen processors, focused on the prosumer / gaming market, this feature is enabled and working but not validated by AMD. You should not have issues creating a whitebox homelab or NAS with ECC memory enabled…

AMD may claim not to validate ECC on Ryzen, but it's working well enough for major motherboard vendors to market Ryzen motherboards with ECC advertised as a feature.

ECC support not being "validated," for all practical purposes, simply means that board vendors can advertise a board lacking ECC support as compatible with AMD's AM4 platform, without getting a nasty letter from AMD's lawyers.

Re: ECC matters

#254
post #174
post #100

Earlier quoted context omitted.

I haven't seen definite details and test results on these (but haven't looked recently). What specific configurations (CPU, MB, RAM) are known to work? Let's say I have a Ryzen system, how can I check if ECC really works? Like, can I see how many bit flips got corrected in, say, last 24h?

All AMD CPUs with integrated memory controllers support ECC. The CPU also exposes an interface usable by the operating system to verify ECC works - the same interface is used to provide monitoring of memory fault data provided by ECC. They aren't tested on it, so it's possible to get a dud, but it's minuscule chance that isn't worth bothering. Now, to actual issues you can encounter: motherboards The problem is that…

> Another issue is motherboard firmware. Even though AMD supplies the memory init code, the configuration can be tweaked by motherboard vendor, and they might simply break ECC support accidentally (even by something as simple as making a toggle default to false then forgot to expose it in configuration menu).

I think some Gigabyte boards are infamous for this in certain circle

OTOH: Gigabyte might have a Threadripper PRO motherboard (WRX80 chipset) coming out in the future

Re: ECC matters

#255
post #209

Earlier quoted context omitted.

TDP matters a fair bit in SFF(Small Form Factor) PCs. For instance the 3700x is a fantastic little CPU since it has a 65W TDP but pretty solid performance. In a sandwich style case you're usually limited to low profile coolers like Noctua L9i/L9a since vertical height is pretty limited.

Performance/watt matters. You can just set TDP to what you want with throttling choices. If you want a 45W TDP from the 3700X, you can just pop into Ryzen Master and ask for a 45W TDP. Boom, you're running in that envelope. I think shopping based on TDP is not the best, because it's not comparable between manufacturers and because it's something you can effectively "choose".

How do you do that? Is it a setting in the bios? Or can it be done runtime? If so, how? It sounds interesting if I can run a beefy rig as a power efficient device, for always-on scenarios, and then boost it when I need.

Re: ECC matters

#256
post #25

Earlier quoted context omitted.

There are situations though, where you’re working on a document and the documents “save” format is a memory dump. Corruption for things of that type (Adobe RAW for example) would remove data. It might present itself as a 1pixel colour difference, but it could be more damaging (incorrect finances, in accounting software for example). Software trusts memory; but memory can lie. That’s dangerous.

Those corner cases might occur rarely but are probably inconsequential given rate of occurrence versus rate of criticalness - it probably doesn't justify the markup for most. In a data center you're processing millions of transactions per minute so occurrence is much more impactful.

I would EASILY pay 12.5% more (that's the bit overhead) for memory that actually works.

If my data is fine being corrupted to save 12.5% on RAM costs, then why am I even bothering processing the data? Apparently it's worthless.

People today weigh the cost of maybe 16 vs 32GB on a mid-tier desktop. ~doubling the cost for twice the RAM. Yes, paying 12.5% more for ECC RAM is a no-brainer.

Re: ECC matters

#257
post #238

Earlier quoted context omitted.

How did you run those tests? From what I understand on the topic, for your results to be statistically significant you need at least hundreds of machines and very rigid testing methodology.

As someone who also ran a similar test myself and haven't seen a bit flip, I'm also skeptical of the 96% figure. I'm too lazy to run the exact numbers right now, but with "4 GB, 96% percent chance, three days" as the hypothesis, I think you'll find that an experimental result of "8 GB, 0% chance, 14 days" is highly statistically significant. Edit: rough back of napkin estimate - you're not seeing an event in roughly…

The 96% figure comes from Google and was obtained in a large scale experiment over many months. I've been in this business long enough to have witnessed adverse effects of cosmic rays an non-ECC memory multiple times myself. I don't think you're sample gets anywhere near statistical significance. Not mentioning testing methodology.

Re: ECC matters

#258

Well shit. I run some large ML models in my home PC and I get NaN's and some out of range floats every month or so. I have spent hours debugging but doing the same computation with the same random seeds does not recreate the problem. How about GPU's and their GDDR SDRAM? Do they have parity bits?

[deleted]

Re: ECC matters

#259

I simply care that my computer executes code perfectly. Let's settle on "one instance of unintended behaviour per hundred years" for that metric. If it needs ECC memory to do that, then fit it with ECC memory. If there are other ways to achieve that (for example deeper dram cells to be more robust to cosmic rays) that's fine too. Just meet the reliability spec - I don't care how.

In "theory" it needs ECC because you must also protect the link between the CPU and the RAM. So with ECC fully in DRAM but no protection on the bus, you risk some errors during the transfer. However maybe this kind of errors are rare enough so that you would have less than one per century. It probably depends on the motherboard design and fabrication quality though, and the environment where it is used.
Post reply on HN