I'm about to write some code that will allocate a random buffer, take a checksum of it, and just sit on the buffer, periodically checksuming it again until a bit flips. Or maybe even allocate a buffer of zeros and wait until a non-zero appears in it.
ECC matters
431–440 of 567 posts
Re: ECC matters
#432I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…
Recent advances have blurred the lines a bit. The ECC memory that we all know and love is mainly side-band EEC, with the memory bus widened to accommodate the ECC bits driven by the memory controller. However as process size shrink, bit flips become more likely to the point that now many types of memory have on-die EEC, where the error correction is handled internally on the DRAM modules themselves. This is present o…
> This is present on some DDR4 and DDR5 modules, but information on this is kept internal by the DRAM makers and not usually public.
On-die ECC is going to be a standard feature for DDR5. I'm not aware of any indication that anyone has implemented on-die ECC for DDR4 DRAM, and Hynix at least has made clear statements that on-die ECC is new for their DDR5 and was not present in their DDR4.
Re: ECC matters
#433Earlier quoted context omitted.
I mean raft and similar algorithms run multiple verification machines because a single point of failure is a single point of failure.
Raft, Paxos, and other consensus algorithms add even more overhead. Imagine running every Google query through Raft and think how long it will take and how much extra hardware would be needed. ECC memory is just as fast as non-ECC memory, and only cost a little more.
Your comment sounded like "your recursive definition is impossible".
I am totally for ECC and was flabbergasted when it went away. But the article makes sense since I remember Intel pushing hard to keep it out of the consumer space. The freaking QX6800 didn't support ECC and it retailed for over a grand.
Re: ECC matters
#434Earlier quoted context omitted.
miclosoft.com is only one bit away from microsoft.com. Used to see these problems all the time when I worked on gmail. At Google even with ECC everywhere there wasn't enough systematic error detection and correction to prevent the global database of monitoring metrics from filling up with garbage. /rpc/server/count was supposed to exist but also in there would be /lpc/server/count and /rpc/sdrver/count and every othe…
Ahaha. Reminds me of when I worked there. One day a large service tanked in some datacenter because BigTable replication in that location just stopped. Digging in, it turned out the BigTable should have been replicating from YQ but had started trying to use QQ instead, which didn't exist. Q being one bit away from Y. Or it was something like that, I don't remember exactly. There'd been a bit flip in the exact part of…
Beautiful.
As long as we're swapping war stories there was once a frontend server at that company which died of bit flip related causes but just before it did it managed to charge some product group for 2^63 bytes worth of network traffic in the internal quota system, which set off every budgetary alarm that capacity planners had.
Re: ECC matters
#435Earlier quoted context omitted.
I think it's available for customer SKUs on AMD and not just for servers like in 'Xeon-land'... How I've wanted an ECC-ready NUC...
Probably not what you meant but https://ark.intel.com/content/www/us/en/ark/products/190108/... has support for Xeon (and ECC). Now how to actually practically source 32GB ECC enabled SO-DIMM sticks ..
Oh thanks! TIL. Then I just saw the price tag, ouch.
Re: ECC matters
#436Earlier quoted context omitted.
I think it's available for customer SKUs on AMD and not just for servers like in 'Xeon-land'... How I've wanted an ECC-ready NUC...
The AMD parts all have the ECC feature but the platform support outside of EPYC may as well not exist. Most motherboards for the Ryzen segment don’t do it properly or don’t do it at all, some support it but aren’t capable of reporting events to the operating system which is dumb. Ryzen laptops don’t have it either. Closest you can come to a nuc with ecc is I think a mini server equipped with one of the four-core i3 p…
Ah thanks for the real-world check on AMD Ryzen and ECC. Sad state of affairs really, when so many things are integrated in the SOC, why skimp on this...
As for NUCs, I thought there were some Atom chips with ECC but not in NUC form factor ? Something shiny from Logic Supply might do then...
Re: ECC matters
#437You don't need to look at kernel crashes to speculate about bus and memory errors -- just check the logs on a few systems that do have ecc. Pretty soon you'll see correctable errors being reported.
I don’t know much about this topic, but is it possible that ECC memory is more prone to single bit errors than non-ECC memory because there is less pressure on companies to minimize such errors? If this were the case, it would skew the data.
There are 12.5% more memory cells for a given module size, which equals more targets to possibly be flipped by cosmic rays. It’s not crazy to think that modules of equivalent quality (same brand, same chip part numbers) would experience a greater incidence of that kind of single bit flip (which would be corrected on the ECC modules). If a manufacturer were shipping chips prone to bit flipping because of slightly radioactive packaging, as happened at times in the past, you might see something similar.
But you’ve got it backwards about the incentives. A manufacturer has less incentive to deliberately ship a defective part in the case of ECC modules. If the modules consistently log ECC errors, they can easily be identified and returned under warranty to the manufacturer. A consumer is much less likely to identify an intermittent problem with a non-ECC part.
Re: ECC matters
#438Earlier quoted context omitted.
You could run memtest on a pc without ecc for a couple of days and to estimate the error rate, or not?
If you manage to replicate bitflips every few days your RAM is broken. It's the "once every other year" type of bitflip that's the problem. The proverbial "cosmic ray" hitting your DRAM and flipping a bit. That will be caught by ECC but it'll most likely remain a total mystery if it causes your non-ECC hardware to crash.
In the mean time I read here the rate to be around 1 bit flip per 1 GB per month. So in an 120 GB System that shd be about 1 flip every 6h. But then this number might be wrong.
Re: ECC matters
#439Earlier quoted context omitted.
ECC memory can't eliminate the chances of these failures entirely. They can still happen. Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. So in theory the behavior of software under random bit flips is well... Random. You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. I…
> Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. There is an OS that pretty much fits the bill here. There was a show where Andrew Tanenbaum had a laptop running Minix 3 hooked up to a button that injected random changes into module code while it was running to demonstrate it's resilience to random bugs. Quite fitting that this discussi…
At some point, whatever's watching the watchers is going to be vulnerable to bitflip and similar problems.
Even with a triple-redundant quorum mechanism, slightly further up that stack you're going to have some bit of code running that processes the three returned results - if the memory that's sitting on gets corrupted, you're back where you started.
Re: ECC matters
#440Earlier quoted context omitted.
Did you consider any off-the-shelf ECC boxes? Found some here -- bottom of the EPYC product line starts at $2849 ...! https://www.velocitymicro.com/wizard.php?iid=337
The TDP on EPYC chips is a lot higher. I think of Threadripper as mid-tier, and EPYC as the high-end. Ryzen is remarkable because you can buy new equipment with ECC, at consumer prices. I am hazy, but don't think that has been possible since the 386 era ('parity ram').
> The TDP on EPYC chips is a lot higher.
EPYC TDP ranges from "a lot lower" (35W embedded, 120W regular) up to "comparable with" (180-240W, a single 280W model) relative to Threadripper (180-250W last gen; current gen is all 280W). It's definitely not a lot higher on the Epyc side.
> I think of Threadripper as mid-tier, and EPYC as the high-end.
This oversimplifies to the point of not being a useful intuition (or is arguably even incorrect). Threadripper is a SKU with a moderate number of cores at relatively high clocks; (high-power) EPYC SKUs have a lot of very efficient cores running at lower clocks. They both have a niche, but Threadripper has unambiguously better single-core performance due to the ~20% higher clocks. And single-core IPC still matters in many applications (to oversimplify: Amdahl's law; but also, latency-sensitive applications).