Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ECC matters

realworldtech.com

Post reply on HN
Print view View original

441–450 of 567 posts

Re: ECC matters

#441

Post by djur »

ksec wrote: post #365
nostrademons wrote: post #237

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

>I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Did they ( Google ) or He ( Craig Silverstein ) ever officially admit it on record? I did a Google search and results that came up were all on HN. Did they at least make a few PR pieces saying that they are using ECC memory now because I dont see any with searching. Admitting they mad…

Your [3] has a footnote quoting a Google book that reads "Modern DRAM systems are a good example of a case in which powerful error correction can be provided at a very low additional cost... The following machine generation at Google did include memory parity detection, and once the price of memory with ECC dropped to competitive levels, all subsequent generations have used ECC DRAM"

Re: ECC matters

#442

Post by jfim »

wolrah wrote: post #420
jfim wrote: post #409

Earlier quoted context omitted.

> Now if we can get ECC at the retail level You can actually, most AMD consumer chips (except the ones with integrated graphics) have ECC support, even though it's not officially supported. See this Reddit thread for more details: https://www.reddit.com/r/Amd/comments/ggmyyg/an_overview_of_...

I think they may have been referring to the actual mainstream retail availability of ECC RAM. I can buy non-ECC RAM at almost any retailer that sells computers. If I need non-ECC RAM right now I can have it in my hands in 30 minutes. ECC on the other hand I pretty much have to buy online. Microcenter stocks a single 4GB stick of PC4-21300, and I can't think of a single use case where I'd want ECC but not more than 4G…

You're right, rereading the parent post with that angle makes it clearer that they were complaining about the unavailability of memory and other hardware.

It would definitely be great to have more reliable hardware generally available and at less of a price premium.

Re: ECC matters

#443

Post by MarkusWandel »

mixmastamyk wrote: post #380
MarkusWandel wrote: post #194

This is one justified Linus rant! My personal history includes data loss twice because of defective RAM, and many more RAMs discarded after the now obligatory overnight run of MemTest86+ (these were all secondhand RAMs - I would never buy a new one without a refund guarantee). My very first "PC" still had the ECC capability and I used it. My own now very dated rant on the subject: http://wandel.ca/homepage/memory_ran…

A few years back memtest86 wouldn’t run on newer machines, has that been fixed?

Wouldn't know, I don't run newer machines. But since it's a boot option on Fedora disks, I imagine it would run.

Re: ECC matters

#444

Post by cmehdy »

jgalentine007 wrote: post #346
jjeaff wrote: post #332

Earlier quoted context omitted.

How in the world would you ever know that problem was caused by a bit flip and not just one of the countless other reasons that a sensor could fail?

I have a TPMS programming tool (ATEQ QuickSet) and reader (Autel TS401), because I like to swap my winter / summer tires on my own. The TPMS light came on one day and inflating tires didn't help - I used the reader and found that one sensor's ID had changed. When I compared the ID (it was in hex) to the last programming - it was a single bit off. I couldn't reprogram the sensor itself, but I was able to update the EC…

IIRC higher altitudes definitely register more radiation, the latitude effect being relatively minor in comparison.

Interestingly enough, average background radiation is actually the highest in the US exactly where you are[0] and I've seen comments (on a few websites talking about it) ranging from "the Rockies are loaded in Uranium" to "what about the Rocky Flats nuclear weapons site". Perhaps you're exactly in the right place to be an advocate for ECC :)

[0] http://www.radiationnetwork.com/

Re: ECC matters

#445

Post by AaronFriel »

tjoff wrote: post #319
AaronFriel wrote: post #294

Earlier quoted context omitted.

It can't eliminate it but: 1. Single bitflip correction along with Google's metrics could help them identify algorithms they've got, customer's VMs that are causing bitflips via rowhammer and machines which have errors regardless of workload 2. Double bitflip detection lets Google decide if they say, want to panic at that point and take the machine out of service, and they can report on what software was running or w…

ECC seems like a trivial thing to log and keep track of. Surely any Fortune 500 could do it and would have enough scale to get meaningful data out of it?

It's not just tracking ECC errors, which as you point out is not hard, but correlating it with the other metrics needed to determine the cause and having the scale to reliably root cause bitflips to software (workloads that inadvertently rowhammer) or hardware or even malicious users (GCP customers that may intentionally run a rowhammer.)

Re: ECC matters

#446

Post by sitkack »

ksec wrote: post #365
nostrademons wrote: post #237

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

>I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Did they ( Google ) or He ( Craig Silverstein ) ever officially admit it on record? I did a Google search and results that came up were all on HN. Did they at least make a few PR pieces saying that they are using ECC memory now because I dont see any with searching. Admitting they mad…

The fact that ECC isn't the default across everything is a failure of human cognition and Capitalism.

Re: ECC matters

#447

Post by CalChris »

DCKing wrote: post #321
CalChris wrote: post #262

Earlier quoted context omitted.

Yeah, the iMac Pro has the Xeon W and ECC. T'would be nice if the Apple Silicon MacBook Pro had it. There's not much of a reason to pay for the Pro over the Air. But like Linus, I'm going to blame Intel for this situation in the market. Maybe Apple will strike out on its own with Apple Silicon but since their dominant use case is phones, I'll not hold my breath.

Unless something weird happens, the next generation of the Apple M-line will use LPDDR5 memory instead of the LPDDR4X used in the Apple M1. While it probably won't support error correction monitoring , LPDDR5 has built in error correction that silently corrects single bit flips. That alone should be a huge reliability improvement. LPDDR5 will enable some much needed level of error correction in a metric ton of other…

Well then, maybe I'll wait for the rumored 14.1 MacBook Pro. MacRumors says second half of 2021.

Re: ECC matters

#448

Post by axiolite »

Jedd wrote: post #439
tomxor wrote: post #403

Earlier quoted context omitted.

> Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. There is an OS that pretty much fits the bill here. There was a show where Andrew Tanenbaum had a laptop running Minix 3 hooked up to a button that injected random changes into module code while it was running to demonstrate it's resilience to random bugs. Quite fitting that this discussi…

At some point, whatever's watching the watchers is going to be vulnerable to bitflip and similar problems. Even with a triple-redundant quorum mechanism, slightly further up that stack you're going to have some bit of code running that processes the three returned results - if the memory that's sitting on gets corrupted, you're back where you started.

> At some point, whatever's watching the watchers is going to be vulnerable to bitflip

One advantage of microkernels is that the "watcher" is so small that it could be run directly from ROM, instead of loaded into RAM. QNX has advocated that route for robotics and such in the past.

Minix may not be the best example of the type. While it is a microkernel, it's real world reliability has been poor in the past. More mature microkernel operating systems like QNX and OpenVMS are better examples.

Re: ECC matters

#449

Post by esgwpl »

jfim wrote: post #409
Animats wrote: post #388

Earlier quoted context omitted.

consumer hardware... That's Intel's PR. Only "enterprise hardware", with a bigger markup, supports ECC memory. Adding ECC today should add only 12% to memory cost. AMD decided to break Intel's pricing model. Good for them. Now if we can get ECC at the retail level... The original IBM PC AT had parity in memory.

> Now if we can get ECC at the retail level You can actually, most AMD consumer chips (except the ones with integrated graphics) have ECC support, even though it's not officially supported. See this Reddit thread for more details: https://www.reddit.com/r/Amd/comments/ggmyyg/an_overview_of_...

Small nit, the PRO version of Ryzen APUs do support ECC[0], also ASRock has been quoted saying that all of their AM4 motherboards support ECC, even the low end offerings with the A320 chipset.

[0] https://www.asrock.com/mb/amd/a320m-hdv%20r4.0/index.asp#Spe...

Re: ECC matters

#450

Post by otterley »

Nebasuke wrote: post #361
jjeaff wrote: post #337

Earlier quoted context omitted.

Which means, assuming google is running very large machines with lots of memory that one might expect a single correctable error once every 6-10 years on your average workstation of small server. That's generously assuming your workstation has 1/3 as much memory as the average google server.

Google does not use very large or even large machines for most of their fleet. You can quickly see in the paper this is for 1, 2, and 4 GB RAM machines (in 2006-2008).

These were state of the art 13 years ago. It’s not safe to extrapolate from this paper that they aren’t using servers having significantly more memory today. Thirteen years is more than 2 full depreciation intervals.
Post reply on HN