Live data from Hacker News

ECC matters

realworldtech.com

421–430 of 567 posts

Re: ECC matters

#421

Earlier quoted context omitted.

I remember reading how someone registered some google domains with a single bit flipped, and saw actual requests coming to them.

If you or anybody can remember the source article, that sounds like an interesting read! Edit: found one with a quick search. https://nakedsecurity.sophos.com/2011/08/10/bh-2011-bit-squa... And https://www.researchgate.net/publication/262273269_Bitsquatt...

Here's original paper: http://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinabur...

But I'm not entirely convinced that most of these requests weren't just typos. Though those requests with mismatched Host header surely were true bitflips.

Re: ECC matters

#422

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

ECC memory can't eliminate the chances of these failures entirely. They can still happen. Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. So in theory the behavior of software under random bit flips is well... Random. You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. I…

There was an interesting challenge at DEF CON CTF a while back that tested this, actually. It turns out that it is possible to write x86 code that is 1-bit-flip tolerant–that is, a bit flip anywhere in its code can be detected and recovered from with the same output. Of course, finding the sequence took (or so I hear) something like 3600 cores running for a day to discover it ;)

Re: ECC matters

#423
post #409

Earlier quoted context omitted.

consumer hardware... That's Intel's PR. Only "enterprise hardware", with a bigger markup, supports ECC memory. Adding ECC today should add only 12% to memory cost. AMD decided to break Intel's pricing model. Good for them. Now if we can get ECC at the retail level... The original IBM PC AT had parity in memory.

> Now if we can get ECC at the retail level You can actually, most AMD consumer chips (except the ones with integrated graphics) have ECC support, even though it's not officially supported. See this Reddit thread for more details: https://www.reddit.com/r/Amd/comments/ggmyyg/an_overview_of_...

The CPU chip can do it. Some motherboards bring out the pins to do it, but they're often called "workstation" boards and cost 2x the price of a standard desktop motherboard. ECC memory itself is overpriced. $60 for 16 GB DDR4 without ECC, $130 for 16 GB DDR4 with ECC.

This is the legacy of Intel's policies.

Re: ECC matters

#424

Earlier quoted context omitted.

Nobody denies that bitflips happen . On the whole, you fail to make a case that preventing bitflips is the solution to a problem. Bitsquatting is not a real problem, it's a curiosity. As for the case of bitflips killing someone: Bitflips are not the root cause here. The root cause is that somebody engineered something life-critical that mistakenly assumed hardware can not fail. Bitflips are just one of many reasons f…

>Bitflips are not the root cause here. So those systems didn't fail when a bitflip happened? > The root cause is that somebody engineered something life-critical that mistakenly assumed hardware can not fail. The systems I am aware of were designed with bitflips in mind. NO software can handle arbitrary amounts of bitflips. ALL software designed to mitigate bitflips only lower the odds via various forms of redundancy…

> So those systems didn't fail when a bitflip happened?

I didn't say that. I'm saying that the root cause (as in "root cause analysis") is not the bitflip. Designating the bitflip as the root cause is like analyzing your drunk driving accident and concluding that the root cause must be ethanol, rather than your drinking habits.

> The systems I am aware of were designed with bitflips in mind. NO software can handle arbitrary amounts of bitflips. ALL software designed to mitigate bitflips only lower the odds via various forms of redundancy.

Of course, and I'm not actually arguing that adding in ECC is completely worthless to that effect, though it is close to worthless. Luckily, ECC is quite cheap, if not free, so throwing it in there makes sense.

However, suppose ECC would increase the cost by several magnitudes, would it still be worth it? Obviously not. Redundancy alone reduces the probability of spurious failure by several magnitudes, and simply increasing redundancy would be far cheaper than adding in ECC.

> If bitflips are not a problem then we don't need ECC ram (or ECC almost anything!) which is clearly used a lot. So bitflips are enough of a problem that a massively widespread technology is in place to handle precisely that problem.

My point is that bitflips either don't really matter, in case data integrity is not mission critical, or they don't actually solve the problem, in case data integrity is mission critical.

If you have solved the problem of data integrity through redundancy, then ECC doesn't make much of a difference anymore. If you haven't solved the problem, then ECC will only prevent a vanishingly small subset of disasters that are awaiting you.

> I guess you've never written a program and watched bits flip on computers you control? You should try it - it's a good exercise to see how often it does happen.

I don't care how often it happens. I care about the odds of a bitflip causing an actual problem. If a computer crashes, that's okay, it'll reboot. If any data were to be corrupted, it would most likely happen at the disk level and not the DRAM level.

> I guess you define something being a problem differently than I or the ECC ram industry do.

Of course, somebody who sells ECC RAM will want to convince you that ECC actually solves a real problem. The same can be said about the nutritional supplement industry, or many other industries that rely on make-belief.

Re: ECC matters

#425
post #77

As someone who has had to read thousands of random game crash reports from all over the interwebs (you know when Windows says you might want to send that crash log? like that), I totally agree. Of all the things to be worried about, like OS bugs, bad hardware configuration, etc. bad memory is one of those really troubling things. You look at the code and say "it's can't make it here, because this was set" but when yo…

[deleted]

Re: ECC matters

#426

Earlier quoted context omitted.

I don't think ECC is going to give anyone a false sense of security. The issue at Google's scale is they had to spend thousands of person-hours implementing in software what they would have gotten for "free" with ECC RAM. Lacking ECC (and generally using consumer-level hardware) compounded scale and reliability problems or at least made them more expensive than they might otherwise had been. Using consumer hardware a…

consumer hardware... That's Intel's PR. Only "enterprise hardware", with a bigger markup, supports ECC memory. Adding ECC today should add only 12% to memory cost. AMD decided to break Intel's pricing model. Good for them. Now if we can get ECC at the retail level... The original IBM PC AT had parity in memory.

When I said consumer hardware I was meaning early Google literally using consumer/desktop components mounted on custom metal racks. While Intel does artificially separate "enterprise" and "consumer" parts, there's still a bit of difference between SuperMicro boards with ECC, LOM features, and data center quality PSUs and the off the shelf consumer stuff Google was using for a while.

I don't know if AMD really intended to break Intel's pricing model. Their higher end Ryzen chips you'd use in servers and capital W Workstations don't seem to have a huge price difference from equivalent Xeons. Even if they're a bit cheaper you still need a motherboard that supports ECC so it seems at first glance to be a wash as far as price.

That being said if I was putting together a machine today it would be Ryzen-based with ECC.

Re: ECC matters

#427

Earlier quoted context omitted.

> validate again Recursion, see recursion.

I mean raft and similar algorithms run multiple verification machines because a single point of failure is a single point of failure.

Raft, Paxos, and other consensus algorithms add even more overhead. Imagine running every Google query through Raft and think how long it will take and how much extra hardware would be needed.

ECC memory is just as fast as non-ECC memory, and only cost a little more.

Re: ECC matters

#428
post #77

As someone who has had to read thousands of random game crash reports from all over the interwebs (you know when Windows says you might want to send that crash log? like that), I totally agree. Of all the things to be worried about, like OS bugs, bad hardware configuration, etc. bad memory is one of those really troubling things. You look at the code and say "it's can't make it here, because this was set" but when yo…

One of my favourite stories about this type if stuff is from Patrick Wyatt on developing Guild Wars: https://www.codeofhonor.com/blog/whose-bug-is-this-anyway

See "Your computer is broken". They essentially inserted a stress test into the game that verified if the hardware was still doing calculations correctly, and if not, inform the user.

Re: ECC matters

#429

Earlier quoted context omitted.

> I suppose that if the memory contains code, the process is killed (if ECC correction failed). Generally, it would make the most sense to kill the process if the corrupted page is data , but if it's code, then maybe re-load that page from the executable file on non-volatile storage. (You might also be able to rescue some data pages from swap space this way.)

If you go that route, you should be able to avoid the code/data distinction entirely; as data pages can also be completly backed by files. I believe the kernel already keeps track of what pages are a clean copy of data from the filesystem, so I would think it would be a simple matter of essentially pageing out the corrupted data. What would be interesting is if userspace could mark a region of memory as recomputable.…

> What would be interesting is if userspace could mark a region of memory as recomputable.

I believe there's already some support for things like this, but intended as a mechanism to gracefully handle memory pressure rather than corruption. Apple has a Purgeable Memory mechanism, but handled through higher-level interfaces rather than something like madvise().

Re: ECC matters

#430
post #319

Earlier quoted context omitted.

It can't eliminate it but: 1. Single bitflip correction along with Google's metrics could help them identify algorithms they've got, customer's VMs that are causing bitflips via rowhammer and machines which have errors regardless of workload 2. Double bitflip detection lets Google decide if they say, want to panic at that point and take the machine out of service, and they can report on what software was running or w…

ECC seems like a trivial thing to log and keep track of. Surely any Fortune 500 could do it and would have enough scale to get meaningful data out of it?

IBM does. They will probably sell you the information if you rent the machines from them.
Post reply on HN