Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ECC matters

realworldtech.com

Post reply on HN
Print view View original

411–420 of 567 posts

Re: ECC matters

#411

Post by thu2111 »

jeffbee wrote: post #81
paulie_a wrote: post #41

There was a great defcon talk a while back regarding using ECC. The concept was called "dns jitter" Basically you can register domains using small bit differences for domains and start getting email and such for that domain If I recall correctly the example given was a variation of microsoft.com All because so much equipment doesn't use ECC

miclosoft.com is only one bit away from microsoft.com. Used to see these problems all the time when I worked on gmail. At Google even with ECC everywhere there wasn't enough systematic error detection and correction to prevent the global database of monitoring metrics from filling up with garbage. /rpc/server/count was supposed to exist but also in there would be /lpc/server/count and /rpc/sdrver/count and every othe…

Ahaha. Reminds me of when I worked there. One day a large service tanked in some datacenter because BigTable replication in that location just stopped. Digging in, it turned out the BigTable should have been replicating from YQ but had started trying to use QQ instead, which didn't exist. Q being one bit away from Y. Or it was something like that, I don't remember exactly. There'd been a bit flip in the exact part of memory that contained the name of the database cluster to replicate from!

Re: ECC matters

#413

Post by grishka »

nostrademons wrote: post #237

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

I remember reading how someone registered some google domains with a single bit flipped, and saw actual requests coming to them.

Re: ECC matters

#414

Post by andrewstuart2 »

grishka wrote: post #413
nostrademons wrote: post #237

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

I remember reading how someone registered some google domains with a single bit flipped, and saw actual requests coming to them.

If you or anybody can remember the source article, that sounds like an interesting read!

Edit: found one with a quick search. https://nakedsecurity.sophos.com/2011/08/10/bh-2011-bit-squa...

And https://www.researchgate.net/publication/262273269_Bitsquatt...

Re: ECC matters

#415

Post by knorker »

skibbityboop wrote: post #310
knorker wrote: post #232

I have multiple times postponed buying new computers for YEARS, because I'm waiting for intel to get their head out of their ass and actually let me buy something that does ECC for desktop. (incl laptops) I would have bought computers when I "wanted one". Now I buy them when I need one. Because buying a non-ECC computer just feels like buying a defective product. In the last 10 years I would have bought TWICE as many…

Have you finally stopped buying Intel? Current Ryzens are a much better CPU anyhow, just dump Intel and be happy with your ECC and everything else.

I'm in the market for a new laptop (since a few years). Is there something like the X1 carbon but with ECC?

Re: ECC matters

#416

Post by NickNameNick »

grishka wrote: post #413
nostrademons wrote: post #237

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

I remember reading how someone registered some google domains with a single bit flipped, and saw actual requests coming to them.

That works for any domain that's busy enough.

Random bit flips happen on client machines and on routers.

If there are enough requests for a domain name, some of those requests will be subject to one one of those bit-flips.

Re: ECC matters

#417

Post by saagarjha »

grishka wrote: post #413
nostrademons wrote: post #237

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

I remember reading how someone registered some google domains with a single bit flipped, and saw actual requests coming to them.

That might just be typos in some cases?

Re: ECC matters

#418

Post by felixhandte »

incrudible wrote: post #149
dijit wrote: post #16

I beg this, every time this conversation comes up it’s the same answer “I don’t see a problem”. It’s so easy to chalk these kind of errors to other issues, a little corruption here, a running program goes bezerk there- could be a buggy program or a little accidental memory overwrite. Reboot will fix it. But I ran many thousands of physical machines, petabytes of RAM, I tracked memory flip errors and they were _common…

When you say bitflips were "common" on thousands of physical machines, does that mean you observed thousands of bitflips? Otherwise, I would think that an unlikely event becoming 1000x more likely by sheer numbers would have warped your perception. I believe that hardware reliability is mostly irrelevant, because software reliability is already far worse. It doesn't matter whether a bitflip (unlikely) or some bug (li…

Yes. I work at Facebook on data compression.

The libraries we maintain (1) are responsible for a non-trivial part of Facebook's overall compute footprint, (2) should basically never fail of their own accord, and (3) have pretty good error monitoring. So my team is operating what is effectively (among other things) a very sensitive detector for hardware failure.

And indeed we see examples all the time of blobs that fail to decompress, and usually when we dig in we find that the blob is only a single bit-flip away from a blob that decompresses successfully into a syntactically correct message. I can't share numbers, but, off the top of my head, I think it's the largest source of failures we see. It happens frequently enough that I wrote a tool to automate checking [0].

So yes. It happens. Pretty frequently, in the sense that if you're doing xillions of operations a day, a one-in-a-xillion failure happens all the time.

[0] https://github.com/facebook/zstd/tree/dev/contrib/diagnose_c...

Re: ECC matters

#419

Post by gigatexal »

colejohnson66 wrote: post #312
gigatexal wrote: post #290

Earlier quoted context omitted.

I mean early on sure at a startup where you’re not printing money I can see how saving on hardware makes sense. But surely you don’t need an MBA to know that hardware will continue to get cheaper whereas developers and their time will only get more expensive: better to let the hardware deal with it than to burden developers with it ... I’d have made the case for ECC but hindsight being what it is ...

But if you can save $1M+ now, then throw the cost of fixing it onto the person who replaces you, why do you care? You already got your bonus and jumped ship.

Such a short sighted incentivisation structure harms the business in the long run I think.

Re: ECC matters

#420

Post by wolrah »

jfim wrote: post #409
Animats wrote: post #388

Earlier quoted context omitted.

consumer hardware... That's Intel's PR. Only "enterprise hardware", with a bigger markup, supports ECC memory. Adding ECC today should add only 12% to memory cost. AMD decided to break Intel's pricing model. Good for them. Now if we can get ECC at the retail level... The original IBM PC AT had parity in memory.

> Now if we can get ECC at the retail level You can actually, most AMD consumer chips (except the ones with integrated graphics) have ECC support, even though it's not officially supported. See this Reddit thread for more details: https://www.reddit.com/r/Amd/comments/ggmyyg/an_overview_of_...

I think they may have been referring to the actual mainstream retail availability of ECC RAM. I can buy non-ECC RAM at almost any retailer that sells computers. If I need non-ECC RAM right now I can have it in my hands in 30 minutes. ECC on the other hand I pretty much have to buy online. Microcenter stocks a single 4GB stick of PC4-21300, and I can't think of a single use case where I'd want ECC but not more than 4GB.
Post reply on HN