Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

blog.robertelder.org

Post reply on HN
Print view View original

11–20 of 79 posts

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#11

Post by woodruffw »

BearOso wrote: post #10
hayst4ck wrote: post #2

Bit flips are totally real, at scale you will definitely see them on large queries. There was a fun talk at DEFCON on bitsquatting, the process of buying 1 bit off domain names and then accepting all incoming connections. Attacks like rowhammer similarly abuse erroneous bit flips. Supposedly microsoft can detect solar activity based on the number of windows crash logs they receive. DEFCON Talk: https://www.youtube.co…

I remember reading somewhere about that talk being debunked. Maybe someone more resourceful than me can find it. It was something about being more likely to be a human typo or a config change that rolled out to a bunch of machines. The statistics didn't add up, and it wasn't plausible that bit flips caused it.

I don't know of a debunking, but more numbers can be found in the associated blog post[1].

[1]: http://dinaburg.org/bitsquatting.html

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#12

Post by muro »

ta988 wrote: post #7

I've had a lot of really strange bugs and data loss with my current build (Ryzen with Gskill memory). After running a memtest for 24h i finally saw that two of the four ram sticks were faulty (two bit flips on each only rarely and on a specific test). The company changed them but now a year later without any issues I have another one that failed in exactly the same way. This is the last time I build a non-ECC system…

Heard a similar story from a friend last week - a faulty RAM stick as well. I'm glad I bought a Threadripper with ECC instead (worth waiting for a Lenovo sale and buy RAM separately)

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#13

Post by silisili »

BearOso wrote: post #10
hayst4ck wrote: post #2

Bit flips are totally real, at scale you will definitely see them on large queries. There was a fun talk at DEFCON on bitsquatting, the process of buying 1 bit off domain names and then accepting all incoming connections. Attacks like rowhammer similarly abuse erroneous bit flips. Supposedly microsoft can detect solar activity based on the number of windows crash logs they receive. DEFCON Talk: https://www.youtube.co…

I remember reading somewhere about that talk being debunked. Maybe someone more resourceful than me can find it. It was something about being more likely to be a human typo or a config change that rolled out to a bunch of machines. The statistics didn't add up, and it wasn't plausible that bit flips caused it.

Unsure about the talk, but I read a research report from someone at I think it was Cisco who purchased a second level tld of a bitsquatted US state( think statenXX.us instead of state.XX.us ). The amount of email they received was staggering, and I'd have trouble believing that many people would make that mistake in typing.

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#14

Post by BearOso »

moremetadata wrote: post #6

Moral of the story? Upgrade to DDR5 ram the latest standard which has on-die ECC memory but is not as good at spotting bit flips unlike proper ECC memory with a separate extra data correction chip. https://en.wikipedia.org/wiki/DDR5_SDRAM#:~:text=Unlike%20DD... . Whilst Proper ECC ram chips and motherboards exist, I'm surprised that a cheaper but equally as good as Proper ECC solution doesn't exist although I know so…

DDR5 has enough ECC on chip to make errors effectively impossible. It doesn't provide error data to the CPU, though, so errors in transit can still occur. This is really unlikely, though, and anything not mission-critical will no longer need the extra ECC computation on the CPU-side. (DDR5 encapsulates the memory controller).

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#16

Post by xwdv »

johnklos wrote: post #4

ECC is good, and I genuinely wish it were more common. Thankfully, Ryzen CPUs support ECC by default (except for pre-7000 series with integrated graphics that aren't "Pro" versions), so long as the motherboard does, too (like all ASRock that I've seen). I'm running several Ryzen servers with ECC. On the other hand, there are many, many systems out there that don't have ECC, nor do they have the option to have ECC. Wh…

I think you overstate the problem here. Chances are, unless you’ve addressed other more pertinent issues, simply using ECC memory isn’t going to stop systems from crashing in the middle of the night.

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#17

Post by hayst4ck »

BearOso wrote: post #10
hayst4ck wrote: post #2

Bit flips are totally real, at scale you will definitely see them on large queries. There was a fun talk at DEFCON on bitsquatting, the process of buying 1 bit off domain names and then accepting all incoming connections. Attacks like rowhammer similarly abuse erroneous bit flips. Supposedly microsoft can detect solar activity based on the number of windows crash logs they receive. DEFCON Talk: https://www.youtube.co…

I remember reading somewhere about that talk being debunked. Maybe someone more resourceful than me can find it. It was something about being more likely to be a human typo or a config change that rolled out to a bunch of machines. The statistics didn't add up, and it wasn't plausible that bit flips caused it.

There are many places that memory exists. Your processor cache is memory. Registers are memor-ish. Hard Drives have memory. If a bit flip happens in your hard drives memory before being written to disk, then it's not unreasonable to think that the bitflip would persist, even through reboots.

I have run queries at large companies and found mistakes most easily explained as bitflips in domain names written to disk. Imagine an environmental variable configuring the use of a proxy without proper whitelists and it's not unimaginable to me that a production machine would be able to speak to machines on the internet at large.

I am open to the idea that what I think is happening might not be the mechanics of what is happening, but I find the talk believable, not based on theory, but actually seeing persisted (and non-persisted) bit flips in domain names queried from data warehoused logs at world scale companies.

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#18

Post by jchw »

muro wrote: post #15

AFAICT, no current Mac comes with ECC - do they have the same issues? If so, one doesn't hear about them too often.

Well, the reason why ECC mattered here is because the RAM was bad, but modern Mac computers do not come with user-serviceable RAM at all, so if you have a problem like this, it's a support ticket anyways, and I'm not even sure there's a true equivalent to Memtest86 for modern Mac computers in the first place. So basically, if it was a RAM problem, there's no point in diagnosing it even if you could; just send your Mac in when you start having issues that seem to be bad RAM.

Even with ECC, it's incredibly hard to know that a given one-off issue isn't a memory error, because even ECC can't detect 100% of memory issues. But without ECC, it's also nearly impossible to know if something is a memory error. If it's bad RAM, the same address will likely continue to exhibit bad behavior, but if it's a solar flare, you're never going to know the difference; you will just get incorrect behavior that may or may not crash, and it will be completely impossible to reproduce.

One big reason you don't hear it as much is there are not nearly as many data centers filled with Macs. There are definitely a few, and I bet if you got an experience report from them, they could give some idea of how visible memory errors are on Macs (although it's hard, because again, if you don't have ECC, there's not really a good way to know if something is a memory error; you can only really postulate.)

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#19

Post by mtlmtlmtlmtl »

Amazing technical write up. But if there's no cause for alarm based on SMART, I would just do the memtest right then because that's always my goto for weird undiagnosed problems. I find it's usually not the problem, although when it has been I've ended up wasting a silly amount of time on it(just like this case!).

And if there was cause for alarm, I would think long and hard about imaging from the original computer at all. With certain failure modes in drives, just reading could cause more corruption; each failed attempt could lose data.

But yeah, happy you did it this way in the end, because I learned a ton from the resulting blog post!

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#20

Post by BearOso »

hayst4ck wrote: post #17
BearOso wrote: post #10

Earlier quoted context omitted.

I remember reading somewhere about that talk being debunked. Maybe someone more resourceful than me can find it. It was something about being more likely to be a human typo or a config change that rolled out to a bunch of machines. The statistics didn't add up, and it wasn't plausible that bit flips caused it.

There are many places that memory exists. Your processor cache is memory. Registers are memor-ish. Hard Drives have memory. If a bit flip happens in your hard drives memory before being written to disk, then it's not unreasonable to think that the bitflip would persist, even through reboots. I have run queries at large companies and found mistakes most easily explained as bitflips in domain names written to disk. Ima…

That was the gist of the response, as far as I remember. It was repeated access from the same set of machines, so it was more likely that one bit flip persisted in a config and was subsequently rolled out to others. So the study grossly overestimated the number of actual bit flips occurring by like 20000x or something.
Post reply on HN