Live data from Hacker News

Google says Rowhammer attacks are gaining range as RAM is getting denser

therecord.media

81–90 of 127 posts

Re: Google says Rowhammer attacks are gaining range as RAM is getting denser

#81

But while there are no known cases where Rowhammer attacks have been used in the real world Not on purpose but I'm sure that either it or phenomenon like it are the causes of a lot of odd "glitchy" behaviour that people encounter, because systems are run so close to their limits that there are bound to be cases when they surpass the limits. I remember many years ago discovering that a system which passed the CPU and…

The problem isn't RowHammer, the problem is lack of ECC. This is mostly Intel's fault, as they cripple their consumer CPU lines to disable ECC for market segmentation purposes. The entire premise that we can store tens of gigabytes of information reliably in tiny silicon capacitors without any error detection or correction whatsoever is ludicrous. Every other storage technology uses advanced error correction - NAND F…

Not here to defend Intel (their decision to remove ECC support from X299 chipset pretty much tanked their own HEDT platform), but I often wonder how much of the current disregard of ECC is the fault of the system integrators.

As of now they would gladly ship configurations with single channel memory and under-par cooling to save a few bucks, while these cut corners could hurt performance by as much as 30% due to insufficient memory bandwidth and thermal throttling.

Even if ECC is available everywhere I doubt any OEM will use it without charging their users a premium. This is kind of what has been going on with AMD MSDT platforms: ECC support is in the CPU but many motherboard does not even implement the necessary memory traces to enable it, and software support is even messier to the point that most people can't be bothered with it.

Re: Google says Rowhammer attacks are gaining range as RAM is getting denser

#82

Earlier quoted context omitted.

Do you think many Ryzen PCs use ECC? I doubt that. It's accepted truth among gamers and power users, that ECC is waste of budget. I don't share this position, but if you'd ask on some computer forums, that's what you'll hear. If ECC would be enabled on all Intel CPUs, nothing fundamentally would change, most users would prefer to save 10% on their RAM.

The price of ECC is artificially inflated because it's not available to consumers.

At minimum ECC will cost you an extra 12.5% for the extra bit of RAM. That part is not inflated.

Re: Google says Rowhammer attacks are gaining range as RAM is getting denser

#83
post #81

Earlier quoted context omitted.

The problem isn't RowHammer, the problem is lack of ECC. This is mostly Intel's fault, as they cripple their consumer CPU lines to disable ECC for market segmentation purposes. The entire premise that we can store tens of gigabytes of information reliably in tiny silicon capacitors without any error detection or correction whatsoever is ludicrous. Every other storage technology uses advanced error correction - NAND F…

Not here to defend Intel (their decision to remove ECC support from X299 chipset pretty much tanked their own HEDT platform), but I often wonder how much of the current disregard of ECC is the fault of the system integrators. As of now they would gladly ship configurations with single channel memory and under-par cooling to save a few bucks, while these cut corners could hurt performance by as much as 30% due to insu…

Maybe while both ECC and non-ECC are both being sold, but that would be a temporary situation. The goal would be to have non-ECC memory be effectively deprecated, so the next revision of the spec enforces ECC as a requirement.

Re: Google says Rowhammer attacks are gaining range as RAM is getting denser

#84

Earlier quoted context omitted.

The problem isn't RowHammer, the problem is lack of ECC. This is mostly Intel's fault, as they cripple their consumer CPU lines to disable ECC for market segmentation purposes. The entire premise that we can store tens of gigabytes of information reliably in tiny silicon capacitors without any error detection or correction whatsoever is ludicrous. Every other storage technology uses advanced error correction - NAND F…

ECC is at best a temporary workaround, and at worst an implicit approval of the creation of defective products. It's purpose is to prevent transient bitflips due to cosmic rays and such, not consistently reproducible errors from what is otherwise perfectly normal operation. Plus you can detect attack attempts as the memory controller will report increased error rates. Now you've turned specific access patterns into "…

> Now you've turned specific access patterns into "attack attempts",

We already have error correction detection available to us and it's in use today.

> RAM should always hold what was last written to it, under all conditions of software accesses.

OK? That's what anyone who is advocating for ECC is already advocating for. ECC is error correcting up to 1 bit and error detecting up to 2 bits. The goal is to ensure that the property you've described holds.

> It only got created because the RAM companies have forsaken correctness,

This problem exists because... physics. Do you actually have a design for RAM that can match today's cell density while somehow providing perfect isolation? Saying this problem didn't exist 15 years ago is like saying that malware didn't impact the abacus - RAM 15 years ago was designed completely differently, and had considerably lower capabilities in terms of storage and latency.

And for reference, hamming codes (which back ECC) were created in the 1950s for error correction in punch cards. This is not a new problem.

ECC is the existing, obvious solution to this problem, it just wasn't built for this threat - but it's an excellent, proven method for correcting bit flips. Even without being designed to prevent rowhammer it already makes the attack much slower, less likely to succeed, and easier to detect (just monitor for a massive spike in ecc corrections).

Trying to design high density RAM that maintains perfect isolation seems pointless. Improving ECC or ECC-like approaches is likely going to be much more effective.

Re: Google says Rowhammer attacks are gaining range as RAM is getting denser

#85
post #51

Earlier quoted context omitted.

I am not familiar with DRAM spec sheets, but are manufacturers specifying that there will be zero errors? Without a specification that says so, I don't think it's necessarily the fault of the manufacturer if they cannot build perfect RAM! Suppose someone builds a car with one these computers in a safety-critical role, and then someone gets injured because of an error that "originated" with the RAM.

They specify timing and when it is followed, RAM should work without fault. But if there are corner cases like this, they should be added to specs. Most likely it would require memory controller to remember last addresses and insert delays if rowhammer attempt is detected. And/or make CPU microoperation scheduler avoid it. No idea how expensive would that be, surely nontrivial.

There are already measures like that where the memory controller attempts to identify "victim rows" and refresh them, but attacks can fool the victim row detection and succeed anyway.

Re: Google says Rowhammer attacks are gaining range as RAM is getting denser

#86

Earlier quoted context omitted.

The problem isn't RowHammer, the problem is lack of ECC. This is mostly Intel's fault, as they cripple their consumer CPU lines to disable ECC for market segmentation purposes. The entire premise that we can store tens of gigabytes of information reliably in tiny silicon capacitors without any error detection or correction whatsoever is ludicrous. Every other storage technology uses advanced error correction - NAND F…

ECC is at best a temporary workaround, and at worst an implicit approval of the creation of defective products. It's purpose is to prevent transient bitflips due to cosmic rays and such, not consistently reproducible errors from what is otherwise perfectly normal operation. Plus you can detect attack attempts as the memory controller will report increased error rates. Now you've turned specific access patterns into "…

> ECC is at best a temporary workaround, and at worst an implicit approval of the creation of defective products. It's purpose is to prevent transient bitflips due to cosmic rays and such, not consistently reproducible errors from what is otherwise perfectly normal operation.

No, ECC is there to correct any errors, including normal errors that occur during operation. It allows us to build technologies that have a much higher raw channel error rate, and decrease the system error rate to better than before. In return, we get much higher density, performance, or whatever other metric you want to optimize for.

This is basic information theory. It is extremely inefficient to attempt to build a channel with a low enough raw bit error rate to be usable. Instead we let the bit error rate rise, then correct the errors with error correction codes. This reduces the bandwidth, proportional to error rate for advanced error correction, which is still a negligible amount compared to the total channel bandwidth - the performance improvements from allowing the raw error rate to rise more than make up for this, and result in orders of magnitude better performance.

> Now you've turned specific access patterns into "attack attempts"

Ever heard of files you can't burn on CD-R, or transmit through certain variants of Ethernet? We use technologies that are probabilistic all the time; they assume data is not correlated with a pseudorandom internal characteristic of some sort. These files only exist because someone explicitly targeted them; the chances of someone stumbling onto such data randomly are basically nonexistent (for any well designed system).

That said, this can be mitigated without throwing ECC out the window.

> RAM should always hold what was last written to it, under all conditions of software accesses.

And it will, after ECC. Why are you drawing an arbitrary line for RAM and saying it can't use ECC? Are you also against HDDs (LDPC error correction), Wi-Fi (convolutional codes), NAND storage of all kinds (increasingly advanced ECC), USB (error detection and re-transmission), TCP (error detection and re-transmission), the internet as a whole (flow control by dropping packets), CDs (three layers of error correction for data!), Bluetooth (FEC and/or ARQ), Digital TV (ATSC, DVB, ISDB all use FEC of some kind), QR codes (reed-solomon), and even things like RAID6? The people who designed these systems aren't dumb, they know the only way to get performance and reliability is to use error correction.

It's a damn miracle we get away with non-ECC RAM, today. That stuff should've died 20 years ago, and ECC should be standard. Even the caches inside CPUs often have ECC these days.

> Where manufacturers are trying to sell 2-4x the capacity with 1/4 to 1/16th of the endurance? That's a rant for another thread...

Yes, manufacturers having a race for the bottom to achieve the highest capacity at the expense of all reliability and performance metrics is a completely different problem that doesn't mean we need to throw ECC out the window, which would also hurt all of those metrics. Optimizing for endurance and performance still requires ECC.

The solution is simple: don't buy QLC flash. That stuff's garbage, at least today.

Or would you prefer to go back to the era of 16MB CompactFlash cards (that's MB) using SLC NOR Flash? That stuff doesn't need ECC, certainly, and the endurance is amazing. Let me know how the cost per gigabyte and write performance metrics work out in your NOR world, though :-)

Or perhaps you'd rather have us store all of our data on 16KB EEPROM chips, with even better endurance! I'm sure nobody will mind the 1KB/s write performance (if you're lucky). Maybe if you RAID a few thousand of those it'll work! Oh wait, then random errors will kill your data anyway, without something like RAID-6. Oops!

> This problem didn't exist 15 years ago.

15 years ago the average computer had 256MB of RAM. I had one of those. And I had a bad RAM bit back then, which I had to mask out with a kernel patch (that I had to port from x86 to amd64, as I was an early adopter of 64-bit), which would've been a non-issue with ECC RAM.

Re: Google says Rowhammer attacks are gaining range as RAM is getting denser

#87
post #81

Earlier quoted context omitted.

The problem isn't RowHammer, the problem is lack of ECC. This is mostly Intel's fault, as they cripple their consumer CPU lines to disable ECC for market segmentation purposes. The entire premise that we can store tens of gigabytes of information reliably in tiny silicon capacitors without any error detection or correction whatsoever is ludicrous. Every other storage technology uses advanced error correction - NAND F…

Not here to defend Intel (their decision to remove ECC support from X299 chipset pretty much tanked their own HEDT platform), but I often wonder how much of the current disregard of ECC is the fault of the system integrators. As of now they would gladly ship configurations with single channel memory and under-par cooling to save a few bucks, while these cut corners could hurt performance by as much as 30% due to insu…

This is a self-fulfilling issue; since Intel does not support ECC, ECC is not mainstream, so mainstream platforms don't care about ECC. It would cost motherboard manufacturers ~nothing to support this properly, so if it were accepted as a thing they would, and it would eventually be a requirement for newer RAM technologies. ECC should've been mandatory since about the DDR2 era, practically speaking. That it isn't is a collective failure of the computer industry, and Intel takes a big part of the blame.

At least I had no issues getting ECC to work on my Threadripper box, but sure, that's HEDT.

Re: Google says Rowhammer attacks are gaining range as RAM is getting denser

#88

Earlier quoted context omitted.

Can Rowhammer bypass ECC and not be detected by an hw_event_mc_err_type ? I don't think so. Why would someone have ECC without a sufficiently sophisticated driver?

> Can Rowhammer bypass ECC and not be detected by an hw_event_mc_err_type? It's definitely possible in theory. You'd need four bit flips rather than three, so you'd probably need more time between accesses to the victim row, but thats a quantitative improvement at best. This can be mitigated by using different ECC bit encodings per memory location[0], so hammered data, with correct ECC for its row, always has wrong E…

We should be using memory encryption with random per-boot keys to prevent cold boot attacks, which would also solve the issue. Then the software doesn't know how its data maps to the encrypted data at the RAM. Rows may map 1:1, but you wouldn't know which bits you're targeting.

Re: Google says Rowhammer attacks are gaining range as RAM is getting denser

#89
post #82

Earlier quoted context omitted.

The price of ECC is artificially inflated because it's not available to consumers.

At minimum ECC will cost you an extra 12.5% for the extra bit of RAM. That part is not inflated.

Yeah I wasn't trying to imply it would cost nothing to add ECC.

Re: Google says Rowhammer attacks are gaining range as RAM is getting denser

#90

Earlier quoted context omitted.

The problem isn't RowHammer, the problem is lack of ECC. This is mostly Intel's fault, as they cripple their consumer CPU lines to disable ECC for market segmentation purposes. The entire premise that we can store tens of gigabytes of information reliably in tiny silicon capacitors without any error detection or correction whatsoever is ludicrous. Every other storage technology uses advanced error correction - NAND F…

ECC is at best a temporary workaround, and at worst an implicit approval of the creation of defective products. It's purpose is to prevent transient bitflips due to cosmic rays and such, not consistently reproducible errors from what is otherwise perfectly normal operation. Plus you can detect attack attempts as the memory controller will report increased error rates. Now you've turned specific access patterns into "…

> I'm sure that either it or phenomenon like it are the causes of a lot of odd "glitchy" behaviour that people encounter, because systems are run so close to their limits that there are bound to be cases when they surpass the limits.

I don't know about that. Rowhammer really emphasizes the hammer. A normal read pattern will basically never hit the lines in a small area that much, it will hit cache instead.

> Now you've turned specific access patterns into "attack attempts", and by discriminating against them as such, entirely destroyed a fundamental part of general-purpose computing.

Can you show me any code that causes this kind of access pattern that wasn't deliberately designed to implement rowhammer?

> Where manufacturers are trying to sell 2-4x the capacity with 1/4 to 1/16th of the endurance?

We could go the other way too and glue together multiple SLC cells to increase endurance and cost. Are you claiming there is a specific correct amount of endurance? Because "as much as possible" isn't feasible and "as much as SLC used to have" could only be correct from an extreme coincidence or from conflating nostalgia with correctness. Is there a different number you have in mind?

> This problem didn't exist 15 years ago. It only got created because the RAM companies have forsaken correctness, and somehow managed to convince everyone else that it's not their own fault.

Old ram had errors too. RAM companies can and should put in mechanisms to prevent rowhammer, but if I was buying a stick of ram and had a choice between ECC and perfect rowhammer prevention I'd choose ECC.

Post reply on HN