Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The mysterious case of the Linux Page Table Isolation patches

pythonsweetness.tumblr.com

Post reply on HN
Print view View original

181–190 of 264 posts

Re: The mysterious case of the Linux Page Table Isolation patches

#181

Post by [deleted] »

jchw wrote: post #177
brians wrote: post #136

Earlier quoted context omitted.

No, but it’s straightforward engineering to prevent many of these problems. Error-correcting codes have been well understood for most of a century. Yes, it costs in performance and in money. For life-safety applications, it shouldn’t be optional. Amazon is hosting life-safety applications in EC2. Commodity x86 hardware is grossly negligent for that environment.

ECC as it is today may not be sufficient. It's mostly useful against single bit flips. Probably better than nothing, though...

[deleted]

Re: The mysterious case of the Linux Page Table Isolation patches

#182

Post by krallja »

pkaye wrote: post #180
ajarmst wrote: post #143

Earlier quoted context omitted.

Alpha particle emissions are common causes of single-bit errors, especially from ceramic enclosure materials in integrated circuits. Mitigating soft errors from circuit packaging is an active area of research in materials science. Parity bits and CRC error checking are needed precisely to reduce the impact of these errors down to manageable levels.

"Alpha particles can be stopped by a sheet of paper, beta particles by aluminum, and gamma radiation by a block of lead." This is what I read online. A typical hardware components is enclosed in a aluminum case so really gamma radiation is of concern right?

GP is saying the radiation is emitted by the component package itself, i.e. the random decay of particles in the ceramic surrounding an IC can cause errors.

Re: The mysterious case of the Linux Page Table Isolation patches

#183

Post by ajarmst »

pkaye wrote: post #180
ajarmst wrote: post #143

Earlier quoted context omitted.

Alpha particle emissions are common causes of single-bit errors, especially from ceramic enclosure materials in integrated circuits. Mitigating soft errors from circuit packaging is an active area of research in materials science. Parity bits and CRC error checking are needed precisely to reduce the impact of these errors down to manageable levels.

"Alpha particles can be stopped by a sheet of paper, beta particles by aluminum, and gamma radiation by a block of lead." This is what I read online. A typical hardware components is enclosed in a aluminum case so really gamma radiation is of concern right?

Lantz, "Soft errors induced by Alpha Particles", 1996. http://ieeexplore.ieee.org/document/510798/?reload=true

Re: The mysterious case of the Linux Page Table Isolation patches

#184

Post by yeukhon »

saas_co_de wrote: post #132
brazzledazzle wrote: post #127

Earlier quoted context omitted.

In this specific scenario what's the mitigation for cloud customers? Or even cloud providers? A customer can't migrate all of their infrastructure before a packaged exploit can be distributed. A provider can't dedicate hardware for every single customer. Let's be realistic here.

Get off the cloud. Which is exactly why the companies involved would want to keep it secret. Not everyone is all in on cloud infrastructure. What about people who are right now deciding whether or not to move critical data to the cloud? Should their security be compromised by hiding the truth about a known exploit in order to "protect" people less concerned about security who already put their data at risk?

How is Netflix going to move off the cloud overnight?

If this vulnerability applies regardless of where it is, what is the mitigation then? Move your machine to where? Your suggestion is not practical and you know it. The right mitigation is one that actually will fix the vulnerability. No one shut down machines and migrated everything to another distro because Debian had a bug in generating private key a decade ago even if the bug was zero-day.

Re: The mysterious case of the Linux Page Table Isolation patches

#185

Post by djsumdog »

amluto wrote: post #62
revelation wrote: post #51

Earlier quoted context omitted.

Thats because Gentoo decided to switch on a new compile flag, then didn't bother to test that the kernel still boots: https://lkml.org/lkml/2017/12/29/449

And, for reasons that are entirely unknown, the issue got worse due to one of the PTI patches (written by, and hence tentatively blamed on, yours truly). Presumably it caused some minor change in code generation causing GCC to go nuts. FWIW, the compile flag that Gentoo enabled activates a seriously busted GCC feature, and I'm a bit surprised that Gentoo gets away with it in user code.

Which compile flag is that? (I run Gentoo)

Edit: nevermind. It's the -fstack-check flag mentioned in the other comments.

Re: The mysterious case of the Linux Page Table Isolation patches

#186

Post by sd8dgf8ds8g8dsg »

acoye wrote: post #156

Ok so all I hear is Intel and no trace of AMD? So X86_64 ISA wise the only diff between modern CPUs are AVX512. I bet you within the foundations of AVX512 lies a nasty one that can't be patched with microcode update.

I'm confused, what makes you think the AMD and Intel cpu internals are the same?

Re: The mysterious case of the Linux Page Table Isolation patches

#187

Post by caf »

amluto wrote: post #62
revelation wrote: post #51

Earlier quoted context omitted.

Thats because Gentoo decided to switch on a new compile flag, then didn't bother to test that the kernel still boots: https://lkml.org/lkml/2017/12/29/449

And, for reasons that are entirely unknown, the issue got worse due to one of the PTI patches (written by, and hence tentatively blamed on, yours truly). Presumably it caused some minor change in code generation causing GCC to go nuts. FWIW, the compile flag that Gentoo enabled activates a seriously busted GCC feature, and I'm a bit surprised that Gentoo gets away with it in user code.

Is there anywhere to read up on the bustedness of the stack probing feature? (apart from the obvious incompatibility with trying to do that for kernel code).

Re: The mysterious case of the Linux Page Table Isolation patches

#188

Post by foobarbazetc »

saas_co_de wrote: post #166
foobarbazetc wrote: post #152

Earlier quoted context omitted.

As someone who shares your skepticism of the cloud, I can say that people don’t switch from bare metal hosting (something like SoftLayer) to AWS/GCP for the cost. If you do the math like “we have 1000 cores and 2048Gb of RAM and 10Tb of RAID’ed SSD” and then plug that in to the GCP calculator... it’s going to be at minimum 1.5-2x your bare metal cost. That’s not even including bandwidth which is pretty much free at b…

Totally agree. Cloud makes tons of sense if your workload is really dynamic. Lots of small players are running static workloads though because actually setting up dynamic workloads is pretty complex. I use GCE for DNS, Storage, CDN (for fronting storage backed files), dynamic workloads that can run on preemptible instances, and scalable instances to serve published static content, but I use dedicated servers for data…

Yeah we’re medium size but still bare metal at IBM/SoftLayer.

We keep looking at GCP waiting for the pricing to make sense and still trying to figure out how people run low latency Postgres on there. :)

Re: The mysterious case of the Linux Page Table Isolation patches

#189

Post by brians »

jchw wrote: post #177
brians wrote: post #136

Earlier quoted context omitted.

No, but it’s straightforward engineering to prevent many of these problems. Error-correcting codes have been well understood for most of a century. Yes, it costs in performance and in money. For life-safety applications, it shouldn’t be optional. Amazon is hosting life-safety applications in EC2. Commodity x86 hardware is grossly negligent for that environment.

ECC as it is today may not be sufficient. It's mostly useful against single bit flips. Probably better than nothing, though...

ECC alone is absolutely insufficient. But ECC can be part of a system design that includes active monitoring and response. I’d expect that system design to also include measurement of ECC events under ordinary conditions, regular re-measurement, and funding for an analysis of the changes and explanation of the difference—just like you’d find in safety engineering in a coal plant, an MRI machine, any sort of engineering that has a professional scientist or engineer on site supervising all operations.

Of course, you’ll also find a tendency there towards specified hardware. They bend or break it to use COTS x86 machines, but—as I think I heard from a comment here last week—nearly nobody ever specified wanting AMT in the initial design, so it’s pretty weird that we’re all buying and deploying it.

Re: The mysterious case of the Linux Page Table Isolation patches

#190

Post by Rebellos »

lolc wrote: post #146
hathawsh wrote: post #106

Earlier quoted context omitted.

OTOH, a rowhammer attack on ECC memory will likely flip 1 bit before it flips 2, making attacks theoretically detectable. Without ECC, there's no clear way to detect an attack.

I'd assume that parity is checked on access, which may give enough time to flip more than one bit before it's detected.

ECC memory controller performs memory scrubbing periodically, in the background, during which it checks parity and corrects any bitflips. Otherwise ECC would not work nearly as well as it does.
Post reply on HN