Earlier quoted context omitted.
> live patching should become part of the linux kernel Services where uptime matters tend to be designed so they can tolerate the reboot of a single node for other reasons besides kernel maintenance. I can't imagine a situation where I can't tolerate the downtime of a reboot but I would be willing to risk the system locking up with brain surgery gone wrong.
> Services where uptime matters tend to be designed so they can tolerate the reboot of a single node for other reasons besides kernel maintenance. I can't imagine a situation where I can't tolerate the downtime of a reboot but I would be willing to risk the system locking up with brain surgery gone wrong. I've run systems with live code updates for userland, and would have considered live kernel updates if it was rea…
Copy Fail, Dirty Frag, and Fragnesia kernel vulnerabilities
41–50 of 55 posts
Re: Copy Fail, Dirty Frag, and Fragnesia kernel vulnerabilities
#42Expanding on gentoo's recommendations: I wonder if we should just universally accept that live patching should become part of the linux kernel? An automatic job that updates (much like some system packages in some distros) that installs (signed) live patches from upstream? Of course we would run into a problem where a malicious patch can now be distributed reliably to hundreds of thousands of machines, but we already…
> live patching should become part of the linux kernel Services where uptime matters tend to be designed so they can tolerate the reboot of a single node for other reasons besides kernel maintenance. I can't imagine a situation where I can't tolerate the downtime of a reboot but I would be willing to risk the system locking up with brain surgery gone wrong.
Consider a hyper-converged cluster with many nodes serving distributed block storage, say at N=3 replication. This can tolerate exactly one N=1 node of outage for the reboot. It would seem preferable to drain the nodes in a way that allows for more parallelism in the per-node kernel-reboot process, but draining is expensive and its cheaper to reboot and hope the data comes back to the pool within some period of time after the reboot. This gets worse linearly as the cluster grows.
A non-trivial size cluster facing this can have a reboot rollout easily stretch from hours into days and even weeks. It is further made slower when the roll-out itself is repeatedly paused when any other production issue is detected, or some other in-cluster event is happening and distributed storage health is degraded or unavailable. If a single (additional) node goes out during the reboot roll-out, data goes unavailable and storage must wait and heal. It also simply takes time for the cluster to reconcile when the storage eventually comes back from reboot to make sure it is all still there.
If your systems are large enough, things will go so slow that things fall into the trap where the target release changes mid-deployment: to benefit from everything learned in the last many days or weeks, security, performance, crashes, whatever! There is benefit because the fixes you cared about most got onto a portion of the cluster sooner than later. There is also penalty, as this resets the time it takes to deploy, elongating the perceived end-to-end deployment time. This negatively affects OKRs and similarly displaces the release of anything that was queued for upcoming releases.
So yeah, live patching is great to get priority fixes out in a matter of minutes or hours. I also think it is the best tool to get oneself out of this rollout-reset trap and onto the next release sooner. Faster than rollback or rollover.
Re: Copy Fail, Dirty Frag, and Fragnesia kernel vulnerabilities
#43Expanding on gentoo's recommendations: I wonder if we should just universally accept that live patching should become part of the linux kernel? An automatic job that updates (much like some system packages in some distros) that installs (signed) live patches from upstream? Of course we would run into a problem where a malicious patch can now be distributed reliably to hundreds of thousands of machines, but we already…
> live patching should become part of the linux kernel Services where uptime matters tend to be designed so they can tolerate the reboot of a single node for other reasons besides kernel maintenance. I can't imagine a situation where I can't tolerate the downtime of a reboot but I would be willing to risk the system locking up with brain surgery gone wrong.
Re: Copy Fail, Dirty Frag, and Fragnesia kernel vulnerabilities
#44> We recommend exploring ways to automate upgrading your kernel Like, running emerge -u @world on a regular basis, or ... /me searches Okay, so https://wiki.gentoo.org/wiki/Live_patching exists but says, > A note of caution: Kernel live patching is risky. Count on hard freezing or panics to become normal... That's not encouraging. --- Another approach: Can we make the kernel vulns less important? Has anyone had luck…
If we are looking at things like gvisor or firecracker, SELinux might be an alternative. From what I can see, SELinux prevented both copy fail and dirty frag, and maybe also fragnesia but I couldn't find any definitive answer on that one. Last time I tried it was a pain to setup and a pain to use, but as a sysadmin there is a lot of thing that share those attributes. The only question if its worth it. If the current…
I presume you are referring to the GrapheneOS post/thread about this[0], although this implementation is not the same implementation we see on Fedora or Debian for example and it appears these distros were (and are) still vulnerable to this exploit, with the out of the box configuration of SELinux on these systems.
Re: Copy Fail, Dirty Frag, and Fragnesia kernel vulnerabilities
#45Earlier quoted context omitted.
>but then package those modules into separate packages. If you don't need somedriver.ko, then you don't `apt install linux-driver-somedriver But I don't want to know what drivers I need and will need next. Tomorrow I could buy a different wifi module and then what? Spend 3 hours googling which rtl378326973268632aahaxhabt.ko to install? Thanks but no thanks.
On older versions of Windows you used to get popups saying new hardware is detected, would you like to install the driver now?
Re: Copy Fail, Dirty Frag, and Fragnesia kernel vulnerabilities
#46Earlier quoted context omitted.
Why would the source of the patches be less trusted than the source of (updated) kernels? I expect it to be the same, your distro. $200/year is peanuts for any commercial use worth the name. The problem, of course, is the whole non-free infrastructure it has to introduce. I wonder when large and critical OSS projects will start to be seen as a public good they are, with large corporations willingly financing them bec…
Public goods are not generally funded by large corporations.
Re: Copy Fail, Dirty Frag, and Fragnesia kernel vulnerabilities
#47> We recommend exploring ways to automate upgrading your kernel Like, running emerge -u @world on a regular basis, or ... /me searches Okay, so https://wiki.gentoo.org/wiki/Live_patching exists but says, > A note of caution: Kernel live patching is risky. Count on hard freezing or panics to become normal... That's not encouraging. --- Another approach: Can we make the kernel vulns less important? Has anyone had luck…
Local kernel vulns are totally unimportant in any vaguely reasonable environment.
Re: Copy Fail, Dirty Frag, and Fragnesia kernel vulnerabilities
#48Earlier quoted context omitted.
> live patching should become part of the linux kernel Services where uptime matters tend to be designed so they can tolerate the reboot of a single node for other reasons besides kernel maintenance. I can't imagine a situation where I can't tolerate the downtime of a reboot but I would be willing to risk the system locking up with brain surgery gone wrong.
> Services where uptime matters tend to be designed so they can tolerate the reboot of a single node for other reasons besides kernel maintenance. I can't imagine a situation where I can't tolerate the downtime of a reboot but I would be willing to risk the system locking up with brain surgery gone wrong. I've run systems with live code updates for userland, and would have considered live kernel updates if it was rea…
As a sibling says, you need a "reconnect now" in the protocol. (GOAWAY, in HTTP.)
In addition to what the sibling says, if you have some sort of cordoning/graceful drain facility at the traffic level, you can also prevent the "several times" bit: bring new, patched nodes online. Disallow new connections to the outgoing nodes. Drain the outgoing nodes. Decommission them.
(I.e., only permit reconnects to patched nodes.)
Re: Copy Fail, Dirty Frag, and Fragnesia kernel vulnerabilities
#49> We recommend exploring ways to automate upgrading your kernel Like, running emerge -u @world on a regular basis, or ... /me searches Okay, so https://wiki.gentoo.org/wiki/Live_patching exists but says, > A note of caution: Kernel live patching is risky. Count on hard freezing or panics to become normal... That's not encouraging. --- Another approach: Can we make the kernel vulns less important? Has anyone had luck…
> Can we make the kernel vulns less important? Local kernel vulns are totally unimportant in any vaguely reasonable environment.
Then they are forgetting how much untrusted software they are running as their user account that can do much damage without need to do privilege escalation to root.
Re: Copy Fail, Dirty Frag, and Fragnesia kernel vulnerabilities
#50> We recommend exploring ways to automate upgrading your kernel Like, running emerge -u @world on a regular basis, or ... /me searches Okay, so https://wiki.gentoo.org/wiki/Live_patching exists but says, > A note of caution: Kernel live patching is risky. Count on hard freezing or panics to become normal... That's not encouraging. --- Another approach: Can we make the kernel vulns less important? Has anyone had luck…
> Can we make the kernel vulns less important? Local kernel vulns are totally unimportant in any vaguely reasonable environment.