Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Ubuntu kernel bug causes container crashes

lwn.net

Post reply on HN
Print view View original

1–10 of 133 posts

Re: An Ubuntu kernel bug causes container crashes

#3

Post by markstos »

This was even worse than the headline made it sound.

If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up.

First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed at boot, since the containers share the kernel with the host.

At that point, you can't SSH in and have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run.

And then of course if you revert the kernel upgrade, you were once again vulnerable to whatever problem the security update was fixing...

Re: An Ubuntu kernel bug causes container crashes

#5

Post by mroche »

Copy-pasta of Jonathan Corbet:

It's nice to see LWN on HN ... but please remember: it is only LWN subscribers that make this kind of writing possible. If you are enjoying it, please consider becoming a subscriber yourself — or, even better, getting your employer to subscribe.

https://news.ycombinator.com/item?id=31852477

If you're interested in detailed commentary on and investigations of the FOSS space, I can't recommend a subscription to LWN enough!

Re: An Ubuntu kernel bug causes container crashes

#6

Post by up6w6 »

I'm using Oracle's ARM servers and I thought it was some weird patch they did to the kernel, the bug only disappeared when I force upgraded it to 22.04. Ubuntu/Canonical itself would be the last place I would have thought to be the source of a problem like that.

Re: An Ubuntu kernel bug causes container crashes

#7

Post by Spivak »

markstos wrote: post #3

This was even worse than the headline made it sound. If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed…

I know it’s too late for a bunch of shops but for gods sake please don’t use unattended upgrades to do your patching unless you want to hate you life and chase down hard to find hard to undo bugs.

Build your images in CI job and have your deploy version be (code version, image version) so patching runs through all the same tests your code does and you have a trivial roll-forward to undo any mess you find yourself in.

Re: An Ubuntu kernel bug causes container crashes

#8

Post by yjftsjthsd-h »

Spivak wrote: post #7
markstos wrote: post #3

This was even worse than the headline made it sound. If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed…

I know it’s too late for a bunch of shops but for gods sake please don’t use unattended upgrades to do your patching unless you want to hate you life and chase down hard to find hard to undo bugs. Build your images in CI job and have your deploy version be (code version, image version) so patching runs through all the same tests your code does and you have a trivial roll-forward to undo any mess you find yourself in.

> don’t use unattended upgrades

> Build your images in CI job

I know container images should generally be immutable, but I would expect unattended upgrades to be mostly used on the host, not in a container, in which that management system doesn't really work (unless you're doing VMs where you can deploy immutable root images to the VMs as well, or some fun bare metal + PXE combination).

Post reply on HN