An Ubuntu kernel bug causes container crashes • Hacker News Board

Re: An Ubuntu kernel bug causes container crashes

#101

Post by markstos » Wed, Jul 06, 2022, 2:03 PM UTC

Spivak wrote: post #7

markstos wrote: post #3

This was even worse than the headline made it sound. If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed…

I know it’s too late for a bunch of shops but for gods sake please don’t use unattended upgrades to do your patching unless you want to hate you life and chase down hard to find hard to undo bugs. Build your images in CI job and have your deploy version be (code version, image version) so patching runs through all the same tests your code does and you have a trivial roll-forward to undo any mess you find yourself in.

Sorry, not my experience.

My experience has been that by the time I notice some serious vulnerability is in the news, my servers have already patched themselves. I have never "hated life" or had a "hard to find and undo bug" due to automatic security patching. I pretty quickly found what caused this and had a clear path to resolution.

This is the first security update that caused a boot failure in about a decade. It was bad, but it didn't change my mind about unattended-upgrades. My takeaway that if that maybe I should have upgraded my 20.04 servers to 22.04 server sooner.

Re: An Ubuntu kernel bug causes container crashes

#102

Post by markstos » Wed, Jul 06, 2022, 2:08 PM UTC

jve wrote: post #84

atoav wrote: post #16

The cost of complexity showing itself. A sysadmin friend of mine is totally against docker and his reason is that he wants as little complexity as is needed on his systems. Complexity, he says, leads to emergent behavior.

Docker actually helps managing complexity, by putting bits and pieces, scattered on the floor and putting them into a single cardboard box. - If you throw the box out, you know you did no harm to other boxes. - If you change your floor, you know you didn't wipe out something useful. - Aaand you can `git switch` to a well known state Ofcourse it's not 100% like that, in reality you still have to have some kind of cons…

Containers help tame complexity, but the container itself could be run without the docker runtime, using systemd-nspawn, or run as a regular systemd service using podman.

It seems less complex to manage a bunch of systemd services than one pile of systemd services that are managed and logged one way and bunch of docker services that are managed and logged another way.

Re: An Ubuntu kernel bug causes container crashes

#103

Post by _ea1k » Wed, Jul 06, 2022, 2:08 PM UTC

jve wrote: post #84

atoav wrote: post #16

The cost of complexity showing itself. A sysadmin friend of mine is totally against docker and his reason is that he wants as little complexity as is needed on his systems. Complexity, he says, leads to emergent behavior.

Docker actually helps managing complexity, by putting bits and pieces, scattered on the floor and putting them into a single cardboard box. - If you throw the box out, you know you did no harm to other boxes. - If you change your floor, you know you didn't wipe out something useful. - Aaand you can `git switch` to a well known state Ofcourse it's not 100% like that, in reality you still have to have some kind of cons…

Exactly, you get many of the benefits of separate boxes without some of the added complexity. Docker and containers in general can reduce complexity significantly in a lot of cases.

Re: An Ubuntu kernel bug causes container crashes

#104

Post by markstos » Wed, Jul 06, 2022, 2:10 PM UTC

Fnoord wrote: post #50

atoav wrote: post #27

Earlier quoted context omitted.

Or it could not be triggered if you don't use containers.

Would it have occurred on Podman as well?

Some of my servers were running Ubuntu with systemd+podman to manage services, and none of them had a problem.

Re: An Ubuntu kernel bug causes container crashes

#105

Post by throwaway787544 » Wed, Jul 06, 2022, 2:14 PM UTC

klqr wrote: post #33

throwaway787544 wrote: post #26

Earlier quoted context omitted.

Complexity is just another part of natural systems. It's not something to be avoided for it's own sake, in as much as we should avoid having eyeballs, because they are crazily complex. Yet we tend to like ours and find them worth whatever cost they incur. The emergent behavior of containerization has had an overall positive effect, even if it has annoying costs.

For the end user there is no positive effect. Many websites were better in 2005 and had greater uptime. They were also better organized. Ebay and Amazon were leaner and more pleasant to use.

We can now push 15 different apps to prod on the same server with completely different base distributions and dependencies, and they'll run the same as they did on a dev's laptop. The apps & servers don't crash as much as they used to when the app or packages would break on an update, and somebody had to juggle actual dependencies or run separate VMs or physical servers to run all those apps, and the configuration management that used to hose the box when it was misconfigured now no longer exists. The site is much more reliable now, and more dynamic, as it can be updated more frequently with random tech.

When was the last time you saw a weekly "Our website is down for maintenance for the next 2 days" message, other than for some government website still running COBOL on a mainframe? When was the last time you saw 500 errors? Used to be a daily thing.

Re: An Ubuntu kernel bug causes container crashes

#106

Post by zerop » Wed, Jul 06, 2022, 2:53 PM UTC

wronglyprepaid wrote: post #34

zerop wrote: post #14

Any good documentation that talks about how big open source software manage code changes, releases cadences, given contributors from across the world.

I'm fairly sure this differs for different project/organization, not sure there is a rule, and not sure there are really any considerations that are specific to open source, good practices are good practices regardless. That being said, I rate Canonical's practices as rather poor.

Taking an example, how are Linux Kernel releases are planned and managed

Re: An Ubuntu kernel bug causes container crashes

#107

Post by bravetraveler » Wed, Jul 06, 2022, 2:58 PM UTC

capableweb wrote: post #58

akx wrote: post #25

Earlier quoted context omitted.

> have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run. Or add `systemd.mask=docker.service` to your boot parameters to prevent Docker from starting.

Which, if your server is stuck in a infinitive "boot -> docker starting -> container starting -> crashing kernel -> reboot" loop, you won't ever get a chance of actually adding anything to your boot parameters.

Out of band management is common and highly recommended

Re: An Ubuntu kernel bug causes container crashes

#108

Post by bravetraveler » Wed, Jul 06, 2022, 3:00 PM UTC

withinboredom wrote: post #42

_y5hn wrote: post #21

Earlier quoted context omitted.

Wouldn't rollback of kernel be a choice in grub menu? It's pretty standard for all distros to have that choice.

That usually requires physical access to the server to select it during boot.

Not really, console access through IPMI found on most servers

Exceptions tend to be white boxes built with desktop components, at which point, yea. The proverbial You asked for this problem

Re: An Ubuntu kernel bug causes container crashes

#109

Post by treesknees » Wed, Jul 06, 2022, 3:04 PM UTC

baggy_trough wrote: post #19

symlinkk wrote: post #9

Wow, another buggy Ubuntu patch breaks something. Why don’t they just stick to what’s upstream?

Yeah, I don't really get why they don't use the stable kernel releases, of which there are many, rather than rolling their own.

Agreed. Ubuntu does use a stable kernel by default for LTS, at least for ISO installs. This problem occurred within the HWE (hardware enablement) release train, where they backport non-LTS kernels and features, which for some reason they use as a default in various places like their official cloud images.

Re: An Ubuntu kernel bug causes container crashes

#110

Post by solarkraft » Wed, Jul 06, 2022, 3:32 PM UTC

markstos wrote: post #102

jve wrote: post #84

Earlier quoted context omitted.

Docker actually helps managing complexity, by putting bits and pieces, scattered on the floor and putting them into a single cardboard box. - If you throw the box out, you know you did no harm to other boxes. - If you change your floor, you know you didn't wipe out something useful. - Aaand you can `git switch` to a well known state Ofcourse it's not 100% like that, in reality you still have to have some kind of cons…

Containers help tame complexity, but the container itself could be run without the docker runtime, using systemd-nspawn, or run as a regular systemd service using podman. It seems less complex to manage a bunch of systemd services than one pile of systemd services that are managed and logged one way and bunch of docker services that are managed and logged another way.

Deviation from standards is what I find to cause a lot (the most?) of problems. Complexity generally makes things more error-prone, but it tends to become less of a problem when in a well maintained system a lot of people use (someone, maybe yourself, has already had and solved a problem you're having).

If your standard is dealing with systemd units, it may make sense to make your containers conform to that (thereby deviating from the most common way of managing containers). Maybe it's what I'd do in a larger operation. For my personal use I find it the most pragmatic to just use Docker since it's reasonably well documented, has reasonably low friction in usage and is very easy to set up.