Wow, another buggy Ubuntu patch breaks something. Why don’t they just stick to what’s upstream?
According to the article, looks reasonable modification (though it's hard) but should be tested.
An Ubuntu kernel bug causes container crashes
11–20 of 133 posts
Re: An Ubuntu kernel bug causes container crashes
#12This was even worse than the headline made it sound. If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed…
Sounds about right. And not the first time it happens either. I recall getting a few of those instant unit 3 panic over the past few years with Ubuntu. Often with things not as common out there in production, like tc (which in our case we were using in production to work around conntrack race conditions), and sometimes we also got non-panicking but absolutely production/nerve wrecking issues like TCP window size calculation overflows after the window went to zero due to a temporary slow consumer - freezing the window size to a few bytes only instead of getting a prompt full window recovery.
Not to mention we’ve also had our fair share of production triple faults from bugs in the Intel firmware patches for Spectre, which took weeks to investigate & fix between ourselves struggling to keep our exchange up & running, Intel, and AWS.
And that is why there’s value in the CoreOS/ContainerLinux-like solutions we designed & implemented nearly a decade ago now. Being able to promptly rollback any kernel/system/package upgrades at once - either manually or either after it’s detected a few panics in quick successions is actually quite awesome. Not to mention the slow update rollout strategy baked into the Omaha controller.
But the reality is that the what-ifs are always the hardest to market, nearly always after-thoughts and with fast-spiking/fast-decaying traction after major events.
Re: An Ubuntu kernel bug causes container crashes
#13Earlier quoted context omitted.
I know it’s too late for a bunch of shops but for gods sake please don’t use unattended upgrades to do your patching unless you want to hate you life and chase down hard to find hard to undo bugs. Build your images in CI job and have your deploy version be (code version, image version) so patching runs through all the same tests your code does and you have a trivial roll-forward to undo any mess you find yourself in.
> don’t use unattended upgrades > Build your images in CI job I know container images should generally be immutable, but I would expect unattended upgrades to be mostly used on the host, not in a container, in which that management system doesn't really work (unless you're doing VMs where you can deploy immutable root images to the VMs as well, or some fun bare metal + PXE combination).
Idea is that your deploy step should handle both deploying code as well as upgrading OS, so all changes go through same pipeline.
Re: An Ubuntu kernel bug causes container crashes
#14Any good documentation that talks about how big open source software manage code changes, releases cadences, given contributors from across the world.
Re: An Ubuntu kernel bug causes container crashes
#15This seems like the kind of thing that automated testing should have been able to catch. It’s not like running Docker is a small use-case these days.
My thoughts exactly, the details of what this bug is about technically are interesting and fascinating but the key take away is that something went terribly wrong with Ubuntu's testing processes. This should not have shipped without more scrutiny. Somebody presumably cut some corners there and it's worrying that that is possible at all.
I actually rolled out Ubuntu 22.04 to a few servers a few weeks ago. Pretty uneventful update, all my Ansible scripts for 20.04 worked without modification against these new servers. So, I guess I dodge this bug for now. One reason I've always preferred Ubuntu over Red Hat for servers is that with Red Hat/Centos essentially everything I care about is perpetually and hopelessly out of date and obsolete. So, it just creates a lot of hassle to work around that and get reasonably current versions of things I actually need my servers to run. With Ubuntu that was always a lot more straightforward.
I currently write this on a laptop with Manjaro and Linux 5.18. I'm glad I don't have to deal with about a year of long fixed issues with hardware, bluetooth, GPUs, performance, etc. IMHO there's very little value in sticking with older kernels on desktop machines. Especially when that involves a convoluted process of back-porting and integrating lots of complicated patches. I recently put Ubuntu on an old imac (secure boot prevents booting Manjaro) and I promptly ran into hardware issues that I recall having with Manjaro a few months ago that were fixed by simply upgrading the kernel. Bluetooth especially seems way more flaky. And that's not exactly flawless on 5.18 either. I get the if it ain't broke don't fix it thing; my point is that with modern Desktop Linux things being broken is a constant. The least broken version of Linux is usually the kernel that was just released that has all the cumulative fixes for all the issues addressed in previous kernel releases. Opting out of a few years of those fixes seems misguided.
Even on servers, I suspect simply updating the kernel more regularly would not be the end of the world for most users. With an incubation period to catch bugs/blocking issues of course, the more people use a kernel version, the more stable it gets. I doubt many users would experience any regressions. And it's a lot cheaper to support. If I had the option, I don't think I would opt to run 2-3 year old kernels on any of my servers if I had a different choice. I don't see the value of opting out of 2-3 years worth of known & fixed stability, performance, and other issues.
Re: An Ubuntu kernel bug causes container crashes
#16The cost of complexity showing itself.
A sysadmin friend of mine is totally against docker and his reason is that he wants as little complexity as is needed on his systems. Complexity, he says, leads to emergent behavior.
Re: An Ubuntu kernel bug causes container crashes
#17The cost of complexity showing itself. A sysadmin friend of mine is totally against docker and his reason is that he wants as little complexity as is needed on his systems. Complexity, he says, leads to emergent behavior.
This is not a docker bug, it is a kernel bug.
It could be triggered by other complex applications that use kernel container features.
Re: An Ubuntu kernel bug causes container crashes
#18This was even worse than the headline made it sound. If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed…
And while I haven't had this happen to me yet; the fear of something like this or even worse is why I try to stay one step behind the update paths on linux distros.
Security patches matter, but I'm no one important, so I should be fine to wait a week or month...
Anyone else who is important though... servers for example...
Re: An Ubuntu kernel bug causes container crashes
#19Wow, another buggy Ubuntu patch breaks something. Why don’t they just stick to what’s upstream?
Yeah, I don't really get why they don't use the stable kernel releases, of which there are many, rather than rolling their own.
Re: An Ubuntu kernel bug causes container crashes
#20They are just trying to convert docker users to snap enthusiasts.