This was even worse than the headline made it sound. If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed…
Sounds about right. And not the first time it happens either. I recall getting a few of those instant unit 3 panic over the past few years with Ubuntu. Often with things not as common out there in production, like tc (which in our case we were using in production to work around conntrack race conditions), and sometimes we also got non-panicking but absolutely production/nerve wrecking issues like TCP window size calc…
An Ubuntu kernel bug causes container crashes
71–80 of 133 posts
Re: An Ubuntu kernel bug causes container crashes
#72Auto update strikes again. Really, we need to re-think this.
There is no need to rethink it; it's never been a good idea to leave it turned on.
Re: An Ubuntu kernel bug causes container crashes
#73Earlier quoted context omitted.
Or a real server with Lights Out Management.
That’s why “usually” is in the sentence. :) Most smaller teams usually don’t prioritize physical access — they usually only need it for one-off events. While this would be a one-off event, it would be one that affects many servers.
I'm not sure I've ever worked with any (2008-present) that don't in any case.
Re: An Ubuntu kernel bug causes container crashes
#74Wow, another buggy Ubuntu patch breaks something. Why don’t they just stick to what’s upstream?
Yeah, I don't really get why they don't use the stable kernel releases, of which there are many, rather than rolling their own.
But I assume as Ubuntu follows an April release schedule, it doesn't always match with an appropriate LTS kernel.
Re: An Ubuntu kernel bug causes container crashes
#75Earlier quoted context omitted.
Sounds about right. And not the first time it happens either. I recall getting a few of those instant unit 3 panic over the past few years with Ubuntu. Often with things not as common out there in production, like tc (which in our case we were using in production to work around conntrack race conditions), and sometimes we also got non-panicking but absolutely production/nerve wrecking issues like TCP window size calc…
It really seems like there’s no good non-redhat (but still “production capable”) alternative to CoreOS nowadays, right? It’s pretty much Fedora / Redhat CoreOS or go directly to things such as k3os?
Elemental is pretty close to coreos: https://github.com/rancher/elemental/
They even have a way to build arbitrary os images: https://github.com/rancher/elemental-toolkit
It's pretty great
Re: An Ubuntu kernel bug causes container crashes
#76For example, looking at the package for postgresql-14, an update still hasn't been released for the unscheduled mid-June release version 14.4, which fixed possible index corruption.
http://changelogs.ubuntu.com/changelogs/pool/main/p/postgres...
I would have thought this would have been packaged earlier, as I would expect the Ubuntu + postgresql would be a common combination.
It makes me wonder exactly how much of a resource is behind creating Ubuntu distributions.
Re: An Ubuntu kernel bug causes container crashes
#77I like Ubuntu, but in the last few months I have been following how things are packaged more closely. For example, looking at the package for postgresql-14, an update still hasn't been released for the unscheduled mid-June release version 14.4, which fixed possible index corruption. http://changelogs.ubuntu.com/changelogs/pool/main/p/postgres... I would have thought this would have been packaged earlier, as I would e…
Re: An Ubuntu kernel bug causes container crashes
#78Copy-pasta of Jonathan Corbet: It's nice to see LWN on HN ... but please remember: it is only LWN subscribers that make this kind of writing possible. If you are enjoying it, please consider becoming a subscriber yourself — or, even better, getting your employer to subscribe. https://news.ycombinator.com/item?id=31852477 If you're interested in detailed commentary on and investigations of the FOSS space, I can't reco…
Re: An Ubuntu kernel bug causes container crashes
#79Earlier quoted context omitted.
Sounds about right. And not the first time it happens either. I recall getting a few of those instant unit 3 panic over the past few years with Ubuntu. Often with things not as common out there in production, like tc (which in our case we were using in production to work around conntrack race conditions), and sometimes we also got non-panicking but absolutely production/nerve wrecking issues like TCP window size calc…
It really seems like there’s no good non-redhat (but still “production capable”) alternative to CoreOS nowadays, right? It’s pretty much Fedora / Redhat CoreOS or go directly to things such as k3os?
Re: An Ubuntu kernel bug causes container crashes
#80Earlier quoted context omitted.
There is no need to rethink it; it's never been a good idea to leave it turned on.
Well, you're between a rock and a hard place. No auto-update = security risk exposure, auto-update = stability risk exposure (and sometimes security risk exposure thrown in for free as well).