Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Ubuntu kernel bug causes container crashes

lwn.net

Post reply on HN
Print view View original

111–120 of 133 posts

Re: An Ubuntu kernel bug causes container crashes

#111

Post by solarkraft »

dncornholio wrote: post #89
jve wrote: post #84

Earlier quoted context omitted.

Docker actually helps managing complexity, by putting bits and pieces, scattered on the floor and putting them into a single cardboard box. - If you throw the box out, you know you did no harm to other boxes. - If you change your floor, you know you didn't wipe out something useful. - Aaand you can `git switch` to a well known state Ofcourse it's not 100% like that, in reality you still have to have some kind of cons…

What you are saying is actual complexity. A box can be a machine instead of container and all your points will still stand but with less complexity.

Having all of your services in the same box opens you up to a ton of possible unwanted cross-interaction, which is complexity.

(just not from the system perspective, but from the "reasoning about it" perspective)

Re: An Ubuntu kernel bug causes container crashes

#112

Post by compsciphd »

ubuntu 22.04 also broke many IBM laptops. Took them 2 months to fix it, without any acknowledgement that the bug existed.

number of tickets in launchpad such as https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1970957

It makes me Q the value in my org looking into an ubuntu advantage subscription. When there are tickets that have lots of "me too" that result in unusable laptops, one should at least triage them / consolidate them into a single ticket and then be able to mark when fixed.

Re: An Ubuntu kernel bug causes container crashes

#113

Post by lproven »

scns wrote: post #28
jillesvangurp wrote: post #15

Earlier quoted context omitted.

My thoughts exactly, the details of what this bug is about technically are interesting and fascinating but the key take away is that something went terribly wrong with Ubuntu's testing processes. This should not have shipped without more scrutiny. Somebody presumably cut some corners there and it's worrying that that is possible at all. I actually rolled out Ubuntu 22.04 to a few servers a few weeks ago. Pretty uneve…

For a more recent Kernel on Ubuntu i can recommend Xanmod. https://xanmod.org

That is specifically and explicitly intended for workstations, i.e. desktops and laptops... not for servers.

Re: An Ubuntu kernel bug causes container crashes

#115

Post by AtlasBarfed »

sofixa wrote: post #39
markstos wrote: post #3

This was even worse than the headline made it sound. If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed…

> If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. Isn't the common wisdom that you should have them enabled, but staggered across hours/days?

Probably better you have rolling A/B replacements that stop the replacement run if the replacement doesn't come up.

This is mostly an in-place upgrade issue?

Re: An Ubuntu kernel bug causes container crashes

#116

Post by lproven »

heurisko wrote: post #74
baggy_trough wrote: post #19

Earlier quoted context omitted.

Yeah, I don't really get why they don't use the stable kernel releases, of which there are many, rather than rolling their own.

22.04 uses the 5.15 LTS kernel. https://www.kernel.org/category/releases.html But I assume as Ubuntu follows an April release schedule, it doesn't always match with an appropriate LTS kernel.

Ubuntu was originally designed as a desktop OS and its release cycle was synched with the GNOME 2.x release cycle.

Re: An Ubuntu kernel bug causes container crashes

#117

Post by Spivak »

markstos wrote: post #101
Spivak wrote: post #7

Earlier quoted context omitted.

I know it’s too late for a bunch of shops but for gods sake please don’t use unattended upgrades to do your patching unless you want to hate you life and chase down hard to find hard to undo bugs. Build your images in CI job and have your deploy version be (code version, image version) so patching runs through all the same tests your code does and you have a trivial roll-forward to undo any mess you find yourself in.

Sorry, not my experience. My experience has been that by the time I notice some serious vulnerability is in the news, my servers have already patched themselves. I have never "hated life" or had a "hard to find and undo bug" due to automatic security patching. I pretty quickly found what caused this and had a clear path to resolution. This is the first security update that caused a boot failure in about a decade. It…

You’re conflating unattended-upgrades (server mutability, hard to roll back) with automated patching in general. Do automated patching but also run the changes though your CI so you can catch breaking changes and roll them out in a way that’s easy to debug (you can diff images) and revert.

I bet when you update your software dependencies you run those changes through your tests but your OS is a giant pile of code that usually gets updated differently and independently because mostly historical reasons.

Re: An Ubuntu kernel bug causes container crashes

#118

Post by Spivak »

yjftsjthsd-h wrote: post #8
Spivak wrote: post #7

Earlier quoted context omitted.

I know it’s too late for a bunch of shops but for gods sake please don’t use unattended upgrades to do your patching unless you want to hate you life and chase down hard to find hard to undo bugs. Build your images in CI job and have your deploy version be (code version, image version) so patching runs through all the same tests your code does and you have a trivial roll-forward to undo any mess you find yourself in.

> don’t use unattended upgrades > Build your images in CI job I know container images should generally be immutable, but I would expect unattended upgrades to be mostly used on the host, not in a container, in which that management system doesn't really work (unless you're doing VMs where you can deploy immutable root images to the VMs as well, or some fun bare metal + PXE combination).

> or some fun bare metal + PXE combination

This is actually what I implemented for our hypervisor tier, it’s not as scary as it sounds. I could legit completely rebuild our entire stack down to the metal in about 3 hours.

Kick off a new hypervisor version, the inactive side PXE boots all the nodes, installs and configures a Proxmox cluster, slaves itself to our Ceph cluster, and then either does a hot migration of all the VMs or kicks off a full deploy which rebuilds all the infra (Consul, Rabbit, Redis, LDAP, Elastic, PowerDNS, etc) along with the app servers. The hardest part (which really isn’t) is maintaining the clusters across the blue/green sides.

With this setup our only mutable infrastructure was our Ceph cluster (because replacing OSDs takes unacceptably long) and our DB (for performance the writers lived on dedicated servers, the read replicas lived on the VMs.).

Re: An Ubuntu kernel bug causes container crashes

#119

Post by topranks »

laumars wrote: post #55
withinboredom wrote: post #47

Earlier quoted context omitted.

Sounds like a VM and not a physical server.

Nope. Back before VMs were thing it was common to do "lights out" style remote management via a console server. That console server would then have a serial connection (the old 9 pin d-sub plug[1]) to your individual physical servers. You could then connect to your remote servers local TTY via the console server a little like jumping to remote servers via an SSH bastion. However it did sometimes require a little bit…

Still common on network devices (Cisco, Juniper, Arista etc.). No IPMI or similar on those.

Console servers from the likes of OpenGear and Lantronix still heavily used for those.

Re: An Ubuntu kernel bug causes container crashes

#120

Post by topranks »

withinboredom wrote: post #52
phillu wrote: post #51

Earlier quoted context omitted.

That is really not my experience at all. Every professional smaller team I worked with "usually" had this figured out and set up. In times of home office, no one wants to be at the office for just pressing a single button on some server. Oh well, I guess experiences differ.

My experiences for ops is all pre-2012 and with teams numbering less than 3 for the whole org. So I’m sure things have changed or gotten cheaper? I can’t see a team of 3-4 having the budget to get something that allows them to be “lazy”, especially when that budget can go towards something useful. But I guess the pandemic probably changed things there?

Depends on the infra and how it’s set up.

If you can afford to have something down for an extended period then fine. But even with a small team some services are built such that certain device outages cannot be tolerated, at least for an extended period.

So out-of-band/console servers or whatever still make a lot of sense and a relatively high priority.

Post reply on HN