Auto update strikes again. Really, we need to re-think this.
An Ubuntu kernel bug causes container crashes
61–70 of 133 posts
Re: An Ubuntu kernel bug causes container crashes
#62Earlier quoted context omitted.
> have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run. Or add `systemd.mask=docker.service` to your boot parameters to prevent Docker from starting.
Which, if your server is stuck in a infinitive "boot -> docker starting -> container starting -> crashing kernel -> reboot" loop, you won't ever get a chance of actually adding anything to your boot parameters.
That's not correct. If you stop the boot you can add 'single' to the boot statement which will drop you in a single user shell from where you can do quite a bit of maintenance.
Re: An Ubuntu kernel bug causes container crashes
#63Earlier quoted context omitted.
Sounds like a VM and not a physical server.
Nope. Back before VMs were thing it was common to do "lights out" style remote management via a console server. That console server would then have a serial connection (the old 9 pin d-sub plug[1]) to your individual physical servers. You could then connect to your remote servers local TTY via the console server a little like jumping to remote servers via an SSH bastion. However it did sometimes require a little bit…
And quite a few implementations actually emulate the serial console allowing for the exact same access. (Serial Over Lan or SOL for short.)
Re: An Ubuntu kernel bug causes container crashes
#64Earlier quoted context omitted.
Or a real server with Lights Out Management.
That’s why “usually” is in the sentence. :) Most smaller teams usually don’t prioritize physical access — they usually only need it for one-off events. While this would be a one-off event, it would be one that affects many servers.
You can do this kind of thing across the network if you have to.
Re: An Ubuntu kernel bug causes container crashes
#65Earlier quoted context omitted.
> have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run. Or add `systemd.mask=docker.service` to your boot parameters to prevent Docker from starting.
Which, if your server is stuck in a infinitive "boot -> docker starting -> container starting -> crashing kernel -> reboot" loop, you won't ever get a chance of actually adding anything to your boot parameters.
If you have access to the console (local physical machine, VM on a system that can expose the console, physical box that you have console access to via IPMI or other means), can you not specify that directive to be passed through via grub's interactive menu?
Failing that you could try the “single” directive and poke other configurations once booted in that mode.
A faf to be sure, but hopefully viable options (assuming the interactive menu hasn't been disabled to save a few seconds off boot time!).
Re: An Ubuntu kernel bug causes container crashes
#66The cost of complexity showing itself. A sysadmin friend of mine is totally against docker and his reason is that he wants as little complexity as is needed on his systems. Complexity, he says, leads to emergent behavior.
This is not a docker bug, it is a kernel bug. It could be triggered by other complex applications that use kernel container features.
That pretty much makes the GPs point: emergent behavior arising from complexity.
Re: An Ubuntu kernel bug causes container crashes
#67Perhaps add “June 2022” to the title to reduce panic? Updated packages that resolve the issue were released on 2022-06-10, so this article is a post-mortem not an alert of a new problem that could affect people now.
Re: An Ubuntu kernel bug causes container crashes
#68Auto update strikes again. Really, we need to re-think this.
There is no need to rethink it; it's never been a good idea to leave it turned on.
Re: An Ubuntu kernel bug causes container crashes
#69The cost of complexity showing itself. A sysadmin friend of mine is totally against docker and his reason is that he wants as little complexity as is needed on his systems. Complexity, he says, leads to emergent behavior.
I don't do much by way of containers myself¹ but some teams in DayJob do and some other contact also. Some run containers in VMs² to separate out some of the complexity due to boot bugs like this - there is a little performance hit from the VM but failures in the container parts of the kernel can't cause the whole machine not to boot so it is easier to get in to revert things back to a last-known-good state.
----
[1] I have a couple of bits running via LCX but otherwise use VMs to split services out
[2] One large VM running many containers³, or sometimes a couple of VMs, perhaps separating them performance-wise across drives or with CPU core affinity where that was/seemed easier, or just so in case of disaster they could concentrate on getting the higher priority VM+containers restored and back up first.
[3] Obviously one VM per container would defeat the container benefits, though I've seen this done where docker was the only officially supported install option and they wanted to run a service in a VM.
Re: An Ubuntu kernel bug causes container crashes
#70This was exceptionally annoying for me, some ec2 instances are used only during the day and we stop/start them with an in house scheduling application outside office hours. Also automatic security upgrades are enabled. Came in to work one day last week and all of our UAT environment was down. It is possible to ssh in for about 2 seconds before the kernel panic so I solved it by doing this: while true; do ssh sudo mv…
I love the approach. Nowadays you can even
sleep 0.1