Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Ubuntu kernel bug causes container crashes

lwn.net

Post reply on HN
Print view View original

31–40 of 133 posts

Re: An Ubuntu kernel bug causes container crashes

#31

Post by mrintegrity »

This was exceptionally annoying for me, some ec2 instances are used only during the day and we stop/start them with an in house scheduling application outside office hours. Also automatic security upgrades are enabled. Came in to work one day last week and all of our UAT environment was down.

It is possible to ssh in for about 2 seconds before the kernel panic so I solved it by doing this:

while true; do ssh sudo mv /usr/bin/containerd /usr/bin/containerd.backup ; sleep 1; done

On the next reboot i was able to ssh in and change to the (then just released within the past hour) kernel that doesn't have this stupid bug. After another reboot you can move containderd back and it should be working again

affected: linux-image-5.13.0-1028 not affected: >linux-image-5.13.0-1029

Re: An Ubuntu kernel bug causes container crashes

#32

Post by CuriousCosmic »

yjftsjthsd-h wrote: post #8
Spivak wrote: post #7

Earlier quoted context omitted.

I know it’s too late for a bunch of shops but for gods sake please don’t use unattended upgrades to do your patching unless you want to hate you life and chase down hard to find hard to undo bugs. Build your images in CI job and have your deploy version be (code version, image version) so patching runs through all the same tests your code does and you have a trivial roll-forward to undo any mess you find yourself in.

> don’t use unattended upgrades > Build your images in CI job I know container images should generally be immutable, but I would expect unattended upgrades to be mostly used on the host, not in a container, in which that management system doesn't really work (unless you're doing VMs where you can deploy immutable root images to the VMs as well, or some fun bare metal + PXE combination).

alternatively I suppose depending on the size of your operation, you should consider having a dummy prod using at least one of each of the servers in your environment and using that to validate host upgrades. after that you can push an unattended upgrade via a self-hosted package+upgrade server.

Let things be automatic to the maximum degree possible but give yourself a single hard human checkpoint and some minimum level of validation in a dummy environment first.

Re: An Ubuntu kernel bug causes container crashes

#33

Post by klqr »

throwaway787544 wrote: post #26
atoav wrote: post #16

The cost of complexity showing itself. A sysadmin friend of mine is totally against docker and his reason is that he wants as little complexity as is needed on his systems. Complexity, he says, leads to emergent behavior.

Complexity is just another part of natural systems. It's not something to be avoided for it's own sake, in as much as we should avoid having eyeballs, because they are crazily complex. Yet we tend to like ours and find them worth whatever cost they incur. The emergent behavior of containerization has had an overall positive effect, even if it has annoying costs.

For the end user there is no positive effect. Many websites were better in 2005 and had greater uptime.

They were also better organized. Ebay and Amazon were leaner and more pleasant to use.

Re: An Ubuntu kernel bug causes container crashes

#34

Post by wronglyprepaid »

zerop wrote: post #14

Any good documentation that talks about how big open source software manage code changes, releases cadences, given contributors from across the world.

I'm fairly sure this differs for different project/organization, not sure there is a rule, and not sure there are really any considerations that are specific to open source, good practices are good practices regardless.

That being said, I rate Canonical's practices as rather poor.

Re: An Ubuntu kernel bug causes container crashes

#35

Post by rawoke083600 »

markstos wrote: post #3

This was even worse than the headline made it sound. If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed…

That sounds downright horrible !

Re: An Ubuntu kernel bug causes container crashes

#36

Post by lllkbcxdd »

ladyanita22 wrote: post #29
ungamedplayer wrote: post #22

Earlier quoted context omitted.

> . One reason I've always preferred Ubuntu over Red Hat for servers is that with Red Hat/Centos essentially everything I care about is perpetually and hopelessly out of date and obsolete This is exactly why you choose it. Lesser chance of insanity.

Ýou can always choose Fedora Server if you want a more up-to-date server OS.

Fedora Server.. is RedHat...

There's no "Fedora Server" product and never has been. Do you mean the rolling release CentOS?

Re: An Ubuntu kernel bug causes container crashes

#37

Post by nix23 »

Spivak wrote: post #7
markstos wrote: post #3

This was even worse than the headline made it sound. If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed…

I know it’s too late for a bunch of shops but for gods sake please don’t use unattended upgrades to do your patching unless you want to hate you life and chase down hard to find hard to undo bugs. Build your images in CI job and have your deploy version be (code version, image version) so patching runs through all the same tests your code does and you have a trivial roll-forward to undo any mess you find yourself in.

>I know it’s too late for a bunch of shops but for gods sake please don’t use unattended upgrades to do your patching unless you want to hate you life and chase down hard to find hard to undo bugs

Some years ago everyone said the same about windows-servers ;)

Re: An Ubuntu kernel bug causes container crashes

#38

Post by ntp85 »

lllkbcxdd wrote: post #36
ladyanita22 wrote: post #29

Earlier quoted context omitted.

Ýou can always choose Fedora Server if you want a more up-to-date server OS.

Fedora Server.. is RedHat... There's no "Fedora Server" product and never has been. Do you mean the rolling release CentOS?

There is indeed a Fedora Server: https://getfedora.org/en/server/

Re: An Ubuntu kernel bug causes container crashes

#39

Post by sofixa »

markstos wrote: post #3

This was even worse than the headline made it sound. If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed…

> If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up.

Isn't the common wisdom that you should have them enabled, but staggered across hours/days?

Re: An Ubuntu kernel bug causes container crashes

#40

Post by nwh5jg56df »

klqr wrote: post #33
throwaway787544 wrote: post #26

Earlier quoted context omitted.

Complexity is just another part of natural systems. It's not something to be avoided for it's own sake, in as much as we should avoid having eyeballs, because they are crazily complex. Yet we tend to like ours and find them worth whatever cost they incur. The emergent behavior of containerization has had an overall positive effect, even if it has annoying costs.

For the end user there is no positive effect. Many websites were better in 2005 and had greater uptime. They were also better organized. Ebay and Amazon were leaner and more pleasant to use.

> Many websites were better in 2005 and had greater uptime.

Source? Sounds bullshit

Post reply on HN