Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Ubuntu kernel bug causes container crashes

lwn.net

Post reply on HN
Print view View original

51–60 of 133 posts

Re: An Ubuntu kernel bug causes container crashes

#51

Post by phillu »

withinboredom wrote: post #48
taspeotis wrote: post #43

Earlier quoted context omitted.

Or a real server with Lights Out Management.

That’s why “usually” is in the sentence. :) Most smaller teams usually don’t prioritize physical access — they usually only need it for one-off events. While this would be a one-off event, it would be one that affects many servers.

That is really not my experience at all. Every professional smaller team I worked with "usually" had this figured out and set up. In times of home office, no one wants to be at the office for just pressing a single button on some server.

Oh well, I guess experiences differ.

Re: An Ubuntu kernel bug causes container crashes

#52

Post by withinboredom »

phillu wrote: post #51
withinboredom wrote: post #48

Earlier quoted context omitted.

That’s why “usually” is in the sentence. :) Most smaller teams usually don’t prioritize physical access — they usually only need it for one-off events. While this would be a one-off event, it would be one that affects many servers.

That is really not my experience at all. Every professional smaller team I worked with "usually" had this figured out and set up. In times of home office, no one wants to be at the office for just pressing a single button on some server. Oh well, I guess experiences differ.

My experiences for ops is all pre-2012 and with teams numbering less than 3 for the whole org. So I’m sure things have changed or gotten cheaper? I can’t see a team of 3-4 having the budget to get something that allows them to be “lazy”, especially when that budget can go towards something useful. But I guess the pandemic probably changed things there?

Re: An Ubuntu kernel bug causes container crashes

#53

Post by trasz »

sofixa wrote: post #41
klqr wrote: post #33

Earlier quoted context omitted.

For the end user there is no positive effect. Many websites were better in 2005 and had greater uptime. They were also better organized. Ebay and Amazon were leaner and more pleasant to use.

> Many websites were better in 2005 and had greater uptime. Citation needed. There were a lot less sites, maintenance windows in the hours weren't uncommon, there was no security to speak of (SQL injections, no SSL/TLS, etc.), and sites could do maybe 1% of what today's sites can (not saying that all of it is good or necessary, bit i quite like a non-insignificant amount of those new features like native video, audio…

It's quite noticeable tbh. Nowadays companies don't care about actual reliability, they only care about "apparent reliability", which is a bullshit statistic, and it shows.

Re: An Ubuntu kernel bug causes container crashes

#54

Post by akx »

withinboredom wrote: post #47
akx wrote: post #46

Earlier quoted context omitted.

Not necessarily. With good timing and some luck, you can connect the serial/"recovery" console before GRUB's timeout ends and either change the running kernel or add the `systemd.mask=docker.service` boot parameter to prevent Docker from starting.

Sounds like a VM and not a physical server.

Sure. For a physical server, you'd use its lights-out management to the same effect.

Re: An Ubuntu kernel bug causes container crashes

#55

Post by laumars »

withinboredom wrote: post #47
akx wrote: post #46

Earlier quoted context omitted.

Not necessarily. With good timing and some luck, you can connect the serial/"recovery" console before GRUB's timeout ends and either change the running kernel or add the `systemd.mask=docker.service` boot parameter to prevent Docker from starting.

Sounds like a VM and not a physical server.

Nope. Back before VMs were thing it was common to do "lights out" style remote management via a console server. That console server would then have a serial connection (the old 9 pin d-sub plug[1]) to your individual physical servers. You could then connect to your remote servers local TTY via the console server a little like jumping to remote servers via an SSH bastion. However it did sometimes require a little bit of prior configuration, depending on your distro[2].

This wasn't just limited to Linux either. It was a common UNIX trick :)

This is a bit of a lost art these days though. iLo, IPMI have replaced the need for serial. Then virtualisation and, to a lesser extent, containerisation have lowered the bar even further plus also moving the industry towards more ephemeral systems that can be destroyed and rebuilt automatically rather than the old habits of nursing failed hosts back to health.

[1] https://duckduckgo.com/?q=9+pin+d-sub+plug&t=newext&atb=v316...

[2] https://www.kernel.org/doc/html/v5.3/admin-guide/serial-cons... (a lot of distros at the time did ship a kernel with this support compiled in. I don't know how common it is now).

Re: An Ubuntu kernel bug causes container crashes

#56

Post by kuschku »

atoav wrote: post #27
lixtra wrote: post #17

Earlier quoted context omitted.

This is not a docker bug, it is a kernel bug. It could be triggered by other complex applications that use kernel container features.

Or it could not be triggered if you don't use containers.

It could also be triggered by application sandboxing solutions that aren't containers.

Re: An Ubuntu kernel bug causes container crashes

#57

Post by laumars »

withinboredom wrote: post #52
phillu wrote: post #51

Earlier quoted context omitted.

That is really not my experience at all. Every professional smaller team I worked with "usually" had this figured out and set up. In times of home office, no one wants to be at the office for just pressing a single button on some server. Oh well, I guess experiences differ.

My experiences for ops is all pre-2012 and with teams numbering less than 3 for the whole org. So I’m sure things have changed or gotten cheaper? I can’t see a team of 3-4 having the budget to get something that allows them to be “lazy”, especially when that budget can go towards something useful. But I guess the pandemic probably changed things there?

Serial connections will only cost you a Raspberry Pi (there's probably some really cheap console servers on eBay too).

I don't think the issue is so much cost but more this kind of systems administration is becoming a forgotten art because 99% of the time modern tooling removes the need for it. So younger sysadmins are never taught how to do these kinds things. However when I started out, I worked in a few small companies that had their physical hosts connected to a console server (which was a Cisco device like a network switch) via serial cables and you'd then connect to that console server remotely.

Re: An Ubuntu kernel bug causes container crashes

#58

Post by capableweb »

akx wrote: post #25
markstos wrote: post #3

This was even worse than the headline made it sound. If you had `unattended-upgrades` running and had the "automatic reboot" option enabled, then all your Ubuntu 20.04 servers running Docker would reboot themselves and not come back up. First, the bug was in a security branch. Second, it wasn't just the containers that crashed. If you booted containers on boot via Docker, then the host OS kernel-panicked and crashed…

> have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run. Or add `systemd.mask=docker.service` to your boot parameters to prevent Docker from starting.

Which, if your server is stuck in a infinitive "boot -> docker starting -> container starting -> crashing kernel -> reboot" loop, you won't ever get a chance of actually adding anything to your boot parameters.

Re: An Ubuntu kernel bug causes container crashes

#59

Post by capableweb »

speedgoose wrote: post #23
atoav wrote: post #16

The cost of complexity showing itself. A sysadmin friend of mine is totally against docker and his reason is that he wants as little complexity as is needed on his systems. Complexity, he says, leads to emergent behavior.

I think it’s more complex to ask people to package software in a good way without software containers.

Is it really? Windows have .exe files, macOS has .app files, Linux has .AppImage (or even tarballs with binaries), and that's just on the host level. Java has .jar files, and so on.

Not to mention, if you want to "natively" pack something for Windows and macOS, containers won't even solve that problem, as they only run on Linux. Only reason you can use Docker on macOS is because of virtualization.

Re: An Ubuntu kernel bug causes container crashes

#60

Post by RedShift1 »

atoav wrote: post #16

The cost of complexity showing itself. A sysadmin friend of mine is totally against docker and his reason is that he wants as little complexity as is needed on his systems. Complexity, he says, leads to emergent behavior.

Although Docker may be another layer, from sysadmin point of view containers are not very complex. What I absolutely love about containers is that it shifts the responsibility of making the software run back to the developers. No more ridiculous installation requirements and long winded instructions just to get something going. Just start container, maybe add some mount volumes and env variables for configuration, boom, done. Way less complicated to set up and manage containers than having to learn what each piece of software does special just to get it going.
Post reply on HN