Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Ubuntu kernel bug causes container crashes

lwn.net

Post reply on HN
Print view View original

121–130 of 133 posts

Re: An Ubuntu kernel bug causes container crashes

#121

Post by jve »

dncornholio wrote: post #89
jve wrote: post #84

Earlier quoted context omitted.

Docker actually helps managing complexity, by putting bits and pieces, scattered on the floor and putting them into a single cardboard box. - If you throw the box out, you know you did no harm to other boxes. - If you change your floor, you know you didn't wipe out something useful. - Aaand you can `git switch` to a well known state Ofcourse it's not 100% like that, in reality you still have to have some kind of cons…

What you are saying is actual complexity. A box can be a machine instead of container and all your points will still stand but with less complexity.

Perhaps, if we talk about deployent.

Howevre looking at the whole chain of process:

I can deploy service/app on my local machine, including dependencies, along with other independent services/apps.

So, I can reuse what I have for production. But ofcourse, local development stuff will have different env variables, some docker-compose etc.

But it is a joy, when you can:

1. git clone something

2. set some env variables

3. docker-compose up -d

And your app, along with database, elasticsearch, whatnot - is running

So for development purposes it really helps. For deployment purposes - if that box is dedicated to something, then yeah, many of the good use cases not necessary.

Re: An Ubuntu kernel bug causes container crashes

#122

Post by markstos »

solarkraft wrote: post #110
markstos wrote: post #102

Earlier quoted context omitted.

Containers help tame complexity, but the container itself could be run without the docker runtime, using systemd-nspawn, or run as a regular systemd service using podman. It seems less complex to manage a bunch of systemd services than one pile of systemd services that are managed and logged one way and bunch of docker services that are managed and logged another way.

Deviation from standards is what I find to cause a lot (the most?) of problems. Complexity generally makes things more error-prone, but it tends to become less of a problem when in a well maintained system a lot of people use (someone, maybe yourself, has already had and solved a problem you're having). If your standard is dealing with systemd units, it may make sense to make your containers conform to that (thereby…

Agreed. When I'm picking "the right tool for the job", something I factor in what are the tools and languages my team knows. If my team knows JavaScript, I'm going to weigh that when I'm choosing the choice of coding languages for the next project.

Re: An Ubuntu kernel bug causes container crashes

#123

Post by jve »

dncornholio wrote: post #90
atoav wrote: post #16

The cost of complexity showing itself. A sysadmin friend of mine is totally against docker and his reason is that he wants as little complexity as is needed on his systems. Complexity, he says, leads to emergent behavior.

Docker hasn't solved any problem for me, so I don't see any use in using it. Meanwhile I have multiple junior devs asking me docker stuff so they can run stuff localy. When I ask them why even use docker, they always say that some tutorial told them to. This is when I introduce them to something called VirtualBox and then their eyes go bright with wonder on how simple that works.

> I introduce them to something called VirtualBox and then their eyes go bright with wonder on how simple that works.

Well, If I'd have a workhorse with loads of RAM... I'd still choose docker, because how FAST it starts/restarts. And because it is easy to recreate everything with docker - a VM may get messy when installing stuff for APP #1, #2, #x, "works on my machine!" etc.

Re: An Ubuntu kernel bug causes container crashes

#124

Post by markstos »

Spivak wrote: post #117
markstos wrote: post #101

Earlier quoted context omitted.

Sorry, not my experience. My experience has been that by the time I notice some serious vulnerability is in the news, my servers have already patched themselves. I have never "hated life" or had a "hard to find and undo bug" due to automatic security patching. I pretty quickly found what caused this and had a clear path to resolution. This is the first security update that caused a boot failure in about a decade. It…

You’re conflating unattended-upgrades (server mutability, hard to roll back) with automated patching in general. Do automated patching but also run the changes though your CI so you can catch breaking changes and roll them out in a way that’s easy to debug (you can diff images) and revert. I bet when you update your software dependencies you run those changes through your tests but your OS is a giant pile of code tha…

> I bet when you update your software dependencies you run those changes through your tests but your OS is a giant pile of code that usually gets updated differently and independently because mostly historical reasons.

Close. We are moving towards defining our server states through Ansible, but the project is not close to completion. Perhaps once that's further along, we could use Ansible Molecule + CI to test a new server state when there's a new patch available, but that's not an option on the table today.

The system we had in place for /today/ worked: Lower priority or redundant servers were set to auto-reboot after applying security updates, while other critical servers require manual reboot at low-risk times. By then, the patch has already been tested on lower-risk servers.

As a result, this issue caused no user-visible downtime for us, and due to the staggered runs of unattended-upgrades affected a minimal number of servers.

And this was the first time in 10+ years that something like this happened and we have to choose to write to prioritize spending our process-improvement time based on likelihood and impact.

Re: An Ubuntu kernel bug causes container crashes

#125

Post by leaflets2 »

amelius wrote: post #80
jacquesm wrote: post #72

Earlier quoted context omitted.

Well, you're between a rock and a hard place. No auto-update = security risk exposure, auto-update = stability risk exposure (and sometimes security risk exposure thrown in for free as well).

If the only externally visible service you run is sshd then how important is it to auto-update for security reasons? (Also considering that security risks in sshd are almost guaranteed to end up on the front page of HN, so you won't miss it).

> the front page of HN

What if you're in bed with the flu

But if you're a team, then maybe.

Still, could delay the response with a whole day (checking HN once a day)

Re: An Ubuntu kernel bug causes container crashes

#126

Post by leaflets2 »

kramerger wrote: post #45
baggy_trough wrote: post #19

Earlier quoted context omitted.

Yeah, I don't really get why they don't use the stable kernel releases, of which there are many, rather than rolling their own.

Seems like every department at canonical needs to learn this on their own. After all, they reinvented everything from DE to init system at least once in past. (They also have their own containers, LXD. I actually really like that one, please keep working on that canonical)

What are some other bad mistakes that's been made?

Re: An Ubuntu kernel bug causes container crashes

#128

Post by remram »

markstos wrote: post #104
Fnoord wrote: post #50

Earlier quoted context omitted.

Would it have occurred on Podman as well?

Some of my servers were running Ubuntu with systemd+podman to manage services, and none of them had a problem.

You would have to be running this specific kernel version and one of your containers has to memory-map a file (not from a volume?), from what I understand.

Re: An Ubuntu kernel bug causes container crashes

#129

Post by remram »

capableweb wrote: post #59
speedgoose wrote: post #23

Earlier quoted context omitted.

I think it’s more complex to ask people to package software in a good way without software containers.

Is it really? Windows have .exe files, macOS has .app files, Linux has .AppImage (or even tarballs with binaries), and that's just on the host level. Java has .jar files, and so on. Not to mention, if you want to "natively" pack something for Windows and macOS, containers won't even solve that problem, as they only run on Linux. Only reason you can use Docker on macOS is because of virtualization.

Those are a lot like containers though.

Mac .app files are not just binary (MACH-O), they can include libraries, "frameworks", etc that will override what's loaded from the system. AppImage is even worse [1]. .exe are usually setup with an installer that trigger the side-by-side assembly mechanism, pretending that the system is using the version of the libraries that you included (and growing your WinSXS folder forever). JAR files usually include all their required transitive dependencies rather than "dynamic linking" with other JARs.

[1]: AppImage official documentation: "Do not depend on system-provided resources" https://docs.appimage.org/introduction/concepts.html#do-not-...

Re: An Ubuntu kernel bug causes container crashes

#130

Post by bravetraveler »

dspillett wrote: post #65
capableweb wrote: post #58

Earlier quoted context omitted.

Which, if your server is stuck in a infinitive "boot -> docker starting -> container starting -> crashing kernel -> reboot" loop, you won't ever get a chance of actually adding anything to your boot parameters.

If you have access to the console (local physical machine, VM on a system that can expose the console, physical box that you have console access to via IPMI or other means), can you not specify that directive to be passed through via grub's interactive menu? Failing that you could try the “single” directive and poke other configurations once booted in that mode. A faf to be sure, but hopefully viable options (assumin…

Absolutely can, I'm quite surprised at the 'what do' attitude around this. It's routine -- not in all organizations to be sure, but it's a solved problem.

There are options even without out of band management. You can choose to configure your systems with PXE -- if the installation ever fails, it can boot into a recovery environment over the network.

Post reply on HN