Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LXC and LXD: a different container story

lwn.net

Post reply on HN
Print view View original

71–80 of 102 posts

Re: LXC and LXD: a different container story

#71

Post by jlokier »

p4bl0 wrote: post #40
0xbadcafebee wrote: post #37

Docker's easier and does all the same stuff. It's integrated with everything else, has an interface everyone is familiar with, uses OCIs, is extensible, is supported on basically every platform, has a simpler config file, tons of community support, and now runs rootless. Does anyone have a use case where they couldn't use Docker? I'm sure they exist but the list must be tiny. Listed in the article is: - run systemd i…

I teach an introduction to computer security course where I show my students how to write some basic rootkits for Linux. I need them to be able do that on the computer room machines (where they aren't root not sudoers), or on their own computer, but in both case I prefer if it doesn't cause a kernel panic if their code is wrong at some point. As far as I know you cannot load kernel modules in a container. You need an…

VMs are perfect for that teaching scenario.

That said, you're not supposed to be able to load kernel modules from a container.

Which is exactly the sort of idea an enterprising rootkit author might think to take advantage of ;-)

You might consider asking your students if they can figure out how they might go about getting a kernel module installed from in a container despite it not being allowed - and what advantages does that bring to the rootkit author. If necessary, you can test it using an LXD container in a VM.

Re: LXC and LXD: a different container story

#72

Post by discardedrefuse »

kwk1 wrote: post #52
discardedrefuse wrote: post #15

I ran LXD for about a year in my home lab. Unfortunately, the easiest way to get it running is by installing Snap; which I didn't do. Instead I ran it on Alpine Edge (one of the few distros that actually has it in their package manager). LXD kept breaking after system updates and I got tired of troubleshooting. I suppose that's just part of the perils of running bleeding edge. When LXD was running, I found that it fe…

> Unfortunately, the easiest way to get it running is by installing Snap; The Debian package for LXD just entered the archive about 2 weeks ago: https://tracker.debian.org/news/1361535/accepted-lxd-500-1-s... This means the next Debian release will make for stable LXD hosts without needing snap.

This is most excellent news!!!

Re: LXC and LXD: a different container story

#73

Post by em-bee »

jordemort wrote: post #12

Hi, author of the article, pleased to see it here! I'd really like to hear more from folks about how they're using LXC and/or LXD, and what they think their greatest strengths are compared to Docker or Kubernetes.

i introduced proxmox and saltstack to a customer who was running all sorts of applications together on a few linux servers. splitting up the services over multiple containers made maintenance easier because upgrading software in one container would not affect apps running in other containers.

on my own servers i use plain LXC without proxmox, but created and managed through saltstack for similar reasons. and also to allow me to use different linux distributions for different use cases, and to seperate various development environments

Re: LXC and LXD: a different container story

#74

Post by cyphar »

dinvlad wrote: post #69
cyphar wrote: post #68

Earlier quoted context omitted.

Checkpoint/restore is supported by runc and there were demos of live migrating Docker containers several years ago. A better question is why Kubernetes isn't working on integrating it (though I suspect there's a KEP for it somewhere).

Could you share a link please? I've only seen similar demos of a Doom game migration on LXD (I believe). EDIT: very interesting, thanks TIL! Here's at least someone from RH playing with it in the context of K8s: https://youtu.be/DDJxqV98b4U?t=1932 EDIT2: and apparently, at least its minimal implementation has been merged: https://github.com/kubernetes/kubernetes/pull/104907

Adrian is one of the CRIU folks and has been working on integrating CRIU and containers for a while.

Re: LXC and LXD: a different container story

#75

Post by panarky »

jordemort wrote: post #12

Hi, author of the article, pleased to see it here! I'd really like to hear more from folks about how they're using LXC and/or LXD, and what they think their greatest strengths are compared to Docker or Kubernetes.

LXC is how Chromebooks let you install Debian or Ubuntu or Arch in a container, with graphical applications running in the container integrated into the ChromeOS UI.

Re: LXC and LXD: a different container story

#76

Post by bokatu4567 »

If you are interested in this kind of lightweight containerization, the best tool to consider would be FreeBSD jails. They are available in FreeBSD since version 4.0 released on March 14, 2000 and at this point are a very mature solution to this problem. You can even attach dedicated network cards to jails and prevent them from being visible on the host. Moreover their integration with the base os and system utilities is seamless and very mature.

Linux folks love saying: use the right tool for the right job when it comes to telling people why Kubernetes is a strong enough reason for any company to switch to linux, however when it comes to things such as lightweight containers (or ZFS-on-root for the matter) for some reason the same reasoning isn't applied.

LXC and LXD look like a patchwork compared to FreeBSD jails. If you are on the desktop and want to run graphical apps, maybe it can make some kind of sense if you are already using Linux for everything else. But if you want to have a lightweight container setup on a headless server, FreeBSD is by far the best option.

In FreeBSD land, no one boasts of running 5 virtualized environments on a single machine because this is just a common thing for us and part of every workfow. You want to work on some github repository? You create a new jail in 1 minute using a tool such as cbsd.

A server with 32gb of RAM can easily run 1000s of jails, even a small VPS with 1-2gb of RAM can run a dozen jails or more without problems. It is intriguing that so many people dismiss the power of that.

And it is intriguing that people easily accept: "there is no equivalent to dockerfiles but it's fine because things can be scripted easily with basic shell commands" when it comes to LXC/LXD but people dismiss this very argument when FreeBSD folks try to explain why there is no need for Kubernetes on FreeBSD.

Anyway, if you run a startup not operating a resource intensive products (like ML requiring spinning up massive amounts of cores and RAM in an irregular basis) FreeBSD and jails can really save massive amounts of time and allow you to focus on the right thing. just run everything from a single Root VPS (usually hosting companies already handle things such as RAID replication and hardware redundancy transparently) and before you reach scalability problems, you'll already be making millions of dollars in revenue.

Even after, running thousands of jails on a single hosy essentially means doing what AWS and other cloud providers are doing. This doesn't prevent you from builing a small high-availability layer on top to ensure that critical apps run from at least two different physical hosts. This isn't very difficult to build with things like Consul, HA proxy, built-in system utilities and a small custom web interface.

Re: LXC and LXD: a different container story

#77

Post by rcarmo »

jordemort wrote: post #12

Hi, author of the article, pleased to see it here! I'd really like to hear more from folks about how they're using LXC and/or LXD, and what they think their greatest strengths are compared to Docker or Kubernetes.

I use it in a vanilla KVM host to have lightweight VMs for Remote Desktop environments (https://taoofmac.com/space/blog/2022/04/02/2130, https://taoofmac.com/space/blog/2022/04/12/2330). That way I can have full GNOME and XFCE desktops with different sets of developer tools installed, sharing the same projects tree on a bind mount, and to which I can remote via Xrdp (with audio, device sharing, the works). Like the two KVM Windows VMs running alongside, each LXC can have its own IP, outbound VPN connection and even Docker running as well, which is great.

A single i7 with 32GB of RAM can have at least 3 people working on it without hassle, like an old time-sharing system. Myself, I stuck a Raspberry Pi to a monitor (https://taoofmac.com/space/blog/2022/08/14/2030) and use it to access both GNOME and Windows desktops remotely.

All of this works solely with Ubuntu system packages, so I have no third party software to install (yes, ok, snap is a pain, but the host only runs KVM and LXD, nothing else, and all distros have xorgxrdp these days).

It’s like a lightweight Proxmox.

In the past, I used LXC to host several of my services on a single beefy Linode box, and I would go that way again today - I already have a similar setup inside Parallels on my M1 Mac, with one LXC per project.

Re: LXC and LXD: a different container story

#78

Post by rcarmo »

chubot wrote: post #38

Do either of them support the concept of “layers” like Docker does? I think that feature combined with overlayfs2 is quite useful, despite my many criticisms of Docker. It is sort of a middle ground between Nix like granularity which requires rewriting upstream, and big LXC blobs created with shell scripts. Although I also think we need some kind of middle ground between docker and nix :)

Both overlays and snapshots work fine.

Re: LXC and LXD: a different container story

#79

Post by rcarmo »

0xbadcafebee wrote: post #37

Docker's easier and does all the same stuff. It's integrated with everything else, has an interface everyone is familiar with, uses OCIs, is extensible, is supported on basically every platform, has a simpler config file, tons of community support, and now runs rootless. Does anyone have a use case where they couldn't use Docker? I'm sure they exist but the list must be tiny. Listed in the article is: - run systemd i…

It doesn’t do the same thing. LXC is a godsend when you need sandboxing other users or workloads that spill over into the host (even Docker stacks need (Host) resources, like mount points and storage).

I run different Docker Compose stacks inside different LXC containers to avoid collisions between some containers that are in both stacks.

Re: LXC and LXD: a different container story

#80

Post by yobbo »

axytol wrote: post #61

How do you all manage the security side for LXC containers? As I understand it, any local user allowed to run lxc containers can effectively spin up a root level access container on the host, example: run a privileged container and mount / inside the container? Is there anyway to mitigate this?

Only root can start privileged containers.
Post reply on HN