Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LXC and LXD: a different container story

lwn.net

Post reply on HN
Print view View original

31–40 of 102 posts

Re: LXC and LXD: a different container story

#31

Post by locusofself »

My last company had hundreds possibly thousands of LXC containers, and we orchestrated everything via saltstack (which is similar to ansible or puppet if you aren't familiar).

The justification was that we needed our SaaS to also work on-prem for financial companies and government entities, and thus we could not count on kubernetes or any specific cloud vendor to be available, so we rolled our own orchestration built on ubuntu hosts and (for some stupid reasons) centos LXC containers running on top of those.

LXC is a good system and all, but what we were doing with it was a total nightmare, we were basically trying to reinvent kubernetes using traditional configuration management and LXC with two flavors of linux and a million dependencies.

Re: LXC and LXD: a different container story

#32

Post by youainti »

jordemort wrote: post #12

Hi, author of the article, pleased to see it here! I'd really like to hear more from folks about how they're using LXC and/or LXD, and what they think their greatest strengths are compared to Docker or Kubernetes.

I've been considering building a system to spin up containers to run repeatable statistical analyses. LXC was the direction I decided to go.

Re: LXC and LXD: a different container story

#33

Post by dvdkon »

I run everything in LXD containers on my home server(s), with Debian as the host (manually compiled, not the snap). Most of the setup is automated with Ansible that I hacked to read Python scripts instead of YAML.

LXD is a treat to work with, and I feel its container model is perfect for home/small business servers that need to run lots of third-party software, much of which would work poorly in a "proper" stateless container.

Re: LXC and LXD: a different container story

#34

Post by thedougd »

locusofself wrote: post #31

My last company had hundreds possibly thousands of LXC containers, and we orchestrated everything via saltstack (which is similar to ansible or puppet if you aren't familiar). The justification was that we needed our SaaS to also work on-prem for financial companies and government entities, and thus we could not count on kubernetes or any specific cloud vendor to be available, so we rolled our own orchestration built…

I can't speak to supporting these, but there are a few vendors that offer k8s in a box for this exact use case. Replicated is the first that comes to my mind. I've used this one as a customer. It worked fine but felt it necessary to do a bit of poking under the hood to help it understand things like our AZs in a private cloud:

https://www.replicated.com/kubernetes/

Re: LXC and LXD: a different container story

#36

Post by 0xbadcafebee »

locusofself wrote: post #31

My last company had hundreds possibly thousands of LXC containers, and we orchestrated everything via saltstack (which is similar to ansible or puppet if you aren't familiar). The justification was that we needed our SaaS to also work on-prem for financial companies and government entities, and thus we could not count on kubernetes or any specific cloud vendor to be available, so we rolled our own orchestration built…

> we needed our SaaS to also work on-prem for financial companies and government entities, and thus we could not count on kubernetes or any specific cloud vendor

....? kubernetes is not a cloud vendor, it runs on any Linux distribution, and it's FOSS.... ?!

Re: LXC and LXD: a different container story

#37

Post by 0xbadcafebee »

Docker's easier and does all the same stuff. It's integrated with everything else, has an interface everyone is familiar with, uses OCIs, is extensible, is supported on basically every platform, has a simpler config file, tons of community support, and now runs rootless.

Does anyone have a use case where they couldn't use Docker? I'm sure they exist but the list must be tiny. Listed in the article is:

  - run systemd in a container
    - why?? does your system not have systemd or equivalent?
    - bad design; how will you monitor what's running and what's not, or upgrade an individual service in this container? restart everything?
    - apparently systemd can run in docker?? https://medium.com/swlh/docker-and-systemd-381dfd7e4628
  
  - run lxc/docker in a container
    - docker-in-docker

Re: LXC and LXD: a different container story

#38

Post by chubot »

Do either of them support the concept of “layers” like Docker does?

I think that feature combined with overlayfs2 is quite useful, despite my many criticisms of Docker.

It is sort of a middle ground between Nix like granularity which requires rewriting upstream, and big LXC blobs created with shell scripts.

Although I also think we need some kind of middle ground between docker and nix :)

Re: LXC and LXD: a different container story

#39

Post by foresto »

weinzierl wrote: post #30
4oo4 wrote: post #10

I use both lxc and docker and have uses cases for both, I think it really comes down to how stateful something is or how lazy I feel about writing a Dockerfile. I had a really hard time with learning lxd and really only got into using lxc without the daemon. One trend with docker that I personally don't like is that a lot of projects prefer docker-compose over regular Dockerfiles (though some of them support both), a…

> " [..] which are just shell scripts for bootstrapping and configuring your container and pretty common between distros. " I find it a big advantage that this builds the containers basically from scratch and you only have to trust the distro and not any other parties. I'm always feeling uneasy with images which are hard to inspect and whose provenance is opaque and potentially involved multiple parties. That, combin…

I don't recall having to do any uid/gid fixup last time I made an unprivileged container. I did have to prepare the unprivileged host account, of course, by setting its subordinate uids/gids (/etc/sub?id) and virtual network interface limit (/etc/lxc/lxc-usernet).

To create the container, I did this:

lxc-create -t download -n -- -d debian -r bullseye -a amd64

Note that this runs the 'download' template, which (IIRC) is better suited to unprivileged containers than the 'debian' template is. The 'download' template will list its available distros if you do this:

lxc-create -t download -n -- --list

Note that some versions of the 'download' template may fail with a keyserver error because sks-keyservers.net died somewhat recently. Workaround: DOWNLOAD_KEYSERVER=hkp://keyserver.ubuntu.com lxc-create ...

https://github.com/lxc/lxc/issues/3894

Re: LXC and LXD: a different container story

#40

Post by p4bl0 »

0xbadcafebee wrote: post #37

Docker's easier and does all the same stuff. It's integrated with everything else, has an interface everyone is familiar with, uses OCIs, is extensible, is supported on basically every platform, has a simpler config file, tons of community support, and now runs rootless. Does anyone have a use case where they couldn't use Docker? I'm sure they exist but the list must be tiny. Listed in the article is: - run systemd i…

I teach an introduction to computer security course where I show my students how to write some basic rootkits for Linux. I need them to be able do that on the computer room machines (where they aren't root not sudoers), or on their own computer, but in both case I prefer if it doesn't cause a kernel panic if their code is wrong at some point.

As far as I know you cannot load kernel modules in a container. You need an actual VM for that.

Post reply on HN