Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LXC and LXD: a different container story

lwn.net

Post reply on HN
Print view View original

81–90 of 102 posts

Re: LXC and LXD: a different container story

#81

Post by dspillett »

jordemort wrote: post #12

Hi, author of the article, pleased to see it here! I'd really like to hear more from folks about how they're using LXC and/or LXD, and what they think their greatest strengths are compared to Docker or Kubernetes.

I basically use LXC for light-weight VMs (a name server here, a simple web server there, pi-hole on my home router). Where I care about security separation (my containers are privileged) I use real VMs via KVM.

Though I'm an old fashioned pets-not-cattle type. Docker and friends have the disadvantage of being invented (or becoming practical/popular) in/after my mid 30s!

Re: LXC and LXD: a different container story

#82

Post by fuzzy2 »

fariszr wrote: post #50
fuzzy2 wrote: post #27

I fucking love LXC! It allowed me to “virtualize” several physical servers on a single newer server, keeping all of the original setup (except some networking changes). Combined with ipvlan , I could even work around the MAC address restriction of my server provider, with both IPv4 and IPv6. I also use it to host a Jitsi instance. Jitsi is rather picky about the underlying distribution. All this without the overhead…

You are using hetzner right? I'm interested to see how you bypassed the Mac address problem, any blogs, guides ?

Yes, it’s Hetzner. The magic ingredient is ipvlan, a special interface type loosely related to macvlan. An ipvlan interface is tied to a physical interface. Whether traffic goes to the virtual interface is decided based on the IP address alone. The ipvlan virtual interface can be moved to a different network namespace and will still work.

Note that Hetzner added support for multiple MAC addresses in the meantime. So at least for IPv4, you don’t need this.

There’s good info in the Linux kernel docs: https://www.kernel.org/doc/html/latest/networking/ipvlan.htm...

I have a Debian host. In my interfaces file, I have the following code for the Jitsi ipvlan link:

    auto ipvl-meet
    iface ipvl-meet inet manual
       pre-up ip link add link eth0 name ipvl-meet type ipvlan mode l2
       post-down ip link delete ipvl-meet
Then, in my LXC guest config, I have this:

    lxc.net.0.type = phys
    lxc.net.0.link = ipvl-meet
    lxc.net.0.ipv4.address = 192.0.2.46/26
    lxc.net.0.ipv4.gateway = 192.0.2.1
    lxc.net.0.ipv6.address = 2001:0DB8::4/128
    lxc.net.0.ipv6.gateway = fe80::1
A word of advice though: LXC likes to eat network interfaces when it fails to start a container. If you experiment, keep in mind that you may have to recreate the ipvlan interface after errors.

Re: LXC and LXD: a different container story

#83

Post by axytol »

yobbo wrote: post #80
axytol wrote: post #61

How do you all manage the security side for LXC containers? As I understand it, any local user allowed to run lxc containers can effectively spin up a root level access container on the host, example: run a privileged container and mount / inside the container? Is there anyway to mitigate this?

Only root can start privileged containers.

Maybe I misunderstood the requirements, I thought that in order to run containers a user has to belong to the lxc/lxd group. Once they belong to this group they can spawn a privileged container, which will indeed be run under root, but this effectively makes the user root too. Is there any other way for a non-root user to be limited to only unprivileged containers or have I misunderstood the requirements?

Re: LXC and LXD: a different container story

#84

Post by intelVISA »

discardedrefuse wrote: post #15

I ran LXD for about a year in my home lab. Unfortunately, the easiest way to get it running is by installing Snap; which I didn't do. Instead I ran it on Alpine Edge (one of the few distros that actually has it in their package manager). LXD kept breaking after system updates and I got tired of troubleshooting. I suppose that's just part of the perils of running bleeding edge. When LXD was running, I found that it fe…

I really love LXD/LXC conceptually but I always found it broke in mysterious ways is there a suggested base distro for imbeciles like me? I end up retreating back to Podman as well.

Re: LXC and LXD: a different container story

#85

Post by smolder »

jordemort wrote: post #12

Hi, author of the article, pleased to see it here! I'd really like to hear more from folks about how they're using LXC and/or LXD, and what they think their greatest strengths are compared to Docker or Kubernetes.

I have a proxmox system that I host lxcs on to serve as development environments for various web apps or network services that have different Linux based tech stacks. I also run some services inside lxcs on my Linux based router. Local network name services, an NFS volume, etc. The strength is the simplicity. I found it all easy to set up and very low maintenance.

Re: LXC and LXD: a different container story

#86

Post by flemhans »

thanatos519 wrote: post #7

I have been running a handful of Debian systems in containers since OpenVZ was state-of-the art. As the hardware expired, I moved to linux-vserver, then LXC, then LXD ... at which point it all shifted from straightforward text configuration files to SQLite mediated by CLI programs. When I could not figure out the incantation required for some gnarly configuration rotations ("computer says no"), I was forced to shut e…

I miss linux-vserver. Easily the most simple containerization I ever worked with.

Re: LXC and LXD: a different container story

#87

Post by MuffinFlavored »

sreevisakh wrote: post #59
MuffinFlavored wrote: post #49

Earlier quoted context omitted.

> The first is that you could try out multi-node K8s clusters on a single system. Why can't you do this without BTRFS/LXD and just with Docker/OCI?

Yes, that's certainly possible - especially with tools like kind. However, the reason I prefer LXD over Docker/OCI is that the former behaves more like a full OS. LXD containers are system containers which are like VMs except for the lack of guest kernel. So it allows me to experiment with some K8s deployment configurations that's needed on a proper cluster.

> LXD containers are system containers which are like VMs except for the lack of guest kernel.

How is this different than a Docker/OCI container/cgroup thingie?

Re: LXC and LXD: a different container story

#88

Post by fariszr »

fuzzy2 wrote: post #82
fariszr wrote: post #50

Earlier quoted context omitted.

You are using hetzner right? I'm interested to see how you bypassed the Mac address problem, any blogs, guides ?

Yes, it’s Hetzner. The magic ingredient is ipvlan , a special interface type loosely related to macvlan . An ipvlan interface is tied to a physical interface. Whether traffic goes to the virtual interface is decided based on the IP address alone. The ipvlan virtual interface can be moved to a different network namespace and will still work. Note that Hetzner added support for multiple MAC addresses in the meantime. S…

> Note that Hetzner added support for multiple MAC addresses in the meantime. So at least for IPv4, you don’t need this.

Does this apply to the IP supplied with the server? Because I can't find any Mac address option for it.

Re: LXC and LXD: a different container story

#89

Post by discardedrefuse »

intelVISA wrote: post #84
discardedrefuse wrote: post #15

I ran LXD for about a year in my home lab. Unfortunately, the easiest way to get it running is by installing Snap; which I didn't do. Instead I ran it on Alpine Edge (one of the few distros that actually has it in their package manager). LXD kept breaking after system updates and I got tired of troubleshooting. I suppose that's just part of the perils of running bleeding edge. When LXD was running, I found that it fe…

I really love LXD/LXC conceptually but I always found it broke in mysterious ways is there a suggested base distro for imbeciles like me? I end up retreating back to Podman as well.

According to other comments OpenSuse and Arch have it stable now and the next release of Debian will have it.

Re: LXC and LXD: a different container story

#90

Post by antod »

chubot wrote: post #38

Do either of them support the concept of “layers” like Docker does? I think that feature combined with overlayfs2 is quite useful, despite my many criticisms of Docker. It is sort of a middle ground between Nix like granularity which requires rewriting upstream, and big LXC blobs created with shell scripts. Although I also think we need some kind of middle ground between docker and nix :)

From memory, the early versions of Docker were basically just combining LXC with an image implementation using overlayfs. They then switched to spinning up their own container processes directly.

Storage with LXC is just a block device, so you could roll your own using overlayfs. Probably a fair amount of work though. And in the other direction, I have used btrfs instead of overlayfs with Docker.

Post reply on HN