Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LXC and LXD: a different container story

lwn.net

Post reply on HN
Print view View original

51–60 of 102 posts

Re: LXC and LXD: a different container story

#51

Post by leetnewb »

jordemort wrote: post #12

Hi, author of the article, pleased to see it here! I'd really like to hear more from folks about how they're using LXC and/or LXD, and what they think their greatest strengths are compared to Docker or Kubernetes.

I use lxd for running core services at home, such as samba/sftp/webdav or home automation. I find it easier to attach a vpn to the container, which combined with a nat traversing sdn/mesh gives me easy, private, encrypted access to data or services. Also easy to pass a usb device (say zwave controller) to a container. I run it over a btrfs array and have a workflow to snapshot containers and send to a backup host. Using unique namespaces for each unprivileged container, I am reasonably confident that a security oops in one container should be isolated.

I also use lxd on vps hosts, often nesting docker to give more granular control over networking.

Re: LXC and LXD: a different container story

#52

Post by kwk1 »

discardedrefuse wrote: post #15

I ran LXD for about a year in my home lab. Unfortunately, the easiest way to get it running is by installing Snap; which I didn't do. Instead I ran it on Alpine Edge (one of the few distros that actually has it in their package manager). LXD kept breaking after system updates and I got tired of troubleshooting. I suppose that's just part of the perils of running bleeding edge. When LXD was running, I found that it fe…

> Unfortunately, the easiest way to get it running is by installing Snap;

The Debian package for LXD just entered the archive about 2 weeks ago:

https://tracker.debian.org/news/1361535/accepted-lxd-500-1-s...

This means the next Debian release will make for stable LXD hosts without needing snap.

Re: LXC and LXD: a different container story

#53

Post by kwk1 »

ansible wrote: post #45
chubot wrote: post #38

Do either of them support the concept of “layers” like Docker does? I think that feature combined with overlayfs2 is quite useful, despite my many criticisms of Docker. It is sort of a middle ground between Nix like granularity which requires rewriting upstream, and big LXC blobs created with shell scripts. Although I also think we need some kind of middle ground between docker and nix :)

> Do either of them support the concept of “layers” like Docker does? Never used LXD, but LXC does not have layers per-se. I usually run LXC containers on a btrfs filesystem, which easily supports snapshots and sending containers to other container hosts via "btrfs send | ssh otherhost btrfs receive". If you are treating your servers like pets (and not cattle) LXC is a very convenient means to consolidate servers ont…

> Never used LXD, but LXC does not have layers per-se.

It's kind of a similar situation in LXD, where that sort of functionality being available depends on the storage backend/driver in use, for example LVM thin provisioning, various options via ZFS, or btrfs like you mentioned.

Re: LXC and LXD: a different container story

#55

Post by chubot »

kwk1 wrote: post #53
ansible wrote: post #45

Earlier quoted context omitted.

> Do either of them support the concept of “layers” like Docker does? Never used LXD, but LXC does not have layers per-se. I usually run LXC containers on a btrfs filesystem, which easily supports snapshots and sending containers to other container hosts via "btrfs send | ssh otherhost btrfs receive". If you are treating your servers like pets (and not cattle) LXC is a very convenient means to consolidate servers ont…

> Never used LXD, but LXC does not have layers per-se. It's kind of a similar situation in LXD, where that sort of functionality being available depends on the storage backend/driver in use, for example LVM thin provisioning, various options via ZFS, or btrfs like you mentioned.

Hm yeah, in some sense I like the things to be more orthogonal, i.e. container images vs. the file system

On the other hand I think storage and networking should be more unified, and the differential compression and caching you get from layers is important

I suspect that doing the layers at the logical level (files) , like overlayfs, rather than the physical level (blocks) is also better for containers, but I'd be interested to read any comparisons

Re: LXC and LXD: a different container story

#56

Post by kiririn »

systemd-nspawn is worth a look - I found it more intuitive, well designed and security conscious (within reason for containers) than the other container options. It must get some serious use behind closed doors somewhere, because the quality outstrips the amount of publicity it gets

Re: LXC and LXD: a different container story

#57

Post by dspillett »

fariszr wrote: post #50
fuzzy2 wrote: post #27

I fucking love LXC! It allowed me to “virtualize” several physical servers on a single newer server, keeping all of the original setup (except some networking changes). Combined with ipvlan , I could even work around the MAC address restriction of my server provider, with both IPv4 and IPv6. I also use it to host a Jitsi instance. Jitsi is rather picky about the underlying distribution. All this without the overhead…

You are using hetzner right? I'm interested to see how you bypassed the Mac address problem, any blogs, guides ?

I'm not sure what the problem is, but if it is stopping you bridging the physical interface effectively, could you not define a virtual bridge with local only addresses and have the host NAT to/from that and the physical interface? Adds a little extra latency to be sure, more so if you have two or more boxes on the same network with containers that would otherwise talk directly to each other on a private vlan, but if your physical box is being a firewall/router for the tasks in the containers anyway it is already practically doing the job.

Re: LXC and LXD: a different container story

#58

Post by cmurf »

chubot wrote: post #55
kwk1 wrote: post #53

Earlier quoted context omitted.

> Never used LXD, but LXC does not have layers per-se. It's kind of a similar situation in LXD, where that sort of functionality being available depends on the storage backend/driver in use, for example LVM thin provisioning, various options via ZFS, or btrfs like you mentioned.

Hm yeah, in some sense I like the things to be more orthogonal, i.e. container images vs. the file system On the other hand I think storage and networking should be more unified, and the differential compression and caching you get from layers is important I suspect that doing the layers at the logical level (files) , like overlayfs, rather than the physical level (blocks) is also better for containers, but I'd be in…

It depends on the workload. Copy-up is more expensive than snapshots. If it's ext4+overlayfs, then copy-up is really expensive (full file copy), whereas if it's btrfs or XFS then it's reflink copies. Much more efficient. But btrfs snapshots in effect are instant reflink copies. But then overlayfs has the benefit of shared pages.

Also, at least Btrfs snapshots are more logical, like files; than physical, like blocks. Whereas LVM snapshots are separate (logical) block devices, Btrfs snapshots are file b-trees on the same file system.

Maybe the overlay2 graph driver (the driver used in Docker or Podman to interface with the kernel) is getting more usage and thus maturing more quickly, compared to the btrfs graph driver? I think upstream Docker/Moby are defaulting to the btrfs graph driver if /var/lib/docker is on Btrfs. Meanwhile on Fedora where Btrfs is the default for desktops, and Cloud edition, Podman defaults to overlay2 graph driver no matter the underlying file system.

Re: LXC and LXD: a different container story

#59

Post by sreevisakh »

MuffinFlavored wrote: post #49
sreevisakh wrote: post #47

Earlier quoted context omitted.

I use Kubernetes within an LXD container with Btrfs backing storage. This isn't anything special. But it has two advantages. The first is that you could try out multi-node K8s clusters on a single system. The second is that the containers can be deleted and rebuilt easily without affecting the host system. Both are very useful when you're learning multi-node K8s. I plan to expand my homelab to true multi-node setup.…

> The first is that you could try out multi-node K8s clusters on a single system. Why can't you do this without BTRFS/LXD and just with Docker/OCI?

Yes, that's certainly possible - especially with tools like kind. However, the reason I prefer LXD over Docker/OCI is that the former behaves more like a full OS. LXD containers are system containers which are like VMs except for the lack of guest kernel. So it allows me to experiment with some K8s deployment configurations that's needed on a proper cluster.

Re: LXC and LXD: a different container story

#60

Post by winstonschmidt »

divbzero wrote: post #11

Does LXC and LXD support the functional equivalent of Dockerfile or docker-compose.yml? I appreciate how Docker has become widely accepted for defining reproducible environments but have always found it to be rather heavy on resources.

You can use cloudinit with lxd to do something quite similar to a Dockerfile. I don't know anything like compose besides just bash scripts.
Post reply on HN