Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making containers safer

lwn.net

Post reply on HN
Print view View original

31–40 of 52 posts

Re: Making containers safer

#31

Post by nisa »

paulfurtado wrote: post #28
nisa wrote: post #13

The more I use Linux Containers (big fan of LXD) the more I'm convinced Solaris got it right with Zones, Crossbow, ZFS, SMF also for Zone services - Linux is still not there but instead we have lot's of vendor glue-code in go and many asteriks what is not possible. IMHO Linux should implement something like the zone concept that conceals cgroups, network, mount-namespaces - quota is still broken in btrfs, you can't d…

What issue did you face around network namespace awareness in this setup?

br_netfilter is not namespace aware at least (it's fixed in 5.3 afaik) but there are probably other issues.

Re: Making containers safer

#32

Post by nailer »

hardwaresofton wrote: post #8
sansnomme wrote: post #5

The whole container safety story has been a mess/cluster /bag-of-tricks since the very beginning. Unlike BSD jails, security was never number one priority for containers. Just take a look at what GCE/AWS use respectively. The former built an entire syscall proxy with Gvisor while the latter uses a hypervisor based solution with Firecracker (think mini-VMs). Anything in production that touches foriegn code can't reall…

> (The main downside of hypervisors is that they are difficult to run on low-cost commodity cloud like Digital Ocean. You are forced to use bare metal stuff like Hetzner, AWS, Paket Cloud etc. The whole point of cloud is that the hardware/infrastructure is mostly abstracted away. If I have to run my own hypervisor just to ensure the container doesn't get broken out of, what's the point even calling it cloud?) Sorry i…

> The Hypervisor you're running (firecracker, kata containers, gvisor, nabla, etc) are to protect your dangerous workloads from your other workloads.

I think it's more likely that the hypervisor someone iss running on top of their cloud provider's hypervisor exists because many people like building their own cloud providers and like to add layers of unnecessary complexity (like a second layer of hypervisors) at their employer's or investor's expense.

If someone doesn't trust their own workloads, it is vastly simpler and less expensive to run them on separate cloud instance.

Re: Making containers safer

#33

Post by hylaride »

nisa wrote: post #13

The more I use Linux Containers (big fan of LXD) the more I'm convinced Solaris got it right with Zones, Crossbow, ZFS, SMF also for Zone services - Linux is still not there but instead we have lot's of vendor glue-code in go and many asteriks what is not possible. IMHO Linux should implement something like the zone concept that conceals cgroups, network, mount-namespaces - quota is still broken in btrfs, you can't d…

Preach, brother! At a previous company we use zfs snapshots to do with Solaris containers what we now do with docker. We had push-button revertible containers(zones) for developers to muck around and test with circa 2009 and we could send them around do different pieces of hardware with `zfs send`. It tied in very nicely with our continuous integration, too.

Re: Making containers safer

#34

Post by sansnomme »

nailer wrote: post #32
hardwaresofton wrote: post #8

Earlier quoted context omitted.

> (The main downside of hypervisors is that they are difficult to run on low-cost commodity cloud like Digital Ocean. You are forced to use bare metal stuff like Hetzner, AWS, Paket Cloud etc. The whole point of cloud is that the hardware/infrastructure is mostly abstracted away. If I have to run my own hypervisor just to ensure the container doesn't get broken out of, what's the point even calling it cloud?) Sorry i…

> The Hypervisor you're running (firecracker, kata containers, gvisor, nabla, etc) are to protect your dangerous workloads from your other workloads. I think it's more likely that the hypervisor someone iss running on top of their cloud provider's hypervisor exists because many people like building their own cloud providers and like to add layers of unnecessary complexity (like a second layer of hypervisors) at their…

Spawn time is fairly significant for your typical VMs. An entire class of applications (such as serverless) cannot be used as your usual traditional VM creation takes several minutes while the hybrid container-VM types do it in milliseconds, allowing for near real-time applications.

Re: Making containers safer

#35

Post by nailer »

sansnomme wrote: post #34
nailer wrote: post #32

Earlier quoted context omitted.

> The Hypervisor you're running (firecracker, kata containers, gvisor, nabla, etc) are to protect your dangerous workloads from your other workloads. I think it's more likely that the hypervisor someone iss running on top of their cloud provider's hypervisor exists because many people like building their own cloud providers and like to add layers of unnecessary complexity (like a second layer of hypervisors) at their…

Spawn time is fairly significant for your typical VMs. An entire class of applications (such as serverless) cannot be used as your usual traditional VM creation takes several minutes while the hybrid container-VM types do it in milliseconds, allowing for near real-time applications.

It's not clear how what you're saying relates to nested hypervisors, but I don't think anyone does serverless on nested hypervisors. AWS Lambda uses MicroVMs which to my knowledge are not nested.

Re: Making containers safer

#36

Post by e12e »

bscphil wrote: post #3

This was a pretty good read. I use containers quite a lot on my server at home at maintain a bunch of utilities. Mostly I'm using systemd-nspawn. > User namespaces have been around since the 3.12 kernel, but few other container management systems use the feature to isolate their containers. Part of the reason for that is the difficulty in sharing files between containers because of the UID mapping. LXD is currently u…

I'm a little surprised that lxd doesn't cater more to nfs rather than bind mounts etc for file sharing. I mean if you already have private/secure networking - just use a network filesystem for your... Network filesystem needs?

Re: Making containers safer

#37

Post by eyberg »

hardwaresofton wrote: post #8
sansnomme wrote: post #5

The whole container safety story has been a mess/cluster /bag-of-tricks since the very beginning. Unlike BSD jails, security was never number one priority for containers. Just take a look at what GCE/AWS use respectively. The former built an entire syscall proxy with Gvisor while the latter uses a hypervisor based solution with Firecracker (think mini-VMs). Anything in production that touches foriegn code can't reall…

> (The main downside of hypervisors is that they are difficult to run on low-cost commodity cloud like Digital Ocean. You are forced to use bare metal stuff like Hetzner, AWS, Paket Cloud etc. The whole point of cloud is that the hardware/infrastructure is mostly abstracted away. If I have to run my own hypervisor just to ensure the container doesn't get broken out of, what's the point even calling it cloud?) Sorry i…

In the public cloud (aws/gce) we provision unikernels as vms themselves - that is there is no underlying linux instance. It's just the unikernel riding on top of the hypervisor (in the case of google that's kvm, for the t2s on aws it's xen but their c5s are kvm). For small services you use t2-micros (aws) and f1-micros (gce). Both will give you a thread. If you need more you scale more.

This works great if you have a handful of services - if you have hundreds or thousands of services yes you might want to look at running on your own hardware as the public cloud could get expensive at that point or conversely you can re-evaluate if you need hundreds of services and maybe just opt for larger services that use heftier amounts of threads/memory (eg: larger instance sizes). As for running your own hardware we have crappy (eg: $800) servers in our office that have 32 threads and we haven't really stressed it at all but we can easily run thousands of unikernels on those boxes. Also, in the bay area you can grab a cabinet from Hurricane Electric for $400/month. Imagine what you could do with real hardware.

Re: Making containers safer

#38

Post by laurieodgers »

Honestly its horses for courses.

If you want to run your workload in the most secure manner using BSD jails then go for it, but you will soon find that anyone with expertise to maintain it is hard to find. Almost all DevOps/SRE/Systems Engineers want to run their workloads (containers) in Docker.

The same goes for LXD.

Lets get one thing right - none of these systems are a Virtual Machine with their own hardware and kernel. All 3 systems share the hosts kernel, albeit BSD has separation of workloads as a consideration within the kernel itself. The Linux kernel has cgroups as a building block, which to me almost makes it an afterthought.

I would never run a container with hostile code on the same Docker host as my mission critical workload. Hell, not even the same network (physical/vpc/overlay). However, I would give it more consideration if the same workloads were on a BSD machine with Jail separation. This was the premise of what I was employed to build in my previous job - https://github.com/tredly/tredly . A better separation of host and jail workload can be achieved though BSD jails as separation of concerns has been taken into consideration within the kernel, instead of building on top of the kernel.

This article is 2 years old. Linux cgroups and BSD jails have come leaps and bounds since. Docker may have been the "flavour of the month" back then, but it continues to be the DevOps/SRE/Systems Engineer's tool of choice. Its not a silver bullet by any means, and has many shortcoming when used on its own. This why systems continue to be build on top of systems (tar -> apt -> docker -> k8s -> ?)

Of course all of the above assumes someone in my position as a DevOps engineer consuming resources upon either on-prem infrastructure or within the cloud. Workloads from an IaaS perspective are a completely different kettle of fish.

Re: Making containers safer

#39

Post by dragonsh »

paulfurtado wrote: post #26
dragonsh wrote: post #22

Earlier quoted context omitted.

We tried kata the user experience is not as good as lxd. Moreover with lxd just use the same deployment platform, scripts as used for bare-metal and VM. No need to fiddle with mix of shell scripts and DSL for container orchestration and how to integrate with your own code. No need for zombie process like in docker.

Kata is great if you're already using docker, it also is increased isolation versus lxd. If you don't already have a use case for docker, kubernetes, the image repository features, etc then of course kata wouldn't be useful to you over lxc. Also, in case you're dealing with zombie processes in docker again and can't ditch docker, running a container with `docker run --init` gets you a minimal init process that reaps…

Thanks for the information and indeed I looked at Kata again. But just it didn't click like the way LXD did. I found with LXD the image repository are so easy, deployment my team can re-use whatever they learned for bare-metal and virtual machines and last but not the least its performant with nice tooling.

Initially when I started using had few networking issues due to different support in GCP, AWS and Azure. But now my team is proficient enough to use it even on them. Also the constant updates Stephen and his team did is fantastic.

If you have time please look into weekly updates and play with LXD. It works great for reasonable sized container cluster which covers most of the startups.

They won't be good for big google size of kind yet without more tooling like Kubernetes.

Re: Making containers safer

#40

Post by oblio »

wikibob wrote: post #19
scaryclam wrote: post #15

Earlier quoted context omitted.

I'm curious as to how you get your images built. I'd love to remove the over reliance on docker in my team, but the sticking point seems to be building and storing images. I know that Kubernetes can run with different runtimes, and that docker images can be oci compliant, but it's a hard sell to ask everyone to try something different in production when they do everything else with docker. Do you have a workflow that…

Bazel can natively build images without Docker. Bazel has many other benefits too like creating a full dependency build graph and fully reproductible builds. https://bazel.build/ https://github.com/bazelbuild/rules_docker

Both your solution, Bazel, and that of the neighboring comment, Gitlab, involve way too many parts you don't necessarily want or need. I'm not going to rewrite my build process (which is generally a thankless job with very low customer ROI) just to build container images :-)

Same story for Gitlab, nobody's going to migrate from Github or Bitbucket just to get container images builds.

Post reply on HN