Unfashionably secure: why we use isolated VMs • Hacker News Board

Re: Unfashionably secure: why we use isolated VMs

#211

Post by pxc » Fri, Jul 26, 2024, 2:29 PM UTC

sgarland wrote: post #196

pxc wrote: post #152

Earlier quoted context omitted.

This is nice in its uniformity (same tool works for any distro that has an existing AMI to work with), but it's insanely slow compared to just putting a rootfs together and uploading it as an image. I think I'd usually rather just use whatever distro-specific tools for putting together a li'l chroot (e.g., debootstrap, pacstrap, whatever) and building a suitable rootfs in there, then finish it up with amazon-ec2-ami-…

I haven’t played with chroot since Gentoo (which for me, was quite a while ago), so I may be incorrect, but isn’t that approach more limited in its customization? As in, you can install some packages, but if you wanted to add other repos, configure 3rd party software, etc. you’re out of luck.

Nah you can add other repos in a chroot! The only thing you can't really do afaik is test running a different kernel; for that you've got to actually boot into the system.

If you dual-boot multiple Linux systems you can still administer any of the ones you're not currently running via chroot at any time, and that works fine whether you've got third-party repositories or not. A chroot is also what you'd use to reinstall the bootloader on a system where Windows has nuked the MBR or the EFI vars or whatever.

There might be some edge cases like software that requires a physical hardware token to be installed for licensing purposes is very aggressive, so it might also try to check if it's running in a chroot, container, or VM and refuse to play nice or something like that. But generally you can do basically anything in a chroot that you might do in a local container, and 99% of what you might do in a local VM.

Re: Unfashionably secure: why we use isolated VMs

#212

Post by pxc » Fri, Jul 26, 2024, 2:37 PM UTC

mananaysiempre wrote: post #208

pxc wrote: post #151

Earlier quoted context omitted.

> so (from my perspective), they punted the entire concept of making what I would refer to as a "layer aware linux distribution" Would you consider there to be any 'layer-aware Linux distributions' today, e.g., NixOS, GuixSD, rpm-ostree-based distros like Fedora CoreOS, or distri? > so much duplicate layers out in the world Have you seen this, which lets existing container systems understand a Linux package manager's…

(Not GP.) NixOS can share its Nix store with child (systemd-nspawn) containers. That is, if you go all in, package everything using Nix, and then carefully ensure you don’t have differing (transitive build- or run-time) dependency versions anywhere, those dependencies will be shared to the maximum extent possible. The amount of sharing you actually get matches the effort you put into making your containers use the sa…

> On the other hand, Nixpkgs (which NixOS is built on top of) doesn’t really follow a discipline of minimizing package sizes to the extent that, say, Alpine does. You fairly often find documentation and development components living together with the runtime ones, especially for less popular software. (The watchword here is “closure size”, as in the size of a package and all of its transitive runtime dependencies.)

Yep. I remember before Nix even had multi-output derivations! I once broke some packages trying to reduce closure sizes when that feature got added, too. :(

Besides continuing to split off more dev and doc outputs, it'd be cool if somehow Nixpkgs had a `pkgsForAnts` just like it has a `pkgsStatic`, where packages just disable more features and integrations. On the other hand, by the time you're really optimizing your Nix container builds it's probably well worth it to use overrides and build from source anyway, binary cache be damned.

Re: Unfashionably secure: why we use isolated VMs

#213

Post by icedchai » Fri, Jul 26, 2024, 2:44 PM UTC

pxc wrote: post #118

vundercind wrote: post #34

Earlier quoted context omitted.

Docker’s the best cross-distro rolling-release package manager and init system for services—staying strictly out of managing the base system, which is great—that I know of. I don’t know of anything that’s even close, really. All the other stuff about it is way less important to me than that part.

This is wrong in pretty much every way I can imagine. Docker's not a package manager. It doesn't know what packages are, which is part of why the chunks that make up Docker containers (image layers) are so coarse. This is also part of why many Docker images are so huge: you don't know exactly the packages you need, strictly speaking, so you start from a whole OS. This is also why your Dockerfiles all invoke real pack…

When I read the parent comment, I was picturing package manager and init system in quotes. Docker is the "package manager" for people who don't want to package their apps with a real package manager. It's a "service manager" and "init system" that can restart your services (containers) on boot or when they fail.

Re: Unfashionably secure: why we use isolated VMs

#214

Post by trueismywork » Fri, Jul 26, 2024, 2:46 PM UTC

topspin wrote: post #61

PedroBatista wrote: post #2

As a permanent "out of style" curmudgeon in the last ~15 years, I like that people are discovering that maybe VMs are in fact the best approach for a lot of workloads and the LXC cottage industry and Docker industrial complex that developed around solving problems created by themselves or solved decades ago might need to take a hike. Modern "containers" were invented to make things more reproducible ( check ) and sim…

Isn't this discussion based on a false dichotomy? I, too, use VMs to isolate customers, and I use containers within those VMs, either with or without k8s. These tools solve different problems. Containers solve software management, whereas VMs provide a high degree of isolation. Container orchestration is where I see the great mistake in all of this. I consider everything running in a k8s cluster to be one "blast doma…

Which language do you develop in?

Re: Unfashionably secure: why we use isolated VMs

#215

Post by marcosdumay » Fri, Jul 26, 2024, 2:49 PM UTC

smallmancontrov wrote: post #31

neaanopri wrote: post #23

Earlier quoted context omitted.

The modern developer yearns for Java

I had to use eclipse the other day. How the hell is it just as slow and clunky as I remember from 20 years ago? Does it exist in a pocket dimension where Moore's Law doesn't apply?

It's not as slow as it was 20 years ago.

It's only as slow as you remember because the actual performance was so bad that you can't physiologically remember it.

Re: Unfashionably secure: why we use isolated VMs

#216

Post by marcosdumay » Fri, Jul 26, 2024, 2:55 PM UTC

thinkst wrote: post #173

jmprspret wrote: post #146

Earlier quoted context omitted.

DNS tunneling, or smuggling through DNS requests, is like a known malware C2 method. Seems really weird to (ab)use it for ""security""

Product builders can learn loads from malware in terms of deployment and operational ease. Malware needs to operate without any assistance in unknown environments. Nobody is allowing outbound comms deliberately for malware, so tunnel methods were developed. Networks have these capabilities, inherently they're part of the specs. But only malware seems to realise that and use it. We love reusing offensive techniques fo…

The problem is that when you apply the malware lessons to your software, every anti-virus starts to work against you.

Re: Unfashionably secure: why we use isolated VMs

#217

Post by crabbone » Fri, Jul 26, 2024, 3:02 PM UTC

ranger207 wrote: post #112

PedroBatista wrote: post #2

As a permanent "out of style" curmudgeon in the last ~15 years, I like that people are discovering that maybe VMs are in fact the best approach for a lot of workloads and the LXC cottage industry and Docker industrial complex that developed around solving problems created by themselves or solved decades ago might need to take a hike. Modern "containers" were invented to make things more reproducible ( check ) and sim…

Docker's good at packaging, and Kubernetes is good at providing a single API to do all the infra stuff like scheduling, storage, and networking. I think that if someone sat down and tried to create a idealized VM management solution that covered everything between "dev pushes changes" to "user requests website" then it'd probably have a single image for each VM to run (like Docker has a single image for each containe…

I'm sorry, but:

* Docker isn't good at packaging. When people talk about packaging, they usually understand it to include dependency management. For Docker to be good at packaging it should be able to create dependency graphs and allow users to re-create those graphs on their systems. Docker has no way of doing anything close to that. Aside from that, Docker suffers from the lack of reproducible builds, lack of upgrade protocols... It's not good at packaging... maybe it's better than something else, but there's a lot of room for improvement.

* Kubernetes doesn't provide a single API to do all the infra stuff. In fact, it provides so little, it's a mystery why anyone would think that. All those stuff like "storage", "scheduling", "networking" that you mentioned comes as add-ons (eg. CSI, CNI) which aren't developed by Kubernetes, aren't following any particular rules, have their own interfaces... Not only that, Kubernetes' integration with CSI / CNI is very lacking. For example, there's no protocol for upgrading these add-ons when upgrading Kubernetes. There's no generic interface that these add-ons have to expose to the user in order to implement common things. It's really anarchy what's going on there...

There are lots of existing VM management solutions, eg. OpenStack, VSphere -- you don't need to imagine them, they exist. They differ from Kubernetes in many ways. Very superficially, yet importantly, they don't have an easy way to automate them. For very simple tasks Kubernetes offers a very simple solution for automation. I.e. write some short YAML file. Automating eg. ESX comes down to using a library like govmomi (or something that wraps it, like Terraform). But, in the mentioned case, Terraform only managed deployment, and doesn't take care of the post-deployment maintenance... and so on.

However, the more you deal with the infra, the more you realize that the initial effort is an insignificant fraction of the overall complexity of the task you need to deal with. And that's where the management advantages of Kubernetes start to seem less appealing. I.e. you realize that you will have to write code to manage your solution, and there will be a lot of it... and a bunch of YAML files won't cut it.

Re: Unfashionably secure: why we use isolated VMs

#218

Post by ktosobcy » Fri, Jul 26, 2024, 3:08 PM UTC

skydhash wrote: post #201

ktosobcy wrote: post #184

Earlier quoted context omitted.

> Modern "containers" were invented to make things more reproducible ( check ) and simplify dev and deployments ( NOT check ). Why? I have my RPi4 and absolutely love docker(-compose) - deploying stuff/services on in it just a breeze compared to previous clusterf*k of relying on system repository for apps (or if something doesnt work)... with docker compose I have nicely separated services with dedicated databases in…

I’ve been using LXC/Incus as lightweight VMs (my home server is an old mac mini) and I think too many software is over reliant on Docker. It’s very much “ship your computer” with a bunch of odd scripts in addition to the Dockerfiles.

Could you elaborate the "ship your computer"? Majority of the images are base OS (which is as lean as possible) and then just the app...

To that end, full blown VM seems even more "ship your computer" thing?

Btw. isn't LXC base for the Docker as well? It looks somewhat similar to docker and podman?

Re: Unfashionably secure: why we use isolated VMs

#219

Post by vundercind » Fri, Jul 26, 2024, 3:34 PM UTC

icedchai wrote: post #213

pxc wrote: post #118

Earlier quoted context omitted.

This is wrong in pretty much every way I can imagine. Docker's not a package manager. It doesn't know what packages are, which is part of why the chunks that make up Docker containers (image layers) are so coarse. This is also part of why many Docker images are so huge: you don't know exactly the packages you need, strictly speaking, so you start from a whole OS. This is also why your Dockerfiles all invoke real pack…

When I read the parent comment, I was picturing package manager and init system in quotes. Docker is the "package manager" for people who don't want to package their apps with a real package manager. It's a "service manager" and "init system" that can restart your services (containers) on boot or when they fail.

Right, it gives me the key functionality of those systems in a way that’s decoupled from the base OS, so I can just run some version of Debian for five years or however long it’s got security updates, and not have to worry about things like services I want to run needing newer versions of lots of things than some old Debian has. Major version updates of my distro, or trying to get back ported newer packages in, have historically been the biggest single source of “fuck, now nothing works and there goes my weekend” running home servers, to the point of making ROI on them not so great and making me dread trying to add or update services I was running. This approach means I can install and run just about any service at any recent-ish version without touching the state of the OS itself. The underlying tech of docker isn’t directly important to me, nor any of the other features it provides, in this context, just that it’s separate from the base OS and gives me a way to “manage packages” and run services in that decoupled manner.

Re: Unfashionably secure: why we use isolated VMs

#220

Post by solatic » Fri, Jul 26, 2024, 3:44 PM UTC

There's nothing in Kubernetes and containers that prevents you from running single-tenant architectures (one tenant per namespace), or from colocating all single-tenant services on the same VM, and preventing multiple customers from sharing the same VM (pod affinity and anti-affinity).

I'm not sure why the author doesn't understand that he could have his cake and eat it too.