I still don’t know why Google has gvisor and AWS has firecracker. Isn’t the firecracker approach strictly better than Google’s approach?
Faster filesystem access with Directfs
31–40 of 58 posts
Re: Faster filesystem access with Directfs
#32The reason to have this in a separate process is so it can be audited "to death" because the code base is small.
gvisor itself is so big that doing an exhaustive audit is out of the question. Google has mostly switched to fuzzing because the code bases have all become too bloated to audit them properly.
The reason you have gvisor is to contain something you consider dangerous. If that contained code managed to break out and take over gvisor, it is still contained in the kernel level namespaces and still cannot open files unless the broker process agrees. That process better be as small as possible then, so we can trust it to not be compromisable from gvisor.
EDIT: Hmm looks like they aren't removing the broker process, just "reducing round-trips". Never mind then. That reduces the security cost to you not being able to take write access away at run time to a file that was already opened for writing.
Re: Faster filesystem access with Directfs
#33To me this resolves a very narrow use case where you have to run untrusted containers on trusted hosts. This is a very narrow use case. I imagine main target users for this are people that want to offer a service like fargate and run multiple customers on a single host. Why would they want to do that instead of separating customers with VMs? My suspicion is this has something to do with the increasing availability of very energy efficient arm servers that have hundreds of cores per socket. My impression is traditional virtualisation on arm is rarely used (I'm not sure why as kvm supports it, arm since armv8.1 has hw support for it). So "containers to the rescue".
Personally I'd much rather extra security to enable untrusted containers access to the hosts fs is implemented in the container runtime, not as a separate component. Or if the "security issues" it addresses perhaps even in the hosts operating system?
Re: Faster filesystem access with Directfs
#34Accessing local file systems from a container? What heresy is this? Containers must all be stateless webscale single-"process" microservices with no need of local file systems and other obsolescent concepts. Next thing you know someone will run as many as two whole "processes" in a container! Having dispensed with that bit of bitter sarcasm; solving their local filesystem performance/security problems is great and al…
A container, being basically a chroot, consumes a rather small amount of resources, mostly as space in namespace and ipfilter tables. If your containers use many of the same base layers (e.g. the same Node or Python image), the code pages will be shared, as they would be shared with plain OS processes. Running several processes in a container is the norm. First, you run with --init anyway, so there is a `tini` parent…
Re: Faster filesystem access with Directfs
#35Earlier quoted context omitted.
A container, being basically a chroot, consumes a rather small amount of resources, mostly as space in namespace and ipfilter tables. If your containers use many of the same base layers (e.g. the same Node or Python image), the code pages will be shared, as they would be shared with plain OS processes. Running several processes in a container is the norm. First, you run with --init anyway, so there is a `tini` parent…
I quite like containers to limit/reserve the ram/cpu use for certain processes. For example imagine a tiny service used by Few concurrent users that needs a SQL db, a app server, and a reverse proxy (For ssl/caching) in front. I'm quite happy to put stuff like this on a tiny VM with 1vCPU and 1GB RAM. Mnthly cost ~$5 for compute. I typically reserve/limit 64MB/128MB for nginx, 384mb/512mb for mariadb, and 256mb/384mb…
You can use cgroups[1] to do this, because that's what your container runtime is doing. People don't know this because they think these features have something to do with their container runtime and that's what they use, so no one discovers it.
Plus, the user facing tools for cgroups are slightly hideous. And that won't ever get fixed for the reasons previously stated. Sigh.
Also, I'm sure a lot of people would appreciate learning about your tuning techniques, containers or otherwise. Consider writing it up.
[1] circa 2007...
Re: Faster filesystem access with Directfs
#36This article is not very good at explaining what is it they are actually describing. Is directfs just a way to access hosts local fs? If so than my understanding of it is that they used to use rpc to access local fs before (horrible overhead) to sandbox it. Now they've just replaced a part of the operating system filesystem API that resolves paths to file descriptors with their tool so once a file descriptor is obtai…
Isn’t that exactly what the original gofer/RPC solution is? The gvisor container runtime operates in userland to ensure that compromises in the runtime don’t result in an immediate compromise of the system kernel.
But running in userland and intercepting syscalls that do IO always has significant performance implications, because all your IO now needs multiple copy operations to get into the reading process address space, because userland process generally can’t directly interact with each other address space (to ensure process isolation), without asking the kernel to come in to do all the heavy lifting.
So if you want fast local IO, you have find a way to allowing the untrusted processes in the container to make direct syscalls, so that you can avoid all the additional high latency hops in userland, and let the kernel directly copy data into the target processes address space.
To magically allow the container runtime to provide direct host fs access itself, with native level performance, that would require the runtime to be operating as part of the kernel. Which is exactly how normal containers work, comes with a whole load of security risks, and is ultimately the reason gvisor exists.
Re: Faster filesystem access with Directfs
#37Earlier quoted context omitted.
DirectStorage does, thought.
Ah... Okay, I think I see how the comment should have been read now...? I will blame whoever named directfs for using a confounding name one way or the other. :V
Re: Faster filesystem access with Directfs
#38Re: Faster filesystem access with Directfs
#39I still don’t know why Google has gvisor and AWS has firecracker. Isn’t the firecracker approach strictly better than Google’s approach?
Re: Faster filesystem access with Directfs
#40Earlier quoted context omitted.
A container, being basically a chroot, consumes a rather small amount of resources, mostly as space in namespace and ipfilter tables. If your containers use many of the same base layers (e.g. the same Node or Python image), the code pages will be shared, as they would be shared with plain OS processes. Running several processes in a container is the norm. First, you run with --init anyway, so there is a `tini` parent…
I quite like containers to limit/reserve the ram/cpu use for certain processes. For example imagine a tiny service used by Few concurrent users that needs a SQL db, a app server, and a reverse proxy (For ssl/caching) in front. I'm quite happy to put stuff like this on a tiny VM with 1vCPU and 1GB RAM. Mnthly cost ~$5 for compute. I typically reserve/limit 64MB/128MB for nginx, 384mb/512mb for mariadb, and 256mb/384mb…
Just be beware of same problems like swap trap (limiting only memory and not memory and swap will just make apps hitting the memory limit start to swap like hell)