Live data from Hacker News

Faster filesystem access with Directfs

gvisor.dev

31–40 of 58 posts

Re: Faster filesystem access with Directfs

#31

I still don’t know why Google has gvisor and AWS has firecracker. Isn’t the firecracker approach strictly better than Google’s approach?

Firescracker is good and all but if one wants to use it, one has to change its ecosystem, it’s communication with other servers, why change your entire ecosystem for one tool or just build a tool to fit your ecosystem, and really like the concept of firecracker-containerd but still need some modifications and also I wouldn’t expect Google to put their entire Cloud Run, App engine under the hands of aws (even tho it’s FOSS)

Re: Faster filesystem access with Directfs

#32
This is a step back.

The reason to have this in a separate process is so it can be audited "to death" because the code base is small.

gvisor itself is so big that doing an exhaustive audit is out of the question. Google has mostly switched to fuzzing because the code bases have all become too bloated to audit them properly.

The reason you have gvisor is to contain something you consider dangerous. If that contained code managed to break out and take over gvisor, it is still contained in the kernel level namespaces and still cannot open files unless the broker process agrees. That process better be as small as possible then, so we can trust it to not be compromisable from gvisor.

EDIT: Hmm looks like they aren't removing the broker process, just "reducing round-trips". Never mind then. That reduces the security cost to you not being able to take write access away at run time to a file that was already opened for writing.

Re: Faster filesystem access with Directfs

#33
This article is not very good at explaining what is it they are actually describing. Is directfs just a way to access hosts local fs? If so than my understanding of it is that they used to use rpc to access local fs before (horrible overhead) to sandbox it. Now they've just replaced a part of the operating system filesystem API that resolves paths to file descriptors with their tool so once a file descriptor is obtained the container can talk directly to the fs.

To me this resolves a very narrow use case where you have to run untrusted containers on trusted hosts. This is a very narrow use case. I imagine main target users for this are people that want to offer a service like fargate and run multiple customers on a single host. Why would they want to do that instead of separating customers with VMs? My suspicion is this has something to do with the increasing availability of very energy efficient arm servers that have hundreds of cores per socket. My impression is traditional virtualisation on arm is rarely used (I'm not sure why as kvm supports it, arm since armv8.1 has hw support for it). So "containers to the rescue".

Personally I'd much rather extra security to enable untrusted containers access to the hosts fs is implemented in the container runtime, not as a separate component. Or if the "security issues" it addresses perhaps even in the hosts operating system?

Re: Faster filesystem access with Directfs

#34
post #20
post #19

Accessing local file systems from a container? What heresy is this? Containers must all be stateless webscale single-"process" microservices with no need of local file systems and other obsolescent concepts. Next thing you know someone will run as many as two whole "processes" in a container! Having dispensed with that bit of bitter sarcasm; solving their local filesystem performance/security problems is great and al…

A container, being basically a chroot, consumes a rather small amount of resources, mostly as space in namespace and ipfilter tables. If your containers use many of the same base layers (e.g. the same Node or Python image), the code pages will be shared, as they would be shared with plain OS processes. Running several processes in a container is the norm. First, you run with --init anyway, so there is a `tini` parent…

I quite like containers to limit/reserve the ram/cpu use for certain processes. For example imagine a tiny service used by Few concurrent users that needs a SQL db, a app server, and a reverse proxy (For ssl/caching) in front. I'm quite happy to put stuff like this on a tiny VM with 1vCPU and 1GB RAM. Mnthly cost ~$5 for compute. I typically reserve/limit 64MB/128MB for nginx, 384mb/512mb for mariadb, and 256mb/384mb for the app server (PHP etc).Also I have cpu share reservation/limits too. Of course it requires tuning the configs, but it runs great(verified with load testing and actual use). If you put the same software on the same host with no reservations/limits there are situations where latencies grow a lot or the whole thing freezes because one component consumes too much resources. If anyone knows any non-container lie overhead ways to partition a single vcpu and a gig of ram like this I'd be interested to hear about it.

Re: Faster filesystem access with Directfs

#35
post #34
post #20

Earlier quoted context omitted.

A container, being basically a chroot, consumes a rather small amount of resources, mostly as space in namespace and ipfilter tables. If your containers use many of the same base layers (e.g. the same Node or Python image), the code pages will be shared, as they would be shared with plain OS processes. Running several processes in a container is the norm. First, you run with --init anyway, so there is a `tini` parent…

I quite like containers to limit/reserve the ram/cpu use for certain processes. For example imagine a tiny service used by Few concurrent users that needs a SQL db, a app server, and a reverse proxy (For ssl/caching) in front. I'm quite happy to put stuff like this on a tiny VM with 1vCPU and 1GB RAM. Mnthly cost ~$5 for compute. I typically reserve/limit 64MB/128MB for nginx, 384mb/512mb for mariadb, and 256mb/384mb…

> If anyone knows any non-container lie overhead ways to partition a single vcpu and a gig of ram like this I'd be interested to hear about it.

You can use cgroups[1] to do this, because that's what your container runtime is doing. People don't know this because they think these features have something to do with their container runtime and that's what they use, so no one discovers it.

Plus, the user facing tools for cgroups are slightly hideous. And that won't ever get fixed for the reasons previously stated. Sigh.

Also, I'm sure a lot of people would appreciate learning about your tuning techniques, containers or otherwise. Consider writing it up.

[1] circa 2007...

Re: Faster filesystem access with Directfs

#36
post #33

This article is not very good at explaining what is it they are actually describing. Is directfs just a way to access hosts local fs? If so than my understanding of it is that they used to use rpc to access local fs before (horrible overhead) to sandbox it. Now they've just replaced a part of the operating system filesystem API that resolves paths to file descriptors with their tool so once a file descriptor is obtai…

> Personally I'd much rather extra security to enable untrusted containers access to the hosts fs is implemented in the container runtime, not as a separate component. Or if the "security issues" it addresses perhaps even in the hosts operating system?

Isn’t that exactly what the original gofer/RPC solution is? The gvisor container runtime operates in userland to ensure that compromises in the runtime don’t result in an immediate compromise of the system kernel.

But running in userland and intercepting syscalls that do IO always has significant performance implications, because all your IO now needs multiple copy operations to get into the reading process address space, because userland process generally can’t directly interact with each other address space (to ensure process isolation), without asking the kernel to come in to do all the heavy lifting.

So if you want fast local IO, you have find a way to allowing the untrusted processes in the container to make direct syscalls, so that you can avoid all the additional high latency hops in userland, and let the kernel directly copy data into the target processes address space.

To magically allow the container runtime to provide direct host fs access itself, with native level performance, that would require the runtime to be operating as part of the kernel. Which is exactly how normal containers work, comes with a whole load of security risks, and is ultimately the reason gvisor exists.

Re: Faster filesystem access with Directfs

#37
post #26

Earlier quoted context omitted.

DirectStorage does, thought.

Ah... Okay, I think I see how the comment should have been read now...? I will blame whoever named directfs for using a confounding name one way or the other. :V

I think the comment which was in response to mine is pretty ambiguous, it isn’t obvious to me what they mean by “here” (in the article, in the comment I responded to, or in my comment?).

Re: Faster filesystem access with Directfs

#39

I still don’t know why Google has gvisor and AWS has firecracker. Isn’t the firecracker approach strictly better than Google’s approach?

Firecracker does not work with long running process. It's only good for function as a service / serverless stuff.

Re: Faster filesystem access with Directfs

#40
post #34
post #20

Earlier quoted context omitted.

A container, being basically a chroot, consumes a rather small amount of resources, mostly as space in namespace and ipfilter tables. If your containers use many of the same base layers (e.g. the same Node or Python image), the code pages will be shared, as they would be shared with plain OS processes. Running several processes in a container is the norm. First, you run with --init anyway, so there is a `tini` parent…

I quite like containers to limit/reserve the ram/cpu use for certain processes. For example imagine a tiny service used by Few concurrent users that needs a SQL db, a app server, and a reverse proxy (For ssl/caching) in front. I'm quite happy to put stuff like this on a tiny VM with 1vCPU and 1GB RAM. Mnthly cost ~$5 for compute. I typically reserve/limit 64MB/128MB for nginx, 384mb/512mb for mariadb, and 256mb/384mb…

You can set up same limits via systemd units as they are using same interfaces as containers. They can be set up via systemd overrides to existing services so they are pretty straightforward

Just be beware of same problems like swap trap (limiting only memory and not memory and swap will just make apps hitting the memory limit start to swap like hell)

Post reply on HN