[flagged]
We improved the performance of a userspace TCP stack in Go
91–100 of 133 posts
Re: We improved the performance of a userspace TCP stack in Go
#92Earlier quoted context omitted.
TCP overhead is part of the story. Theres 20-40x overhead in syscalls, 20% running a tensorflow project end to end, 50% fewer RPS in redis, etc.
We are still talking about people using runsc/runc. That's not what `coder` is doing. All they did was poach a (popular) networking library from the gVisor codebase. None of this benchmarking has anything to do with their product.
I've already accepted this whole thread is a digression, but I keep getting pulled in. Calling out my dislike for Gvisor on a thread lauding a 5x tcp performance they found in it felt on topic to me at the time.
Re: We improved the performance of a userspace TCP stack in Go
#93Earlier quoted context omitted.
Are you referring to gVisor the container runtime, or gVisor/netstack, the TCP/IP stack? I see more uptick in netstack. I don't see proliferation of gVisor itself. "Security" is much more salient to gVisor than it is to netstack.
In the issue of abysmal performance on cloud-compute/PaaS Im talking about the container runtime (most Paas is gVisor or Firecracker, no?) cloudrun, DO, modal, etc. But given this article is about improving gvisors userland tcp performance significantly, it seems like the netstack stuff causes major performance losses too. I saw a github link in another top article today https://github.com/misprit7/computerraria wher…
Google is my former employer and this statement isn't referring to stuff I heard while employed there.
But after I left, I heard a that alot of the poor performance of Cloud Run is just plain old oversubscribed shared core e2 stuff.
Re: We improved the performance of a userspace TCP stack in Go
#94Earlier quoted context omitted.
> I don't understand - what do you suggest as an alternative to Gvisor? Their performance documents you linked claim vs runc: 20-40x syscall overhead, half of redis' QPS, and a 20% increase in runtime in a sample tenserflow script. Also google "CloudRun slow" and "Digital Ocean Apps slow", both are Gvisor. Literally anything else.
A decent while ago, I was the original author of that performance guide. I tried to lay out the set of performance trade-offs in an objective and realistic way. It is shocking to me that you’re spending so much time commenting on a few figures from there, ostensibly w/o reading it. System call overhead does matter, but it’s not the ultimate measure of anything. If it were, gVisor with the KVM platform would be faster…
> it is shocking to me that you’re spending so much time commenting on a few figures from there
You give me too much credit! They were copy pastes to the same responder who responded to me in a few places in the thread. I did that to avoid spending too much time responding!
> because looking at any one figure in isolation is dumb
So the self-reported performance figures are bad, the are hundreds of web pages and support pages reporting slow performance and low startup time from their first hand experience, there are Google hosted documentation pages about how to improve app performance for cloudrun (probably the largest user and creators of Gvisor, can I assume they know how to run it?) including gems like "delete temporary files" and a blog post recommending "using global variables" (I'm not joking). And the accusation is "dumb" cherry-picking? Huh?
Also, if I'm not wrong CloudRun GCP's main (only? besides managed K8s) PaaS container runtime. Presenting it as a general container runtime with ultra fast scaling when people online are reporting 30 second startup times for basic python/node apps, is a joke. These tradeoffs should also be highlighted somewhere in these sales pages, but they're not.
This is the last I'm responding to this thread. Also my apologies to the Coder folks for going off topic like this.
Re: We improved the performance of a userspace TCP stack in Go
#95Earlier quoted context omitted.
We are still talking about people using runsc/runc. That's not what `coder` is doing. All they did was poach a (popular) networking library from the gVisor codebase. None of this benchmarking has anything to do with their product.
I've already accepted this whole thread is a digression, but I keep getting pulled in. Calling out my dislike for Gvisor on a thread lauding a 5x tcp performance they found in it felt on topic to me at the time.
Ok. I'm only triggered by two things:
1. An argument that a tool using netstack is in any way tainted with gVisor's runtime costs.
2. An argument that shared-kernel multitenant is tenable and thus gVisor addresses no meaningful security concerns.
Re: We improved the performance of a userspace TCP stack in Go
#96Earlier quoted context omitted.
A decent while ago, I was the original author of that performance guide. I tried to lay out the set of performance trade-offs in an objective and realistic way. It is shocking to me that you’re spending so much time commenting on a few figures from there, ostensibly w/o reading it. System call overhead does matter, but it’s not the ultimate measure of anything. If it were, gVisor with the KVM platform would be faster…
> it is shocking to me that you’re spending so much time commenting on a few figures from there You give me too much credit! They were copy pastes to the same responder who responded to me in a few places in the thread. I did that to avoid spending too much time responding! > because looking at any one figure in isolation is dumb So the self-reported performance figures are bad, the are hundreds of web pages and supp…
I don’t think copy/pasting the same response everywhere is better.
IIRC, CloudRun has multiple modes of operation (fully-managed and in a K8s cluster) and different sandboxes for the fully-managed environment (VM-based and gVisor-based). Like everything, performance depends a lot of the specifics — for example, the network depends a lot more on the network path (e.g. are you using a VPC connector?) than it does the specific sandbox or network stack (i.e. if you want to push 40gbps, spin up a dedicated GCE instance.) Similarly, the lack of a persistent disk is a design choice for multiple reasons — if you need a lot of non-tmpfs disk or persistent state, CloudRun might not be the right place for the service.
It sounds like you personally had a bad experience or hit a sharp edge, which sucks and I empathize — but I think you can just be concrete about that rather than projecting with system call times (I’d be happy to give you the reason gen1 sandbox would be slow for a typical heavy python app doing 20,000 stats on startup — and it’s real but not really system calls or anything you’re pointing at,… either way you could just turn on gen2 or use other products, e.g. GCE containers, GKE autopilot, etc.).
I’m not sure what’s wrong with advice re: optimizing for a serverless platform (like global variables). I don’t really think it would be sensible to recompute/rebuild application state on any serverless platform on any provider.
Re: We improved the performance of a userspace TCP stack in Go
#97Earlier quoted context omitted.
I've already accepted this whole thread is a digression, but I keep getting pulled in. Calling out my dislike for Gvisor on a thread lauding a 5x tcp performance they found in it felt on topic to me at the time.
Ok. I'm only triggered by two things: 1. An argument that a tool using netstack is in any way tainted with gVisor's runtime costs. 2. An argument that shared-kernel multitenant is tenable and thus gVisor addresses no meaningful security concerns.
Not gonna lie am also getting 200% triggered whenever he states gVisor Syscall costs lol
Re: We improved the performance of a userspace TCP stack in Go
#98Earlier quoted context omitted.
Yes, we have proven that shared-kernel multitenant is unsafe. The best example (though there are many) is the `waitid` LPE; nobody's container lockdown configuration was blocking `waitid`, which is what you'd have had to do to prevent container code from compromising the kernel. The list of Linux LPEs is long, and syzkaller crashes longer stil.
https://news.ycombinator.com/item?id=40591147 So the PaaS providers mentioned in that comment should be assumed to be compromised?
If they are using multitenant Docker / containerd containers with no additional sandboxing, then yes, then it's only a matter of time and attacker interest before a cross-tenant compromise occurs.
Re: We improved the performance of a userspace TCP stack in Go
#99Earlier quoted context omitted.
A TCP/IP stack is not an "implementation of syscalls". The things most netstack users do with netstack have nothing to do with wanting to move the kernel into userland and everything to do with the fact that the kernel features they want to access are either privileged or (in a lot of IP routing cases) not available at all. Netstack (like any user-mode IP stack) allows programs to do things they couldn't otherwise do…
> The gVisor/perf thing is a tendentious argument Interesting to dismiss it as such. The gvisor netstack is a (big) part of gvisor and this article is discussing how the performance of that component was, and could well still be, garbage. These tools bring marginal capability and performance gains, shoved down peoples throat by manufacturing security paranoia. Oh an it all happens to cost you like 10x time, but look…
> The gvisor netstack ... this article is discussing how the performance of that component was ... garbage.
The article and a related GitHub discussion (linked from TFA) points out that the default congestion algorithm (reno) wasn't good for long-distance (over Internet) workloads. The gvisor team never noticed it because they test/tune for in-datacenter usecases.
> These tools bring marginal capability and performance gains
I get your point (ex: app sandbox in Android ruins battery & perf, website sandbox on chrome wastes memory, etc). While 0-days continue to sell for millions, opsec are right to be skeptical about a very critical component (kernel) that runs on 50%+ of all servers & personal devices.
Re: We improved the performance of a userspace TCP stack in Go
#100I met one of the founders of Coder.com, he's a really cool dude. It's a pity that it is a product aimed more at enterprises than individual developers, else it would have far more developer mindshare. Unlike, say, GitHub Codespaces, running something like this on your own infra means your incentives and Coder.com's are aligned, i.e. both of you want to reduce your cloud costs (as opposed to, say, GitHub running on Az…
It seems like a great product. I'm wondering why they don't offer more "startup-oriented" plans. It's like either Self Hosted or "Talk to sales". Is it maybe to not compete against Github codespaces?