Related post from a few weeks ago:
Fork is not my favorite syscall:
How a fix in Go 1.9 sped up our Gitaly service by 30x
11–20 of 72 posts
Re: How a fix in Go 1.9 sped up our Gitaly service by 30x
#12Each Gitaly server instance was fork/exec'ing Git processes about 20 times per second so we seemed to finally have a very promising lead. What's really wrong here is that they're apparently spawning processes like crazy. Do they spawn a new process for each API call? That's like running CGI programs under Apache, like it's 1995.
Nothing wrong with it. In fact, I wish spawning processes was more common. It's beneficial for security.
Re: How a fix in Go 1.9 sped up our Gitaly service by 30x
#13Each Gitaly server instance was fork/exec'ing Git processes about 20 times per second so we seemed to finally have a very promising lead. What's really wrong here is that they're apparently spawning processes like crazy. Do they spawn a new process for each API call? That's like running CGI programs under Apache, like it's 1995.
Nothing wrong with it. In fact, I wish spawning processes was more common. It's beneficial for security.
> It's beneficial for security
... and for RAM usage. Java applications all have a tendency to bloat the longer you keep them running.
Re: How a fix in Go 1.9 sped up our Gitaly service by 30x
#14It's often to remember the other point of view when you see huge performance differences. > A bug in Go <1.9 was causing a 30x slowdown in our Gitaly service.
The author does say fix.
Re: How a fix in Go 1.9 sped up our Gitaly service by 30x
#15Each Gitaly server instance was fork/exec'ing Git processes about 20 times per second so we seemed to finally have a very promising lead. What's really wrong here is that they're apparently spawning processes like crazy. Do they spawn a new process for each API call? That's like running CGI programs under Apache, like it's 1995.
I wonder how much it would speed up if they were using libgit2 directly.
Re: How a fix in Go 1.9 sped up our Gitaly service by 30x
#16Each Gitaly server instance was fork/exec'ing Git processes about 20 times per second so we seemed to finally have a very promising lead. What's really wrong here is that they're apparently spawning processes like crazy. Do they spawn a new process for each API call? That's like running CGI programs under Apache, like it's 1995.
[deleted]
Re: How a fix in Go 1.9 sped up our Gitaly service by 30x
#17"Recompiling with Go 1.9 solved the problem, thanks to the switch to posix_spawn" I never understood why so many people use fork() instead of POSIX spawn(). For example OpenJDK (Java) also does this as the default for starting a process. Which leads to interesting results when you use it on a OS which does do memory over committing like Solaris. Since the process briefly doubles in memory use with fork() your process…
fork() is a pretty simple way to be able to modify the environment for a process you will spawn. fork(), the child can modify its own environment using various orthogonal system calls, e.g. to redirect stdout/stderr or drop permissions, and then exec the target executable.
Threads throw a wrench in things. But fork() existed for decades before threads. O_CLOEXEC etc helps. Lots of command-line utilities don't use threads.
fork() isn't the fastest way - but in many situations it's not a problem, it's just convenient. In that respect it's somewhat like using python when you could have used go.
Re: How a fix in Go 1.9 sped up our Gitaly service by 30x
#18Each Gitaly server instance was fork/exec'ing Git processes about 20 times per second so we seemed to finally have a very promising lead. What's really wrong here is that they're apparently spawning processes like crazy. Do they spawn a new process for each API call? That's like running CGI programs under Apache, like it's 1995.
Nothing wrong with it. In fact, I wish spawning processes was more common. It's beneficial for security.
Process isolation is good for security.
Parsing text data in ad-hoc, non-standardized, not documented, not defined format is really bad for security.
Just spawning a process creates as many security problems as it solves.
If it was done right, it would look like Chrome architecture, where untrusted, isolated processes can do dangerous work but communicate with trusted process via well defined IPC protocol.
Re: How a fix in Go 1.9 sped up our Gitaly service by 30x
#19"Recompiling with Go 1.9 solved the problem, thanks to the switch to posix_spawn" I never understood why so many people use fork() instead of POSIX spawn(). For example OpenJDK (Java) also does this as the default for starting a process. Which leads to interesting results when you use it on a OS which does do memory over committing like Solaris. Since the process briefly doubles in memory use with fork() your process…
I thought of creating a fix myself way back, and the issue was that Go made use of system calls directly. You basically have to re-implement posix_spawn in Go. If you look at their change, it includes updates to chipset specific files, and the fix only seems to work on a CPU that reports as amd64.
Re: How a fix in Go 1.9 sped up our Gitaly service by 30x
#20Each Gitaly server instance was fork/exec'ing Git processes about 20 times per second so we seemed to finally have a very promising lead. What's really wrong here is that they're apparently spawning processes like crazy. Do they spawn a new process for each API call? That's like running CGI programs under Apache, like it's 1995.
It's even worse. Gitaly is a program that takes loosely-validated externally-triggered requests and turns them into Git command lines to be exec()ed. So every API request transmutes its input into one or more Git command lines that are exec()ed, each one invoking fork() on the main massively-parallel Gitaly process (well, used to anyway).
It's like a terrible China router firmware, without the C. Bonus points for every straightforward way of running a throwaway command on Linux invoking fork().
I guess it's a good thing because it sets us up for another blog post once they learn of the latency gains to be had when you are not creating new processes on API requests. Hell, when someone starts looking into how this Git thing works, we might be in for a whole series.