Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How a fix in Go 1.9 sped up our Gitaly service by 30x

about.gitlab.com

Post reply on HN
Print view View original

21–30 of 72 posts

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#21

Post by revelation »

jsiepkes wrote: post #2

"Recompiling with Go 1.9 solved the problem, thanks to the switch to posix_spawn" I never understood why so many people use fork() instead of POSIX spawn(). For example OpenJDK (Java) also does this as the default for starting a process. Which leads to interesting results when you use it on a OS which does do memory over committing like Solaris. Since the process briefly doubles in memory use with fork() your process…

Because every straightforward way of running an external command on Unix involves fork(). So someone wrote that API not thinking much of it.

Then shock horror they realize running a throwaway command is fork()ing the main process. But now everyone is too angsty to change it because someone out there might rely on the environment copy functionality, even when they shouldn't.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#22

Post by Animats »

revelation wrote: post #20
Animats wrote: post #9

Each Gitaly server instance was fork/exec'ing Git processes about 20 times per second so we seemed to finally have a very promising lead. What's really wrong here is that they're apparently spawning processes like crazy. Do they spawn a new process for each API call? That's like running CGI programs under Apache, like it's 1995.

It's even worse. Gitaly is a program that takes loosely-validated externally-triggered requests and turns them into Git command lines to be exec()ed . So every API request transmutes its input into one or more Git command lines that are exec()ed, each one invoking fork() on the main massively-parallel Gitaly process (well, used to anyway). It's like a terrible China router firmware, without the C. Bonus points for ev…

Do they just do that for commands that make changes, or do they do it for pure read commands as well? Most of the volume is in reads, especially since so many build systems now read directly from Github.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#23

Post by revelation »

Animats wrote: post #22
revelation wrote: post #20

Earlier quoted context omitted.

It's even worse. Gitaly is a program that takes loosely-validated externally-triggered requests and turns them into Git command lines to be exec()ed . So every API request transmutes its input into one or more Git command lines that are exec()ed, each one invoking fork() on the main massively-parallel Gitaly process (well, used to anyway). It's like a terrible China router firmware, without the C. Bonus points for ev…

Do they just do that for commands that make changes, or do they do it for pure read commands as well? Most of the volume is in reads, especially since so many build systems now read directly from Github.

Here is the "reference exists" that the blog post alludes to:

https://gitlab.com/gitlab-org/gitaly/blob/master/internal/se...

I'm not an expert in this abomination, but it looks all the world like invoking "git show-ref --verify".

(Of course a Git ref is usually just a file in .git with a SHA1 in it. They don't care about the SHA1, so really they are launching a Git process for a file exists operation. This used to take them 400ms, but now it's only 100ms!)

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#24

Post by sreque »

dchest wrote: post #12
Animats wrote: post #9

Each Gitaly server instance was fork/exec'ing Git processes about 20 times per second so we seemed to finally have a very promising lead. What's really wrong here is that they're apparently spawning processes like crazy. Do they spawn a new process for each API call? That's like running CGI programs under Apache, like it's 1995.

Nothing wrong with it. In fact, I wish spawning processes was more common. It's beneficial for security.

Spawning a process, even if its just vfork() followed by exec(), is very very expensive compared to spawning a thread, and even then most apps created threads on startup and then just cache them for the lifetime of the application. Spawning a process 20 times a second honestly isn't terrible, but even hundreds per second can be noticeably slow IIRC.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#25

Post by ithkuil »

nitwit005 wrote: post #19
jsiepkes wrote: post #2

"Recompiling with Go 1.9 solved the problem, thanks to the switch to posix_spawn" I never understood why so many people use fork() instead of POSIX spawn(). For example OpenJDK (Java) also does this as the default for starting a process. Which leads to interesting results when you use it on a OS which does do memory over committing like Solaris. Since the process briefly doubles in memory use with fork() your process…

I thought of creating a fix myself way back, and the issue was that Go made use of system calls directly. You basically have to re-implement posix_spawn in Go. If you look at their change, it includes updates to chipset specific files, and the fix only seems to work on a CPU that reports as amd64.

I must say I didn't go to look at the sources of the patch, but what you say sounds so odd that I'll take the chance and suggest that perhaps the fact that in golang "amd64" is, for historical reasons, the name of the architecture more neutrally known as "x86_64", is the source of confusion (I.e. it doesn't just work on AMD or on CPUs that claim/report having a specific model/maker etc).

Low level syscall ABI is architecture dependent.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#26

Post by azinman2 »

sreque wrote: post #24
dchest wrote: post #12

Earlier quoted context omitted.

Nothing wrong with it. In fact, I wish spawning processes was more common. It's beneficial for security.

Spawning a process, even if its just vfork() followed by exec(), is very very expensive compared to spawning a thread, and even then most apps created threads on startup and then just cache them for the lifetime of the application. Spawning a process 20 times a second honestly isn't terrible, but even hundreds per second can be noticeably slow IIRC.

I thought Linux treated forks and processes the same, where as the former is just a special case?

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#27

Post by lbotos »

revelation wrote: post #23
Animats wrote: post #22

Earlier quoted context omitted.

Do they just do that for commands that make changes, or do they do it for pure read commands as well? Most of the volume is in reads, especially since so many build systems now read directly from Github.

Here is the "reference exists" that the blog post alludes to: https://gitlab.com/gitlab-org/gitaly/blob/master/internal/se... I'm not an expert in this abomination, but it looks all the world like invoking "git show-ref --verify". (Of course a Git ref is usually just a file in .git with a SHA1 in it. They don't care about the SHA1, so really they are launching a Git process for a file exists operation. This used to t…

Curious, how would you do it?

(disclaimer: I work at GitLab, not on this project though)

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#28

Post by saghm »

azinman2 wrote: post #26
sreque wrote: post #24

Earlier quoted context omitted.

Spawning a process, even if its just vfork() followed by exec(), is very very expensive compared to spawning a thread, and even then most apps created threads on startup and then just cache them for the lifetime of the application. Spawning a process 20 times a second honestly isn't terrible, but even hundreds per second can be noticeably slow IIRC.

I thought Linux treated forks and processes the same, where as the former is just a special case?

`fork` is how you create a new process on Linux. Threads, which GP was talking about, are different, especially Go threads, which are not the same as OS threads.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#29

Post by revelation »

lbotos wrote: post #27
revelation wrote: post #23

Earlier quoted context omitted.

Here is the "reference exists" that the blog post alludes to: https://gitlab.com/gitlab-org/gitaly/blob/master/internal/se... I'm not an expert in this abomination, but it looks all the world like invoking "git show-ref --verify". (Of course a Git ref is usually just a file in .git with a SHA1 in it. They don't care about the SHA1, so really they are launching a Git process for a file exists operation. This used to t…

Curious, how would you do it? (disclaimer: I work at GitLab, not on this project though)

Someone mentioned libgit2, and that is a good first step fix. There is no need to launch a process that calls libgit2 when you can call high-level libgit2 functions yourself. That already eliminates this problem entirely, along with all the other ickiness that comes with running command line tools programmatically.

But ultimately this is the main Git interface for the remainder of the site, and it is apparently already sharded so only has to deal with a limited number of repositories. You can use libgit2 on a low-enough level that you can just keep and mutate repository state in memory. Something like a ref exists should be just a hash table lookup, and there are a bunch of other commands where gains can be had when you are not starting from scratch on every API call.

(This is what github is doing. They started out with grit, which was some parts of Git reimplemented in Ruby and then launching git for heavy-weight stuff. Nowadays they use rugged, the ruby bindings for libgit2.)

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#30

Post by Animats »

lbotos wrote: post #27
revelation wrote: post #23

Earlier quoted context omitted.

Here is the "reference exists" that the blog post alludes to: https://gitlab.com/gitlab-org/gitaly/blob/master/internal/se... I'm not an expert in this abomination, but it looks all the world like invoking "git show-ref --verify". (Of course a Git ref is usually just a file in .git with a SHA1 in it. They don't care about the SHA1, so really they are launching a Git process for a file exists operation. This used to t…

Curious, how would you do it? (disclaimer: I work at GitLab, not on this project though)

Use something like boring old FCGI. FCGI spawns off N workers, which process one request at a time. If a worker crashes, it's restarted. Workers are usually restarted every once in a while to deal with any memory leaks.

Read-only requests with no security implications get done by in the worker process, which has read permissions for public files. More complex requests spawn a Git client program.

It would be so uncool, though.

Post reply on HN