Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How a fix in Go 1.9 sped up our Gitaly service by 30x

about.gitlab.com

Post reply on HN
Print view View original

51–60 of 72 posts

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#51

Post by laumars »

ithkuil wrote: post #25
nitwit005 wrote: post #19

Earlier quoted context omitted.

I thought of creating a fix myself way back, and the issue was that Go made use of system calls directly. You basically have to re-implement posix_spawn in Go. If you look at their change, it includes updates to chipset specific files, and the fix only seems to work on a CPU that reports as amd64.

I must say I didn't go to look at the sources of the patch, but what you say sounds so odd that I'll take the chance and suggest that perhaps the fact that in golang "amd64" is, for historical reasons, the name of the architecture more neutrally known as "x86_64", is the source of confusion (I.e. it doesn't just work on AMD or on CPUs that claim/report having a specific model/maker etc). Low level syscall ABI is arch…

AMD64 refers to both Intel and AMD chipsets that are 64bit x86. While you are right that there is also the term "x86_64" in common use, AMD64 is the actually more standard name (as well as the term specifically used in the Go ecosystem eg $GOARCH env var and build parameters for cross platform sources)

Further to that point, I didn't detect any confusion from others in this thread that AMD64 excluded Intel chips. Where they were talking about AMD64 specific code they were saying that Go code targeting other architectures (eg arm, mips, s390x and ppc - to name a few. Go supports an impressive number of architectures[1]) would still use their respective fork() code rather than this new fix.

[1] https://golang.org/doc/install/source#introduction

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#53

Post by rakoo »

lbotos wrote: post #27
revelation wrote: post #23

Earlier quoted context omitted.

Here is the "reference exists" that the blog post alludes to: https://gitlab.com/gitlab-org/gitaly/blob/master/internal/se... I'm not an expert in this abomination, but it looks all the world like invoking "git show-ref --verify". (Of course a Git ref is usually just a file in .git with a SHA1 in it. They don't care about the SHA1, so really they are launching a Git process for a file exists operation. This used to t…

Curious, how would you do it? (disclaimer: I work at GitLab, not on this project though)

If forking to a process is not good enough, I'd use a native, read-only library such as https://github.com/speedata/gogit

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#55

Post by jacobvosmaer »

CapacitorSet wrote: post #15
Animats wrote: post #9

Each Gitaly server instance was fork/exec'ing Git processes about 20 times per second so we seemed to finally have a very promising lead. What's really wrong here is that they're apparently spawning processes like crazy. Do they spawn a new process for each API call? That's like running CGI programs under Apache, like it's 1995.

I wonder how much it would speed up if they were using libgit2 directly.

I'm on the Gitaly team.

As zegerjan wrote, Gitaly is a Go/Ruby hybrid.

The main Go process doesn't use libgit2 (for now) because we didn't want to have to deal with cgo. We already know how to deal with C extensions in Ruby, and we have a lot of existing Ruby application code that uses libgit2, so we still use it there. And that code works fine so I don't see us removing it.

In practice, sometimes spawning a Git process is faster than using libgit2, so why then not do that. Also for parts of our workload (handling Git push/pull operations), spawning a one-off process (git-upload-pack) is the most boring / tried-and-true approach.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#57

Post by jacobvosmaer »

ploxiln wrote: post #17
jsiepkes wrote: post #2

"Recompiling with Go 1.9 solved the problem, thanks to the switch to posix_spawn" I never understood why so many people use fork() instead of POSIX spawn(). For example OpenJDK (Java) also does this as the default for starting a process. Which leads to interesting results when you use it on a OS which does do memory over committing like Solaris. Since the process briefly doubles in memory use with fork() your process…

fork() is a pretty simple way to be able to modify the environment for a process you will spawn. fork(), the child can modify its own environment using various orthogonal system calls, e.g. to redirect stdout/stderr or drop permissions, and then exec the target executable. Threads throw a wrench in things. But fork() existed for decades before threads. O_CLOEXEC etc helps. Lots of command-line utilities don't use thr…

Another nice example is changing the working directory for the new process. With fork+exec, you can do a chdir after fork but before exec. With posix_spawn you're stuck with the working directory of the parent.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#58

Post by cesarb »

linkregister wrote: post #43
ithkuil wrote: post #25

Earlier quoted context omitted.

I must say I didn't go to look at the sources of the patch, but what you say sounds so odd that I'll take the chance and suggest that perhaps the fact that in golang "amd64" is, for historical reasons, the name of the architecture more neutrally known as "x86_64", is the source of confusion (I.e. it doesn't just work on AMD or on CPUs that claim/report having a specific model/maker etc). Low level syscall ABI is arch…

amd64 is the original name of the instruction set. Intel did beat AMD to a 64-bit instruction set: that of the Itanium processors, IA-64. Itanium had performance issues and lots of errata. Most importantly, IA-64 was not natively backwards-compatible with x86 instructions. amd64 became the standard. x86_64 is a common name for the amd64 architecture, and is a way to describe both the AMD and Intel implementations. In…

It's not just the article title. Follow the two footnotes in the "History" section of that article, to the press releases from AMD announcing the new ISA. They consistently call it "AMD x86-64" or "AMD's x86-64" or just "x86-64". The oldest snapshot I could find of the x86-64 web site (https://web.archive.org/web/20000817014037/http://www.x86-64...) also calls it x86-64. The most recent snapshot of that site, however, calls it AMD64; it seems to have changed sometime in the middle of April 2003.

That is, both x86-64 and AMD64 are historically accurate (2003 was early enough in the ISA's lifetime), but x86-64 is the earlier name.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#59

Post by stonewhite »

> Having solid application monitoring in place allowed us to detect this issue, and start investigating it, far earlier than we otherwise would have been able to.

Yet apparently nobody either caught or investigated the latency spike after the previous deployment.

Post reply on HN