Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How a fix in Go 1.9 sped up our Gitaly service by 30x

about.gitlab.com

Post reply on HN
Print view View original

61–70 of 72 posts

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#61

Post by AnIdiotOnTheNet »

nitwit005 wrote: post #19
jsiepkes wrote: post #2

"Recompiling with Go 1.9 solved the problem, thanks to the switch to posix_spawn" I never understood why so many people use fork() instead of POSIX spawn(). For example OpenJDK (Java) also does this as the default for starting a process. Which leads to interesting results when you use it on a OS which does do memory over committing like Solaris. Since the process briefly doubles in memory use with fork() your process…

I thought of creating a fix myself way back, and the issue was that Go made use of system calls directly. You basically have to re-implement posix_spawn in Go. If you look at their change, it includes updates to chipset specific files, and the fix only seems to work on a CPU that reports as amd64.

Go uses system calls directly because the alternative in UNIX land is linking with libc.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#62

Post by haikuginger »

The article linked to a set of posix-spawn benchmarks[1] that seemed to indicate that while fork/exec time scaled linearly with resident memory on Linux, it did not scale at all on macOS.

First of all, I was somewhat confused by that due to the availability of copy-on-write; I wouldn't have expected fork/exec time to scale up that way.

Second, I was surprised that there wasn't an attempt to explain the behavior difference between the two systems. Can someone familiar with either or both point towards an explanation for why that's the case? It seems very odd.

[1]https://github.com/rtomayko/posix-spawn#benchmarks

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#63

Post by carussell »

romanovcode wrote: post #54
zebra9978 wrote: post #52

Are you guys planning to migrate gitlab to golang ? I think the biggest feature that everyone wants is better performance. Is the migration path that tough ?

They have so many features that I don't see it happening ever.

They wouldn't need to stop the world and do a full rewrite. It would be feasible if they stop writing new components in Ruby and began replacing the existing parts piecemeal.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#64

Post by paulddraper »

Animats wrote: post #9

Each Gitaly server instance was fork/exec'ing Git processes about 20 times per second so we seemed to finally have a very promising lead. What's really wrong here is that they're apparently spawning processes like crazy. Do they spawn a new process for each API call? That's like running CGI programs under Apache, like it's 1995.

I am tired of this "processes are expensive" bullcrap. (At least for Linux.)

    $ time seq 1000 | while read; do sleep 0 & done

    real        0m0.185s
    user        0m0.546s
    sys         0m0.265s
That's less than .2ms to start a process.

Processes give you operational control (CPU, memory, permissions, isolation, monitoring) that other constructs simple cannot. Decades ago when we had far slower computers, people were doing process-oriented development and forking as if it was okay (CGI, make, git).

Somehow, separate processes came to be avoided like the plague, when in reality, they are probably the smallest resource "waste" in 99% of systems.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#65

Post by arghwhat »

paulddraper wrote: post #64
Animats wrote: post #9

Each Gitaly server instance was fork/exec'ing Git processes about 20 times per second so we seemed to finally have a very promising lead. What's really wrong here is that they're apparently spawning processes like crazy. Do they spawn a new process for each API call? That's like running CGI programs under Apache, like it's 1995.

I am tired of this "processes are expensive" bullcrap. (At least for Linux.) $ time seq 1000 | while read; do sleep 0 & done real 0m0.185s user 0m0.546s sys 0m0.265s That's less than .2ms to start a process. Processes give you operational control (CPU, memory, permissions, isolation, monitoring) that other constructs simple cannot. Decades ago when we had far slower computers, people were doing process-oriented devel…

This is a terrible microbenchmark.

First of all, you're only benchmarking the time it takes for fork(2) to return in the parent subshell, nothing else. The new processes don't exist yet at this point, and certainly hasn't exec'd (which tends to be why you're forking).

Second, you're not measuring the cost at all. The forked children will, at some point, start executing on other CPUs, which includes finishing configuration and running exec, which takes time. The cost is the total cycles it takes before the child is executing the intended code.

Fork is damn expensive, but whether they're too expensive depends on the usecase, and the cost of expanding hardware.

Fork time scales with the virtual memory of the forking process, and you're forking from a fresh subshell that hardly has anything allocated. It's even mentioned in the linked post that their issue stemmed from this (specifically fork lock contention spiking as fork time increased).

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#66

Post by arghwhat »

revelation wrote: post #20
Animats wrote: post #9

Each Gitaly server instance was fork/exec'ing Git processes about 20 times per second so we seemed to finally have a very promising lead. What's really wrong here is that they're apparently spawning processes like crazy. Do they spawn a new process for each API call? That's like running CGI programs under Apache, like it's 1995.

It's even worse. Gitaly is a program that takes loosely-validated externally-triggered requests and turns them into Git command lines to be exec()ed . So every API request transmutes its input into one or more Git command lines that are exec()ed, each one invoking fork() on the main massively-parallel Gitaly process (well, used to anyway). It's like a terrible China router firmware, without the C. Bonus points for ev…

I suspect you are misunderstanding "exec" as "shell". China router firmwares call the shell.

Putting arbitrary input into a shell is dangerous, as missed escaping can result in control of the shell.

When you call exec yourself, however, you are passing the individual arguments as NULL-terminated list of strings (char*). There is no shell to abuse. Calling a process this way is about as safe as calling a function that takes strings for arguments. The function can still have vulnerabilities, but the process of calling it is safe.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#67

Post by arghwhat »

mschuster91 wrote: post #13
dchest wrote: post #12

Earlier quoted context omitted.

Nothing wrong with it. In fact, I wish spawning processes was more common. It's beneficial for security.

> It's beneficial for security ... and for RAM usage. Java applications all have a tendency to bloat the longer you keep them running.

That's mostly just a Java thing, though. Java has a weird relationship with memory.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#68

Post by connorshea »

zebra9978 wrote: post #52

Are you guys planning to migrate gitlab to golang ? I think the biggest feature that everyone wants is better performance. Is the migration path that tough ?

There are no plans to migrate all of GitLab to Go. The main Rails app is going to stay a Rails app for the foreseeable future. There are a few reasons for this. For one it'd be such a huge project, but also Rails is working well for us, it's great for our pace of feature development.

We are working on moving the git layer to Gitaly[0] which is written in Go (and is what this blog post is about). It was one of our major bottlenecks and we've seen a lot of benefit from having made the switch. It's not done yet, but a lot of the calls to git that the application makes are now done through Gitaly.

[0]: https://gitlab.com/gitlab-org/gitaly

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#69

Post by paulddraper »

arghwhat wrote: post #65
paulddraper wrote: post #64

Earlier quoted context omitted.

I am tired of this "processes are expensive" bullcrap. (At least for Linux.) $ time seq 1000 | while read; do sleep 0 & done real 0m0.185s user 0m0.546s sys 0m0.265s That's less than .2ms to start a process. Processes give you operational control (CPU, memory, permissions, isolation, monitoring) that other constructs simple cannot. Decades ago when we had far slower computers, people were doing process-oriented devel…

This is a terrible microbenchmark. First of all, you're only benchmarking the time it takes for fork(2) to return in the parent subshell, nothing else. The new processes don't exist yet at this point, and certainly hasn't exec'd (which tends to be why you're forking). Second, you're not measuring the cost at all. The forked children will, at some point, start executing on other CPUs, which includes finishing configur…

(1) The benchmark measured the point of discussion.

(2) Even not using asynchronity (which Go is heralded for), processes take 

    $ time seq 1000 | while read; do sleep 0; done

    real        0m1.644s
    user        0m1.065s
    sys         0m0.672s

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#70

Post by arghwhat »

paulddraper wrote: post #69
arghwhat wrote: post #65

Earlier quoted context omitted.

This is a terrible microbenchmark. First of all, you're only benchmarking the time it takes for fork(2) to return in the parent subshell, nothing else. The new processes don't exist yet at this point, and certainly hasn't exec'd (which tends to be why you're forking). Second, you're not measuring the cost at all. The forked children will, at some point, start executing on other CPUs, which includes finishing configur…

(1) The benchmark measured the point of discussion. (2) Even not using asynchronity (which Go is heralded for), processes take $ time seq 1000 | while read; do sleep 0; done real 0m1.644s user 0m1.065s sys 0m0.672s

1. No, the point was that fork(2)+exec(3)/spawning processes is an expensive way to run code, not how long it takes for the parent to be able to do something else.

2. Your new benchmark is better. However, it is still a useless microbenchmark, as it is an unrealistic best-case scenario. Your spawn of sleep is happening within a fresh subshell started by the pipe you made. fork(2) depends on things like VMM size and open file descriptors of the parent process, and your subshell basically has nothing at all. A real application likely holds at least a few gigabytes of virtual memory (more likely tens of gigabytes—note that virtual memory isn't the same as resident memory), which will make fork(2) take much longer, split between parent and child.

I suspect you might be confusing asynchronicity with concurrency or parallelism. Go is heralded for concurrency, sometimes in the form of parallelism, but not asynchronicity. Concurrency does not have any positive effect on execution time or cost. Parallelism can reduce execution time, but does not decrease execution cost, it simply throws more hardware at the problem.

In fact, Go is a worse-than-average language to call fork(2) in, due to it running fork(2) under a global lock. This is mentioned in the linked article. The lock contention caused by fork(2) execution time as memory consumption increased was what made the process unresponsive.

However, as I also said, whether fork is too expensive depends on the use-case.

Post reply on HN