Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How a fix in Go 1.9 sped up our Gitaly service by 30x

about.gitlab.com

Post reply on HN
Print view View original

31–40 of 72 posts

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#31

Post by sytse »

CapacitorSet wrote: post #15
Animats wrote: post #9

Each Gitaly server instance was fork/exec'ing Git processes about 20 times per second so we seemed to finally have a very promising lead. What's really wrong here is that they're apparently spawning processes like crazy. Do they spawn a new process for each API call? That's like running CGI programs under Apache, like it's 1995.

I wonder how much it would speed up if they were using libgit2 directly.

I'm at GitLab but not on the Gitaly team. I think we are using libgit2 but that it doesn't contain all the calls we need.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#32

Post by paulfurtado »

mschuster91 wrote: post #13
dchest wrote: post #12

Earlier quoted context omitted.

Nothing wrong with it. In fact, I wish spawning processes was more common. It's beneficial for security.

> It's beneficial for security ... and for RAM usage. Java applications all have a tendency to bloat the longer you keep them running.

It's not bloat or memory leaks per se, the JVM just does not return memory to the OS after it is freed. To limit its memory usage, tune the heap size. To fully allocate the heap on startup for consistent usage, use -XX:+AlwaysPreTouch

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#33

Post by _urga »

We currently have the same problem in Node, where fork is still being called synchronously from the event loop instead of asynchronously from the thread pool.

Calling exec() or spawn() in Node is therefore not asynchronous and can block your event loop for hundreds of milliseconds or even seconds as RSS increases.

https://github.com/nodejs/node/issues/14917

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#34

Post by ereyes01 »

saghm wrote: post #28
azinman2 wrote: post #26

Earlier quoted context omitted.

I thought Linux treated forks and processes the same, where as the former is just a special case?

`fork` is how you create a new process on Linux. Threads, which GP was talking about, are different, especially Go threads, which are not the same as OS threads.

I believe GP was referring to the `clone` system call, which both fork and pthread_create use under the covers. The difference is what happens to the memory-resident data- the forked/child program shares the address space of the parent until either writes to a page, in which case the page is copied such that each process gets its own unique copy. This is known as copy-on-write (COW). Pthreads, on the other hand, outright share the process's address space and must implement their own synchronization via locks or whatever.

Obviously, the performance of fork vs. pthread_create could differ dramatically depending on what the program does.

Goroutines are a layer of abstraction above this. They might run on different threads concurrently- the Go runtime controls what happens here and may differ on various architectures / OSes. If you break into a running Go program on Linux with gdb, there's definitely a bunch of pthreads running, maybe for goroutines, and probably for garbage collection, and other stuff. (If you want to actually debug go code, you should of course use something like delve).

http://man7.org/linux/man-pages/man2/clone.2.html

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#35

Post by axaxs »

paulfurtado wrote: post #32
mschuster91 wrote: post #13

Earlier quoted context omitted.

> It's beneficial for security ... and for RAM usage. Java applications all have a tendency to bloat the longer you keep them running.

It's not bloat or memory leaks per se, the JVM just does not return memory to the OS after it is freed. To limit its memory usage, tune the heap size. To fully allocate the heap on startup for consistent usage, use -XX:+AlwaysPreTouch

Java has always been 'use memory to increase speed...sometimes'. You can tune it some, sure, but that's what it's known to do.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#36

Post by CapacitorSet »

sytse wrote: post #31
CapacitorSet wrote: post #15

Earlier quoted context omitted.

I wonder how much it would speed up if they were using libgit2 directly.

I'm at GitLab but not on the Gitaly team. I think we are using libgit2 but that it doesn't contain all the calls we need.

Reading through an old issue (can't link right now, am on mobile), it seems that the main reason for not using libgit2 in Gitaly is performance, since it would read too many unused files.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#37

Post by chinhodado »

paulfurtado wrote: post #32
mschuster91 wrote: post #13

Earlier quoted context omitted.

> It's beneficial for security ... and for RAM usage. Java applications all have a tendency to bloat the longer you keep them running.

It's not bloat or memory leaks per se, the JVM just does not return memory to the OS after it is freed. To limit its memory usage, tune the heap size. To fully allocate the heap on startup for consistent usage, use -XX:+AlwaysPreTouch

Last time I checked, I still couldn't control the max heap free ratio, because apparently that option/flag just doesn't work with Java 8's default GC.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#38

Post by cstrahan »

lbotos wrote: post #27
revelation wrote: post #23

Earlier quoted context omitted.

Here is the "reference exists" that the blog post alludes to: https://gitlab.com/gitlab-org/gitaly/blob/master/internal/se... I'm not an expert in this abomination, but it looks all the world like invoking "git show-ref --verify". (Of course a Git ref is usually just a file in .git with a SHA1 in it. They don't care about the SHA1, so really they are launching a Git process for a file exists operation. This used to t…

Curious, how would you do it? (disclaimer: I work at GitLab, not on this project though)

In their case, why not stat the path directly?

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#39

Post by cstrahan »

f2f wrote: post #6
dboreham wrote: post #4

Earlier quoted context omitted.

Because decades of written material about Unix says fork() is really cool (even though it isn't)?

fork was alright before other people tacked on multiples of cruft like threads and whatnot onto commercial unixes and they became mainstream. the current problem is that you don't want to have to copy all file descriptors if all you're going to do is call "exec" and reduce them to three: in, out, err. for example, here's the caveats section from the macOS fork man page: There are limits to what you can do in the chil…

> the current problem is that you don't want to have to copy all file descriptors if all you're going to do is call "exec" and reduce them to three: in, out, err.

To be clear, exec does not necessarily close all but the first three fds -- by default all fds will be inherited. However, you can set the close-on-exec flag on each individual fd (in fact, that's what the Go stdlib does behind the scenes).

Search for FD_CLOEXEC in fcntl(2) and open(2) and you'll see what I'm referring to.

http://man7.org/linux/man-pages/man2/fcntl.2.html

http://man7.org/linux/man-pages/man2/open.2.html

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#40

Post by nurettin »

paulfurtado wrote: post #32
mschuster91 wrote: post #13

Earlier quoted context omitted.

> It's beneficial for security ... and for RAM usage. Java applications all have a tendency to bloat the longer you keep them running.

It's not bloat or memory leaks per se, the JVM just does not return memory to the OS after it is freed. To limit its memory usage, tune the heap size. To fully allocate the heap on startup for consistent usage, use -XX:+AlwaysPreTouch

Back when I tried running a jruby application on 800mb ram, it bloated, then started throwing "OutOfMemoryError"s and "Insufficient Class Space" or something similar. Apparently jruby was generating too many new types at runtime to accommodate rails framework. Garbage collector was pretty garbage at it's job back in 2011.
Post reply on HN