Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How a fix in Go 1.9 sped up our Gitaly service by 30x

about.gitlab.com

Post reply on HN
Print view View original

41–50 of 72 posts

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#41

Post by readittwice »

nurettin wrote: post #40
paulfurtado wrote: post #32

Earlier quoted context omitted.

It's not bloat or memory leaks per se, the JVM just does not return memory to the OS after it is freed. To limit its memory usage, tune the heap size. To fully allocate the heap on startup for consistent usage, use -XX:+AlwaysPreTouch

Back when I tried running a jruby application on 800mb ram, it bloated, then started throwing "OutOfMemoryError"s and "Insufficient Class Space" or something similar. Apparently jruby was generating too many new types at runtime to accommodate rails framework. Garbage collector was pretty garbage at it's job back in 2011.

AFAIR JRuby creates a Java class for each Ruby method. Classes were part of the Permanent Generation, so sometimes this generation was too small escpecially when using Rails. It was enough to just increase the size of this generation with -XX:MaxPermSize (-Xmx doesn't increase the permanent generation's size).

The Permanent Generation was named that way since objects in there were never collected. For most applications this isn't really a problem. Running JRuby+Rails just allocated a lot of classes in this generation, so the default size was too small. But still, the permanent generation was quite small compared to the heap size.

I wouldn't really call the GC bad because of this, IMHO they were already quite good back then. And in Java 8 the permanent generation was replaced with the Metaspace, objects in there can be free'd and the Metaspace can be expanded at runtime so it's less likely to get these OOM errors for the permanent generation.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#42

Post by pjmlp »

kjksf wrote: post #18
dchest wrote: post #12

Earlier quoted context omitted.

Nothing wrong with it. In fact, I wish spawning processes was more common. It's beneficial for security.

Process isolation is good for security. Parsing text data in ad-hoc, non-standardized, not documented, not defined format is really bad for security. Just spawning a process creates as many security problems as it solves. If it was done right, it would look like Chrome architecture, where untrusted, isolated processes can do dangerous work but communicate with trusted process via well defined IPC protocol.

Yes, but there is no need to spawn them all the time.

A parser service daemon, or a pool of them can be used instead, getting requests from the main application process.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#43

Post by linkregister »

ithkuil wrote: post #25
nitwit005 wrote: post #19

Earlier quoted context omitted.

I thought of creating a fix myself way back, and the issue was that Go made use of system calls directly. You basically have to re-implement posix_spawn in Go. If you look at their change, it includes updates to chipset specific files, and the fix only seems to work on a CPU that reports as amd64.

I must say I didn't go to look at the sources of the patch, but what you say sounds so odd that I'll take the chance and suggest that perhaps the fact that in golang "amd64" is, for historical reasons, the name of the architecture more neutrally known as "x86_64", is the source of confusion (I.e. it doesn't just work on AMD or on CPUs that claim/report having a specific model/maker etc). Low level syscall ABI is arch…

amd64 is the original name of the instruction set. Intel did beat AMD to a 64-bit instruction set: that of the Itanium processors, IA-64. Itanium had performance issues and lots of errata. Most importantly, IA-64 was not natively backwards-compatible with x86 instructions. amd64 became the standard.

x86_64 is a common name for the amd64 architecture, and is a way to describe both the AMD and Intel implementations. In my opinion, amd64 is a less ambiguous name and is more historically accurate.

https://en.m.wikipedia.org/wiki/X86-64

Yes, I am aware that my point is undercut by the fact that the article title is x86-64, but I stand by my statement.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#44

Post by zegerjan »

CapacitorSet wrote: post #15
Animats wrote: post #9

Each Gitaly server instance was fork/exec'ing Git processes about 20 times per second so we seemed to finally have a very promising lead. What's really wrong here is that they're apparently spawning processes like crazy. Do they spawn a new process for each API call? That's like running CGI programs under Apache, like it's 1995.

I wonder how much it would speed up if they were using libgit2 directly.

The Gitaly server has a Ruby component, but also a Go component. The Ruby server uses Rugged[1] and Gollum-lib[2] which both use libgit2.

The Go component doesn't have libgit2 binding yet, although we're looking into adding that later. That or maybe go-git[3]. But for now Gitaly is mainly focussed at migrating all git calls from the Rails monolith. Not introducing a new component now reduces the risks this project has.

[1]: https://github.com/libgit2/rugged/ [2]: https://gitlab.slack.com/archives/C027N716H/p151695430400026... [3]: https://github.com/src-d/go-git/

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#45

Post by chriscool »

revelation wrote: post #23
Animats wrote: post #22

Earlier quoted context omitted.

Do they just do that for commands that make changes, or do they do it for pure read commands as well? Most of the volume is in reads, especially since so many build systems now read directly from Github.

Here is the "reference exists" that the blog post alludes to: https://gitlab.com/gitlab-org/gitaly/blob/master/internal/se... I'm not an expert in this abomination, but it looks all the world like invoking "git show-ref --verify". (Of course a Git ref is usually just a file in .git with a SHA1 in it. They don't care about the SHA1, so really they are launching a Git process for a file exists operation. This used to t…

Git refs can be packed (see https://git-scm.com/docs/git-pack-refs) so it's not just about checking if a file exists.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#46

Post by chriscool »

CapacitorSet wrote: post #15
Animats wrote: post #9

Each Gitaly server instance was fork/exec'ing Git processes about 20 times per second so we seemed to finally have a very promising lead. What's really wrong here is that they're apparently spawning processes like crazy. Do they spawn a new process for each API call? That's like running CGI programs under Apache, like it's 1995.

I wonder how much it would speed up if they were using libgit2 directly.

libgit2 has had bugs related to file locking (for example when you make some ref changes while garbage collecting) and libgit2 does not implement all the git features, so you cannot do everything using libgit2 anyway.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#47

Post by nurettin »

readittwice wrote: post #41
nurettin wrote: post #40

Earlier quoted context omitted.

Back when I tried running a jruby application on 800mb ram, it bloated, then started throwing "OutOfMemoryError"s and "Insufficient Class Space" or something similar. Apparently jruby was generating too many new types at runtime to accommodate rails framework. Garbage collector was pretty garbage at it's job back in 2011.

AFAIR JRuby creates a Java class for each Ruby method. Classes were part of the Permanent Generation, so sometimes this generation was too small escpecially when using Rails. It was enough to just increase the size of this generation with -XX:MaxPermSize (-Xmx doesn't increase the permanent generation's size). The Permanent Generation was named that way since objects in there were never collected. For most applicatio…

That was my experience as well. Increase maxpermsize and reboot every night.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#48

Post by masklinn »

CapacitorSet wrote: post #15
Animats wrote: post #9

Each Gitaly server instance was fork/exec'ing Git processes about 20 times per second so we seemed to finally have a very promising lead. What's really wrong here is that they're apparently spawning processes like crazy. Do they spawn a new process for each API call? That's like running CGI programs under Apache, like it's 1995.

I wonder how much it would speed up if they were using libgit2 directly.

Assuming they're currently pure git, binding to libgit2 would require using cgo, which could have far-ranging (negative) consequences.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#49

Post by masklinn »

cstrahan wrote: post #38
lbotos wrote: post #27

Earlier quoted context omitted.

Curious, how would you do it? (disclaimer: I work at GitLab, not on this project though)

In their case, why not stat the path directly?

Packs, most likely. On my main $work local repo, I have 27 refs and 22240 packed refs.

Re: How a fix in Go 1.9 sped up our Gitaly service by 30x

#50

Post by alecthomas »

_urga wrote: post #33

We currently have the same problem in Node, where fork is still being called synchronously from the event loop instead of asynchronously from the thread pool. Calling exec() or spawn() in Node is therefore not asynchronous and can block your event loop for hundreds of milliseconds or even seconds as RSS increases. https://github.com/nodejs/node/issues/14917

That looked like a very frustrating exchange.
Post reply on HN