Live data from Hacker News

My Go executable files are still getting larger

cockroachlabs.com

51–60 of 174 posts

Re: My Go executable files are still getting larger

#51

I always wonder if this is the flip side of the fast compilation? It would be nice to be able to decide on those trade-offs ourselves. I mostly write web servers in Go, which (as the article says) are executed rarely, so init time really doesn't matter to me. But I've been looking at writing some desktop apps in Go, and then init time will matter.

> I always wonder if this is the flip side of the fast compilation?

Go is cheating a bit on this one by heavily caching everything. Building in fresh container is quite slow (ok, maybe not c++ slow but still much slower then C).

Always having to build anything due to static linking does not help either.

Re: My Go executable files are still getting larger

#52

It's time for debug info like this to be sent to "onlinesymbolserver.com", encrypted with a hash of the binary. Then, whenever a debugger connects to a binary, it can simply download the symbols as required. And for the 99.9% who don't need debug info, it isn't needlessly shipped. Microsoft invented this in the 90's...

I don't think this information in Go is used for debugging alone. It's also used by mallocgc and other parts of the runtime. This is why you will see "gentraceback" in the profiles of busy Go servers. It's not because your program is printing a lot of stack traces, it's because stack walking is a core part of the GC, and because some logic in the runtime refers to the names of functions to branch around special cases.

Re: My Go executable files are still getting larger

#53
post #9

> In other words, the Go team decided to make executable files larger to save up on initialization time. I mean... Im genuinely curious if this is a "we have extra engineering resources and can explore/complain about this" or "we have a client who is running cockroachdb and can't handle a 172mb binary install for a database server". Is there really someone out there who installs Cockroach (a global distributed auto-s…

It affects the download and instantiation time for containers.

172MB for a Docker image that contains a DB is pretty small, Go Docker image are among the smallest because Go binary can run with minimal deps.

Re: My Go executable files are still getting larger

#54
post #17

Earlier quoted context omitted.

Is it clear that the “dark bytes” are “useless” debug information? It sounds like it’s just stuff not in the symbol table.

I did keep waiting for some point in the blog post where they would null out the "dark bytes" and see if/how the binary runs.

It won't. If you try to remove this information your program will crash as soon as the GC runs or someone calls systemstack, i.e. pretty much instantly.

Re: My Go executable files are still getting larger

#55
This article is full of misinformation. Just a few representative things:

- The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint, by letting the OS demand-page this critical table that is used any time a stack must be walked (in particular, during garbage collection). See https://golang.org/s/go12symtab for details.

- We (the Go team) did not “recompress” pclntab in Go 1.15. We did not remove pclntab in Go 1.16. Nor do we have plans to do either. Consequently, we never claimed “pclntab has been reduced to zero”, which is presented in the article as if a direct quote.

- If the 73% of the binary diagnosed as “not useful” were really not useful, a reasonable demonstration would be to delete it from the binary and see the binary still run. It clearly would not.

- The big table seems to claim that a 40 MB Go 1.8 binary has grown to a 289 MB Go 1.16 binary. That’s certainly not the case. More is changing from line to line in that table than the Go version.

Overall, the claim of “dark bytes” or “non-useful bytes” strikes me as similar to the claims of “junk DNA”. They’re not dark or non-useful. It turns out that having the necessary metadata for garbage collection and reflection in a statically-compiled language takes up a significant amount of space, which we’ve worked over time at reducing. But the dynamic possibilities in reflection and interface assertions mean that fewer bytes can be dropped than you’d hope. We track binary size work in https://golang.org/issue/6853.

An unfortunate article.

Re: My Go executable files are still getting larger

#56
post #9

> In other words, the Go team decided to make executable files larger to save up on initialization time. I mean... Im genuinely curious if this is a "we have extra engineering resources and can explore/complain about this" or "we have a client who is running cockroachdb and can't handle a 172mb binary install for a database server". Is there really someone out there who installs Cockroach (a global distributed auto-s…

> Is there really someone out there who installs Cockroach (a global distributed auto-sharded database) and thinks twice about 172mb of disk space?

I think it is exactly this mindset that caused the current situation (that the 2/3 of the compiled binaries are useless to the users).

Re: My Go executable files are still getting larger

#58

Go static linking is a great happy idea for a Java guy trapped in the Classpath Dependency Hell (or C# / DLL Hell). It is a very annoying thing for a C++ programmer, which can dynamically link operating system libraries at will.

Go's static linking idea comes from plan9 C compilers, a few years before Java. We owe a lot from plan9:

- Go's design, based on both C compilers and Inferno's limbo

- /proc

- utf-8

- 9p

Post reply on HN