Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Go executable files are still getting larger

cockroachlabs.com

Post reply on HN
Print view View original

41–50 of 174 posts

Re: My Go executable files are still getting larger

#41

Post by jerf »

marcus_holmes wrote: post #3

I always wonder if this is the flip side of the fast compilation? It would be nice to be able to decide on those trade-offs ourselves. I mostly write web servers in Go, which (as the article says) are executed rarely, so init time really doesn't matter to me. But I've been looking at writing some desktop apps in Go, and then init time will matter.

More likely the flip side of statically compiling every library in. Even if the Go compiler did the equivalent of -Os you'd still be getting binaries that people were complaining about.

I know for a good long time Go wasn't even stripping out unused functions, e.g., if you used any of the "strings" standard module you got the whole thing compiled in. I don't know if that's still the case, but it would be another source of size problems.

I'm also not sure why you're talking about init time; binary size doesn't necessarily factor in to that very much. The things that make the Go binaries large also tend to lean in favor of faster initialization anyhow. A lot of these binaries won't even necessarily be read in to memory, or have to stay there in the event of swap pressure, since they're already disk-backed memory pages that can be never loaded in the first place, or cheaply evicted by the virtual memory system if space is needed.

Re: My Go executable files are still getting larger

#42

Post by jerf »

marcus_holmes wrote: post #37
throwaway894345 wrote: post #14

Earlier quoted context omitted.

> I mostly write web servers in Go, which (as the article says) are executed rarely, so init time really doesn't matter to me. Presumably a few mb of disk usage also doesn't matter then? I also write web services, but I do care about the init time precisely because I want to be deployments to take as little time as possible so that we can deploy (and rollback) many times per day with relatively simple automation. Tha…

Yeah, it's a trade-off. At the moment the longest part of my deploy is copying the new executable to the server. I'd trade a couple of seconds of init time for a smaller executable because that would result in a faster deploy. I don't use Docker to deploy, because it's just a single executable file (and a bunch of templates, though I'm looking at embedding those). One of the reasons I'm reluctant to go down the Docke…

I still use Docker images to deploy several of my Go systems. For one, it's nice to have it integrated into other ecosystems where a "Docker image" is just the base level of functionality. The additional security (even if not necessarily perfect) and isolation isn't all bad either.

It's perfectly fine to compile a Go binary and stick it into a Docker container on its own; it is not obligatory for a Docker container to contain a full Linux installation. I've got a couple of Docker files that are pretty much just

    FROM scratch

    ADD my_binary /
(Usually my bind mounts and the command executed are set up elsewhere for logging and such.)

It is also a convenient way of knowing exactly what your dependencies are... for instance I have several Go containers that also have to include a trusted cert store so they can access remote HTTPS resources correctly. Since you don't need a full Linux install to run a Go binary, it's very easy to use Docker as a complete, human-comprehensible manifest of exactly what is in that container.

Re: My Go executable files are still getting larger

#43

Post by oefrha »

gen220 wrote: post #27
Touche wrote: post #24

Earlier quoted context omitted.

Go binary size makes it not an option for wasm.

There are go variants for this. See tinygo [1], which targeted embedded originally (iirc) but now also targets wasm. So you’re definitely correct that core Go is not an option, but options exist within the “greater metropolitan area” that’s built up around downtown. These are among the benefits of having a relatively simple language and a 1.0 compatibility commitment, I think. [1]: https://tinygo.org/ their FAQ is qu…

Tinygo is severely limited, e.g. it doesn't support encoding/json. Want to deal with any JSON API, or anything that indirectly uses JSON serialization? Forget about it.

A current side project of mine uses golang's wasm. Not a big codebase, but the wasm is 2.7MB brotli'ed. Certainly huge to me (I'm sure it's almost in the lean and mean camp compared to the average website today, though).

Re: My Go executable files are still getting larger

#44

Post by rob74 »

blinkingled wrote: post #29

> We can call this the “dark file usage” of Go binaries, and it occupies between 15% and 33% of the total file size inside CockroachDB. > Sadly, the removal of pclntab in Go 1.16 actually transferred the payload to the “dark” bytes. I surely would have expect better from programming language designers/developers than this. Sounds like they just moved the problem from one place to another.

I'll hold my judgement until someone manages to actually find out what this "dark" space is (quote from the article: "At this time, I do not have a satisfying explanation for this “dark” file usage").

Re: My Go executable files are still getting larger

#45

Post by speedgoose »

gen220 wrote: post #27
Touche wrote: post #24

Earlier quoted context omitted.

Go binary size makes it not an option for wasm.

There are go variants for this. See tinygo [1], which targeted embedded originally (iirc) but now also targets wasm. So you’re definitely correct that core Go is not an option, but options exist within the “greater metropolitan area” that’s built up around downtown. These are among the benefits of having a relatively simple language and a 1.0 compatibility commitment, I think. [1]: https://tinygo.org/ their FAQ is qu…

I tried tinygo on web assembly. I very quickly decided to use Rust instead.

Re: My Go executable files are still getting larger

#46

Post by patrickaljord »

Touche wrote: post #24
parhamn wrote: post #9

> In other words, the Go team decided to make executable files larger to save up on initialization time. I mean... Im genuinely curious if this is a "we have extra engineering resources and can explore/complain about this" or "we have a client who is running cockroachdb and can't handle a 172mb binary install for a database server". Is there really someone out there who installs Cockroach (a global distributed auto-s…

Go binary size makes it not an option for wasm.

Why not? If you use wasm for a small library need in your web app, then sure, go doesn't make sense. But if your whole app is coded in wasm like a game, then it's probably ok I guess as the app will be heavy to load regardless.

Re: My Go executable files are still getting larger

#47

Post by kstenerud »

daitangio wrote: post #35

Go static linking is a great happy idea for a Java guy trapped in the Classpath Dependency Hell (or C# / DLL Hell). It is a very annoying thing for a C++ programmer, which can dynamically link operating system libraries at will.

"DLL Hell" was coined in the early days of Windows (before C#), and originally referred to C and C++ dynamic library problems.

Re: My Go executable files are still getting larger

#49

Post by icholy »

the_mitsuhiko wrote: post #16
londons_explore wrote: post #4

It's time for debug info like this to be sent to "onlinesymbolserver.com", encrypted with a hash of the binary. Then, whenever a debugger connects to a binary, it can simply download the symbols as required. And for the 99.9% who don't need debug info, it isn't needlessly shipped. Microsoft invented this in the 90's...

Sadly go doesn’t care about DWARF which would allow them to reuse already existing infrastructure.

Go uses DWARF.

Re: My Go executable files are still getting larger

#50

Post by gray_-_wolf »

oefrha wrote: post #43
gen220 wrote: post #27

Earlier quoted context omitted.

There are go variants for this. See tinygo [1], which targeted embedded originally (iirc) but now also targets wasm. So you’re definitely correct that core Go is not an option, but options exist within the “greater metropolitan area” that’s built up around downtown. These are among the benefits of having a relatively simple language and a 1.0 compatibility commitment, I think. [1]: https://tinygo.org/ their FAQ is qu…

Tinygo is severely limited, e.g. it doesn't support encoding/json. Want to deal with any JSON API, or anything that indirectly uses JSON serialization? Forget about it. A current side project of mine uses golang's wasm. Not a big codebase, but the wasm is 2.7MB brotli'ed. Certainly huge to me (I'm sure it's almost in the lean and mean camp compared to the average website today, though).

encoding/json sucks anyway, so using something 3rd party would likely be better option anyway
Post reply on HN