Couldn't the dark bytes just be shipped as a separate file - for those who need it?
Is it clear that the “dark bytes” are “useless” debug information? It sounds like it’s just stuff not in the symbol table.
My Go executable files are still getting larger
11–20 of 174 posts
Re: My Go executable files are still getting larger
#12It's time for debug info like this to be sent to "onlinesymbolserver.com", encrypted with a hash of the binary. Then, whenever a debugger connects to a binary, it can simply download the symbols as required. And for the 99.9% who don't need debug info, it isn't needlessly shipped. Microsoft invented this in the 90's...
Around 1992 Borland C++ would use macro based code generation for generic code in BIDS 1.0, sounds familiar?
I guess it is just a tradition with Go, rediscovering history.
Re: My Go executable files are still getting larger
#13> In other words, the Go team decided to make executable files larger to save up on initialization time. I mean... Im genuinely curious if this is a "we have extra engineering resources and can explore/complain about this" or "we have a client who is running cockroachdb and can't handle a 172mb binary install for a database server". Is there really someone out there who installs Cockroach (a global distributed auto-s…
It affects the download and instantiation time for containers.
Re: My Go executable files are still getting larger
#14I always wonder if this is the flip side of the fast compilation? It would be nice to be able to decide on those trade-offs ourselves. I mostly write web servers in Go, which (as the article says) are executed rarely, so init time really doesn't matter to me. But I've been looking at writing some desktop apps in Go, and then init time will matter.
> I mostly write web servers in Go, which (as the article says) are executed rarely, so init time really doesn't matter to me.
Presumably a few mb of disk usage also doesn't matter then? I also write web services, but I do care about the init time precisely because I want to be deployments to take as little time as possible so that we can deploy (and rollback) many times per day with relatively simple automation. That said, the bottleneck to fast deployments isn't the binary starting up, but the machine pulling the executable artifact, so the binary sizes do matter to me. That said, very often these executable artifacts are a Docker image, which tend to come with a lot more bloat than one will find in a Go binary, so step 1 is getting your Go binary on a scratch image.
Re: My Go executable files are still getting larger
#15> Every time, 70% of a couple hundred megabytes are copied around for no good reason and someone needs to pay ingress/egress networking costs for these file copies. That is quite some money being burned for no good reason!
Meanwhile half of the world is pushing images by nvidia, intel and amd around for their machine learning software:
Intel OneAPI runtime libraries: 4.74GB (or 18.4GB for compilers) CUDA runtime libraries: 1.92GB (or 4.2GB for compilers)
These go binaries are still relatively small
Re: My Go executable files are still getting larger
#16It's time for debug info like this to be sent to "onlinesymbolserver.com", encrypted with a hash of the binary. Then, whenever a debugger connects to a binary, it can simply download the symbols as required. And for the 99.9% who don't need debug info, it isn't needlessly shipped. Microsoft invented this in the 90's...
Sadly go doesn’t care about DWARF which would allow them to reuse already existing infrastructure.
Re: My Go executable files are still getting larger
#17Couldn't the dark bytes just be shipped as a separate file - for those who need it?
Is it clear that the “dark bytes” are “useless” debug information? It sounds like it’s just stuff not in the symbol table.
I did keep waiting for some point in the blog post where they would null out the "dark bytes" and see if/how the binary runs.
Re: My Go executable files are still getting larger
#18Is there a bug report for this?
Rob Pike filed this issue for it after the first incarnation of the article was released: https://github.com/golang/go/issues/36313
Re: My Go executable files are still getting larger
#19> In other words, the Go team decided to make executable files larger to save up on initialization time. I mean... Im genuinely curious if this is a "we have extra engineering resources and can explore/complain about this" or "we have a client who is running cockroachdb and can't handle a 172mb binary install for a database server". Is there really someone out there who installs Cockroach (a global distributed auto-s…
It affects the download and instantiation time for containers.
One word: trade-offs.
Re: My Go executable files are still getting larger
#20> In other words, the Go team decided to make executable files larger to save up on initialization time. I mean... Im genuinely curious if this is a "we have extra engineering resources and can explore/complain about this" or "we have a client who is running cockroachdb and can't handle a 172mb binary install for a database server". Is there really someone out there who installs Cockroach (a global distributed auto-s…
It affects the download and instantiation time for containers.
I worry about that for 10gb omnibus containers, not so much for <500mb.