Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Go executable files are still getting larger

cockroachlabs.com

Post reply on HN
Print view View original

81–90 of 174 posts

Re: My Go executable files are still getting larger

#81

Post by hobbified »

rsc wrote: post #73
knz42 wrote: post #67

Earlier quoted context omitted.

> But still, the sum of all the bytes advertised in symbol tables for the non-DWARF data does not sum up to the stripped size. What's the remainder about? if you do know, then pray, what is the answer to this question?

Maybe another day I will take the time to write a full-length blog post examining the bytes in a Go binary. Today I have other work planned and still intend to do it. My points today are only that: 1. Go binary size has not gotten dramatically better or worse in any particular release and is mostly unchanged since Go 1.3. 2. Many claimed facts in the blog post are incorrect. 3. The linker is not "embedding random dat…

It's a mystery, not a lynch mob. Everyone reading is interested in knowing "huh, what is this stuff then?"

Re: My Go executable files are still getting larger

#83

Post by arp242 »

> Moreover, consider that these executable files fly around as container images, and/or are copied between VMs in the cloud, thousands of times per day! Every time, 70% of a couple hundred megabytes are copied around for no good reason and someone needs to pay ingress/egress networking costs for these file copies. That is quite some money being burned for no good reason!

Does the author think the Go authors are stupid blubbering idiots who someone missed this huge elephant-sized low-hanging fruit? Binary sizes have been a point of attention for years, and somehow missing 70% wasted space would be staggeringly incompetent.

Reminds me of the time in high school when one of the my classmates ended up with a 17A doorbell in some calculations. I think he used the wrong formula or swapped some numbers. The teacher, quite rightfully, berated him for not actually looking at the result of his calculation and judging if it's roughly in the right ballpark, as 17A is a ludicrous amount of power for a doorbell. Anyone can see that's just widely wrong.

If this story had ended up with 0.7%, sure, I can believe that. 7%? Unlikely and I'd be skeptical, but still possible I suppose. *70%* Yeah nah, that's just as silly as a 17A doorbell.

This huge 70% number should have been a clue to the author themselves too that they've missed something.

Re: My Go executable files are still getting larger

#84

Post by knz42 »

rsc wrote: post #55

This article is full of misinformation. Just a few representative things: - The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint, by letting the OS demand-page this critical table that is used any time a stack must be walked (in particular, during garbage collection). See https://golang.org/s/go12symtab for details. - We (the Go team) did not “recompress” pclntab in Go 1.…

> The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint [...]

yes this is acknowledged in the OP

> We (the Go team) did not “recompress” pclntab in Go 1.15.

There's now less redundancy in the funcdata, so in my book less redundancy = more compression.

> We did not remove pclntab in Go 1.16.

Correct; it is not "removed"; instead the advertised size of the symbol has been reduced to zero. Maybe the data is still there, but it's not accounted any more.

Changed in the text. (The correction is already present in the original version of the analysis, and the cockroach labs copy should be updated soon)

> we never claimed “pclntab has been reduced to zero”, which is presented in the article as if a direct quote.

Correct, there was indeed no such claim. Removed the quotes and rephrased that paragraph.

> If the 73% of the binary diagnosed as “not useful” were really not useful, a reasonable demonstration would be to delete it from the binary and see the binary still run. It clearly would not.

1) the phrase "non-useful" was a mistake. There is no definite proof it is non-useful, as you pointed out. Corrected in the text.

2) see discussion below - the demonstration is more complicated than that, as removing 100 bytes where just 1 byte is necessary will break the executable in the same way as removing 100 necessary bytes.

I think the proper next step here is to acknowledge that the problem is not "usefulness" but rather accountability.

> The big table seems to claim that a 40 MB Go 1.8 binary has grown to a 289 MB Go 1.16 binary. That’s certainly not the case. More is changing from line to line in that table than the Go version.

Correct. Added a note to emphasize this fact.

> Overall, the claim of “dark bytes” or “non-useful bytes” strikes me as similar to the claims of “junk DNA”. They’re not dark or non-useful.

Let's forget about "non-useful", this was a mistake and will be corrected. The word 'dark' is still relevant however. The adequate comparison is not "junk DNA", but instead "dark silicon":

https://ieeexplore.ieee.org/abstract/document/6307773

We're talking about a general % of executable size that's necessary for a smaller % of use cases in program function.

I'm all for trade-offs, but IMHO they should be transparent.

Re: My Go executable files are still getting larger

#86

Post by superdisk »

arp242 wrote: post #83

> Moreover, consider that these executable files fly around as container images, and/or are copied between VMs in the cloud, thousands of times per day! Every time, 70% of a couple hundred megabytes are copied around for no good reason and someone needs to pay ingress/egress networking costs for these file copies. That is quite some money being burned for no good reason! Does the author think the Go authors are stupi…

In a world where electron reigns supreme, 70% waste isn't unthinkable

Re: My Go executable files are still getting larger

#87

Post by cristaloleg »

marcus_holmes wrote: post #37
throwaway894345 wrote: post #14

Earlier quoted context omitted.

> I mostly write web servers in Go, which (as the article says) are executed rarely, so init time really doesn't matter to me. Presumably a few mb of disk usage also doesn't matter then? I also write web services, but I do care about the init time precisely because I want to be deployments to take as little time as possible so that we can deploy (and rollback) many times per day with relatively simple automation. Tha…

Yeah, it's a trade-off. At the moment the longest part of my deploy is copying the new executable to the server. I'd trade a couple of seconds of init time for a smaller executable because that would result in a faster deploy. I don't use Docker to deploy, because it's just a single executable file (and a bunch of templates, though I'm looking at embedding those). One of the reasons I'm reluctant to go down the Docke…

Have you tried UPX ?

Can be a good start https://blog.filippo.io/shrink-your-go-binaries-with-this-on...

Re: My Go executable files are still getting larger

#88

Post by nappy-doo »

knz42 wrote: post #68
bradfitz wrote: post #64

To promote my own tool, https://github.com/bradfitz/shotizam lets you drill down into why Go binaries are large without having to make up terms like "dark bytes".

This code, if I'm reading it right, uses the symbol table and ELF section headers. As explained in OP, the sum of sizes advertised in the symtable and ELF section headers does not add up to the final binary size. The shotizam tool is thus blind to that difference.

This sounds like a bug to me, not nefarious intent.

Re: My Go executable files are still getting larger

#89

Post by ngrilly »

hobbified wrote: post #81
rsc wrote: post #73

Earlier quoted context omitted.

Maybe another day I will take the time to write a full-length blog post examining the bytes in a Go binary. Today I have other work planned and still intend to do it. My points today are only that: 1. Go binary size has not gotten dramatically better or worse in any particular release and is mostly unchanged since Go 1.3. 2. Many claimed facts in the blog post are incorrect. 3. The linker is not "embedding random dat…

It's a mystery, not a lynch mob. Everyone reading is interested in knowing "huh, what is this stuff then?"

Russ is totally right. Pretending the linker is embedding “random data” is just trolling.

Re: My Go executable files are still getting larger

#90

Post by wlll »

gray_-_wolf wrote: post #50
oefrha wrote: post #43

Earlier quoted context omitted.

Tinygo is severely limited, e.g. it doesn't support encoding/json. Want to deal with any JSON API, or anything that indirectly uses JSON serialization? Forget about it. A current side project of mine uses golang's wasm. Not a big codebase, but the wasm is 2.7MB brotli'ed. Certainly huge to me (I'm sure it's almost in the lean and mean camp compared to the average website today, though).

encoding/json sucks anyway, so using something 3rd party would likely be better option anyway

How so?
Post reply on HN