Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Go executable files are still getting larger

cockroachlabs.com

Post reply on HN
Print view View original

61–70 of 174 posts

Re: My Go executable files are still getting larger

#61

Post by rsc »

rsc wrote: post #55

This article is full of misinformation. Just a few representative things: - The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint, by letting the OS demand-page this critical table that is used any time a stack must be walked (in particular, during garbage collection). See https://golang.org/s/go12symtab for details. - We (the Go team) did not “recompress” pclntab in Go 1.…

An easily obtained apples-to-apples¹ table:

    $ for i in $(seq 3 16); do
        curl -sLo go1.$i.tgz https://golang.org/dl/go1.$i.linux-amd64.tar.gz
        tar xzf go1.$i.tgz go/bin/gofmt
        size=$(ls -l go/bin/gofmt | awk '{print $5}')
        strip go/bin/gofmt
        size2=$(ls -l go/bin/gofmt | awk '{print $5}')
        echo go1.$i $size $size2
    done
    
    go1.3  3496520 2528664
    go1.4² 14398336 13139184
    go1.5  3937888 2765696
    go1.6  3894568 2725376
    go1.7  3036195 1913704
    go1.8  3481554 2326760
    go1.9  3257829 2190792
    go1.10 3477807 2166536
    go1.11 3369391 2441288
    go1.12 3513529 2506632
    go1.13 3543823 2552632
    go1.14 3587746 2561208
    go1.15 3501176 2432248
    go1.16 3448663 2443736
    $
Size fluctuates from release to release, but the overall trendline is flat: Go 1.16 binaries are roughly where Go 1.3 binaries were.

At the moment, it looks like Go 1.17 binaries will get a bit smaller thanks to the new register ABI making executable code smaller (and faster).

¹ Well, not completely. The gofmt code itself was changing from release to release, but not much. Most of the binary is the libraries and runtime, though, so it's still accurate for trends.

² Turns out we shipped the go 1.4 gofmt binary built with the race detector enabled! Oops.

Re: My Go executable files are still getting larger

#62

Post by knz42 »

rsc wrote: post #55

This article is full of misinformation. Just a few representative things: - The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint, by letting the OS demand-page this critical table that is used any time a stack must be walked (in particular, during garbage collection). See https://golang.org/s/go12symtab for details. - We (the Go team) did not “recompress” pclntab in Go 1.…

As explained in another comment below, it's a good thing when all the byte usage is represented in the ELF section headers or the symbol tables.

The DWARF data is currently so represented, and so was pclntab prior to 1.16.

Today, the DWARF data is still present; the symbol for pclntab has an advertized size of zero, and the amount of data not represented in the symbol table and ELF section headers has grown.

> If the 73% of the binary diagnosed as “not useful” were really not useful, a reasonable demonstration would be to delete it from the binary and see the binary still run. It clearly would not.

Probably not if all of it was removed at once.

It could be that just 5% of it is useful and removing all of it would produce a non-working binary. What does the experiment prove exactly?

Look you can claim that "having the necessary metadata for garbage collection and reflection in a statically-compiled language takes up a significant amount of space" but without clear evidence of how much space that is, with proper accounting of byte usage, this claim is non-falsifiable and thus of doubtful value.

Re: My Go executable files are still getting larger

#63

Post by knz42 »

rsc wrote: post #61
rsc wrote: post #55

This article is full of misinformation. Just a few representative things: - The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint, by letting the OS demand-page this critical table that is used any time a stack must be walked (in particular, during garbage collection). See https://golang.org/s/go12symtab for details. - We (the Go team) did not “recompress” pclntab in Go 1.…

An easily obtained apples-to-apples¹ table: $ for i in $(seq 3 16); do curl -sLo go1.$i.tgz https://golang.org/dl/go1.$i.linux-amd64.tar.gz tar xzf go1.$i.tgz go/bin/gofmt size=$(ls -l go/bin/gofmt | awk '{print $5}') strip go/bin/gofmt size2=$(ls -l go/bin/gofmt | awk '{print $5}') echo go1.$i $size $size2 done go1.3 3496520 2528664 go1.4² 14398336 13139184 go1.5 3937888 2765696 go1.6 3894568 2725376 go1.7 3036195 1…

Strip removes the symbol tables and DWARF information.

But still, the sum of all the bytes advertised in symbol tables for the non-DWARF data does not sum up to the stripped size. What's the remainder about?

I am reminded of how early versions of MSWord were embedding pages of heap space in save files that were not relevant to the document being saved, just because it made the saving algorithm simpler. For all we know, the go linker could be embedding random data.

Re: My Go executable files are still getting larger

#65

Post by rsc »

knz42 wrote: post #63
rsc wrote: post #61

Earlier quoted context omitted.

An easily obtained apples-to-apples¹ table: $ for i in $(seq 3 16); do curl -sLo go1.$i.tgz https://golang.org/dl/go1.$i.linux-amd64.tar.gz tar xzf go1.$i.tgz go/bin/gofmt size=$(ls -l go/bin/gofmt | awk '{print $5}') strip go/bin/gofmt size2=$(ls -l go/bin/gofmt | awk '{print $5}') echo go1.$i $size $size2 done go1.3 3496520 2528664 go1.4² 14398336 13139184 go1.5 3937888 2765696 go1.6 3894568 2725376 go1.7 3036195 1…

Strip removes the symbol tables and DWARF information. But still, the sum of all the bytes advertised in symbol tables for the non-DWARF data does not sum up to the stripped size. What's the remainder about? I am reminded of how early versions of MSWord were embedding pages of heap space in save files that were not relevant to the document being saved, just because it made the saving algorithm simpler. For all we kno…

> For all we know, the go linker could be embedding random data.

I do know, and it is not.

Re: My Go executable files are still getting larger

#66

Post by throwaway894345 »

marcus_holmes wrote: post #37
throwaway894345 wrote: post #14

Earlier quoted context omitted.

> I mostly write web servers in Go, which (as the article says) are executed rarely, so init time really doesn't matter to me. Presumably a few mb of disk usage also doesn't matter then? I also write web services, but I do care about the init time precisely because I want to be deployments to take as little time as possible so that we can deploy (and rollback) many times per day with relatively simple automation. Tha…

Yeah, it's a trade-off. At the moment the longest part of my deploy is copying the new executable to the server. I'd trade a couple of seconds of init time for a smaller executable because that would result in a faster deploy. I don't use Docker to deploy, because it's just a single executable file (and a bunch of templates, though I'm looking at embedding those). One of the reasons I'm reluctant to go down the Docke…

> I don't use Docker to deploy, because it's just a single executable file (and a bunch of templates, though I'm looking at embedding those). One of the reasons I'm reluctant to go down the Docker road is because it's going to add more time to my deployment.

Yeah, I don't advocate Docker for its own sake, but my organization deploys everything via Kubernetes because it's simpler than having a bespoke orchestration strategy/toolchain for each project, but if you're a one-project shop then containerization probably doesn't add a lot of value (although I still haven't figured out a streamlined, reliable way to provision/configure VMs).

Re: My Go executable files are still getting larger

#67

Post by knz42 »

rsc wrote: post #65
knz42 wrote: post #63

Earlier quoted context omitted.

Strip removes the symbol tables and DWARF information. But still, the sum of all the bytes advertised in symbol tables for the non-DWARF data does not sum up to the stripped size. What's the remainder about? I am reminded of how early versions of MSWord were embedding pages of heap space in save files that were not relevant to the document being saved, just because it made the saving algorithm simpler. For all we kno…

> For all we know, the go linker could be embedding random data. I do know, and it is not.

> But still, the sum of all the bytes advertised in symbol tables for the non-DWARF data does not sum up to the stripped size. What's the remainder about?

if you do know, then pray, what is the answer to this question?

Re: My Go executable files are still getting larger

#68

Post by knz42 »

bradfitz wrote: post #64

To promote my own tool, https://github.com/bradfitz/shotizam lets you drill down into why Go binaries are large without having to make up terms like "dark bytes".

This code, if I'm reading it right, uses the symbol table and ELF section headers.

As explained in OP, the sum of sizes advertised in the symtable and ELF section headers does not add up to the final binary size. The shotizam tool is thus blind to that difference.

Re: My Go executable files are still getting larger

#69

Post by rsc »

knz42 wrote: post #62
rsc wrote: post #55

This article is full of misinformation. Just a few representative things: - The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint, by letting the OS demand-page this critical table that is used any time a stack must be walked (in particular, during garbage collection). See https://golang.org/s/go12symtab for details. - We (the Go team) did not “recompress” pclntab in Go 1.…

As explained in another comment below, it's a good thing when all the byte usage is represented in the ELF section headers or the symbol tables. The DWARF data is currently so represented, and so was pclntab prior to 1.16. Today, the DWARF data is still present; the symbol for pclntab has an advertized size of zero, and the amount of data not represented in the symbol table and ELF section headers has grown. > If the…

> Look you can claim that "having the necessary metadata for garbage collection and reflection in a statically-compiled language takes up a significant amount of space" but without clear evidence of how much space that is, with proper accounting of byte usage, this claim is non-falsifiable and thus of doubtful value.

I respectfully disagree. I believe there is value in pointing out the false claims in a long blog post even when there isn't time to write a full-length rebuttal.

Re: My Go executable files are still getting larger

#70

Post by synergy20 »

skrebbel wrote: post #39
daitangio wrote: post #35

Go static linking is a great happy idea for a Java guy trapped in the Classpath Dependency Hell (or C# / DLL Hell). It is a very annoying thing for a C++ programmer, which can dynamically link operating system libraries at will.

You mean a C++ programmer who does not care whether their program works on anybody else's computer?

in that case you can happily static-link c++ programs, which probably will be smaller than go binary.
Post reply on HN