Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Go executable files are still getting larger

cockroachlabs.com

Post reply on HN
Print view View original

91–100 of 174 posts

Re: My Go executable files are still getting larger

#91

Post by OskarS »

rsc wrote: post #73
knz42 wrote: post #67

Earlier quoted context omitted.

> But still, the sum of all the bytes advertised in symbol tables for the non-DWARF data does not sum up to the stripped size. What's the remainder about? if you do know, then pray, what is the answer to this question?

Maybe another day I will take the time to write a full-length blog post examining the bytes in a Go binary. Today I have other work planned and still intend to do it. My points today are only that: 1. Go binary size has not gotten dramatically better or worse in any particular release and is mostly unchanged since Go 1.3. 2. Many claimed facts in the blog post are incorrect. 3. The linker is not "embedding random dat…

I have no dog in this fight either way, I'm just very curious about the answer: if something like 30-40% in a Go executable that clocks in at more than a 100 megabytes is not taken up by either symbols, debug information or the pclntab, what exactly is in it? You mentioned "necessary metadata for garbage collection and reflection in a statically-compiled language" in a previous comment. Can you give some more details on what that means?

Re: My Go executable files are still getting larger

#92

Post by knz42 »

nappy-doo wrote: post #88
knz42 wrote: post #68

Earlier quoted context omitted.

This code, if I'm reading it right, uses the symbol table and ELF section headers. As explained in OP, the sum of sizes advertised in the symtable and ELF section headers does not add up to the final binary size. The shotizam tool is thus blind to that difference.

This sounds like a bug to me, not nefarious intent.

What in my reply suggests I assumed nefarious intent?

Brad is a good person, it's the code we're talking about here.

Re: My Go executable files are still getting larger

#93

Post by rsc »

OskarS wrote: post #91
rsc wrote: post #73

Earlier quoted context omitted.

Maybe another day I will take the time to write a full-length blog post examining the bytes in a Go binary. Today I have other work planned and still intend to do it. My points today are only that: 1. Go binary size has not gotten dramatically better or worse in any particular release and is mostly unchanged since Go 1.3. 2. Many claimed facts in the blog post are incorrect. 3. The linker is not "embedding random dat…

I have no dog in this fight either way, I'm just very curious about the answer: if something like 30-40% in a Go executable that clocks in at more than a 100 megabytes is not taken up by either symbols, debug information or the pclntab, what exactly is in it? You mentioned "necessary metadata for garbage collection and reflection in a statically-compiled language" in a previous comment. Can you give some more details…

You can see the true size of the Go pclntab in ELF binaries using "readelf -S" and in Mac binaries using "otool -l". Its not zero.

One thing that did change from Go 1.15 to Go 1.16 is that we broke up the pclntab into a few different pieces. Again, it's all in the section headers. But the pieces are not in the actual binary's symbol table anymore, because they don't need to be. And since the format is different, we would have removed the old "runtime.pclntab" symbol entirely, except some old tools got mad if the symbol was missing. So we left the old symbol table entry present, with a zero length.

Clearly, we could emit many more symbols accurately describing all these specific pieces of the binary. But ironically that would just make the binary even larger. Leaving them out is probably the right call for nearly all use case.

Except perhaps trying to analyze binary size, although even there even symbols don't paint a full picture. OK, the pclntab is large. Why is it large? What are the specific things in it that are large? Symbols don't help there at all. You need to analyze the actual data, add debug prints to the linker, and so on.

That would make an interesting post, and perhaps we will write one like that. But not today.

Re: My Go executable files are still getting larger

#94

Post by gen220 »

oefrha wrote: post #43
gen220 wrote: post #27

Earlier quoted context omitted.

There are go variants for this. See tinygo [1], which targeted embedded originally (iirc) but now also targets wasm. So you’re definitely correct that core Go is not an option, but options exist within the “greater metropolitan area” that’s built up around downtown. These are among the benefits of having a relatively simple language and a 1.0 compatibility commitment, I think. [1]: https://tinygo.org/ their FAQ is qu…

Tinygo is severely limited, e.g. it doesn't support encoding/json. Want to deal with any JSON API, or anything that indirectly uses JSON serialization? Forget about it. A current side project of mine uses golang's wasm. Not a big codebase, but the wasm is 2.7MB brotli'ed. Certainly huge to me (I'm sure it's almost in the lean and mean camp compared to the average website today, though).

Yeah, tinygo was originally intended for embedded, so the good news is that many significant wasm binary size optimizations are still to come, I think.

Today, there are ways to get the binary smaller, but they involve avoiding certain libraries or preferring certain implementations of common libraries over others.

It’s definitely an art that requires conscientious practice these days. A run of the mill go application will not compile down to a tiny wasm with tinygo, without refactoring.

But it’s possible. Like I said, it’s on the outskirts, not downtown. Still within reach at least.

Re: My Go executable files are still getting larger

#95

Post by harikb »

hobbified wrote: post #81
rsc wrote: post #73

Earlier quoted context omitted.

Maybe another day I will take the time to write a full-length blog post examining the bytes in a Go binary. Today I have other work planned and still intend to do it. My points today are only that: 1. Go binary size has not gotten dramatically better or worse in any particular release and is mostly unchanged since Go 1.3. 2. Many claimed facts in the blog post are incorrect. 3. The linker is not "embedding random dat…

It's a mystery, not a lynch mob. Everyone reading is interested in knowing "huh, what is this stuff then?"

You must know that breaking down and verifying someone else's analysis is more time consuming than writing your own. Just like dealing with a bug in another person's code.

Given them the benefit of doubt that Go team is cautious about binary size. People have dug in to this. Sure, they could do a better job giving some breakdown, but claiming that they are careless deserves that kind of response.

Given a choice of their time, I would rather have them work on some other Go language problem. Most low hanging fruit has already been had. See [1] [2] [3]

[1] https://dave.cheney.net/2020/05/09/ensmallening-go-binaries-...

[2] https://dave.cheney.net/tag/performance

[3] https://dave.cheney.net/2016/04/02/go-1-7-toolchain-improvem...

Re: My Go executable files are still getting larger

#96

Post by gen220 »

wlll wrote: post #90
gray_-_wolf wrote: post #50

Earlier quoted context omitted.

encoding/json sucks anyway, so using something 3rd party would likely be better option anyway

How so?

I think it’s well demonstrated to be slow (ns/op) and wasteful (allocs/op) in benchmarks, especially in bigger payloads.

The HTTP library is similar.

Most 3rd party libs make some trade offs that are pretty reasonable to achieve superior perf, but it’s worth understanding them to make sure it’s best for your use case.

Re: My Go executable files are still getting larger

#97

Post by cristaloleg »

bradfitz wrote: post #64

To promote my own tool, https://github.com/bradfitz/shotizam lets you drill down into why Go binaries are large without having to make up terms like "dark bytes".

Sadly it doesn't work with Go 1.16 :(( https://github.com/bradfitz/shotizam/issues/10

Re: My Go executable files are still getting larger

#98

Post by arp242 »

knz42 wrote: post #62
rsc wrote: post #55

This article is full of misinformation. Just a few representative things: - The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint, by letting the OS demand-page this critical table that is used any time a stack must be walked (in particular, during garbage collection). See https://golang.org/s/go12symtab for details. - We (the Go team) did not “recompress” pclntab in Go 1.…

As explained in another comment below, it's a good thing when all the byte usage is represented in the ELF section headers or the symbol tables. The DWARF data is currently so represented, and so was pclntab prior to 1.16. Today, the DWARF data is still present; the symbol for pclntab has an advertized size of zero, and the amount of data not represented in the symbol table and ELF section headers has grown. > If the…

> Look you can claim that "having the necessary metadata for garbage collection and reflection in a statically-compiled language takes up a significant amount of space" but without clear evidence of how much space that is, with proper accounting of byte usage, this claim is non-falsifiable and thus of doubtful value.

The article made the claim that 70% of space is wasted "dark bytes". The article should prove the claim, which it did not. It's an extraordinary claim that really requires more evidence than just an off-hand napkin calculation and talk about mysterious "dark bytes".

It takes very little time to write up something that's wrong.

It takes much more time to write a detailed rebuttal.

What you're doing here is pretty much the same trick quacks, young-earth creationists, and purveyors of all sorts of pseudo-scientific claims pull whenever they're challenged. Any fool can claim the earth is 6,000 years old. Proving that it's actually several billions years old requires deep insight in multiple branches of science. People stopped doing this after a while as it's so much effort and pointless as they're not listening anyway, so now they pull the "aha, you didn't falsify my claim about this or that bullshit I pulled out of my ass therefore I am right" zinger and think they look Very Smart And Scientific™.

But that's not how it works.

Also just outright disbelieving people like this is rather rude. You're coming off really badly here and your comment has the strong implication that Russ is outright lying. Yikes!

Re: My Go executable files are still getting larger

#99

Post by returningfory2 »

knz42 wrote: post #84
rsc wrote: post #55

This article is full of misinformation. Just a few representative things: - The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint, by letting the OS demand-page this critical table that is used any time a stack must be walked (in particular, during garbage collection). See https://golang.org/s/go12symtab for details. - We (the Go team) did not “recompress” pclntab in Go 1.…

> The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint [...] yes this is acknowledged in the OP > We (the Go team) did not “recompress” pclntab in Go 1.15. There's now less redundancy in the funcdata, so in my book less redundancy = more compression. > We did not remove pclntab in Go 1.16. Correct; it is not "removed"; instead the advertised size of the symbol has been re…

In this case, what is the point the blog post is trying to make?

The title of the post is "Go Executable Files Are Still Getting Larger". Upon further reading and conversation here it seems this is possibly not true, nor what the post is about. If we believe Russ's comments, Go executable sizes haven't increased much in general. Perhaps the reason you're seeing increases in Cockroach DB is because you keep writing more code for Cockroach DB?

Now the point has shifted to this notion of "dark bytes". So the article is about ... how the way you previously diagnosed the contents of binaries doesn't work anymore? That's fine and legitimate, but it seems like the point is over-extrapolated to become a criticism of the Go team.

Re: My Go executable files are still getting larger

#100

Post by papaf »

marcus_holmes wrote: post #37
throwaway894345 wrote: post #14

Earlier quoted context omitted.

> I mostly write web servers in Go, which (as the article says) are executed rarely, so init time really doesn't matter to me. Presumably a few mb of disk usage also doesn't matter then? I also write web services, but I do care about the init time precisely because I want to be deployments to take as little time as possible so that we can deploy (and rollback) many times per day with relatively simple automation. Tha…

Yeah, it's a trade-off. At the moment the longest part of my deploy is copying the new executable to the server. I'd trade a couple of seconds of init time for a smaller executable because that would result in a faster deploy. I don't use Docker to deploy, because it's just a single executable file (and a bunch of templates, though I'm looking at embedding those). One of the reasons I'm reluctant to go down the Docke…

At the moment the longest part of my deploy is copying the new executable to the server.

Do you use rsync [1]? I ask because most of my collegues don't and they take minutes to deploy what I usually do in seconds.

[1] https://man7.org/linux/man-pages/man1/rsync.1.html

Post reply on HN