Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Go executable files are still getting larger

cockroachlabs.com

Post reply on HN
Print view View original

131–140 of 174 posts

Re: My Go executable files are still getting larger

#131

Post by hatuio »

rsc wrote: post #55

This article is full of misinformation. Just a few representative things: - The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint, by letting the OS demand-page this critical table that is used any time a stack must be walked (in particular, during garbage collection). See https://golang.org/s/go12symtab for details. - We (the Go team) did not “recompress” pclntab in Go 1.…

In env without swap, the binary size should/might block relative amount of ram.

Might it be possible to stream binaries or to detect junks which could be unloaded like an json parser which is only needed when reading json

Re: My Go executable files are still getting larger

#132

Post by mort96 »

knz42 wrote: post #115
mort96 wrote: post #110

Earlier quoted context omitted.

It's not clear that the overhead is due to Go itself producing bigger binaries over time though. If you recompiled all the different CockroachDB versions with Go 1.8 (if that was feasible), it's quite probable that the tables you would end up with would look fairly similar to the ones you're actually showing. If there is superlinear growth in binary sizes as the project grows – for example, if some part is O(n^2) in…

> tried to blame it partly on the Go compiler producing more bloated code over time Where? The argument is _precisely_ that the growth is occurring in non-code areas. > partly on a mystical "dark area" which you don't understand The _observation_ is that the growth is happening in an area of the file that's not accounted for in the symtable. That's what makes it "dark". It's not mythical: it's _there_ and you can obs…

Hmm, I think I understand where we have a misunderstanding. I - and presumably many others - interpreted the article to make the claim that newer Go versions are producing more bloated Go executables. There are multiple parts of the article which can be read that way. But if you're not doing that, and you're just trying to investigate why the CockroachDB binary is getting bigger over time, then that's a different matter.

I'm not going to respond point by point because those points are kind of moot if my accusations were based on an incorrect reading.

Re: My Go executable files are still getting larger

#133

Post by hobbified »

ngrilly wrote: post #89
hobbified wrote: post #81

Earlier quoted context omitted.

It's a mystery, not a lynch mob. Everyone reading is interested in knowing "huh, what is this stuff then?"

Russ is totally right. Pretending the linker is embedding “random data” is just trolling.

I mean on everyone else's part.

Re: My Go executable files are still getting larger

#134

Post by jerf »

knz42 wrote: post #62
rsc wrote: post #55

This article is full of misinformation. Just a few representative things: - The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint, by letting the OS demand-page this critical table that is used any time a stack must be walked (in particular, during garbage collection). See https://golang.org/s/go12symtab for details. - We (the Go team) did not “recompress” pclntab in Go 1.…

As explained in another comment below, it's a good thing when all the byte usage is represented in the ELF section headers or the symbol tables. The DWARF data is currently so represented, and so was pclntab prior to 1.16. Today, the DWARF data is still present; the symbol for pclntab has an advertized size of zero, and the amount of data not represented in the symbol table and ELF section headers has grown. > If the…

Go has a complete accounting of what all the bytes are. You can read them for yourself in the compiler, as it is open source, and trace through where every byte comes from, if you like. It isn't exactly in a broken down table precisely to the specs of what knz42 wants, but the info is all there. There's nothing hidden here.

Oh, you don't want to do that? That's not a surprise. Neither do I. Definitely not on my agenda today.

But I'm not making wild accusations. If you're going to, you probably shouldn't be surprised when we're not exactly impressed.

The compiler is all open, and as such things go, fairly easy to read. "As such things go" is definitely doing some heavy lifting there, but, still, for a compiler of a major language, it's a relatively easy read.

The vague intimation that there's something hidden and sneaky going on is hard to give much credence to.

Re: My Go executable files are still getting larger

#135

Post by arp242 »

knz42 wrote: post #129
arp242 wrote: post #127

Earlier quoted context omitted.

So you did make the claim, but just retracted it, in spite of you saying you never made the claim, a claim which is still in the article linked here . I am now supposed to argue against some revised article published elsewhere? This is a very vexing way to have a conversation. This is also a trick peddlers of pseudoscience pull by the way. Honestly, you're coming off even worse now and this is reflecting pretty badly…

> a claim which is still in the article linked here. I am now supposed to argue against some revised article published elsewhere? The article linked in this thread is a syndicated copy of an original article published elsewhere, as clearly stated by the attribution section at the bottom. It's reasonable to expect that changes to the original will only be updated in the copy with a delay.

This is nowhere near reasonable. You said that "the article made the claim that 70% of space is wasted dark bytes" was "incorrect" with no further details. Only when pressed and provided the quote where you literally said exactly that did you start talking about some retraction.

But whatever, this is pointless. Russ was right and it's hard to take any of this in good faith. It feels like you're going to great lengths to avoid saying "oops, I was wrong".

Re: My Go executable files are still getting larger

#136

Post by remram »

hatuio wrote: post #131
rsc wrote: post #55

This article is full of misinformation. Just a few representative things: - The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint, by letting the OS demand-page this critical table that is used any time a stack must be walked (in particular, during garbage collection). See https://golang.org/s/go12symtab for details. - We (the Go team) did not “recompress” pclntab in Go 1.…

In env without swap, the binary size should/might block relative amount of ram. Might it be possible to stream binaries or to detect junks which could be unloaded like an json parser which is only needed when reading json

Without swap, you mean without virtual memory or without a swap partition/file?

Because even without a swap partition/file, the whole executable will not block physical memory, but will page in/out as needed. And whole sections of it will never be loaded at all.

Re: My Go executable files are still getting larger

#137

Post by gray_-_wolf »

wlll wrote: post #90
gray_-_wolf wrote: post #50

Earlier quoted context omitted.

encoding/json sucks anyway, so using something 3rd party would likely be better option anyway

How so?

I've run into few issues:

1. Performance: lot of reflection and allocations, so not the fastest thing ever.

2. Case insensitivity: https://play.golang.org/p/LtwChO_tp0 this can be pretty unpleasant when it bites you

3. Unicode: It replaces invalid utf8 with unicode questions marks (dunno how exactly is the replacement character called) but does not provide any API to detect that it happened. So that can lead to silent corruption of the data.

Re: My Go executable files are still getting larger

#138

Post by anthk »

synergy20 wrote: post #70
skrebbel wrote: post #39

Earlier quoted context omitted.

You mean a C++ programmer who does not care whether their program works on anybody else's computer?

in that case you can happily static-link c++ programs, which probably will be smaller than go binary.

happily and "statically link c++" don't match together well.

Re: My Go executable files are still getting larger

#139

Post by Footkerchief »

knz42 wrote: post #63
rsc wrote: post #61

Earlier quoted context omitted.

An easily obtained apples-to-apples¹ table: $ for i in $(seq 3 16); do curl -sLo go1.$i.tgz https://golang.org/dl/go1.$i.linux-amd64.tar.gz tar xzf go1.$i.tgz go/bin/gofmt size=$(ls -l go/bin/gofmt | awk '{print $5}') strip go/bin/gofmt size2=$(ls -l go/bin/gofmt | awk '{print $5}') echo go1.$i $size $size2 done go1.3 3496520 2528664 go1.4² 14398336 13139184 go1.5 3937888 2765696 go1.6 3894568 2725376 go1.7 3036195 1…

Strip removes the symbol tables and DWARF information. But still, the sum of all the bytes advertised in symbol tables for the non-DWARF data does not sum up to the stripped size. What's the remainder about? I am reminded of how early versions of MSWord were embedding pages of heap space in save files that were not relevant to the document being saved, just because it made the saving algorithm simpler. For all we kno…

> For all we know, the go linker could be embedding random data

To all the people reading this as a literal accusation instead of hyperbole: plase consider that this reading is only possible under the same bad faith that you're attributing to knz42.

Re: My Go executable files are still getting larger

#140

Post by knz42 »

arp242 wrote: post #135
knz42 wrote: post #129

Earlier quoted context omitted.

> a claim which is still in the article linked here. I am now supposed to argue against some revised article published elsewhere? The article linked in this thread is a syndicated copy of an original article published elsewhere, as clearly stated by the attribution section at the bottom. It's reasonable to expect that changes to the original will only be updated in the copy with a delay.

This is nowhere near reasonable. You said that "the article made the claim that 70% of space is wasted dark bytes" was "incorrect" with no further details. Only when pressed and provided the quote where you literally said exactly that did you start talking about some retraction. But whatever, this is pointless. Russ was right and it's hard to take any of this in good faith. It feels like you're going to great lengths…

> You said that " the article made the claim that 70% of space is wasted dark bytes" was "incorrect" with no further details

I wrote this because there was no mention of "waste" anywhere in OP.

Post reply on HN