Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Go executable files are still getting larger

cockroachlabs.com

Post reply on HN
Print view View original

121–130 of 174 posts

Re: My Go executable files are still getting larger

#121

Post by returningfory2 »

knz42 wrote: post #115
mort96 wrote: post #110

Earlier quoted context omitted.

It's not clear that the overhead is due to Go itself producing bigger binaries over time though. If you recompiled all the different CockroachDB versions with Go 1.8 (if that was feasible), it's quite probable that the tables you would end up with would look fairly similar to the ones you're actually showing. If there is superlinear growth in binary sizes as the project grows – for example, if some part is O(n^2) in…

> tried to blame it partly on the Go compiler producing more bloated code over time Where? The argument is _precisely_ that the growth is occurring in non-code areas. > partly on a mystical "dark area" which you don't understand The _observation_ is that the growth is happening in an area of the file that's not accounted for in the symtable. That's what makes it "dark". It's not mythical: it's _there_ and you can obs…

> Where? The argument is _precisely_ that the growth is occurring in non-code areas.

But how is this important? If the thing you're optimizing for is "total Go binary size", then all that matters is the total size of binary! How bytes are organized internally is irrelevant to this metric.

You should redo the analysis where you compile an old version of Cockroach DB (say v1.0.0) with Go versions 1.8 through 1.16, and then see what the numbers say. Your current analysis, which doesn't account for growth in the code base at all, or tries to account for it by deep-diving into the internal organization of the binary, is not sound.

Re: My Go executable files are still getting larger

#122

Post by knz42 »

arp242 wrote: post #118
knz42 wrote: post #108

Earlier quoted context omitted.

> The article made the claim that 70% of space is wasted "dark bytes" This is incorrect. The claim is that the bytes are either non-accounted, or motivated by technical choices specific to Go. > What you're doing here is pretty much the same trick quacks [...] Look the article has some measurements with numbers which you can readily reproduce on your own computer, and the methodology is even described. The main claim…

> The claim is that the bytes are either non-accounted, or motivated by technical choices specific to Go. That's not what it says; with "70% of a couple hundred megabytes are copied around for no good reason" written in bold no less: > That’s right! More than two thirds of the file on disk is taken by… bits of dubious value to the software product. > > Moreover, consider that these executable files fly around as cont…

> That's not what it says [...]

That claim was retracted a while ago already on the original version; the syndicated copy on the crl web site will be updated at some point.

Re: My Go executable files are still getting larger

#123

Post by knz42 »

Scarbutt wrote: post #117
knz42 wrote: post #104

Earlier quoted context omitted.

> Go executable sizes haven't increased much in general. Russ's example was just the "gofmt" program. > Perhaps the reason you're seeing increases in Cockroach DB is because you keep writing more code for Cockroach DB? If that was the only reason, then the % overhead would remain constant-ish. But it is increasing. So there is a non-linear factor for _some_ go programs (like cockroachdb) and it's still unclear what t…

CockroachDB has always had a reputation of being slow, 10x-20x slower than the same operation being made in Postgres, with this and the issues about binary size, was a GC language like Go the right choice for CK? Would you have pick something else today if starting a new?

For being somewhat familiar with the CockroachDB project, I doubt that that claimed performance difference is linked to the programming language. It's more something about mandatory 3-way (or more) replication upon every write, and several additional layers of protection against hardware failures, network problems etc which postgres do not have.

Re: My Go executable files are still getting larger

#124

Post by vvern »

Scarbutt wrote: post #117
knz42 wrote: post #104

Earlier quoted context omitted.

> Go executable sizes haven't increased much in general. Russ's example was just the "gofmt" program. > Perhaps the reason you're seeing increases in Cockroach DB is because you keep writing more code for Cockroach DB? If that was the only reason, then the % overhead would remain constant-ish. But it is increasing. So there is a non-linear factor for _some_ go programs (like cockroachdb) and it's still unclear what t…

CockroachDB has always had a reputation of being slow, 10x-20x slower than the same operation being made in Postgres, with this and the issues about binary size, was a GC language like Go the right choice for CK? Would you have pick something else today if starting a new?

My intuition is that later versions of crdb are more like 1/3rd the efficiency of Postgres per core. GC is some of that but I don’t think it’s all that much.

Everything has trade offs. Go is not the easiest language in which to write highly efficient code. Cockroach generates some code to help here. Certainly at this point there’s pain around tracking memory usage so as to not OOM and there’s pain around just controlling scheduling priorities. But then again, had it been C++ or Rust perhaps getting to table stakes correctness would have taken so long it wouldn’t have mattered.

Some cost just comes from running distributed replication and concurrency control. That’s unavoidable. Some also comes from lack of optimization. Postgres has been around and has some very optimized things in its execution engineer.

Also, if you run Postgres in SERIALIZABLE, it actually does quite badly, largely because that isolation level was bolted on and the concurrency control isn’t optimized for it. Crdb was core-for-core competitive in serializable on some workloads last time I checked.

Re: My Go executable files are still getting larger

#125

Post by knz42 »

returningfory2 wrote: post #121
knz42 wrote: post #115

Earlier quoted context omitted.

> tried to blame it partly on the Go compiler producing more bloated code over time Where? The argument is _precisely_ that the growth is occurring in non-code areas. > partly on a mystical "dark area" which you don't understand The _observation_ is that the growth is happening in an area of the file that's not accounted for in the symtable. That's what makes it "dark". It's not mythical: it's _there_ and you can obs…

> Where? The argument is _precisely_ that the growth is occurring in non-code areas. But how is this important? If the thing you're optimizing for is "total Go binary size", then all that matters is the total size of binary! How bytes are organized internally is irrelevant to this metric. You should redo the analysis where you compile an old version of Cockroach DB (say v1.0.0) with Go versions 1.8 through 1.16, and…

> all that matters is the total size of binary! How bytes are organized internally is irrelevant to this metric.

Not quite so if the task is to work on reducing the metric.

When the size is attributed to data/code that's linked to the source code, then we know how to reduce the final file size (by removing data/code from the source code, or reducing them).

When the size is non-attributed and/or non-explained (“dark”) we are lacking a control to make the size smaller over time.

Re: My Go executable files are still getting larger

#126

Post by dvfjsdhgfv »

parhamn wrote: post #113
dvfjsdhgfv wrote: post #56

Earlier quoted context omitted.

> Is there really someone out there who installs Cockroach (a global distributed auto-sharded database) and thinks twice about 172mb of disk space? I think it is exactly this mindset that caused the current situation (that the 2/3 of the compiled binaries are useless to the users).

The alternative isn't avoiding fixing it. The alternative is a boring bug report on github with actual stakeholders discussing the best strategies and trade offs. This is only on HN's homepage because of the langauge flame wars. It's a garbage post.

I agree 100% and I sincerely hope the author did that already. Just complaining about it to vent off one's frustration will accomplish nothing.

Re: My Go executable files are still getting larger

#127

Post by arp242 »

knz42 wrote: post #122
arp242 wrote: post #118

Earlier quoted context omitted.

> The claim is that the bytes are either non-accounted, or motivated by technical choices specific to Go. That's not what it says; with "70% of a couple hundred megabytes are copied around for no good reason" written in bold no less: > That’s right! More than two thirds of the file on disk is taken by… bits of dubious value to the software product. > > Moreover, consider that these executable files fly around as cont…

> That's not what it says [...] That claim was retracted a while ago already on the original version; the syndicated copy on the crl web site will be updated at some point.

So you did make the claim, but just retracted it, in spite of you saying you never made the claim, a claim which is still in the article linked here. I am now supposed to argue against some revised article published elsewhere? This is a very vexing way to have a conversation.

This is also a trick peddlers of pseudoscience pull by the way. Honestly, you're coming off even worse now and this is reflecting pretty badly on all of CockroachDB to be honest. I don't know what your relationship with CockroachDB is exactly (if any), but it's on their website, and it's not a good look. If I was a manager there then I'd back-pedal, unpublish the entire thing, and issue an apology.

Re: My Go executable files are still getting larger

#128

Post by haberman »

rsc wrote: post #93
OskarS wrote: post #91

Earlier quoted context omitted.

I have no dog in this fight either way, I'm just very curious about the answer: if something like 30-40% in a Go executable that clocks in at more than a 100 megabytes is not taken up by either symbols, debug information or the pclntab, what exactly is in it? You mentioned "necessary metadata for garbage collection and reflection in a statically-compiled language" in a previous comment. Can you give some more details…

You can see the true size of the Go pclntab in ELF binaries using "readelf -S" and in Mac binaries using "otool -l". Its not zero. One thing that did change from Go 1.15 to Go 1.16 is that we broke up the pclntab into a few different pieces. Again, it's all in the section headers. But the pieces are not in the actual binary's symbol table anymore, because they don't need to be. And since the format is different, we w…

> OK, the pclntab is large. Why is it large? What are the specific things in it that are large?

Is it reasonably easy to attribute individual entries in pclntab to specific symbols? If so I'd love to add this capability to https://github.com/google/bloaty which already tries to do per-symbol analysis of many other sections (eh_frame, rela.dyn, etc).

Re: My Go executable files are still getting larger

#129

Post by knz42 »

arp242 wrote: post #127
knz42 wrote: post #122

Earlier quoted context omitted.

> That's not what it says [...] That claim was retracted a while ago already on the original version; the syndicated copy on the crl web site will be updated at some point.

So you did make the claim, but just retracted it, in spite of you saying you never made the claim, a claim which is still in the article linked here . I am now supposed to argue against some revised article published elsewhere? This is a very vexing way to have a conversation. This is also a trick peddlers of pseudoscience pull by the way. Honestly, you're coming off even worse now and this is reflecting pretty badly…

> a claim which is still in the article linked here. I am now supposed to argue against some revised article published elsewhere?

The article linked in this thread is a syndicated copy of an original article published elsewhere, as clearly stated by the attribution section at the bottom. It's reasonable to expect that changes to the original will only be updated in the copy with a delay.

Re: My Go executable files are still getting larger

#130

Post by masklinn »

kissgyorgy wrote: post #48

Python is still more wildly used / popular language, but I never seen a Python container for a real project which was less than 1GB.

Alpine containers for P3 go down to about 60MB for the baseline. Everything else is what you added in.

CPython doesn't generate anything, and the binary installers (including docs and all) are about 30MB. Hell, the 64b embeddable package for windows (https://docs.python.org/3/using/windows.html#the-embeddable-...) is 16MB uncompressed.

Post reply on HN