Live data from Hacker News

My Go executable files are still getting larger

cockroachlabs.com

151–160 of 174 posts

Re: My Go executable files are still getting larger

#151
post #148
post #147

Earlier quoted context omitted.

What's the use case where a 200MB binary size is a problem?

The use case where said binary is shipped to GCE instances hundreds/thousands times per day, for stress testing and unit testing of cockroachdb.

If most of the bytes are indeed cruft, can't you just send binary diffs around? I imagine they should change that much between compilations? It seems to be an issue for db developers but not for users.

Re: My Go executable files are still getting larger

#152
post #103

Python is still more wildly used / popular language, but I never seen a Python container for a real project which was less than 1GB.

I'm guessing you're doing ML here? Because the core python runtime is far smaller than that.

Yeah, ML libraries are a huge offender. The Torch 1.8.1 manylinux wheel is 1.6G unpacked, with 1.1G going to CUDA support.

Re: My Go executable files are still getting larger

#153
post #125

Earlier quoted context omitted.

> Where? The argument is _precisely_ that the growth is occurring in non-code areas. But how is this important? If the thing you're optimizing for is "total Go binary size", then all that matters is the total size of binary! How bytes are organized internally is irrelevant to this metric. You should redo the analysis where you compile an old version of Cockroach DB (say v1.0.0) with Go versions 1.8 through 1.16, and…

> all that matters is the total size of binary! How bytes are organized internally is irrelevant to this metric. Not quite so if the task is to work on reducing the metric. When the size is attributed to data/code that's linked to the source code, then we know how to reduce the final file size (by removing data/code from the source code, or reducing them). When the size is non-attributed and/or non-explained (“dark”)…

You keep saying it’s unexplained as if it’s intentionally kept secret. You pretend you have no control over it, but if you reduced your own source code, you would find that the “dark” space shrunk.

The Go source code is available to you. Russ has pointed out there’s no existing tool to break down those “dark” bytes but that they do serve a purpose, but perhaps you could work on that tool instead of complaining that it’s not covered by the symbol table.

Re: My Go executable files are still getting larger

#154
post #24

Earlier quoted context omitted.

Go binary size makes it not an option for wasm.

Why not? If you use wasm for a small library need in your web app, then sure, go doesn't make sense. But if your whole app is coded in wasm like a game, then it's probably ok I guess as the app will be heavy to load regardless.

Ok, sure. If Go wasm is only useful for games... and I'm pretty sure game developers are picking C++ or Rust, then that's a severe limitation.

Re: My Go executable files are still getting larger

#155
post #62
post #55

This article is full of misinformation. Just a few representative things: - The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint, by letting the OS demand-page this critical table that is used any time a stack must be walked (in particular, during garbage collection). See https://golang.org/s/go12symtab for details. - We (the Go team) did not “recompress” pclntab in Go 1.…

As explained in another comment below, it's a good thing when all the byte usage is represented in the ELF section headers or the symbol tables. The DWARF data is currently so represented, and so was pclntab prior to 1.16. Today, the DWARF data is still present; the symbol for pclntab has an advertized size of zero, and the amount of data not represented in the symbol table and ELF section headers has grown. > If the…

You are getting ridiculous at this point. Probably some things could be done to improve the binary size, but the maintainers of Go don’t have unlimited time. Russ showed that Go is not growing binary sizes. Your article is misleading and at best poorly worded. When you add more code your binaries get bigger. Is that a surprise to you? If Go binary size is a critical concern for you, you could be helping solve the problem, but you are just complaining.

Re: My Go executable files are still getting larger

#156
post #58

Go static linking is a great happy idea for a Java guy trapped in the Classpath Dependency Hell (or C# / DLL Hell). It is a very annoying thing for a C++ programmer, which can dynamically link operating system libraries at will.

Go's static linking idea comes from plan9 C compilers, a few years before Java. We owe a lot from plan9: - Go's design, based on both C compilers and Inferno's limbo - /proc - utf-8 - 9p

/proc does not come from plan9: http://dtrace.org/blogs/eschrock/2004/06/25/a-brief-history-...

Re: My Go executable files are still getting larger

#157
post #149
post #136

Earlier quoted context omitted.

Without swap, you mean without virtual memory or without a swap partition/file? Because even without a swap partition/file, the whole executable will not block physical memory, but will page in/out as needed. And whole sections of it will never be loaded at all.

Without swap. You have to disable it for k8s

So you did mean partition/file, and you are misinformed.

Re: My Go executable files are still getting larger

#158
post #55

This article is full of misinformation. Just a few representative things: - The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint, by letting the OS demand-page this critical table that is used any time a stack must be walked (in particular, during garbage collection). See https://golang.org/s/go12symtab for details. - We (the Go team) did not “recompress” pclntab in Go 1.…

Yeah, I do not understand those numbers. I just compiled one of my Go projects with version 1.13.8 and version 1.16.3. The size was 8.3 MB (6.5 MB after stripping), and 8.0 MB (5.8 MB after stripping) respectively.

Re: My Go executable files are still getting larger

#159
post #9

> In other words, the Go team decided to make executable files larger to save up on initialization time. I mean... Im genuinely curious if this is a "we have extra engineering resources and can explore/complain about this" or "we have a client who is running cockroachdb and can't handle a 172mb binary install for a database server". Is there really someone out there who installs Cockroach (a global distributed auto-s…

[deleted]

Re: My Go executable files are still getting larger

#160
post #9

> In other words, the Go team decided to make executable files larger to save up on initialization time. I mean... Im genuinely curious if this is a "we have extra engineering resources and can explore/complain about this" or "we have a client who is running cockroachdb and can't handle a 172mb binary install for a database server". Is there really someone out there who installs Cockroach (a global distributed auto-s…

It affects the download and instantiation time for containers.

This isn't an interesting or convincing argument because it revolves around a non-satisfiable metric. No matter the size of an artifact, this line of reasoning can always be used to claim it's too big and needs to be smaller.
Post reply on HN