Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Better Go Linker

golang.org

Post reply on HN
Print view View original

51–60 of 104 posts

Re: Building a Better Go Linker

#51

Post by azylman »

benesch wrote: post #48
geodel wrote: post #46

Earlier quoted context omitted.

> If five highly-qualified engineers spent five years trying to speed up gollvm compiles without success, we'd have strong evidence that something about LLVM prohibits the fast compiles that are possible with the gc toolchain. But that's not the situation. At some point people are going to need to put the money where their mouth is. Or they can live with the software which open source developers developed on their ow…

I spent six weeks, unpaid, at the Recurse Center this year hacking on gccgo to see if I could improve the situation. I didn't get very far—it's hard! And I don't have much experience with compilers. So, truly, I did as much as my finances could reasonably support. (I did manage to write up a blog post about where specifically gccgo needs the most work [0].) I did learn enough in those six weeks to feel comfortable as…

If you're going to rely on personal experience in an argument, "in six weeks I learned enough to criticize many engineers with decades of experience" is not a strong argument. That's not even claiming to be a 10x engineer, that's claiming to be a 1000x engineer.

I don't intend this to sound harsh, apologies if it does. I strongly recommend you introspect on your own confidence and experience levels. I've mentored a lot of junior engineers with similar viewpoints and often there's a lot of hidden complexity they simply lack the experience to understand.

This isn't to say you should take everything anyone with more experience says at blind faith, simply that dismissing without understanding is counter to a growth mentality.

Re: Building a Better Go Linker

#52

Post by apta »

zeeboo wrote: post #47
benesch wrote: post #43

Earlier quoted context omitted.

> being able to throw money and people at LLVM development doesn't explain why it's still much slower than the Go compiler toolchain. Right, you need to specifically throw money and people at the problem of making LLVM faster, not just at LLVM in general. Neither Swift nor Objective-C have "fast compiles" as part of their pitch. Much of the work on LLVM goes into producing the highest quality object code possible, wh…

Maybe you should trust the judgement of the highly-qualified engineers then when they decided to go with their own toolchain, and not just assume that they’re doing it because plan 9 or some other vacuous reason.

The same people who created a language with poor design decisions where superior solutions existed in other languages they could have simply used?

Re: Building a Better Go Linker

#53

Post by hu3 »

apta wrote: post #52
zeeboo wrote: post #47

Earlier quoted context omitted.

Maybe you should trust the judgement of the highly-qualified engineers then when they decided to go with their own toolchain, and not just assume that they’re doing it because plan 9 or some other vacuous reason.

The same people who created a language with poor design decisions where superior solutions existed in other languages they could have simply used?

What you vaguely label as poor design decisions are actually trade-offs and they make sense to a plethora of highly educated engineers.

Re: Building a Better Go Linker

#54

Post by jbarham »

benesch wrote: post #43
jbarham wrote: post #32

Earlier quoted context omitted.

Given that Apple are the major sponsor of LLVM and they're not exactly cash poor, I think it's reasonable to conclude that being able to throw money and people at LLVM development doesn't explain why it's still much slower than the Go compiler toolchain. In hindsight the Go team made the right call to use their own toolchain. As enneff alludes, Ken Thompson's antipathy towards C++ is well documented: https://bryanpen…

> being able to throw money and people at LLVM development doesn't explain why it's still much slower than the Go compiler toolchain. Right, you need to specifically throw money and people at the problem of making LLVM faster, not just at LLVM in general. Neither Swift nor Objective-C have "fast compiles" as part of their pitch. Much of the work on LLVM goes into producing the highest quality object code possible, wh…

> Right, you need to specifically throw money and people at the problem of making LLVM faster

I repeat, Apple is the major sponsor of LLVM and is sitting on ~$250 billion in cash. If LLVM could be made significantly faster by simply throwing money at it, presumably this has also occurred to Apple management. I can't believe they're not doing it because it doesn't align with their marketing pitch.

Re: Building a Better Go Linker

#55

Post by tylerl »

benesch wrote: post #42
enneff wrote: post #28

Earlier quoted context omitted.

> the state of compilers would be much better off if the folks on the Go team had invested more in improving the compile and link times of LLVM or GCC It never would have happened. How do you motivate people whose principal frustration is the state of C++ to work on a large C++ codebase? Heterogeneity is a huge benefit to any ecosystem. Improving existing things is great, but building new things is also very importan…

Let me refine my point a bit. Apologies; my original post had gotten a bit long. I agree that enthusiasm is important! And indeed, for the Go creators, their particular leanings might have been such that they couldn't get excited about building an LLVM/GCC frontend, and adapting the Plan 9 toolchain is literally the only way those three could have Go gotten off the ground. As a member of the Go team, you'd certainly…

> it is several years past due for Google to start investing seriously in a Go toolchain based on a mature compiler stack.

That's a bit presumptuous, prescribing specific implementation details based on the fact that it's "several years past due" that they replace their tech stack with one you would like to see improved.

Remember there already is a mature C compiler alternative for Go: gccgo. There's also already a first-party LLVM based Go toolchain created by Google (gollvm). Whatever hypothetical benefits you might presume would emerge from this kind of synergy already exist. But the community mostly isn't interested.

Also, Google also already invests a massive amount of resource into LLVM. In fact, the principal author of LLVM and Clang works at Google. But even when he was at Apple they were already shoveling resource into the project.

Re: Building a Better Go Linker

#56

Post by seebs »

jstimpfle wrote: post #9

I don't get it. A linker's task should be straightforward. In essence, it looks up addresses from strings, whose number is bounded by the lines of code written by humans. I think that there must be a lot of incidental complexity if that task somehow becomes a bottleneck. And how can it be that a binary called "cmd/compile" has 170k symbols (that's like, global definitions, right?). Not that that's a huge number in te…

I too often don't see why a thing would not be simple and straightforward, until I've done it a few times.

Re: Building a Better Go Linker

#57

Post by jbarham »

apta wrote: post #52
zeeboo wrote: post #47

Earlier quoted context omitted.

Maybe you should trust the judgement of the highly-qualified engineers then when they decided to go with their own toolchain, and not just assume that they’re doing it because plan 9 or some other vacuous reason.

The same people who created a language with poor design decisions where superior solutions existed in other languages they could have simply used?

Imagine what Ken Thompson could have accomplished if he hadn't made all those poor design decisions!

And yet hundreds of thousands of working programmers around the world are productively using Go while still continuing to ignore the supposedly superior solutions.

Re: Building a Better Go Linker

#58

Post by hollerith »

jbarham wrote: post #54
benesch wrote: post #43

Earlier quoted context omitted.

> being able to throw money and people at LLVM development doesn't explain why it's still much slower than the Go compiler toolchain. Right, you need to specifically throw money and people at the problem of making LLVM faster, not just at LLVM in general. Neither Swift nor Objective-C have "fast compiles" as part of their pitch. Much of the work on LLVM goes into producing the highest quality object code possible, wh…

> Right, you need to specifically throw money and people at the problem of making LLVM faster I repeat, Apple is the major sponsor of LLVM and is sitting on ~$250 billion in cash. If LLVM could be made significantly faster by simply throwing money at it, presumably this has also occurred to Apple management. I can't believe they're not doing it because it doesn't align with their marketing pitch.

Apple embraced LLVM because Apple's lawyers are scared of version 3 of the GPL and consequently of GCC, and they need something to compile Objective C.

That is different from a belief that making LLVM better is worth management's attention or a slice of Apple's budget once it proved a satisfactory replacement for GCC.

Re: Building a Better Go Linker

#59

Post by insulanus »

luuio wrote: post #4
cpeterso wrote: post #2

"The original linker was also simpler than it is now and its implementation fit in one Turing award winner’s head, so there’s little abstraction or modularity. Unfortunately, as the linker grew and evolved, it retained its lack of structure, and our sole Turing award winner retired." :)

For those out of the loop, who are they talking about here?

Definitely Ken Thompson. :)

https://en.wikipedia.org/wiki/Ken_Thompson

Re: Building a Better Go Linker

#60

Post by benesch »

azylman wrote: post #51
benesch wrote: post #48

Earlier quoted context omitted.

I spent six weeks, unpaid, at the Recurse Center this year hacking on gccgo to see if I could improve the situation. I didn't get very far—it's hard! And I don't have much experience with compilers. So, truly, I did as much as my finances could reasonably support. (I did manage to write up a blog post about where specifically gccgo needs the most work [0].) I did learn enough in those six weeks to feel comfortable as…

If you're going to rely on personal experience in an argument, "in six weeks I learned enough to criticize many engineers with decades of experience" is not a strong argument. That's not even claiming to be a 10x engineer, that's claiming to be a 1000x engineer. I don't intend this to sound harsh, apologies if it does. I strongly recommend you introspect on your own confidence and experience levels. I've mentored a l…

I invoked the personal experience only as evidence that I did what (little) I could to push along the project that I think has the most long term potential. I'm not criticizing without at least trying to do what I can.

In short, the reasons gccgo produces worse code than gc are the result of fairly basic optimizations that are/were missing. For example, open-coding string slicing [0] and string equality [1] is enough to close the gap in the TimeParse benchmark in the blog post above. In fact, with those changes, gccgo actually produces better code than gc, because the full power of GCC can be unleashed on the IR, and GCC has more aggressive optimizations than gc.

> This isn't to say you should take everything anyone with more experience says at blind faith, simply that dismissing without understanding is counter to a growth mentality.

Quite honestly, I think you're more guilty of dismissing without understanding than I am. It's true that I haven't been programming for decades, but I've spent two years dealing with the Go toolchain's shortcomings on a large Go project, and six weeks specifically on a passion project to improve gccgo. Of course it's possible that I'm wrong! But I've spent a while thinking about it, and often just as relevant as experience is a fresh, outside perspective. It's easy for a team to get stuck in tunnel vision or groupthink.

[0]: https://github.com/golang/gofrontend/commit/62e3a8cc0a862b0a... [1]: https://github.com/golang/gofrontend/commit/89b442a0100286ee...

Post reply on HN