Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on the Go translation of Reposurgeon (2020)

gitlab.com

Post reply on HN
Print view View original

21–30 of 155 posts

Re: Notes on the Go translation of Reposurgeon (2020)

#21

Post by db48x »

nicoburns wrote: post #12

The article mentions that Go, OCaml, or a compiled lisp were considered for this project. I wonder why Rust wasn't on that list. It seems to cover every concern that is raised here except for keyword arguments (which are high on my list of desired Rust features too). I guess maybe they were worried about the lack of GC, but my experience has been that's it's generally quite easy to port code from dynamic languages li…

Async wasn’t in yet at the time. He did investigate Rust to a certain extent, but bounced hard. His chosen program didn’t really show off Rust’s strengths, because he started by trying to call select and write essentially the same program that he would have written in C. That’s a pretty painful way to go.

Incidentally, I helped with the port to Go, and I spent some time polishing the code, and finding and fixing performance problems. When we were helping GCC convert their SVN repository to Git (a repository with 287k commits, btw), we reduced the memory usage by 50% (from over 250GB to under 128GB), and the run time by quite a lot as well (down to just around 2 hours to read in the SVN repository and convert it to a basic Git repository).

Now that we’ve done that work, Reposurgeon spends 50–60% of its cpu time scanning the heap for garbage. There is often garbage to find, but just as often there is not. GC is useful, but for Reposurgeon it has become a bottleneck.

My preliminary work on a Rust port shows that it is around 4× faster than the Go version. I personally think that Rust is the future, but I haven’t been able to put as much effort into the port as I would like.

Re: Notes on the Go translation of Reposurgeon (2020)

#22

Post by marcus_holmes »

wpietri wrote: post #9

What a delight to read, even as somebody who has barely used Go. I really appreciate the author's self-awareness as demonstrated in things like "Expected problems that weren’t". The bits about exceptions and typing remind me of an open question I have about Go: to me it mainly looks like a language for well-understood problems. Static typing and the lack of ability to do broad, high-level exception catching seem to m…

Honestly, you don't notice it once you get into the Go mindset.

The exception handling looks onerous, and is a pain at the start. But it doesn't take long to get used to it, and then (for me anyway) it becomes second-nature. Everything returns an error, and you have to handle that error (even if only passing it up again).

Static typing is more "fun" when you're exploring new concepts, but interfaces are the key. Defining the interface at the beginning ("what do I need this thing to do?") and then implementing it with a struct/whatever, feels like the right way to do this.

Re: Notes on the Go translation of Reposurgeon (2020)

#23

Post by oefrha »

marcus_holmes wrote: post #22
wpietri wrote: post #9

What a delight to read, even as somebody who has barely used Go. I really appreciate the author's self-awareness as demonstrated in things like "Expected problems that weren’t". The bits about exceptions and typing remind me of an open question I have about Go: to me it mainly looks like a language for well-understood problems. Static typing and the lack of ability to do broad, high-level exception catching seem to m…

Honestly, you don't notice it once you get into the Go mindset. The exception handling looks onerous, and is a pain at the start. But it doesn't take long to get used to it, and then (for me anyway) it becomes second-nature. Everything returns an error, and you have to handle that error (even if only passing it up again). Static typing is more "fun" when you're exploring new concepts, but interfaces are the key. Defi…

> But it doesn't take long to get used to it

I did get used to it, but all the

  if err != nil {
    return nil, err
  }
sometimes taking up half or more of the vertical space of a function is still an eyesore to me, even after writing a quite substantial amount of Go.

Re: Notes on the Go translation of Reposurgeon (2020)

#24

Post by stouset »

db48x wrote: post #21
nicoburns wrote: post #12

The article mentions that Go, OCaml, or a compiled lisp were considered for this project. I wonder why Rust wasn't on that list. It seems to cover every concern that is raised here except for keyword arguments (which are high on my list of desired Rust features too). I guess maybe they were worried about the lack of GC, but my experience has been that's it's generally quite easy to port code from dynamic languages li…

Async wasn’t in yet at the time. He did investigate Rust to a certain extent, but bounced hard. His chosen program didn’t really show off Rust’s strengths, because he started by trying to call select and write essentially the same program that he would have written in C. That’s a pretty painful way to go. Incidentally, I helped with the port to Go, and I spent some time polishing the code, and finding and fixing perf…

Appreciate your thoughtful reply! One question. Regarding this bit:

> He did investigate Rust to a certain extent, but bounced hard. His chosen program didn’t really show off Rust’s strengths, because he started by trying to call select and write essentially the same program that he would have written in C. That’s a pretty painful way to go.

Is this stating that he was writing the Rust version similarly to how he would have written it in C because that was the most natural way to do it in Rust or simply because it was the first path he went down? In other words, was your point that Rust is a fundamentally poor fit for this problem, or was it that his lack of experience with the language took him down the wrong path?

Thanks!

Re: Notes on the Go translation of Reposurgeon (2020)

#25

Post by wpietri »

marcus_holmes wrote: post #22
wpietri wrote: post #9

What a delight to read, even as somebody who has barely used Go. I really appreciate the author's self-awareness as demonstrated in things like "Expected problems that weren’t". The bits about exceptions and typing remind me of an open question I have about Go: to me it mainly looks like a language for well-understood problems. Static typing and the lack of ability to do broad, high-level exception catching seem to m…

Honestly, you don't notice it once you get into the Go mindset. The exception handling looks onerous, and is a pain at the start. But it doesn't take long to get used to it, and then (for me anyway) it becomes second-nature. Everything returns an error, and you have to handle that error (even if only passing it up again). Static typing is more "fun" when you're exploring new concepts, but interfaces are the key. Defi…

Not noticing it is exactly my fear.

Passing an error manually up several levels when it may only be a theoretical concern is a ton of expressive duplication. If I'm in the mindset, that would feel like valuable work, even though it isn't actually making things better for users.

It reminds me of one team I dealt with years ago. They were Java experts used to doing enterprise stuff; their overlords had put them on a scrappy, startup-like thing that was intended to be open source. As was in fashion for Java at the time, they had written things in many layers, the goal of which was to provide scaling cut-points. However valuable in theory, the project ever needing to scale was uncertain. What really mattered was finding something that served user needs. But the layer-by-layer duplication increased the cost of change significantly, lowering the odds we'd find the right product.

Re: Notes on the Go translation of Reposurgeon (2020)

#26

Post by kubb »

mwcampbell wrote: post #13
nicoburns wrote: post #12

The article mentions that Go, OCaml, or a compiled lisp were considered for this project. I wonder why Rust wasn't on that list. It seems to cover every concern that is raised here except for keyword arguments (which are high on my list of desired Rust features too). I guess maybe they were worried about the lack of GC, but my experience has been that's it's generally quite easy to port code from dynamic languages li…

Here is what esr wrote about Rust in early 2017: http://esr.ibiblio.org/?p=7294 http://esr.ibiblio.org/?p=7303

It sounds that it was hard to learn and use for him, that's why he preferred Go and Python. It's a frequent complaint and the step from Python to Go is indeed easier than from Python to Rust. He also had some philosophical concerns about the crate system.

Re: Notes on the Go translation of Reposurgeon (2020)

#27

Post by db48x »

stouset wrote: post #24
db48x wrote: post #21

Earlier quoted context omitted.

Async wasn’t in yet at the time. He did investigate Rust to a certain extent, but bounced hard. His chosen program didn’t really show off Rust’s strengths, because he started by trying to call select and write essentially the same program that he would have written in C. That’s a pretty painful way to go. Incidentally, I helped with the port to Go, and I spent some time polishing the code, and finding and fixing perf…

Appreciate your thoughtful reply! One question. Regarding this bit: > He did investigate Rust to a certain extent, but bounced hard. His chosen program didn’t really show off Rust’s strengths, because he started by trying to call select and write essentially the same program that he would have written in C. That’s a pretty painful way to go. Is this stating that he was writing the Rust version similarly to how he wou…

He went down the wrong path, but at the same time Rust didn’t do anything to make async code easier to write. You just had to call libc::select and std::thread::spawn and whatnot, with all of the unsafe blocks that implies. These days there are about 47 different crates that can help you do it, all of them a lot nicer than that.

Re: Notes on the Go translation of Reposurgeon (2020)

#28

Post by kubb »

> What I really missed was generic map-function-over-slice

Me too, and mapping a map, a channel, etc. With generics, things will be easier, we can have iterators, even lazy iterators, but there will still be no type inference in lambdas for arguments and return values and no easy currying, so things will still be awkward, but better.

> Keyword arguments

You can kind of do that by passing a struct as an argument, and represent the optionalness by having a pointer in the struct. That's still awkward, you can't easily make a pointer to an integer or string literal, you need a separate helper function for each primitive type. Overall it could be nicer.

> Annoying limitations on const

I think the Go constants are good, the fact that you can perform calculations on them in compile time also. But I wish there was some way to express immutability (and non-nilability) in the language. Programs are simpler with less moving parts.

> 14KLOC -> 21KLOC

That's to be expected, Go could be nicer if it was more concise, obviously that's a hard thing to balance, because you can end up with too obscure code, I just wish that there were some small improvements from time to time.

> Absence of sum/discriminated-union types

Yeah, I refer to it as Algebraic Data Types, where you can for example have your tree type to be a Node with children or a Leaf with value. And a function can accept a Tree that can be one of these. IIRC, the Go people claim that you can do kind of something like this with interfaces, but it's not very nice to use interfaces in this way and has some drawbacks.

> Catchable exceptions require silly contortions

I don't like exceptions, they disrupt the flow of the program - any function call could "return early", so you need to be always careful and account for that possibility. In C++ this was solved with RAAI, and it was a source of so many bugs that some companies just disallow exceptions internally.

> Aesthetic doubt

Yeah, it's there, some things in Go just come out ugly.

> Absence of iterators

Yes. Hopefully generics will enable implementing them.

Re: Notes on the Go translation of Reposurgeon (2020)

#30

Post by autarch »

kubb wrote: post #28

> What I really missed was generic map-function-over-slice Me too, and mapping a map, a channel, etc. With generics, things will be easier, we can have iterators, even lazy iterators, but there will still be no type inference in lambdas for arguments and return values and no easy currying, so things will still be awkward, but better. > Keyword arguments You can kind of do that by passing a struct as an argument, and…

> You can kind of do that by passing a struct as an argument, and represent the optionalness by having a pointer in the struct. That's still awkward, you can't easily make a pointer to an integer or string literal, you need a separate helper function for each primitive type. Overall it could be nicer.

I think using a builder is a better option in Go if you have more than 2-3 arguments.

> Yeah, I refer to it as Algebraic Data Types, where you can for example have your tree type to be a Node with children or a Leaf with value. And a function can accept a Tree that can be one of these. IIRC, the Go people claim that you can do kind of something like this with interfaces, but it's not very nice to use interfaces in this way and has some drawbacks.

Except you can't really do this with interfaces. If you need to get back to the concrete implementation of the interface, there's no way to know what all possible implementations might be. This is especially true since any type can implement an interface, not just the types you create initially. So someone else could implement `Tree` and you would not know to account for that in your function that accepts the `Tree` interface.

I don't think there's any substitute for proper sum types in Go.

Post reply on HN