Live data from Hacker News

Experience report on a large Python-to-Go translation

gitlab.com

11–20 of 99 posts

Re: Experience report on a large Python-to-Go translation

#11

> The problem directed the choice of Go, not the other way around. I seriously considered OCaml or a compiled Lisp as alternatives. I concluded that in either case the semantic gap between Python and the target language was so large that translation would be impractical. Only Go offered me any practical hope. I wish they expanded more on this point. Do they mean that rewriting in, say, Lisp would be longer because it…

I'd be interested in reading more about that, too.

I'm familiar with both Python and Common Lisp, and to me they've always felt very similar.

Re: Experience report on a large Python-to-Go translation

#12
Pretty interesting. It is scary to make your "learn a new language" task to port 14,000 lines of code, but with that in mind, this all seems to have gone well. Some random thoughts:

> I had to write my own set-of-int and set-of-string classes

map[int]struct{}, map[string]struct{}

   ints[42] = struct{}{} // insert
   delete(ints, 42) // delete 
   for i := range ints { ... } // iterate
   if _, ok := ints[42]; ok { ... } // exists?
> Catchable exceptions require silly contortions

I am not sure why go has panic/recover, but it's not something to use. panic means "programming error", recover means "well, the code is broken but I'm just a humble generic web framework and I guess maybe the next request won't be broken, so let's keep running". It is absolutely not for things like "timeout waiting for webserver" or "no rows in the database" as other languages use exceptions for. For those, you return an error and either wrap it with fmt.Errorf("waiting for webserver: %w", err) or check it with errors.Is and move on. Yup, you have to remember to do that or your program will run weirdly. It's just how it is. There is not something better that maybe with some experimentation you will figure out. You have to just do the tedious, boring, and simple thing.

I have used recover exactly once in my career. I wrote a program that accepted mini-programs (more like rules) via a config file that could be reloaded at runtime. We tried to prove them safe, but recover was there so that we could disable the faulty rule and keep running in the event some sort of null pointer snuck in. (I don't think one ever did!)

> Pass-by-reference vs. pass-by-value

I feel like the author wants []*Type instead of []Type here.

> Absence of sum/discriminated-union types

True. Depending on what your goals are, there are many possibilities:

   type IntOrString struct { i int; s string; iok, sok bool }
   func (i IntOrString) String() (string, error) { if i.sok { return i.s, nil } else { return "", errors.New("is not a string") }}
   func NewInt(x int) IntOrString { return IntOrString{i: x, iok: true} }
   ...

This uses more memory than interface{}, but it's also very clear what you intend for this thing to be.

I will also point out that switch can bind the value for you:

   switch x := foo.(type) {
   case int:
      return x + 1
   case string:
      i, err := strconv.Atoi(x)
      return i + 1
   }
And that you need not name types merely to call methods on them:

   if x, ok := foo.(interface { IntValue() int }); ok { return x.IntValue() }

You can also go crazy like the proto compiler does for implementations of "oneof" and have infinite flexibility. It is not very ergonomic, but it is reliable.

> Keyword arguments

   type Point struct { X, Y float64 }

   func EuclideanDistance(a, b Point) float64 { ... }

   EuclideanDistance(Point{X: 1, Y: 2}, Point{3, 4})
> No map over slices

This one is like returning errors. You will press a lot of buttons on your keyboard. It is how it is.

I personally hate typing the average "simple" for loop:

   func fooToInterface(foos []foo) []interface{} {
       var result []interface{}
       for _, f := range foos {
           result = append(result, f)
       }
       return result
   }
But it's also not that hard. I used to be a Python readability reviewer at Google. I always had the hardest time reading people's very aggressive list comprehensions. It was like they HAD to get their entire program into one line of code, or people wouldn't think they were smart. The result was that the line became a black box; nobody would read it, it was just assumed to work.

I really like seeing the word "for" twice when you're iterating over two things.

Re: Experience report on a large Python-to-Go translation

#14
Here's another experience report: I ported a small 1 KLOC PHP project to Go this week (in some spare time between large C++ compile times). The primary goal was to reduce the number of supported languages we use.

The port happened in the mechanical line-by-line way, copying each PHP file to a *.go and fixing all the syntax. The project was small enough that automation wasn't interesting.

I agree with the "1/3 time spent debugging the result". Another complicated facet was the lack of insertion-order preserving maps, that PHP applications end up relying on heavily. The error/exception impedance mismatch was not a problem in practice at all.

According to cloc, the original PHP project (excluding vendor) is 1.0 KLOC, the resulting Go application is 1.2 KLOC. I imagined Go would have been more verbose than this, but actually most lines remained 1:1 conversions, and the Go standard library happened to cover a lot of small utility functions that had to be separately written in PHP (e.g. for string suffix matching).

Another interesting point is the number of comment lines in cloc appeared to drop dramatically, since real type annotations are much less verbose than PHPDoc.

Re: Experience report on a large Python-to-Go translation

#15

Interesting. I'd have expected more than a 50% code expansion going to Go, maybe even 3x or 5x. Similarly, he's using 40x speedup as a rule of thumb. I usually think of Python as 20x slower than C. Personally I'd be loathe to convert a working Python system to Go, but it sounds like he had good reasons. I do wonder a bit whether divide-and-conquer or a C extension might not have worked instead.

"Interesting. I'd have expected more than a 50% code expansion going to Go, maybe even 3x or 5x."

This has been my extensive experience as well. I wouldn't be able to use Go if it was that much more verbose than Python. It certainly isn't as succinct as Python by any means, but it's not the night-and-day nightmare a lot of HN posters seem to think it is... provided you actually learn the language.

In fact, one of the questions that I've found coming up in my head is... if you take the huge, huge pile of features that Python or other languages bring to the table that Go doesn't, and all their corresponding disadvantages in terms of having to learn them all, and how they all interact, etc.... and that's all you get in real, production code... is it really worth it? Because let's not mince words... it's a long list of features, all of which superficially seem awesome. And I can craft one-liners that would be a dozen lines or more of Go... but usually those one-liners have a lot of single-letter variables in them to focus on all the awesome syntax and features. When I get into real code with real variable names, the advantage fades fast.

I find this to be food for thought. I still haven't fully integrate it into my worldview. But I can definitely say I feel like the cost/benefit matrix I had in my head even three years ago has shifted a lot. Perhaps it would be fair to say I haven't necessarily lowered my estimate of the benefits of all the fancy features, but my estimates of their costs have gone significantly up. A lot of them are really cheap in the moment you're writing them down for the first time and using them, but carry hidden long-term costs that I feel like younger me was not accounting for properly, especially if you are not the only developer.

Re: Experience report on a large Python-to-Go translation

#16

I do not get why people do these total rewrites, especially for working Python systems. Why throw out the baby with the bathwater ? Python is fundamentally a composing toolkit. Rewrite the slow bits in C++/Rust/Go and wrap it. That's how all major Python components like Numpy, Scipy, Tensorflow, PyTorch etc. does it. And that's a major reason why Python dominates today. Align with the core strengths of Python's philo…

I think it is fine when people are doing rewrites on their time and dime. It is certainly better than situation where language fans just make drive-by comments to authors on github etc to rewrite stuff in xyz-lang because it is so much better.

Re: Experience report on a large Python-to-Go translation

#17
post #15

Interesting. I'd have expected more than a 50% code expansion going to Go, maybe even 3x or 5x. Similarly, he's using 40x speedup as a rule of thumb. I usually think of Python as 20x slower than C. Personally I'd be loathe to convert a working Python system to Go, but it sounds like he had good reasons. I do wonder a bit whether divide-and-conquer or a C extension might not have worked instead.

"Interesting. I'd have expected more than a 50% code expansion going to Go, maybe even 3x or 5x." This has been my extensive experience as well. I wouldn't be able to use Go if it was that much more verbose than Python. It certainly isn't as succinct as Python by any means, but it's not the night-and-day nightmare a lot of HN posters seem to think it is... provided you actually learn the language. In fact, one of the…

I've been saying for a long time that every abstraction has a cost. Sometimes that cost is hard to quantify or externalized, but our inability to quantify it doesn't mean the cost doesn't exist.

Abstractions have to at least pay for themselves many times over to be worth the extra cognitive burden and we need to get better at measuring these trade-offs. The success of languages like Go hint that there's more costs than we have traditionally been willing to acknowledge.

Re: Experience report on a large Python-to-Go translation

#18
post #14

Here's another experience report: I ported a small 1 KLOC PHP project to Go this week (in some spare time between large C++ compile times). The primary goal was to reduce the number of supported languages we use. The port happened in the mechanical line-by-line way, copying each PHP file to a *.go and fixing all the syntax. The project was small enough that automation wasn't interesting. I agree with the "1/3 time sp…

Sounds about right! Here's another: https://benhoyt.com/writings/learning-go/ ... I ported a medium-sized web backend in Python to Go.

"Due to Go’s static typing and because I was using fewer libraries, I expected that the code would end up being more than twice as many lines of code. However, it was only 1900 lines of Go (about 50% more than the 1300 lines of Python)."

"The porting effort was very smooth, and a lot of the business logic was almost mechanical, line-for-line porting of the original Python. I was surprised how well many Python concepts translate to Go, right down to the things[:20] slice notation."

Re: Experience report on a large Python-to-Go translation

#19

I'd like to compliment the author on the quality of this post. It's very well written, data/example driven, fair, and educational. Overall, a joy to read. Thank you!

Came here to say this. Engaging, thoughtful, truly well written... This excellent piece is a real contribution to the body of knowledge of language design, and I'm grateful I got to read it.

I freakin' love python, but also it feels like the community is top notch among languages.

Re: Experience report on a large Python-to-Go translation

#20

I do not get why people do these total rewrites, especially for working Python systems. Why throw out the baby with the bathwater ? Python is fundamentally a composing toolkit. Rewrite the slow bits in C++/Rust/Go and wrap it. That's how all major Python components like Numpy, Scipy, Tensorflow, PyTorch etc. does it. And that's a major reason why Python dominates today. Align with the core strengths of Python's philo…

"Rewrite the slow bits" only works when you have a solid hot loop that you can rewrite. But what if most of your program is the hot loop?

This particular problem is not particularly numerically oriented. There's nothing he could feed into a external library to speed it up. It is highly algorithmic code, pretty much the worst case for Python.

Post reply on HN