Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experience report on a large Python-to-Go translation

gitlab.com

Post reply on HN
Print view View original

51–60 of 99 posts

Re: Experience report on a large Python-to-Go translation

#51

Post by gameswithgo »

DougBTX wrote: post #46

There would probably be a much longer list of issues if ESR had converted to Rust instead, but the syntax for error returns is quite interesting. Rust and Go both opt not to have exceptions, instead they use error return values. The original Python code using exceptions was: sink = transform3(transform2(transform1(source))) Making that use error return values looks quite verbose in Go, but Rust has syntax specificall…

> if ESR had converted to Rust instead,

Rust would have solved three of his complaints with Go: Sum types, iterators, and generics.

Borrow checking can get especially tricky with graph algorithms though, so perhaps that would have been a new issue.

Re: Experience report on a large Python-to-Go translation

#52

Post by peteretep »

downerending wrote: post #2

Interesting. I'd have expected more than a 50% code expansion going to Go, maybe even 3x or 5x. Similarly, he's using 40x speedup as a rule of thumb. I usually think of Python as 20x slower than C. Personally I'd be loathe to convert a working Python system to Go, but it sounds like he had good reasons. I do wonder a bit whether divide-and-conquer or a C extension might not have worked instead.

> it sounds like he had good reasons

Meh.

> Subversion-to-git conversion of the Gnu Compiler Collection history, at over 280K commits (over 1.6M individual change actions), was the straw that broke the camel’s back. Using PyPy with every optimization on semi-custom hardware tuned for this job still yielded test conversion times of over 9 hours, which is death on the recipe-debugging cycle.

Just how often does one need to convert the Gnu Compiler Collection from Subversion to Git in under 9 hours?

Re: Experience report on a large Python-to-Go translation

#53

Post by gameswithgo »

downerending wrote: post #2

Interesting. I'd have expected more than a 50% code expansion going to Go, maybe even 3x or 5x. Similarly, he's using 40x speedup as a rule of thumb. I usually think of Python as 20x slower than C. Personally I'd be loathe to convert a working Python system to Go, but it sounds like he had good reasons. I do wonder a bit whether divide-and-conquer or a C extension might not have worked instead.

When you have a system that is at big scale, even a 2x speedup can relieve a huge amount of problems.

I think that Go, which is very inexpressive for modern languages, only expanded the code size that much, should make people consider whether we should ever be using interpreted, dynamic typed languages. Perhaps various JITted/compiled and statically typed languages can offer the same productivity on medium to large sized projects while just being a lot faster on top of it.

People who love Python may especially enjoy Nim, or F#

Re: Experience report on a large Python-to-Go translation

#54

Post by dpratt71 »

rob74 wrote: post #49
mappu wrote: post #14

Here's another experience report: I ported a small 1 KLOC PHP project to Go this week (in some spare time between large C++ compile times). The primary goal was to reduce the number of supported languages we use. The port happened in the mechanical line-by-line way, copying each PHP file to a *.go and fixing all the syntax. The project was small enough that automation wasn't interesting. I agree with the "1/3 time sp…

Small nitpick: in a statically typed language, you don't have "type annotations", since an annotation is usually optional. If you're talking about a function declaration, I think it's called a parameter declaration?

Small nitpick: some statically typed programming languages have optional type annotations.

Re: Experience report on a large Python-to-Go translation

#55

Post by rauhl »

Insanity wrote: post #50

The missing 'keyword arguments' could have been replaced with a struct passed to a function, no? Unless I'm missing something from Python, in Go you could replace this type of function: func f(x int, y int, c string) with something like this: type funcOptions struct { x, y int c string } func f(o funcOptions) {} f(funcOptions{x:3, y:-1, c: "hello"}) So the readability hit would have been more 'minimal.

The problem there is that you then have a struct populating the namespace, which means it’s an entity to track, something to think about, something which populates autocomplete buffers. That’s not necessarily terrible, but it does impose complexity that simple keyword arguments do not.

Re: Experience report on a large Python-to-Go translation

#56

Post by rauhl »

Izkata wrote: post #31
AnimalMuppet wrote: post #23

Earlier quoted context omitted.

I wish he had been a bit more explicit there, TBH. What's better about Go strings?

Skimming over a post from a quick search [0], it looks like Go strictly uses byte strings and manipulates them with various keywords and functions. No encoding/decoding between byte strings (and picking an encoding) and unicode strings like in python. [0] https://blog.golang.org/strings

I think that’s not really the right way to put it. Rather, go offers UTF-8 strings and byte slices, with a simple typecast in either way, with various keywords and functions to DTRT to each. One still must worry about encoding & decoding if one doesn’t want UTF-8, and one must worry about invalid UTF-8 in a byte slice when casting, but in general Go does what one would expect with a minimum of fuss.

Re: Experience report on a large Python-to-Go translation

#57

Post by room271 »

Insanity wrote: post #50

The missing 'keyword arguments' could have been replaced with a struct passed to a function, no? Unless I'm missing something from Python, in Go you could replace this type of function: func f(x int, y int, c string) with something like this: type funcOptions struct { x, y int c string } func f(o funcOptions) {} f(funcOptions{x:3, y:-1, c: "hello"}) So the readability hit would have been more 'minimal.

Yes, though you also have to consider default values as a possible source of error.

Re: Experience report on a large Python-to-Go translation

#58

Post by Insanity »

room271 wrote: post #57
Insanity wrote: post #50

The missing 'keyword arguments' could have been replaced with a struct passed to a function, no? Unless I'm missing something from Python, in Go you could replace this type of function: func f(x int, y int, c string) with something like this: type funcOptions struct { x, y int c string } func f(o funcOptions) {} f(funcOptions{x:3, y:-1, c: "hello"}) So the readability hit would have been more 'minimal.

Yes, though you also have to consider default values as a possible source of error.

True, but you could provide default values as well if you use something like this: https://medium.com/@meeusdylan/go-reduce-function-parameters...

You'd need extra logic to know a value has not been set at all though. At which point the complexity might not outweigh simply..not using them.

Re: Experience report on a large Python-to-Go translation

#59

Post by xvilka »

gameswithgo wrote: post #51
DougBTX wrote: post #46

There would probably be a much longer list of issues if ESR had converted to Rust instead, but the syntax for error returns is quite interesting. Rust and Go both opt not to have exceptions, instead they use error return values. The original Python code using exceptions was: sink = transform3(transform2(transform1(source))) Making that use error return values looks quite verbose in Go, but Rust has syntax specificall…

> if ESR had converted to Rust instead, Rust would have solved three of his complaints with Go: Sum types, iterators, and generics. Borrow checking can get especially tricky with graph algorithms though, so perhaps that would have been a new issue.

For graph algorithms there is a good Petgraph[1][2] crate, so many common operations can be done in no time.

[1] https://crates.io/crates/petgraph

[2] https://github.com/petgraph/petgraph

Re: Experience report on a large Python-to-Go translation

#60

Post by zbentley »

jrockway wrote: post #12

Pretty interesting. It is scary to make your "learn a new language" task to port 14,000 lines of code, but with that in mind, this all seems to have gone well. Some random thoughts: > I had to write my own set-of-int and set-of-string classes map[int]struct{}, map[string]struct{} ints[42] = struct{}{} // insert delete(ints, 42) // delete for i := range ints { ... } // iterate if _, ok := ints[42]; ok { ... } // exist…

I feel about list comprehensions the way I feel about regular expressions. Below a certain point of complexity, both are vastly superior ways of expressing what's going on. Above that point, comprehensibility drops off fast, and they immediately become inferior tools.

For example, something like:

    [some_func(y) for x, y in some_dict.items() if some_condition(x)]
...is, at least in my mind, eminently more readable than the imperative equivalent, and less PEBCAK-risky (i.e. what if you have to do two similar iterations and forget to use a different accumulator between the two?).

However, I totally agree with you re: "don't get too clever". Pretty much the instant you have nested comprehensions, or try to get clever iterating over multiple data structures in a single expression, it immediately becomes much worse than the imperative form, and you should feel a little bashful and bust out some loops, functions, and accumulators.

Same is true for regex. Something like:

    ($match) =~ qr/\A(?:foo|bar)[.]com (\d+)-baz$/
...while it does require you to know regex, is way simpler and more robust than writing the equivalent stack of many string slicing conditions, and less error prone. However, once the "too clever" rubicon is crossed (subjective, but my personal rule of thumb is: more than 50chrs of non-literals or more than one lookaround expression), like list comprehensions, it rapidly becomes much harder to understand than the equivalent long, explicit, string-slicing form.

As with many things, I think that skill in these areas is a matter of knowing when to stop.

Post reply on HN