Experience report on a large Python-to-Go translation • Hacker News Board

Re: Experience report on a large Python-to-Go translation

#81

Post by downerending » Wed, Feb 12, 2020, 5:39 PM UTC

jerf wrote: post #72

stubish wrote: post #30

Earlier quoted context omitted.

You wouldn't take the huge pile of features in an all or nothing. You get to pick and choose. I think the authors wish list is similar to most people who are experienced with more expressive languages. I don't see calls for Python f-strings or async syntax to end up in Go. But something like a list-comprehension syntax I think would be really popular (based on what I saw when it was introduced to Python, originally '…

"Interesting how Python is now considered a huge pile of features" I've been tracking Python since 1.5.2 was common and 2.0 was just coming out. For me, it's not been a problem because the new features were added incrementally. But I've seen new programmers try to come at it in 2019 and 2020, swallowing what I got spread out in ~15 major releases over ~20 years all in one big chunk, and learning even the core languag…

That's a good point. One of the original appeals of Python is that it started as a pedagogical language, and it shows. Even Python2 today still has most of that feel.

Python3 on the other hand, just isn't. You're pretty much required to constantly deal with Unicode and its related issues, and this is a burden for beginners. In Python2, you can kick that can down the road almost indefinitely.

Re: Experience report on a large Python-to-Go translation

#82

Post by downerending » Wed, Feb 12, 2020, 5:41 PM UTC

nimmer wrote: post #76

gameswithgo wrote: post #53

Earlier quoted context omitted.

When you have a system that is at big scale, even a 2x speedup can relieve a huge amount of problems. I think that Go, which is very inexpressive for modern languages, only expanded the code size that much, should make people consider whether we should ever be using interpreted, dynamic typed languages. Perhaps various JITted/compiled and statically typed languages can offer the same productivity on medium to large s…

Nim is the closest language to Python. Translating to Python is a breeze and you get speeds comparable with pure C.

You seem like a Nimmer. What would you describe as the biggest downsides of Nim wrt Python? (aside from the fact that it's currently a niche language.)

Re: Experience report on a large Python-to-Go translation

#83

Post by rrrrreee » Wed, Feb 12, 2020, 5:46 PM UTC

why not be a real man and then translate back to python?

Re: Experience report on a large Python-to-Go translation

#84

Post by [deleted] » Wed, Feb 12, 2020, 6:24 PM UTC

steveklabnik wrote: post #73

DougBTX wrote: post #46

There would probably be a much longer list of issues if ESR had converted to Rust instead, but the syntax for error returns is quite interesting. Rust and Go both opt not to have exceptions, instead they use error return values. The original Python code using exceptions was: sink = transform3(transform2(transform1(source))) Making that use error return values looks quite verbose in Go, but Rust has syntax specificall…

ESR tried Rust a while back, and decided to go with Go in general.

[deleted]

Re: Experience report on a large Python-to-Go translation

#85

Post by ASalazarMX » Wed, Feb 12, 2020, 6:54 PM UTC

DougBTX wrote: post #46

There would probably be a much longer list of issues if ESR had converted to Rust instead, but the syntax for error returns is quite interesting. Rust and Go both opt not to have exceptions, instead they use error return values. The original Python code using exceptions was: sink = transform3(transform2(transform1(source))) Making that use error return values looks quite verbose in Go, but Rust has syntax specificall…

the final ")?)?)?" is kinda unsettling, but I can't explain why.

Re: Experience report on a large Python-to-Go translation

#86

Post by mappu » Wed, Feb 12, 2020, 7:03 PM UTC

rob74 wrote: post #49

mappu wrote: post #14

Here's another experience report: I ported a small 1 KLOC PHP project to Go this week (in some spare time between large C++ compile times). The primary goal was to reduce the number of supported languages we use. The port happened in the mechanical line-by-line way, copying each PHP file to a *.go and fixing all the syntax. The project was small enough that automation wasn't interesting. I agree with the "1/3 time sp…

Small nitpick: in a statically typed language, you don't have "type annotations", since an annotation is usually optional. If you're talking about a function declaration, I think it's called a parameter declaration?

You're right, that is the word I was looking for.

Re: Experience report on a large Python-to-Go translation

#87

Post by luord » Wed, Feb 12, 2020, 8:41 PM UTC

I liked this summation because the migration happened for a truly valid reason: Python really was a bottleneck. Not that I expected ESR to succumb to hype driven development, but it's nice to see for sure.

On the article itself: I just knew that error handling would have the biggest write-up, even when the one writing was someone like ESR. Gods, the error handling in Go is odious.

Now my obligatory opinion: If only [insert language here, Go in my current job]'s promise of producing more maintainable code was true; the reality is that it's just the same nigh unmaintainable hell I've found in nearly every other project I've worked on. At least Python is nice to read, even (mostly) when awfully written. Oh, how I miss it.

Re: Experience report on a large Python-to-Go translation

#88

Post by transfire » Wed, Feb 12, 2020, 8:47 PM UTC

transfire wrote: post #45

If you want a real surprise, try a rewrite in Elixir.

Hmm down votes... Too bad b/c I'm serious. I ported a corpus generator some years ago from Ruby to both Go and Elixir. To my surprise it was easier to port to Go, but the Elixir version ran much faster.

Re: Experience report on a large Python-to-Go translation

#89

Post by andrewzah » Wed, Feb 12, 2020, 9:03 PM UTC

devmop wrote: post #48

DougBTX wrote: post #46

There would probably be a much longer list of issues if ESR had converted to Rust instead, but the syntax for error returns is quite interesting. Rust and Go both opt not to have exceptions, instead they use error return values. The original Python code using exceptions was: sink = transform3(transform2(transform1(source))) Making that use error return values looks quite verbose in Go, but Rust has syntax specificall…

It's ugly either way but for clarity you can move the error checks into the transformation leaving the call point clean. i.e transform1 returns (result, error) and transform2 accepts (result, error) and short-circuits if err is not nil. It allows the expressive and succinct description of a transformation list but with a bunch of messiness hidden.

This isn't ideal, because it forces each successive function that takes a Result type to add handling code in the case of an error.

fn(fn(fn()?)?)? is a bit gnarly but better than duplicate code like that, imo.

Re: Experience report on a large Python-to-Go translation

#90

Post by pdonis » Wed, Feb 12, 2020, 9:05 PM UTC

peteretep wrote: post #52

downerending wrote: post #2

Interesting. I'd have expected more than a 50% code expansion going to Go, maybe even 3x or 5x. Similarly, he's using 40x speedup as a rule of thumb. I usually think of Python as 20x slower than C. Personally I'd be loathe to convert a working Python system to Go, but it sounds like he had good reasons. I do wonder a bit whether divide-and-conquer or a C extension might not have worked instead.

> it sounds like he had good reasons Meh. > Subversion-to-git conversion of the Gnu Compiler Collection history, at over 280K commits (over 1.6M individual change actions), was the straw that broke the camel’s back. Using PyPy with every optimization on semi-custom hardware tuned for this job still yielded test conversion times of over 9 hours, which is death on the recipe-debugging cycle. Just how often does one nee…

> Just how often does one need to convert the Gnu Compiler Collection from Subversion to Git in under 9 hours?

For this particular repository, once the conversion is complete, it's complete. But there are plenty of other old projects out there with large repositories in old version control systems. Reposurgeon is a general tool.

Also, one repository conversion does not mean one run of reposurgeon and you're done. Reposurgeon is meant to be run multiple times since each run is likely to uncover issues that have to be addressed, and the way to address them is to do another run with updated parameters. Reposurgeon also has an interactive mode where you can explore a repository and test the results of various possible transformations.