Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experience report on a large Python-to-Go translation

gitlab.com

Post reply on HN
Print view View original

31–40 of 99 posts

Re: Experience report on a large Python-to-Go translation

#31

Post by Izkata »

AnimalMuppet wrote: post #23
mistrial9 wrote: post #13

significant mastery ahead! This is a success story and a teaching document. .. have to point to this : "Now that I’ve seen Go strings… holy hell, Python 3 unicode strings sure look like a nasty botch in retrospect. " (!)

I wish he had been a bit more explicit there, TBH. What's better about Go strings?

Skimming over a post from a quick search [0], it looks like Go strictly uses byte strings and manipulates them with various keywords and functions. No encoding/decoding between byte strings (and picking an encoding) and unicode strings like in python.

[0] https://blog.golang.org/strings

Re: Experience report on a large Python-to-Go translation

#32

Post by dboreham »

jrockway wrote: post #12

Pretty interesting. It is scary to make your "learn a new language" task to port 14,000 lines of code, but with that in mind, this all seems to have gone well. Some random thoughts: > I had to write my own set-of-int and set-of-string classes map[int]struct{}, map[string]struct{} ints[42] = struct{}{} // insert delete(ints, 42) // delete for i := range ints { ... } // iterate if _, ok := ints[42]; ok { ... } // exist…

Much of what passes for wisdom in this field is just people's way of feeling good about themselves because they're convinced they're smarter than others. The list comprehension or whatever it is is just a construct to enable that thought pattern.

Re: Experience report on a large Python-to-Go translation

#33

Post by hibbelig »

3fe9a03ccd14ca5 wrote: post #22

> The man barrier to translation was that, while at 14KLOC of Python reposurgeon was not especially large, the code is very dense. Reasoning about somebody else’s dense code is probably the least favorite activities. When I hear about a language being “expressive” or having “flexible syntax” I shudder.

I think the author was reasoning about his own dense code, though.

Re: Experience report on a large Python-to-Go translation

#34

Post by patrec »

Adding lookbehinds to the regexp library is a terrible idea.

> The regexp implementation provided by this package is guaranteed to run in time linear in the size of the input.

Python's is exponential, because it inherits all the non-regular "regular" expression mess (such as lookbehind and backrefs) from perl.

One would assume esr would have marinated in unix culture for long enough to be aware of this.

Re: Experience report on a large Python-to-Go translation

#35

Post by erik_seaberg »

bmurphy1976 wrote: post #17
jerf wrote: post #15

Earlier quoted context omitted.

"Interesting. I'd have expected more than a 50% code expansion going to Go, maybe even 3x or 5x." This has been my extensive experience as well. I wouldn't be able to use Go if it was that much more verbose than Python. It certainly isn't as succinct as Python by any means, but it's not the night-and-day nightmare a lot of HN posters seem to think it is... provided you actually learn the language. In fact, one of the…

I've been saying for a long time that every abstraction has a cost. Sometimes that cost is hard to quantify or externalized, but our inability to quantify it doesn't mean the cost doesn't exist. Abstractions have to at least pay for themselves many times over to be worth the extra cognitive burden and we need to get better at measuring these trade-offs. The success of languages like Go hint that there's more costs th…

The one-time cost of learning a good abstraction is strictly less than the ongoing cost of understanding and then continually reimplementing it by hand. The purpose of a high-level language is to make programs more concise and clear; a language that doesn't do this may somehow become popular but that shouldn't be mistaken for successful.

Re: Experience report on a large Python-to-Go translation

#36

Post by [deleted] »

patrec wrote: post #34

Adding lookbehinds to the regexp library is a terrible idea. > The regexp implementation provided by this package is guaranteed to run in time linear in the size of the input. Python's is exponential, because it inherits all the non-regular "regular" expression mess (such as lookbehind and backrefs) from perl. One would assume esr would have marinated in unix culture for long enough to be aware of this.

[deleted]

Re: Experience report on a large Python-to-Go translation

#37

Post by tejinderss »

xiaodai wrote: post #24

If it was too slow in Python and now moving to Go. Could there a time when there is a need to move to Rust/C/C++ for even faster performance? Go seems an odd choice based on performance consideration alone.

Especially with node.js/TypeScript one can reach similar performance to Go with arguably much nicer programming language to work with.

Re: Experience report on a large Python-to-Go translation

#38

Post by gwd »

xiaodai wrote: post #24

If it was too slow in Python and now moving to Go. Could there a time when there is a need to move to Rust/C/C++ for even faster performance? Go seems an odd choice based on performance consideration alone.

As he said, given the size and complexity of the transition, one of his primary goals was to do as much of a "literal" translation first as possible. IOW, this wasn't so much of a "rewrite from scratch" (which is almost always a bad idea), as "do a translation first; then do refactoring to be more idiomatic / performant later". Obviously some parts will need to be rewritten, but the lower you can keep the rewrite:translation ratio, the better.

I'm not an expert in Rust, but from what I know it seems like moving to Rust would require much more rewriting than Go.

The other thing, as he says in the post, is that you get diminishing returns: The Go code was 20x faster than the Python code, but the whole 7-hour operation was only 10x times faster, because at some point the external SVM calls start to dominate. So even if a rewrite into Rust could gain him another dozen percentage points in speed, it's unlikely that a rewrite into Rust would have much of an impact on his end-to-end performance; and it would almost certainly make the code less accessible.

Re: Experience report on a large Python-to-Go translation

#39

Post by wittekm »

patrec wrote: post #34

Adding lookbehinds to the regexp library is a terrible idea. > The regexp implementation provided by this package is guaranteed to run in time linear in the size of the input. Python's is exponential, because it inherits all the non-regular "regular" expression mess (such as lookbehind and backrefs) from perl. One would assume esr would have marinated in unix culture for long enough to be aware of this.

Some projects need performant regexes, and some honestly just don't. I agree that keeping the base regex library linear is admirable, but it's be nice if they offered a well-marked thing like regex.slow_and_perl_like in the stdlib

Re: Experience report on a large Python-to-Go translation

#40

Post by marcus_holmes »

erik_seaberg wrote: post #35
bmurphy1976 wrote: post #17

Earlier quoted context omitted.

I've been saying for a long time that every abstraction has a cost. Sometimes that cost is hard to quantify or externalized, but our inability to quantify it doesn't mean the cost doesn't exist. Abstractions have to at least pay for themselves many times over to be worth the extra cognitive burden and we need to get better at measuring these trade-offs. The success of languages like Go hint that there's more costs th…

The one-time cost of learning a good abstraction is strictly less than the ongoing cost of understanding and then continually reimplementing it by hand. The purpose of a high-level language is to make programs more concise and clear; a language that doesn't do this may somehow become popular but that shouldn't be mistaken for successful .

> The one-time cost of learning a good abstraction is strictly less than the ongoing cost of understanding and then continually reimplementing it by hand.

There is a danger of a "no true Scotsman" fallacy here - it's easy to define a "good" abstraction as one that is worth the one-time cost of learning it.

So, given that there are both "good" and "bad" abstractions in every language, is the net gain from learning them all greater than the cognitive effort of having to learn them all?

I'd argue that no, it's not. Absolutely reducing the total number of abstractions in the language reduces the cognitive load of working in that language, even if it requires us to be more verbose.

Post reply on HN