I'd like to compliment the author on the quality of this post. It's very well written, data/example driven, fair, and educational. Overall, a joy to read. Thank you!
He's been slightly famous for technical writing for a long time
Experience report on a large Python-to-Go translation
21–30 of 99 posts
Re: Experience report on a large Python-to-Go translation
#22> The man barrier to translation was that, while at 14KLOC of Python reposurgeon was not especially large, the code is very dense.
Reasoning about somebody else’s dense code is probably the least favorite activities. When I hear about a language being “expressive” or having “flexible syntax” I shudder.
Re: Experience report on a large Python-to-Go translation
#23significant mastery ahead! This is a success story and a teaching document. .. have to point to this : "Now that I’ve seen Go strings… holy hell, Python 3 unicode strings sure look like a nasty botch in retrospect. " (!)
I wish he had been a bit more explicit there, TBH. What's better about Go strings?
Re: Experience report on a large Python-to-Go translation
#24If it was too slow in Python and now moving to Go. Could there a time when there is a need to move to Rust/C/C++ for even faster performance? Go seems an odd choice based on performance consideration alone.
Re: Experience report on a large Python-to-Go translation
#25I do not get why people do these total rewrites, especially for working Python systems. Why throw out the baby with the bathwater ? Python is fundamentally a composing toolkit. Rewrite the slow bits in C++/Rust/Go and wrap it. That's how all major Python components like Numpy, Scipy, Tensorflow, PyTorch etc. does it. And that's a major reason why Python dominates today. Align with the core strengths of Python's philo…
Because nine hour runtime is really long when you're trying out different rules to make sure that the conversion happens the way you want. It's almost like a REPL with a nine-hour response time (not quite, but it's in that direction). That's so completely unworkable that a rewrite into something faster is basically your only option.
Re: Experience report on a large Python-to-Go translation
#26Pretty interesting. It is scary to make your "learn a new language" task to port 14,000 lines of code, but with that in mind, this all seems to have gone well. Some random thoughts: > I had to write my own set-of-int and set-of-string classes map[int]struct{}, map[string]struct{} ints[42] = struct{}{} // insert delete(ints, 42) // delete for i := range ints { ... } // iterate if _, ok := ints[42]; ok { ... } // exist…
> I really like seeing the word "for" twice when you're iterating over two things.
You will if you write a Python list or generator comprehension that iterates over two things, won't you?
Re: Experience report on a large Python-to-Go translation
#27Interesting. I'd have expected more than a 50% code expansion going to Go, maybe even 3x or 5x. Similarly, he's using 40x speedup as a rule of thumb. I usually think of Python as 20x slower than C. Personally I'd be loathe to convert a working Python system to Go, but it sounds like he had good reasons. I do wonder a bit whether divide-and-conquer or a C extension might not have worked instead.
"Interesting. I'd have expected more than a 50% code expansion going to Go, maybe even 3x or 5x." This has been my extensive experience as well. I wouldn't be able to use Go if it was that much more verbose than Python. It certainly isn't as succinct as Python by any means, but it's not the night-and-day nightmare a lot of HN posters seem to think it is... provided you actually learn the language. In fact, one of the…
Well... fancy features/abstractions are pretty low cost, if you use just one. It's when they interact that they really get expensive unless they are almost flawlessly designed to work together (and even then, they just get somewhat more expensive).
It's like the first donut has 100 calories, the second has 200, the third has 400, and the fourth has 800. If you eat the whole box, it's really going to hurt. Rather than eat them all, you need to pick the one or two that you really want, and leave the rest in the box.
Re: Experience report on a large Python-to-Go translation
#28I do not get why people do these total rewrites, especially for working Python systems. Why throw out the baby with the bathwater ? Python is fundamentally a composing toolkit. Rewrite the slow bits in C++/Rust/Go and wrap it. That's how all major Python components like Numpy, Scipy, Tensorflow, PyTorch etc. does it. And that's a major reason why Python dominates today. Align with the core strengths of Python's philo…
> Rewrite the slow bits in C++/Rust/Go and wrap it.
For reposurgeon, you can't. Not the author, but I have done some hacking on it. "The slow bits" are not things you can rewrite in a different language and wrap. They are much too integral to the code.
> That's how all major Python components like Numpy, Scipy, Tensorflow, PyTorch etc. does it.
And what all of these have in common is that the "slow bits" are not like those in reposurgeon. The wrapped speeded-up things are basically fast implementations of appropriate basic data types, like Numpy arrays and vectors. Those aren't the kinds of things that are slow in reposurgeon.
Re: Experience report on a large Python-to-Go translation
#29I do not get why people do these total rewrites, especially for working Python systems. Why throw out the baby with the bathwater ? Python is fundamentally a composing toolkit. Rewrite the slow bits in C++/Rust/Go and wrap it. That's how all major Python components like Numpy, Scipy, Tensorflow, PyTorch etc. does it. And that's a major reason why Python dominates today. Align with the core strengths of Python's philo…
> Align with the core strengths of Python's philosophy and its toolset and get the benefit. Why fight it ?
In this particular case, it's worth noting that reposurgeon highlights some core weaknesses in Python's philosophy and its toolset. The two biggest are the amount of memory required for even basic objects and the GIL.
Re: Experience report on a large Python-to-Go translation
#30Interesting. I'd have expected more than a 50% code expansion going to Go, maybe even 3x or 5x. Similarly, he's using 40x speedup as a rule of thumb. I usually think of Python as 20x slower than C. Personally I'd be loathe to convert a working Python system to Go, but it sounds like he had good reasons. I do wonder a bit whether divide-and-conquer or a C extension might not have worked instead.
"Interesting. I'd have expected more than a 50% code expansion going to Go, maybe even 3x or 5x." This has been my extensive experience as well. I wouldn't be able to use Go if it was that much more verbose than Python. It certainly isn't as succinct as Python by any means, but it's not the night-and-day nightmare a lot of HN posters seem to think it is... provided you actually learn the language. In fact, one of the…
You wouldn't take the huge pile of features in an all or nothing. You get to pick and choose. I think the authors wish list is similar to most people who are experienced with more expressive languages. I don't see calls for Python f-strings or async syntax to end up in Go. But something like a list-comprehension syntax I think would be really popular (based on what I saw when it was introduced to Python, originally 'why' but now many people's favorite feature).
Interesting how Python is now considered a huge pile of features, considering how long it took for things like a ternary operator to end up in the language. Maybe Go is in it's Python 1.5.2 lifecycle stage ;)