Faster Python with Guido van Rossum

I'm excited to listen to the full podcast. The highlights look really solid.

I think I would like to get into some of the optimization work on Python. I've contributed to some of the DS/ML/NLP libraries in the past, but the core language would be super interesting.

Anyone have good resources on getting started on that sort of thing?

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decisions in the language from two decades ago that are quite difficult to fix.

Unless you go all-in on typing -- which is difficult today with any meaningfully sized existing codebase -- maintenance is largely a "hope manual testing and unit tests catch anything resembling type errors" which is a major challenge for, say, structural change to a code base like refactoring. Plus typing is still young, the tooling somewhat immature, and can lead to false senses of security if you aren't very careful and opt into the strictest modes. This makes large number of developers and codebase size a major stumbling block.

The interpreter performance and GIL are fundamental issues as well. Multiprocessing and hacks around the GIL are quite painful if you even glance at any native threading code (say in a C++ library) and even when you stick to pure Python, you have a debugging mess when anything goes wrong.

But if someone can improve performance, that'd be great, and has massive impact potential. It is incremental though and doesn't solve fundamental issues with the language. I'm also skeptical of the "5x" plan referenced in the blog, and very skeptical we can ever see meaningful removal of the GIL due to library and baked in design decisions in existing code. This means performance will fall further and further behind compiled languages.

(I write all of this having been a part of supporting Python at massive scale for over two decades, including at two FAANG companies who invest heavily in it. I've seen the curve and the pain it's caused, and would never use it for any code that needs to be performant or actively developed on the multi-month or year timescale).

Given Python has the multiprocessing module, I always get confused when people talk about Python lack of support for multicore. What are the shortcomings of the multiprocessing module, that cause people to disregard it?

I'm curious, as you seem very versed in the Python ecosystem, where you stand on projects like Cython and integration of C libraries with Python using projects like Cython? Or would you consider that more of a stop-gap measure and not "real" Python (which would be valid)?

Python's multiprocessing module forces you to serialize all your data, or use ctypes. This means it's either going to be slow, or it's like writing code in a hybrid of Pythonic and C-style code. Which of course goes against the idea of using the simplest tool for the job.