how about getting rid of GIL?
Faster Python with Guido van Rossum
11–20 of 251 posts
Re: Faster Python with Guido van Rossum
#12I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…
Also, typing in python has a "kluge" feel about it. It's like most things in python: it works nicely on toy examples, but is rather frail on the edges (if you're not convinced try to define the json type). I also think they chose the wrong strategy (type inference would have been way better)
To use a metaphor: Python is the duplo of programming languages.
Re: Faster Python with Guido van Rossum
#13I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…
I'm not terribly familiar with the options for typing, mostly because I'm salty that Python rolled out annotations without a clear plan for how they'd be useful*. Aside from Cython, does typing provide any mechanism to improve performance, or is it only there for static analysis? Worse yet, is it being used to do dynamic analysis that slows the whole thing even further?
* specifically, this happened long after the tragically many-ways-to-do-it of packaging systems and virtual environments, which survive to this day.
Re: Faster Python with Guido van Rossum
#14I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…
Do you have the opposite experience? A language used by FAANG that doesn’t have performance issues at their scale. They seem to invest a lot in JavaScript which is not what I’d call a performant language and as you’ve said Python.
I’ve used Python for more scripting type tasks and creating basic APIs to get specific jobs done. I’m very weary to go “all in” on Python for an app based on all the feedback about performance.
All that being said I’ve never heard someone say we used X language and it scaled remarkably without issues. I’ve noticed lots of companies eventually go to Java once they get big but nobody really likes to brag about Java apps.
Re: Faster Python with Guido van Rossum
#15I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…
You may be right but I’m convinced most problems don’t need to scale.
Re: Faster Python with Guido van Rossum
#16I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…
Do you have the opposite experience? A language used by FAANG that doesn’t have performance issues at their scale. They seem to invest a lot in JavaScript which is not what I’d call a performant language and as you’ve said Python. I’ve used Python for more scripting type tasks and creating basic APIs to get specific jobs done. I’m very weary to go “all in” on Python for an app based on all the feedback about performa…
Everything has performance issues but Python is at least one, if not two, orders of magnitude off of compiled languages like C++, Rust, and Go (despite Go having a garbage collector). They have various usability tradeoffs but once you get things working, at least you're not spending 10x as many cycles to do the same work as another language would need.
(Java probably is also fine; I've less personal experience there so I can't say).
Re: Faster Python with Guido van Rossum
#17I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…
I'm not terribly familiar with the options for typing, mostly because I'm salty that Python rolled out annotations without a clear plan for how they'd be useful*. Aside from Cython, does typing provide any mechanism to improve performance, or is it only there for static analysis? Worse yet, is it being used to do dynamic analysis that slows the whole thing even further? * specifically, this happened long after the tr…
There are some tools that use types for optimization, but I'd say it's still pretty early.
For example, mypyc is a python to C compiler that is part of mypy. It uses types for both correctness and optimization:
https://mypyc.readthedocs.io/en/latest/using_type_annotation...
Re: Faster Python with Guido van Rossum
#18I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…
Thanks for this comment, sums up exactly my experience with python. Do you have a pointer to a larger write up of these issues? I really enjoy python, but agree with you that it isn't well suited for performance or scale necessarily. I find that people really get attached to it though and I'd like to have a good reference for them to help explain why issues like interpreter performance, GIL and typing make a difference on large projects even if not their pet ones.
Re: Faster Python with Guido van Rossum
#19how about getting rid of GIL?
If you get rid of the GIL you have to replace one big lock with a bunch of very small locks, which actually hurts performance for programs which only need 1 core, which is most programs.
Re: Faster Python with Guido van Rossum
#20I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…
I'm curious, as you seem very versed in the Python ecosystem, where you stand on projects like Cython and integration of C libraries with Python using projects like Cython? Or would you consider that more of a stop-gap measure and not "real" Python (which would be valid)?
Things like Cython are great until they aren't, so yes, I think of them as a stop gap (albeit a practical and useful one). Any time you change interpreters or the execution model, you risk compatibility challenges with third party libraries and native modules. You also risk getting painted into a corner by their limitations. They definitely can help with hot code paths when their trade-offs don't prevent it but it still ends up being a kluge working around fundamental language issues.
But when Cython fits, it's great, and a good tool to have in your toolbox if you're working with Python.