Is there a website which tracks which of the major libraries; pandas, requests, django, etc support each major version? I remember there was one years ago for python 3.6? Been a long time!
Python 3.11 vs 3.10 performance
371–380 of 460 posts
Re: Python 3.11 vs 3.10 performance
#372There was always a denial of removing the Global Interpreter Lock because it would decrease single threaded Python speed for which most people din’t care. So I remember a guy recently came up with a patch that both removed GIL and also to make it easier for the core team to accept it he added also an equivalent number of optimizations. I hope this release was is not we got the optimizations but ignored the GIL part.…
Removing GIL also breaks existing native packages, and would require wholesale migration across the entire ecosystem, on a scale not dissimilar to what we've seen with Python 3.
> on a scale not dissimilar to what we've seen with Python 3.
If only people had asked for it before the Python 3 migration so it could have been done with all the other breaking and performance harming changes. But no, people only started to ask for it literally yesterday so it just could not ever be done, my bad. /sarcasm
If anything the whole Python 3 migration makes any argument against the removal of the GIL appear dishonest to me. It should have been gone decades ago and still wasn't included in the biggest pile of breaking changes the language went through .
Re: Python 3.11 vs 3.10 performance
#373Is there a website which tracks which of the major libraries; pandas, requests, django, etc support each major version? I remember there was one years ago for python 3.6? Been a long time!
I think this is what you’re looking for: https://pyreadiness.org/3.11/
Re: Python 3.11 vs 3.10 performance
#374Earlier quoted context omitted.
Does the new walrus := in python solve your problem?
The official name is assignment expression, which can be good to know to find documentation. Here's the PEP: https://peps.python.org/pep-0572/
Curiously, searching the docs for "walrus" (https://docs.python.org/3/search.html?q=walrus) yields exactly the results you need, but searching for "assignment expression" yields the previous results mixed a lot of other results due to the word "expression".
Re: Python 3.11 vs 3.10 performance
#375Earlier quoted context omitted.
Or maybe keep things the way they are.If you really need performance python is not the language you should be looking for. Instead of breaking decades of code, maybe use a language like Go or Rust for performance instead.
Python is also the dominant language for machine learning which does care for performance. The person who made recent nogil work is one of the core maintainers of key ML library. The standard workaround is ML libraries, the performance sensitive stuff is written in C/C++ (either manually or with cython) and then uses python bindings. But it would be much friendlier if we could just use python directly. It's also comm…
> Python is also the dominant language for machine learning which does care for performance. The person who made recent nogil work is one of the core maintainers of key ML library. The standard workaround is ML libraries, the performance sensitive stuff is written in C/C++ (either manually or with cython) and then uses python bindings. But it would be much friendlier if we could just use python directly.
Multithreading is not really the reason why things get written in cython etc., you can easily see 100x improvements in single threaded performance (compared to maybe a factor of 2-8x for multithreading). If you care about performance you'd definitely write the performance critical stuff in cython/pythran/c.
Re: Python 3.11 vs 3.10 performance
#376Earlier quoted context omitted.
> As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language. Hmm, how is that so? As a python dev as well, I don't have much complaint with multiprocessing. The API is simple, it works OK, the overall paradigm is simple to grok, you can share transparently with pickle, etc.
Multiprocessing is fine, but the cost of starting another interpreter is pretty visible, so you need to create a pool, and it may not be an overall speedup if the run time is short. It takes more careful planning than async in JS?, say, or goroutines.
Yeah but for JS style async you'd use probably use an event loop in Python, not multi processing.
Re: Python 3.11 vs 3.10 performance
#377One of the biggest features I'm looking forward to is the more specific error messages. I can't tell you how much time I've wasted with cryptic errors that just point to a line that has a list comprehension or similar.
This is the first time I ever heard someone complain about Python's error messages in the 10+ years I've been using it. Even people who just learned it, pick up reading tracebacks after about a day of practice. The only problem I ever see is if a library swallows too much, and gives a generic error, but that's not something the language can fix. I really hope they don't change things too much.
You've clearly been working with the right people. I constantly get screenshots of error messages (not even the text of the traceback so that I could copy it) with questions about what that means. It takes some training for people to be able to read tracebacks correctly, especially if they have no other programming experience.
Re: Python 3.11 vs 3.10 performance
#378Is this speed a feature of python now, or is this performance something that could regress in future if needed for a different feature? As in, will python continue to run this fast (or faster)?
I think it’s hard to answer your question. The development mentality of Python seems to have changed a little, going into 3.11, and you see sort of a shift in the priority of clean vs fast. I think you can assume that Python will continue to care about speed, but really, it sort of always has. Just like it’s going to continue to care about clean implementation. Then there is the part where Python will likely never become a “fast” language in the sense that C is a fast language, but the flip side of this is that Python was probably already fast enough for most uses. I tend to like to use the “we aren’t Netflix” or the “stackoverflow runs on IIS and always has” arguments when it comes to worrying about speed in anyone who isn’t Netflix, but that doesn’t really apply to Python since the backend of Instagram is django, and if Python can power Instagram then it’s probably fast enough for you, unless you’re already working with C/C++/Rust and know exactly why you are working with those.
I know I’m a bit more focused on the business side of things than a lot of techies here on HN, it’s a curse and a gift, but what I read the 10-60% speed increase as is money. The less resources you consume, the less money you burn. Which is really good news for Python in general, because it makes it more competitive to languages like C# for many implementations.
This comes from the perspective of someone who actually things running Typescript on your backend is a good idea because it lets you share developer resources easier so that the people who are really good at react can cover for the people who are really good at the backend and the other way around in smaller teams in non-software-engineering organisations.
Re: Python 3.11 vs 3.10 performance
#379Earlier quoted context omitted.
That’s bizarre, are they implemented as linked lists or something? Why would arrays be that slow?
They are not guaranteed to be contiguous, but they supposedly are if you specify a primative type and initialize them all at once. Tuples have a constant size, are contiguous, and immutable. So lots of optimization opportunity.
How is it an array if it's not guaranteed to be contiguous?
Re: Python 3.11 vs 3.10 performance
#380Earlier quoted context omitted.
In the example from the end of the earlier post: y for x in xs if (y:=x.ComputeSomething()).IsFoo and y.IsBar
Much better would be [(y:=x.ComputeSomething()) for x in xs if y.IsFoo and y.IsBar]
That doesn't work, because that first subexpression is evaluated last, after "if".