If it were ever open sourced, I could see Mojo filling the performance niche for Python programmers. I'm hopeful because Lattner certainly has the track record, if he doesn't move on beforehand. https://en.wikipedia.org/wiki/Mojo_(programming_language)
State of Python 3.13 performance: Free-threading
91–100 of 193 posts
Re: State of Python 3.13 performance: Free-threading
#92Earlier quoted context omitted.
This is a good question, and I think about it as well. My best guess for a simple explanation: Python is very popular; it makes sense to improve performance for python users, given many do not wish to learn to use a more performant language, or to use a more performant Python implementation. Becoming proficient in a range of tools so you can use the right one for the right job is high enough friction that it is not t…
You should really add that Python is also a very good tool for people who know more performant languages. I think one of the sides which often gets forgotten is that a lot of software will never actually need to be very performant and often you’re not going to know the bottlenecks beforehand. If you even get to the bottlenecks it means you’ve succeeded enough to get to the bottlenecks. Somewhere you might not have go…
> it’s extremely easy to replace parts of your Python code with something like C
I tend to use C++, so use SWIG [1] to make python code to interface with C++ (or C). You can nearly just give it a header file, and a python class pops out, with native types and interfaces. It's really magical.
Re: State of Python 3.13 performance: Free-threading
#93Earlier quoted context omitted.
There was a discussion the other day about how Python devs apparently don't care enough for backwards compatibility. I pointed out that I've often gotten Python 2 code running on Python 3 by just changing print to print(). But then a few hours later, I tried running a very small project I wrote last year and it turned out that a bunch of my dependencies had changed their APIs. I've had similar (and much worse) experi…
> Python 2 code running on Python 3 by just changing print to print(). This was very much the opposite of my experience. Consider yourself lucky.
This migration took the industry years because it was not that simple.
Re: State of Python 3.13 performance: Free-threading
#94Earlier quoted context omitted.
Pinning deps is discouraged by years of Python practice. And going back to a an old project and finding versions that work, a year or more later, might be nigh on impossible. Last week I was trying to install snakemake via Conda, and couldn't find any way to satisfy dependencies at all, so it's not just pypi, and pip tends to be one of the more forgiving version dependency managers. It's not just Python, trying to ge…
I’m curious as to which packages you are unable to find older versions for. You mention snakemake, but that doesn’t seem to have any sort of issues. https://pypi.org/project/snakemake/#history
It's not about finding old packages, it's about not finding the magical compatible set of package versions.
Pip is nice in that you can install packages individually to get around some version conflicts. But with conda and npm and CRAN I have always found my stuck without being able to install dependencies after 15 minutes of mucking.
Its rare that somebody has left the equivalent of the output of a `pip freeze` around to document their state.
With snakemake, I abandoned conda and went with pip in a venv, without filing an issue. Perhaps it was user error from being unfamiliar with conda, but I did not have more time to spend on the issue, much less doing the research to be able to file a competent issue and follow up later on.
Re: State of Python 3.13 performance: Free-threading
#95I'm not smart nor have any university title butmy opinion is this it's very good, but efforts should also go into remove features, not just python, i get it, it would breake anything.
Re: State of Python 3.13 performance: Free-threading
#96I don't really have a dog in this race as I don't use Python much, but this sort of thing always seemed to be of questionable utility to me. Python is never really going to be 'fast' no matter what is done to it because its semantics make most important optimizations impossible, so high performance "python" is actually going to always rely on restricted subsets of the language that don't actually match language's "re…
> so high performance "python" is actually going to always rely on restricted subsets of the language that don't actually match language's "real" semantics.
I don't even understand what this means. If I write `def foo(x):` versus `def foo(x: int) -> float:`, one is a restricted subset of the other, but both are the language's "real" semantics. Restricted subsets of languages are wildly popular in programming languages, and for very varied reasons. Why should that be a barrier here?
Personally, if I have to annotate some of my code that run with C style semantics, but in return that part runs with C speed, for example, then I just don't really mind it. Different tools for different jobs.
Re: State of Python 3.13 performance: Free-threading
#97I don't really have a dog in this race as I don't use Python much, but this sort of thing always seemed to be of questionable utility to me. Python is never really going to be 'fast' no matter what is done to it because its semantics make most important optimizations impossible, so high performance "python" is actually going to always rely on restricted subsets of the language that don't actually match language's "re…
Given the scale at which python is run, how much energy are we saving by improving its performance by 1%?
Re: State of Python 3.13 performance: Free-threading
#98I don't really have a dog in this race as I don't use Python much, but this sort of thing always seemed to be of questionable utility to me. Python is never really going to be 'fast' no matter what is done to it because its semantics make most important optimizations impossible, so high performance "python" is actually going to always rely on restricted subsets of the language that don't actually match language's "re…
> Python is never really going to be 'fast' no matter what is done to it because its semantics make most important optimizations impossible Scientific computing community have a bunch of code calling numpy or whatever stuff. They are pretty fast because, well, numpy isn't written in Python. However, there is a scalability issue: they can only drive so many threads (not 1, but not many) in a process due to GIL. Okay,…
This is misleading. Most of the compute intensive work in Numpy releases the GIL, and you can use traditional multithreading. That is the case for many other compute intensive compiled extensions as well.
Re: State of Python 3.13 performance: Free-threading
#99Earlier quoted context omitted.
It's hard to comment on this without knowing more about the dependencies and when/how they changed their APIs. I would say if it was a major version change, that isn't too shocking. For a minor version change, it should be. Stuff that is actually included with Python tends to be more stable than random Pypi packages, though. NPM packages also sometimes change. That's the world.
The big difference is that npm will automatically (since 2017) save a version range to the project metadata, and will automatically create this metadata file if it doesn't exist. Same for other package managers in the Node world. I just installed Python 3.13 with pip 24.2, created a venv and installed a package - and nothing, no file was created and nothing was saved. Even if I touch requirements.txt and pyproject.to…
> Of course there are other package managers for Python that do this better
I think if you are comparing with what NPM does then you would have to say that native pip can do that too. It is just one command
`pip freeze > requirements.txt`
It does include everything in the venv (or in you environment in general) but if you stick to only add required things (one venv for each project) then you will get requirements.txt files
Re: State of Python 3.13 performance: Free-threading
#100I don't really have a dog in this race as I don't use Python much, but this sort of thing always seemed to be of questionable utility to me. Python is never really going to be 'fast' no matter what is done to it because its semantics make most important optimizations impossible, so high performance "python" is actually going to always rely on restricted subsets of the language that don't actually match language's "re…
As a Java dev I think people don’t always appreciate the flexiblity of threads for manage parallelism, concurrency, and memory. In particular with threads you can have a large number of thread share a large data structure. Say you have an ML inference model that takes 1GB of RAM. No way can you let 25 Celery workers each have a copy of that model, so if you are using Python you have to introduce a special worker class. It’s one more thing to worry about and more parameters to tune. With Java all your “workers” could be in one process, even the same process as your web server and that will be the case in Python.
Threads break down so many bottlenecks of CPU resources, memory, data serialization, waiting for communications, etc. I have a 16-core computer on my desk and getting a 12x speed-up is possible for many jobs and often worth worse less efficient use of the individual CPU. Java has many primitives for threads that include tools for specialized communications (e.g. barrier synchronization) that are necessary to really get those speed-ups and I hope Python gets those too.