Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

State of Python 3.13 performance: Free-threading

codspeed.io

Post reply on HN
Print view View original

101–110 of 193 posts

Re: State of Python 3.13 performance: Free-threading

#101

Post by PaulHoule »

willseth wrote: post #98
rfoo wrote: post #40

Earlier quoted context omitted.

> Python is never really going to be 'fast' no matter what is done to it because its semantics make most important optimizations impossible Scientific computing community have a bunch of code calling numpy or whatever stuff. They are pretty fast because, well, numpy isn't written in Python. However, there is a scalability issue: they can only drive so many threads (not 1, but not many) in a process due to GIL. Okay,…

This is misleading. Most of the compute intensive work in Numpy releases the GIL, and you can use traditional multithreading. That is the case for many other compute intensive compiled extensions as well.

It’s an Amdahl’s law sort of thing, you can extract some of the parallelism with scikit-learn but what’s left is serialized. Particularly for those interactive jobs where you might write plain ordinary Python snippets that could get a 12x speedup (string parsing for a small ‘data lake’)

In so far as it is all threaded for C and Python you can parallelize it all with one paradigm that also makes a mean dynamic web server.

Re: State of Python 3.13 performance: Free-threading

#102

Post by salomonk_mur »

andai wrote: post #24
eigenspace wrote: post #3

I don't really have a dog in this race as I don't use Python much, but this sort of thing always seemed to be of questionable utility to me. Python is never really going to be 'fast' no matter what is done to it because its semantics make most important optimizations impossible, so high performance "python" is actually going to always rely on restricted subsets of the language that don't actually match language's "re…

There was a discussion the other day about how Python devs apparently don't care enough for backwards compatibility. I pointed out that I've often gotten Python 2 code running on Python 3 by just changing print to print(). But then a few hours later, I tried running a very small project I wrote last year and it turned out that a bunch of my dependencies had changed their APIs. I've had similar (and much worse) experi…

What APIs were broken? They couldn't be in the standard library.

If the dependency was in external modules and you didn't have pinned versions, then it is to be expected (in almost any active language) that some APIs will break.

Re: State of Python 3.13 performance: Free-threading

#103

Post by maleldil »

dbsmith83 wrote: post #33
dagmx wrote: post #23

Earlier quoted context omitted.

To your last point: it’s neither the language nor the packages but rather it’s the ABI. Python isn’t fully ABI stable (though it’s improved greatly) so you can’t just intermix compiled dependencies between different versions of Python. This is true for many packages in your distro as well.

There have been many breaking changes throughout python 3.x releases: - standard library modules removed - zip error handling behaves differently - changes to collections module - new reserved keywords (async, await, etc.) You can argue how big of a deal it is or isn't, but there were definitely breakages that violate semantic versioning

Python doesn't follow SemVer, that's why.

https://peps.python.org/pep-2026/

Re: State of Python 3.13 performance: Free-threading

#105

Post by selcuka »

dietr1ch wrote: post #93
dataflow wrote: post #84

Earlier quoted context omitted.

> Python 2 code running on Python 3 by just changing print to print(). This was very much the opposite of my experience. Consider yourself lucky.

This migration took the industry years because it was not that simple.

> This migration took the industry years because it was not that simple.

It was not that simple, but it was not that hard either.

It took the industry years because Python 2.7 was still good enough, and the tangible benefits of migrating to Python 3 didn't justify the effort for most projects.

Also some dependencies such as MySQL-python never updated to Python 3, which was also an issue for projects with many dependencies.

Re: State of Python 3.13 performance: Free-threading

#106

Post by nas »

the5avage wrote: post #104

Can someone share insight into what was technically done to enable this? What replaced the global lock? Is the GC stopping all threads during collection or an other locking mechanism?

The key enabling tech is thread safe reference counting. There are many other problems that Sam Gross solved in order to make it happen but the reference counting was one of the major blockers.

Re: State of Python 3.13 performance: Free-threading

#108

Post by tightbookkeeper »

the5avage wrote: post #104

Can someone share insight into what was technically done to enable this? What replaced the global lock? Is the GC stopping all threads during collection or an other locking mechanism?

Lots of little locks littered all over the place.

Re: State of Python 3.13 performance: Free-threading

#109

Post by __mharrison__ »

Timon3 wrote: post #58
adamc wrote: post #34

Earlier quoted context omitted.

It's hard to comment on this without knowing more about the dependencies and when/how they changed their APIs. I would say if it was a major version change, that isn't too shocking. For a minor version change, it should be. Stuff that is actually included with Python tends to be more stable than random Pypi packages, though. NPM packages also sometimes change. That's the world.

The big difference is that npm will automatically (since 2017) save a version range to the project metadata, and will automatically create this metadata file if it doesn't exist. Same for other package managers in the Node world. I just installed Python 3.13 with pip 24.2, created a venv and installed a package - and nothing, no file was created and nothing was saved. Even if I touch requirements.txt and pyproject.to…

I recommend to start using UV.

It is very fast and tracks the libraries you are using.

After years of venv/pip, I'm not going back (unless a client requires it).

Re: State of Python 3.13 performance: Free-threading

#110

Post by Someone »

eigenspace wrote: post #3

I don't really have a dog in this race as I don't use Python much, but this sort of thing always seemed to be of questionable utility to me. Python is never really going to be 'fast' no matter what is done to it because its semantics make most important optimizations impossible, so high performance "python" is actually going to always rely on restricted subsets of the language that don't actually match language's "re…

I think the reasoning is like this:

- People choose Python to get ease of programming, knowing that they give up performance.

- With multi-core machines now the norm, they’re relatively giving up more performance to get the same amount of ease of programming.

- so, basically, the price of ease of programming has gone up.

- economics 101 is that rising prices will decrease demand, in this case demand for programming in Python.

- that may be problematic for the long-term survival of Python, especially with new other languages aiming to provide python’s ease of use while supporting multi-processing.

So, Python must get even easier to use and/or it must get faster.

Post reply on HN