Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The first year of free-threaded Python

labs.quansight.org

Post reply on HN
Print view View original

151–160 of 302 posts

Re: The first year of free-threaded Python

#151

Post by lenerdenator »

bgwalter wrote: post #32

This is just an advertisement for the company. Fact is, free-threading is still up to 50% slower, the tail call interpreter isn't much faster at all, and free-threading is still flaky. Things they won't tell you at PyCon.

I don't see how any of that's a problem given that it's not the default for how people run Python.

It's a big project that's going to take lots of time by lots of people to finish. Keep it behind opt-in, keep accepting pull requests after rigorous testing, and it's fine.

Re: The first year of free-threaded Python

#153

Post by biorach »

nothrabannosir wrote: post #148
biorach wrote: post #127

Earlier quoted context omitted.

> A classic example is ActiveX. Nah, even that was based on earlier MS technologies - OLE and COM A good starter list of EEE plays is on the wikipedia page: https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...

Funny you linked that page because that’s where I got activex from :D > Examples by Microsoft > Browser incompatibilities > The plaintiffs in an antitrust case claimed Microsoft had added support for ActiveX controls in the Internet Explorer Web browser to break compatibility with Netscape Navigator, which used components based on Java and Netscape's own plugin system.

ah ok, sorry. I thought you were saying that they tried an EEE play on ActiveX.

You meant they used ActiveX in an EEE play in the browser wars.

Re: The first year of free-threaded Python

#154

Post by breadwinner »

jaoane wrote: post #91
breadwinner wrote: post #64

Earlier quoted context omitted.

When the language is dynamic there is less rigor. Statically checked code is more likely to be correct. When you add threads to "fast and loose" code things get really bad.

Unless your claim is that the same error can happen more times per minute because threading can execute more code in the same timespan, this makes no sense.

Some statically checked languages and tools can catch potential data races at compile time. Example: Rust's ownership and borrowing system enforces thread safety at compile time. Statically typed functional languages like Haskell or OCaml encourage immutability, which reduces shared mutable state — a common source of concurrency bugs. Statically typed code can enforce usage of thread-safe constructs via types (e.g., Sync/Send in Rust or ConcurrentHashMap in Java).

Re: The first year of free-threaded Python

#155

Post by frollogaston »

AlexanderDhoore wrote: post #2

Am I the only one who sort of fears the day when Python loses the GIL? I don't think Python developers know what they’re asking for. I don't really trust complex multithreaded code in any language. Python, with its dynamic nature, I trust least of all.

What reliance did you have in mind? All sorts of calls in Python can release the GIL, so you already need locking, and there are race conditions just like in most languages. It's not like JS where your code is guaranteed to run in order until you "await" something.

I don't fully understand the challenge with removing it, but thought it was something about C extensions, not something most users have to directly worry about.

Re: The first year of free-threaded Python

#156

Post by wongarsu »

isignal wrote: post #133
sgarland wrote: post #48

> Instead, many reach for multiprocessing, but spawning processes is expensive Agreed. > and communicating across processes often requires making expensive copies of data SharedMemory [0] exists. Never understood why this isn’t used more frequently. There’s even a ShareableList which does exactly what it sounds like, and is awesome. [0]: https://docs.python.org/3/library/multiprocessing.shared_mem...

Processes can die independently so the state of a concurrent shared memory data structure when a process dies while modifying this under a lock can be difficult to manage. Postgres which uses shared memory data structures can sometimes need to kill all its backend processes because it cannot fully recover from such a state. In contrast, no one thinks about what happens if a thread dies independently because the failu…

> In contrast, no one thinks about what happens if a thread dies independently because the failure mode is joint.

In Rust if a thread holding a mutex dies the mutex becomes poisoned, and trying to acquire it leads to an error that has to be handled. As a consequence every rust developer that touches a mutex has to think about that failure mode. Even if in 95% of cases the best answer is "let's exit when that happens".

The operating system tends to treat your whole process as one and shot down everything or nothing. But a thread can still crash in its own due to unhandled oom, assertion failures or any number of other issues

Re: The first year of free-threaded Python

#157

Post by dhruvrajvanshi »

modeless wrote: post #109
sgarland wrote: post #48

> Instead, many reach for multiprocessing, but spawning processes is expensive Agreed. > and communicating across processes often requires making expensive copies of data SharedMemory [0] exists. Never understood why this isn’t used more frequently. There’s even a ShareableList which does exactly what it sounds like, and is awesome. [0]: https://docs.python.org/3/library/multiprocessing.shared_mem...

Yeah I've had great success sharing numpy arrays this way. Explicit sharing is not a huge burden, especially when compared with the difficulty of debugging problems that occur when you accidentally share things between threads. People vastly overstate the benefit of threads over multiprocessing and I don't look forward to all the random segfaults I'm going to have to debug after people start routinely disabling the G…

> I wonder why people never complained so much about JavaScript not having shared-everything threading. Maybe because JavaScript is so much faster that you don't have to reach for it as much. I wish more effort was put into baseline performance for Python.

This is a fair observation.

I think a part of the problem is that the things that make GIL less python hard are also the things that make faster baseline performance hard. I.e. an over reliance of the ecosystem on the shape of the CPython data structures.

What makes python different is that a large percentage of python code isn't python, but C code targeting the CPython api. This isn't true for a lot of other interpreted languages.

Re: The first year of free-threaded Python

#158

Post by jcalvinowens »

isignal wrote: post #133
sgarland wrote: post #48

> Instead, many reach for multiprocessing, but spawning processes is expensive Agreed. > and communicating across processes often requires making expensive copies of data SharedMemory [0] exists. Never understood why this isn’t used more frequently. There’s even a ShareableList which does exactly what it sounds like, and is awesome. [0]: https://docs.python.org/3/library/multiprocessing.shared_mem...

Processes can die independently so the state of a concurrent shared memory data structure when a process dies while modifying this under a lock can be difficult to manage. Postgres which uses shared memory data structures can sometimes need to kill all its backend processes because it cannot fully recover from such a state. In contrast, no one thinks about what happens if a thread dies independently because the failu…

This is a solvable problem though, the literature is overflowing with lock-free implementations of common data structures. The real question is how much performance you have to sacrifice for the guarantee...

Re: The first year of free-threaded Python

#159

Post by kevingadd »

bratao wrote: post #55
AlexanderDhoore wrote: post #2

Am I the only one who sort of fears the day when Python loses the GIL? I don't think Python developers know what they’re asking for. I don't really trust complex multithreaded code in any language. Python, with its dynamic nature, I trust least of all.

This is a common mistake and very badly communicated. The GIL do not make the Python code thread-safe. It only protect the internal CPython state. Multi-threaded Python code is not thread-safe today.

This should not have been downvoted. It's true that the GIL does not make python code thread-safe implicitly, you have to either construct your code carefully to be atomic (based on knowledge of how the GIL works) or make use of mutexes, semaphores, etc. It's just memory-safe and can still have races etc.

Re: The first year of free-threaded Python

#160

Post by henry700 »

I find it peculiar how, in a language so riddled with simple concurrency architectural issues, the approach is to painstankingly fix every library after fixing the runtime, instead of just using some better language. Why does the community insist on such a bad language when literally even fucking Javascript has a saner execution model?
Post reply on HN