Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The first year of free-threaded Python

labs.quansight.org

Post reply on HN
Print view View original

41–50 of 302 posts

Re: The first year of free-threaded Python

#41

Post by bgwalter »

HDThoreaun wrote: post #36
jillesvangurp wrote: post #23

Earlier quoted context omitted.

You are not the only one who is afraid of changes and a bit change resistant. I think the issue here is that the reasons for this fear are not very rational. And also the interest of the wider community is to deal with technical debt. And the GIL is pure technical debt. Defensible 30 years ago, a bit awkward 20 years ago, and downright annoying and embarrassing now that world + dog does all their AI data processing w…

> But a simple solution to address your fears: simply don't use threads. You'll be fine. Im not worried about new code. Im worried about stuff written 15 years ago by a monkey who had no idea how threads work and just read something on stack overflow that said to use threading. This code will likely break when run post-GIL. I suspect there is actually quite a bit of it.

If it is C-API code: Implicit protection of global variables by the GIL is a documented feature, which makes writing extensions much easier.

Most C extensions that will break are not written by monkeys, but by conscientious developers that followed best practices.

Re: The first year of free-threaded Python

#42

Post by bayindirh »

HDThoreaun wrote: post #36
jillesvangurp wrote: post #23

Earlier quoted context omitted.

You are not the only one who is afraid of changes and a bit change resistant. I think the issue here is that the reasons for this fear are not very rational. And also the interest of the wider community is to deal with technical debt. And the GIL is pure technical debt. Defensible 30 years ago, a bit awkward 20 years ago, and downright annoying and embarrassing now that world + dog does all their AI data processing w…

> But a simple solution to address your fears: simply don't use threads. You'll be fine. Im not worried about new code. Im worried about stuff written 15 years ago by a monkey who had no idea how threads work and just read something on stack overflow that said to use threading. This code will likely break when run post-GIL. I suspect there is actually quite a bit of it.

Software rots, software tools evolve. When Intel released performance primitives libraries which required recompilation to analyze multi-threaded libraries, we were amazed. Now, these tools are built into processors as performance counters and we have way more advanced tools to analyze how systems behave.

Older code will break, but they break all the time. A language changes how something behaves in a new revision, suddenly 20 year old bedrock tools are getting massively patched to accommodate both new and old behavior.

Is it painful, ugly, unpleasant? Yes, yes and yes. However change is inevitable, because some of the behavior was rooted in inability to do some things with current technology, and as hurdles are cleared, we change how things work.

My father's friend told me that length of a variable's name used to affect compile/link times. Now we can test whether we have memory leaks in Rust. That thing was impossible 15 years ago due to performance of the processors.

Re: The first year of free-threaded Python

#43

Post by kfrane »

immibis wrote: post #24
rowanG077 wrote: post #18

Earlier quoted context omitted.

Does it? The GIL only ensured each interpreter instruction is atomic. But any group of instruction is not protected. This makes it very hard to rely on the GIL for synchronization unless you really know what you are doing.

AFAIK a group of instructions is only non-protected if one of the instructions does I/O. Explicit I/O - page faults don't count.

If I understand that correctly, it would mean that running a function like this on two threads f(1) and f(2) would produce a list of 1 and 2 without interleaving.

  def f(x):
      for _ in range(N):
          l.append(x)
I've tried it out and they start interleaving when N is set to 1000000.

Re: The first year of free-threaded Python

#44

Post by falcor84 »

pjmlp wrote: post #40

On the other news, Microsoft dumped the whole faster Python team, apparently the 2025 earnings weren't enough to keep the team around. https://www.linkedin.com/posts/mdboom_its-been-a-tough-coupl... Lets see whatever performance improvements still land on CPython, unless other company sponsors the work. I guess Facebook (no need to correct me on the name) is still sponsoring part of it.

It wouldn't have bothered me if you just said "Facebook" - I probably wouldn't have even noticed it. But I'm really curious why you chose to write "Facebook", then apparently noticed the issue, and instead of replacing it with "Meta" decided to add the much longer "(no need to correct me on the name)". What axe are you grinding?

Re: The first year of free-threaded Python

#46

Post by rich_sasha »

pjmlp wrote: post #40

On the other news, Microsoft dumped the whole faster Python team, apparently the 2025 earnings weren't enough to keep the team around. https://www.linkedin.com/posts/mdboom_its-been-a-tough-coupl... Lets see whatever performance improvements still land on CPython, unless other company sponsors the work. I guess Facebook (no need to correct me on the name) is still sponsoring part of it.

Ah that's very, very sad. I guess they have embraced and extended, there's only one thing left to do.

Re: The first year of free-threaded Python

#47

Post by bgwalter »

pjmlp wrote: post #40

On the other news, Microsoft dumped the whole faster Python team, apparently the 2025 earnings weren't enough to keep the team around. https://www.linkedin.com/posts/mdboom_its-been-a-tough-coupl... Lets see whatever performance improvements still land on CPython, unless other company sponsors the work. I guess Facebook (no need to correct me on the name) is still sponsoring part of it.

They were quite a bit behind the schedule that was promised five years ago.

Additionally, at this stage the severe political and governance problems cannot have escaped Microsoft. I imagine that no competent Microsoft employee wants to give his expertise to CPython, only later to suffer group defamation from a couple of elected mediocre people.

CPython is an organization that overpromises, allocates jobs to the obedient and faithful while weeding out competent dissenters.

It wasn't always like that. The issues are entirely self-inflicted.

Re: The first year of free-threaded Python

#48

Post by sgarland »

> Instead, many reach for multiprocessing, but spawning processes is expensive

Agreed.

> and communicating across processes often requires making expensive copies of data

SharedMemory [0] exists. Never understood why this isn’t used more frequently. There’s even a ShareableList which does exactly what it sounds like, and is awesome.

[0]: https://docs.python.org/3/library/multiprocessing.shared_mem...

Re: The first year of free-threaded Python

#49

Post by actinium226 »

HDThoreaun wrote: post #36
jillesvangurp wrote: post #23

Earlier quoted context omitted.

You are not the only one who is afraid of changes and a bit change resistant. I think the issue here is that the reasons for this fear are not very rational. And also the interest of the wider community is to deal with technical debt. And the GIL is pure technical debt. Defensible 30 years ago, a bit awkward 20 years ago, and downright annoying and embarrassing now that world + dog does all their AI data processing w…

> But a simple solution to address your fears: simply don't use threads. You'll be fine. Im not worried about new code. Im worried about stuff written 15 years ago by a monkey who had no idea how threads work and just read something on stack overflow that said to use threading. This code will likely break when run post-GIL. I suspect there is actually quite a bit of it.

If code has been unmaintained for more than a few years, it's usually such a hassle to get it working again that 99% of the time I'll just write my own solution, and that's without threads.

I feel some trepidation about threads, but at least for debugging purposes there's only one process to attach to.

Re: The first year of free-threaded Python

#50

Post by pjmlp »

falcor84 wrote: post #44
pjmlp wrote: post #40

On the other news, Microsoft dumped the whole faster Python team, apparently the 2025 earnings weren't enough to keep the team around. https://www.linkedin.com/posts/mdboom_its-been-a-tough-coupl... Lets see whatever performance improvements still land on CPython, unless other company sponsors the work. I guess Facebook (no need to correct me on the name) is still sponsoring part of it.

It wouldn't have bothered me if you just said "Facebook" - I probably wouldn't have even noticed it. But I'm really curious why you chose to write "Facebook", then apparently noticed the issue, and instead of replacing it with "Meta" decided to add the much longer "(no need to correct me on the name)". What axe are you grinding?

Yes, because I am quite certain someone without anything better to do would correct me on that.

For me Facebook will always be Facebook, and Twitter will always be Twitter.

Post reply on HN