Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Free-threaded CPython is ready to experiment with

labs.quansight.org

Post reply on HN
Print view View original

341–350 of 398 posts

Re: Free-threaded CPython is ready to experiment with

#341

Post by Uptrenda »

wokwokwok wrote: post #238
eigenvalue wrote: post #12

Really excited for this. Once some more time goes by and the most important python libraries update to support no GIL, there is just a tremendous amount of performance that can be automatically unlocked with almost no incremental effort for so many organizations and projects. It's also a good opportunity for new and more actively maintained projects to take market share from older and more established libraries if th…

> there is just a tremendous amount of performance that can be automatically unlocked with almost no incremental effort for so many organizations and projects This just isn’t true. This does not improve single threaded performance (it’s worse) and concurrent programming is already available. This will make it less annoying to do concurrent processing. It also makes everything slower (arguable where that ends up, curr…

Why would it make single threaded performance slower? Sorry, but that's kind of ridiculous. You're just making shit up at this point.

Re: Free-threaded CPython is ready to experiment with

#342

Post by rbanffy »

Certhas wrote: post #260
Derbasti wrote: post #191

Earlier quoted context omitted.

A thought experiment: A piece of code takes 6h to develop in C++, and 1h to run. The same algorithm takes 3h to code in Python, but 6h to run. If I could thread-spam that Python code on my 24 core machine, going Python would make sense. I've certainly been in such situations a few times.

C++ and python are not the only options though. Julia is one that is gaining a lot of use in academia, but any number of modern, garbage collected compiled high level languages could probably do.

Fair. Add a couple hours to learn enough Julia to write the code.

Re: Free-threaded CPython is ready to experiment with

#343

Post by cozzyd »

vhcr wrote: post #90
GTP wrote: post #46

Earlier quoted context omitted.

Then we have very different ideas of what proper typing is :D Look at this function, can you tell me what it does? def plus(x, y): return x+y If your answer is among the lines of "It returns the sum x and y" then I would ask you who said that x and y are numbers. If these are strings, it concatenates them. If instead you pass a string and a number, you will get a runtime exception. So not only you can't tell what a f…

In theory it's nice that the compiler would catch those kinds of problems, but in practice it doesn't matters.

It's very hard to write long-running daemons in python partially for this reason, when you make a typo on a variable or method name in an uncommon code path.

Re: Free-threaded CPython is ready to experiment with

#344

Post by skissane »

lyu07282 wrote: post #324
skissane wrote: post #237

Earlier quoted context omitted.

Just the other day I was trying to do two things in parallel in Python using threads - and then I switched to multiprocessing - why? I wanted to immediately terminate one thing whenever the other failed. That’s straightforwardly supported with multiprocessing. With threads, it gets a lot more complicated and can involve things with dubious supportability

There is a reason why it's "complicated" in threads, because doing it correctly just IS complicated, and the same reason applies to child processes, you just ignored that reason. That's one example of a footgun in using multiprocessing, people write broken code but they don't know that because it appears to work... until it doesn't (in production on friday night).

I don't agree. A big reason why abruptly terminating threads at an arbitrary point is risky is it can corrupt shared memory. If you aren't using shared memory in a multiprocess solution, that's not an issue. Another big reason is it can lead to resource leaks (e.g. thread gets terminated in a finally clause to close resources and hence the resource doesn't get closed). Again, that's less of an issue for processes, since many resources (file descriptors, network connections) get automatically closed by the OS kernel when the process exits.

Abruptly terminating a child process still can potentially cause issues, but there are whole categories of potential issues which exist for abrupt thread termination but not for abrupt process termination.

Re: Free-threaded CPython is ready to experiment with

#345

Post by QkdhagA »

Uptrenda wrote: post #341
wokwokwok wrote: post #238

Earlier quoted context omitted.

> there is just a tremendous amount of performance that can be automatically unlocked with almost no incremental effort for so many organizations and projects This just isn’t true. This does not improve single threaded performance (it’s worse) and concurrent programming is already available. This will make it less annoying to do concurrent processing. It also makes everything slower (arguable where that ends up, curr…

Why would it make single threaded performance slower? Sorry, but that's kind of ridiculous. You're just making shit up at this point.

What is "it"?

If you assume two completely separate implementations where there is an #ifdef every 10 lines and atomics and locking only occur with --disable-gil, there is no slowdown for the --enable-gil build.

I don't think that is entirely the case though!

If the --enable-gil build becomes the default in the future, then peer pressure and packaging discipline will force everyone to use it. Then you have the OBVIOUS slowdown of atomics and of locking the reference counting and in other places.

The advertised figures were around 20%, which would be offset by minor speedups in other areas. But if you compare against Python 3.8, for instance, the slowdowns are still there (i.e., not offset by anything). Further down on the second page of this discussion numbers of 30-40% have been measured by the submitter of this blog post.

Actual benchmarks of Python tend to be suppressed or downvoted, so they are not on the first page. The Java HotSpot VM had a similar policy that forbid benchmarks.

Re: Free-threaded CPython is ready to experiment with

#346

Post by mixmastamyk »

blumomo wrote: post #273
vegabook wrote: post #236

Clearly the Python 2 to 3 war was so traumatising (and so badly handled) that the core Python team is too scared to do the obvious thing, and call this Python 4. This is a big fundamental and (in many cases breaking) change, even if it's "optional".

Did Python as the language change which justified that version bump?

When on, there are incompatibilities yes.

There were a lot of smaller breaking changes over the years, especially 3.10 that probably should have been a 4.0.

Re: Free-threaded CPython is ready to experiment with

#347

Post by [deleted] »

eigenvalue wrote: post #12

Really excited for this. Once some more time goes by and the most important python libraries update to support no GIL, there is just a tremendous amount of performance that can be automatically unlocked with almost no incremental effort for so many organizations and projects. It's also a good opportunity for new and more actively maintained projects to take market share from older and more established libraries if th…

[deleted]

Re: Free-threaded CPython is ready to experiment with

#348

Post by coldtea »

eigenvalue wrote: post #12

Really excited for this. Once some more time goes by and the most important python libraries update to support no GIL, there is just a tremendous amount of performance that can be automatically unlocked with almost no incremental effort for so many organizations and projects. It's also a good opportunity for new and more actively maintained projects to take market share from older and more established libraries if th…

>using simple threads instead of dealing with the massive overhead and complexity and bugs of using something like multiprocessing

I've never heard threading described as "simple", even less so as simpler than multiprocessing.

Threads means synchronization issues, shared memory, locking, and other complexities.

Re: Free-threaded CPython is ready to experiment with

#349

Post by coldtea »

bausgwi678 wrote: post #249
pizza234 wrote: post #231

Earlier quoted context omitted.

> using simple threads instead of dealing with the massive overhead and complexity and bugs of using something like multiprocessing. Depending on the domain, the reality can be the reverse. Multiprocessing in the web serving domain, as in "spawning separate processes", is actually simpler and less bug-prone, because there is considerably less resource sharing. The considerably higher difficulty of writing, testing an…

Using multiple processes is simpler in terms of locks etc, but python libraries like multiprocessing or even subprocess.popen[1] which make using multiple processes seem easy are full of footguns which cause deadlocks due to fork-safe code not being well understood. I’ve seen this lead to code ‘working’ and being merged but then triggering sporadic deadlocks in production after a few weeks. The default for multiproce…

>which make using multiple processes seem easy are full of footguns which cause deadlocks due to fork-safe code not being well understood. I’ve seen this lead to code ‘working’ and being merged but then triggering sporadic deadlocks in production after a few weeks

Compared to theads being "pain free"?

Re: Free-threaded CPython is ready to experiment with

#350

Post by Galanwe »

imtringued wrote: post #318
Galanwe wrote: post #213

Earlier quoted context omitted.

Maybe I am biased, because I learned these things so long ago and I don't realize that it's a pain to learn. But what exactly is so confusing about virtualenvs ? They really not that different from any other packaging system like JS or Rust. The only difference is instead of relying on your current directory to find the the libraries / binaries (and thus requiring you to wrap binaries call with some wrapper to search…

Only python demands you to source an activation script before doing anything.

Yes, though just to illustrate that it's a matter of taste, I do prefer the solution of virtualenvs requiring to source a script that append to PATH, rather than a solution requiring the use of a wrapper that executes in its PATH.

I never remember how to run Javascript binaries. Is it npm run ? npm run script ? npx ? I always end up running the links in node_modules/bin

Post reply on HN