Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Free-threaded CPython is ready to experiment with

labs.quansight.org

Post reply on HN
Print view View original

91–100 of 398 posts

Re: Free-threaded CPython is ready to experiment with

#91

Post by wongarsu »

phkahler wrote: post #25
eigenvalue wrote: post #12

Really excited for this. Once some more time goes by and the most important python libraries update to support no GIL, there is just a tremendous amount of performance that can be automatically unlocked with almost no incremental effort for so many organizations and projects. It's also a good opportunity for new and more actively maintained projects to take market share from older and more established libraries if th…

I feel like most things that will benefit from moving to multiple cores for performance should probably not be written in Python. OTH "most" is not "all" so it's gonna be awesome for some.

I often reach for python multiprocessing for code that will run $singleDigit number of times but is annoyingly slow when run sequentially. I could never justify the additional development time for using a more performant language, but I can easily justify spending 5-10 minutes making the embarrassingly parallel stuff execute in parallel.

Re: Free-threaded CPython is ready to experiment with

#92

Post by hot_gril »

BugsJustFindMe wrote: post #78
hot_gril wrote: post #72

Earlier quoted context omitted.

ThreadPoolExecutor is the most similar thing to asyncio: It hands out promises, and when you call .result(), it's the same as await. JS even made its own promises implicitly compatible with async/await. I'm mentioning what JS does because you're describing a very common JS use case, and Python isn't all that different. If you have async stuff happening all over the place, what do you use, a global ThreadPoolExecutor?…

> Python isn't all that different Again, Python's implementation of asyncio does not allow you to background worker code without explicitly altering that worker code to be aware of asyncio. Threads do. They just don't occupy the same space. > Also, I was originally asking about free threads...there's multiprocessing Eh, the obvious reason to not want to use separate processes is a desire for some kind of shared state…

Basically true in JS too. You're not supposed to do blocking calls in async code. You also can't "await" an async call inside a non-async func, though you could fire-and-forget it.

Right, but how often does a Python program have complex shared state across threads, rather than some simple fan-out-fan-in, and also need to take advantage of multiple cores?

Re: Free-threaded CPython is ready to experiment with

#93

Post by adgjlsfhk1 »

__MatrixMan__ wrote: post #83
jolux wrote: post #9

Earlier quoted context omitted.

The successful languages without efficient dependency management are painful to manage dependencies in, though. I think Python should be shooting for a better package management user experience than C++.

Python's dependency management sucks because they're audacious enough to attempt packaging non-python dependencies. People always bring Maven up as a system that got it right, but Maven only does JVM things. I think the real solution here is to just only use python dependency management for python things and to use something like nix for everything else.

Julia's package manager (for one) works great and can manage non Julia packages. the problem with python's system is that rejecting semver makes writing a package manager basically impossible since there is no way to automatically resolve packages.

Re: Free-threaded CPython is ready to experiment with

#94

Post by geekone »

stavros wrote: post #85
teaearlgraycold wrote: post #70

Earlier quoted context omitted.

It's kept behind a flag. Hopefully will be forever.

Oh, very interesting, that's a great solution then.

Looks like according to the PEP it may eventually be default in 4-6 releases down the road: https://peps.python.org/pep-0703/#python-build-modes

Re: Free-threaded CPython is ready to experiment with

#95

Post by arp242 »

vhcr wrote: post #86
arp242 wrote: post #56

Earlier quoted context omitted.

I guess that depends from your perspective. I'm not a Python developer, but like many people I do want to run Python programs from time to time. I don't really know Rust, or Cargo, but I never have trouble building any Rust program: "cargo build [--release]" is all I need to know. Easy. Even many C programs are actually quite easy: "./configure", "make", and optionally "make install". "./configure" has a nice "--help…

Do you have a problem with Node.js too because it creates a node_modules folder, or is the problem that it is not handled automatically?

I don't care about the internals. I care about "just" being able to run it.

I find that most JS projects work fairly well: "npm install" maybe followed by "npm run build" or the like. This isn't enforced by npm and I don't think npm is perfect here, but practical speaking as a non-JS dev just wanting to run some JS projects: it works fairly well for almost all JS projects I've wanted to run in the last five years or so.

A "run_me.py" that would *Just Work™" is fine. I don't overly care what it does internally as long as it's not hugely slow or depends on anything other than "python". Ideally this should be consistent throughout the ecosystem.

To be honest I can't imagine shipping any project intended to be run by users and not have a simple, fool-proof, and low-effort way of running it by anyone of any skill level, which doesn't depend on any real knowledge of the language.

Re: Free-threaded CPython is ready to experiment with

#96

Post by metadat »

teaearlgraycold wrote: post #70
stavros wrote: post #68

Earlier quoted context omitted.

Well, I'm not looking forward to the day when I upgrade my Python and suddenly I have to debug a ton of fun race conditions.

It's kept behind a flag. Hopefully will be forever.

The article states the goal is to eventually (after some years of working out the major kinks and performance regressions) promote Free-Threaded Python to be the default cPython distribution.

Re: Free-threaded CPython is ready to experiment with

#97

Post by dagenix »

stavros wrote: post #68
ameliaquining wrote: post #31

Earlier quoted context omitted.

The semantic changes are negligible for authors of Python code. All the complexity falls on the maintainers of the CPython interpreter and on authors of native extension modules.

Well, I'm not looking forward to the day when I upgrade my Python and suddenly I have to debug a ton of fun race conditions.

As I understand it, if your code would have race conditions with free threaded python, than it probably already has them.

Re: Free-threaded CPython is ready to experiment with

#98

Post by [deleted] »

hot_gril wrote: post #77
nine_k wrote: post #76

Earlier quoted context omitted.

I used to think like that until I tried. There are areas where typing is more important: public interfaces. You don't have to make every piece of your program well-typed. But signatures of your public functions / methods matter a lot, and from them types of many internal things can be inferred. If your code has a well-typed interface, it's pleasant to work with. If interfaces of the libraries you use are well-typed,…

I shouldn't have said it's a waste of time period, cause every project I work on does have static typing in two very important places: the RPC or web API (OpenAPI, gRPC, whatever it is), and the relational database. But not in the main JS or Py code. That's all I've ever needed. I did try migrating a NodeJS backend to TS along with a teammate driving that effort. The type-checking never ended up catching any bugs, an…

[deleted]

Re: Free-threaded CPython is ready to experiment with

#99

Post by rty32 »

MBCook wrote: post #51
phkahler wrote: post #25

Earlier quoted context omitted.

I feel like most things that will benefit from moving to multiple cores for performance should probably not be written in Python. OTH "most" is not "all" so it's gonna be awesome for some.

But it would give you more headroom before rewriting for performance would make sense right? That alone could be beneficial to a lot of people.

I think it is beneficial to some people, but not a lot. My guess is that most Python users (from beginners to advanced users, including many professional data scientists) have never heard of GIL or thought of doing any parallelization in Python. Code that needs performance and would benefit from multithreading, usually written by professional software engineers, likely isn't written in Python in the first place. It would make sense for projects that can benefit from disabling GIL without a ton of changes. Remember it is not trivial to update single threaded code to use multithreading correctly.

in Python language specifically. Their library may have already done some form of parallelization under the hood

Re: Free-threaded CPython is ready to experiment with

#100

Post by kristjansson »

BugsJustFindMe wrote: post #66
kristjansson wrote: post #65

Earlier quoted context omitted.

You don’t? concurrent.futures.ThreadPoolExecutor can get a lot done without touching async code.

I am a big advocate for ThreadPoolExecutor. I'm saying it's superior to asyncio. The person I'm responding to was asking why use threads when you can use asyncio instead.

Ach, I posted before I saw the rest of your thread, apologies.

Totally agree, concurrent.futures strikes a great balance. Enough to get work done, a bit more constrained than threads on their own.

Asyncio is a lot of cud to chew if you just want a background task in an otherwise sync application

Post reply on HN