Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's up, Python? The GIL removed, a new compiler, optparse deprecated

bitecode.dev

Post reply on HN
Print view View original

271–280 of 302 posts

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#271

Post by mardifoufs »

kzrdude wrote: post #206
nickcw wrote: post #74

Earlier quoted context omitted.

I've tried to love argparse but it is so complicated. I always have to read the docs each time I use it. getopt has its own brutal simplicity.

I think argparse works fine. What worries me is that it's also "soft-deprecated", because devs have said it should get no further development. I hope it stays around, because I use it by default as a no-dependencies solution that I know how it works.

Wait.. source? If argparse isn't getting more development, what's the current alternative?

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#272

Post by [deleted] »

mardifoufs wrote: post #271
kzrdude wrote: post #206

Earlier quoted context omitted.

I think argparse works fine. What worries me is that it's also "soft-deprecated", because devs have said it should get no further development. I hope it stays around, because I use it by default as a no-dependencies solution that I know how it works.

Wait.. source? If argparse isn't getting more development, what's the current alternative?

[deleted]

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#273

Post by quietbritishjim »

gjulianm wrote: post #259
woodrowbarlow wrote: post #246

Earlier quoted context omitted.

any existing async/await code.

async/await code already runs in threads, so that's not really a change.

What do you mean it already runs in threads? It does so if you specify it with run_in_executor [1], or if you run multiple event loops at once, but it doesn't automatically.

[1] https://docs.python.org/3/library/asyncio-eventloop.html#asy...

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#274

Post by quietbritishjim »

gjulianm wrote: post #236
crabbone wrote: post #215

Earlier quoted context omitted.

> your average python dev can just ignore it if they want to. Oh, so naive... All the mutation code in Python which "worked" because Python didn't really have any real concurrency. Add to it -- there's no real plan about what to do with Python concurrency. Removing GIL is only one "half" of the problem, you need to give developers some sort of a framework to use to deal with concurrency. Python's threads are extremel…

Which code is automatically going to run in threads? As you say, basically nobody uses Python threads. So even enabling no-gil, nothing is going to change because sequential code will still be sequential.

> basically nobody uses Python threads

Not true at all. Plenty of people (including me) use threads in Python for:

* Blocking I/O

* CPU heavy libraries written in C (as those release the GIL)

They work fine, even with the GIL. They only work badly if you want to run a lot of pure-Python (non-I/O) code in multiple threads - which, fair enough, sometimes you might want to do, and the GIL is a problem for that.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#275

Post by gjulianm »

quietbritishjim wrote: post #273
gjulianm wrote: post #259

Earlier quoted context omitted.

async/await code already runs in threads, so that's not really a change.

What do you mean it already runs in threads? It does so if you specify it with run_in_executor [1], or if you run multiple event loops at once, but it doesn't automatically. [1] https://docs.python.org/3/library/asyncio-eventloop.html#asy...

Woops, yeah, you're right. I thought the default executor was a threadpool, my mistake. However, in that case, I assume that the default executor will not change to multithreaded when no-gil comes.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#276

Post by masklinn »

tgv wrote: post #196
School-Cotton wrote: post #15

Earlier quoted context omitted.

I feel like orchestrating thousands of GPUs is such a niche use case that it’s fair to expect the people wanting to do it to learn a more suited language, rather than ruining Python for everyone else.

You are completely right. Why don't they write their stuff in another language? They've got the resources. Now the rest of the world will suffer the consequences, one of which may be that the devs of native libs will simply abandon the work, or that those libs will become too difficult to use for the casual or starting programmer, completely defeating the purpose. I'm fine with two builds, but not a single non-GIL bu…

> Now the rest of the world will suffer the consequences, one of which may be that the devs of native libs will simply abandon the work, or that those libs will become too difficult to use for the casual or starting programmer, completely defeating the purpose.

Or get the benefits, so casual or starting programmers won't be wondering why their python program refuses to go above 100% CPU, or have to deal with the bullshit of multiprocessing.

> I'm fine with two builds, but not a single non-GIL build.

The "no-GIL" build has GIL machinery included, it just runs with the GIL disabled by default. You can force it on (https://peps.python.org/pep-0703/#pythongil-environment-vari...), and it will automatically enable itself when loading a non-no-gil library (https://peps.python.org/pep-0703/#py-mod-gil-slot).

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#277

Post by alfalfasprout »

chlorion wrote: post #119
alfalfasprout wrote: post #93

Earlier quoted context omitted.

Nowadays multiprocessing is rarely the answer. Between all the gotchas (memory usage can be horrific, have to be careful what you modify, etc.) it's almost never the right answer. Nowadays numba is usually a better solution for when you want to run some computationally expensive python code that itself calls numpy, etc. For the parent commenter's use case though that wouldn't be a great solution either. In general, P…

You have to be much more careful about what you modify when using multithreading, so I'm not sure what you mean by that. A lot of people here mention that sharing data is much easier with multithreading, but doing this without races is not easy. You can't just use the values from difference threads like you would in normal code, you need to synchronize access with locks, which can be difficult to do correctly and can…

Not sure why this is directed at my comment-- I didn't touch on synchronization.

Yes, locks like mutexes, semaphores, etc. and approaches like atomics, lockfree datastructures come into play when writing multithreaded code. There's no getting around that.

> In my experience, most problems are inherently synchronous with lots of mutable state and complex data dependencies, or inherently parallel with lots of tasks that can run independently. Problems that can be easily parallelized already work fine with multiprocessing!

This is a hot take though-- most problems that are truly embarrassingly parallel don't work as well as you'd think w/ multiprocessing. There's a ton of overhead there and when you do need synchronization steps (eg; in reductions) it can get pretty messy.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#278

Post by vorticalbox »

wmwmwm wrote: post #14

Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed somethin…

Why not load the data into sqlite dB and let the clients query that? Is there a reason you're loading 10s/100s gb into memory?

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#279

Post by mixmastamyk »

kzrdude wrote: post #186
mixmastamyk wrote: post #179

Earlier quoted context omitted.

The guy who was smart enough/motivated to do it showed up only a couple of years ago.

That guy shows up every couple of years, almost like a prophecy https://github.com/larryhastings/gilectomy

Only Mr. Gross succeeded, the others are not relevant.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#280

Post by tgv »

masklinn wrote: post #276
tgv wrote: post #196

Earlier quoted context omitted.

You are completely right. Why don't they write their stuff in another language? They've got the resources. Now the rest of the world will suffer the consequences, one of which may be that the devs of native libs will simply abandon the work, or that those libs will become too difficult to use for the casual or starting programmer, completely defeating the purpose. I'm fine with two builds, but not a single non-GIL bu…

> Now the rest of the world will suffer the consequences, one of which may be that the devs of native libs will simply abandon the work, or that those libs will become too difficult to use for the casual or starting programmer, completely defeating the purpose. Or get the benefits, so casual or starting programmers won't be wondering why their python program refuses to go above 100% CPU, or have to deal with the bull…

That might have been the original intention. The latest notice from the SC says:

Our base assumptions are:

* Long-term (probably 5+ years), the no-GIL build should be the only build. We do not want to create a permanent split between with-GIL and no-GIL builds (and extension modules).

They repeat it later. It looks as if they really want to remove it.

> have to deal with the bullshit of multiprocessing

The problems multi-threading introduces outweigh that by far.

Post reply on HN