Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's up, Python? The GIL removed, a new compiler, optparse deprecated

bitecode.dev

Post reply on HN
Print view View original

261–270 of 302 posts

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#261

Post by da39a3ee »

IshKebab wrote: post #178
xcv123 wrote: post #149

Earlier quoted context omitted.

Pedantic nerd nitpick: it gives you concurrency but not parallelism. (Concurrent threads can be time sliced on one core)

It was clear from the context that he meant concurrently running not concurrently in progress. I wish nerds would give up on this parallelism/concurrency pedantry or at least choose some new nomenclature that didn't conflict so massively with the English meaning of "concurrent". I mean it's not even right. Most parallel/concurrent pedants would consider multithreaded code to be "parallel" even if it is running on a s…

> I mean it's not even right. Most parallel/concurrent pedants would consider multithreaded code to be "parallel" even if it is running on a single core.

Hm, I think running on a single core is the exact definition of what the "pedants" say is not parallel. If all you have is one core then you can't achieve parallelism under their definition.

I think the terminology is pretty well established now. But I do agree with you that it's a bad choice of words and that it's annoying, intelligent even, for people to pick a particular unintuitive definition and then go around brow-beating people for not using/understanding their definition.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#262

Post by zbentley »

da39a3ee wrote: post #258
zbentley wrote: post #250

Earlier quoted context omitted.

> every process keeps its own copy of hundreds of gigabytes of stuff. May be okay, depending on how many processes you spawn That depends on how you're using multiprocessing. If you're using the "spawn" multiprocessing-start method (which was set to the default on MacOS a few years ago[1], unfortunately), then every process re-starts python from the beginning of your program and does indeed have its own copy of anyth…

Is what you're describing only true of the "Framework" Python build on MacOS? It sounds like that's the case from a quick read of the issue you linked. I would say that people should basically never use the "Framework" Python on MacOS. (There's some insanity IIRC where matplotlib wants you to use the Framework build? But that's matplotlib)

> Is what you're describing only true of the "Framework" Python build on MacOS?

No. This behavior is present on any Python 3.8 or greater running on MacOS, enforced via "platform == darwin" runtime check: https://github.com/python/cpython/pull/13626/files#diff-6836...

You can check the default process-start method of your Python's multiprocessing by running this command: "python -c 'import multiprocessing; print(multiprocessing.get_start_method())'"

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#263

Post by jjoonathan »

akvadrako wrote: post #229
jjoonathan wrote: post #81

Earlier quoted context omitted.

Yep, multiprocessing is a cope. If processes were a universal substitute for threads we wouldn't have threads. That reasoning only gets stronger when you apply python's heavy limitations, but it gets the most strength when you experience the awkwardness of multiprocessing firsthand.

There isn't much difference on Linux between threads and processes that share memory. Multiprocessing is fine, it's just slightly more isolated threads.

That's why I took special care to mention how python's multiprocessing module was particularly poor.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#264

Post by thiht »

wodenokoto wrote: post #140

When it was an in dev project, I felt the consensus on HN was that it was amazing work and a shame that it looked like the steering committee wouldn’t adopt it. Now they have and everyone seems to hate it.

Almost as if there was more than 1 person on the internet

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#265

Post by thiht »

fbdab103 wrote: post #11

Still not encouraged by the no-GIL, "We don't want another Python 2->3 situation", yet very little proffered on how to avoid that scenario. More documentation on writing thread-safe code, suggested tooling to lint for race conditions (for whatever it is worth), discussions with popular C libraries, dedicated support channels for top tier packages, what about the enormous long-tail of abandoned extensions which still…

> Note that if the program imports one single C-extension that uses the GIL on the no-GIL build, it's designed to switch back to the GIL automatically. So this is not a 2=>3 situation where non-compatible code breaks.

Sounds good enough to me, am I missing something?

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#266

Post by thiht »

swyx wrote: post #127
BiteCode_dev wrote: post #2

Summary: - Python without the GIL, for good - LPython: a new Python Compiler - Pydantic 2 is getting usable - PEP 387 defines "Soft Deprecation", getopt and optparse soft deprecated - Cython 3.0 released with better pure Python support - PEP 722 – Dependency specification for single-file scripts - Python VSCode support gets faster - Paint in the terminal

great recap thanks!

It's literally the summary at the top of the article

Doesn't anyone click on links anymore?

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#267

Post by kzrdude »

schemescape wrote: post #51

The title says "GIL removed", but the article says "This means in the coming years, Python will have its GIL removed." I'm assuming the article is correct and the GIL has not been removed yet (but there is a plan to remove it in the future). If that's not the case, please correct me!

There's been an announcement that they are probably going to decide to start a development plan that can eventually lead to removing the GIL later, if it works out.

That plan is called PEP 703 and this is the factual basis: "We intend to accept PEP 703, although we’re still working on the acceptance details."

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#268

Post by pjmlp »

__d wrote: post #249
pjmlp wrote: post #210

Earlier quoted context omitted.

We can add Smalltalk, SELF, Dylan, JavaScript into the discussion then.

And maybe Strongtalk

Kind of, I left it out on purpose, as it was designed with strong typing in mind, and I only wanted to list dynamic languages with good JIT support.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#269

Post by danpalmer »

kayodelycaon wrote: post #260
danpalmer wrote: post #243

Earlier quoted context omitted.

These don't really apply to the parent commenter's scenario. 1) gunicorn or any solution with multiple processes is going to just multiply the RAM usage. Using 10-100GB of RAM per effective thread makes this sort of problem very RAM bound, to the point that it can be hard to find hardware or VM support. 2) This isn't I/O bound. 3) If your service is fundamentally just looking up data in a huge in-memory data store, a…

For #1, would copy on write help? Or does python store the counters on the objects?

Ha! Yes! Unfortunately I know this because of terrible reasons. Python is reference counted so copy-on-write doesn't work for this with Python objects (note: if your Python object is actually just a reference to a native object in a library all bets are off, may work or may not).

We had an issue with the service I mentioned above where VMs with ~6GB RAM weren't working, because at the point that gunicorn forked there was instantaneously >10GB RAM usage because everything got copied. We had to make sure that the data file was only loaded after the daemon fork, which unfortunately limits the benefits of that fork, part of the idea is that you do all your setup before forking so that you know you've started cleanly.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#270

Post by swyx »

thiht wrote: post #266
swyx wrote: post #127

Earlier quoted context omitted.

great recap thanks!

It's literally the summary at the top of the article Doesn't anyone click on links anymore?

just trying to compliment the author on a useful blogpost, calm down.
Post reply on HN