Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's up, Python? The GIL removed, a new compiler, optparse deprecated

bitecode.dev

Post reply on HN
Print view View original

151–160 of 302 posts

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#151

Post by quickthrower2 »

doctoboggan wrote: post #148
quickthrower2 wrote: post #145

Earlier quoted context omitted.

But you need to pick your horse. In 5 years time, Python will either be GIL or no GIL, and it is hard to tell which. It might be a setting (which might be more ideal). If you assume nogil, you need to choose dependencies that support that. You may need to trade off: eschew dependencies that aren't looking like they will be nogil compatible by the deadline. You are stuck on Python 3.18 maintenance branch or whatever,…

I don’t really understand this. Unless I am missing something you should always pick the “no GIL” version as that will work with or without a GIL. Thread safe No GIL code would be totally fine to run on python compiled with the GIL with zero modifications. Because of this I don’t expect there to be multiple versions of any library. Once a library does the (admittedly heavy) lift to no GIL it will just be the main ver…

Each library maintainer (probably mostly volunteers) has to decide whether to put effort into making their code thread safe. Clearly it won't be 100% of libraries that "upgrade".

Then on top of that, they know their effort might be for nothing if the decision is made to keep Python GIL-only all along (one of the possible 3 outcomes at the end of the 5 years: ["gil", "nogil", "both supported").

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#152

Post by empthought »

KolenCh wrote: post #138
empthought wrote: post #120

Earlier quoted context omitted.

You may be interested in the concurrent.futures library, available for over a decade now. It keeps you from shooting yourself in the foot like that. https://docs.python.org/3/library/concurrent.futures.html

Why do you think it would help? It provides a nice interface but is using multiprocessing or multi threading under the hood depending on which executioner you use: > The ProcessPoolExecutor class is an Executor subclass that uses a pool of processes to execute calls asynchronously. ProcessPoolExecutor uses the multiprocessing module, which allows it to side-step the Global Interpreter Lock but also means that only pi…

Your trouble seems to involve not understanding how to set up signal handlers, which ProcessPoolExecutor handles for you and exposes via a BrokenProcessPool exception.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#153

Post by Jtsummers »

wtetzner wrote: post #142
dragonwriter wrote: post #136

Earlier quoted context omitted.

Because at the time of the 2-3 migration, parallelism wasn’t viewed as being as important as it is today.

Is that really true? We already had multicore machines, and Herb Sutter's "The free lunch is over" article had been published for years by then.

> Herb Sutter's "The free lunch is over" article had been published for years by then.

Python 3.0 was released in 2008 with the work starting in early 2006 (maybe earlier, going by PEP 3000 which was published in April 2006). Herb Sutter's "The Free Lunch is Over" was first published in 2005. I don't think a year between its publishing and the work on Python 3 beginning qualifies as "years".

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#154

Post by doctoboggan »

quickthrower2 wrote: post #151
doctoboggan wrote: post #148

Earlier quoted context omitted.

I don’t really understand this. Unless I am missing something you should always pick the “no GIL” version as that will work with or without a GIL. Thread safe No GIL code would be totally fine to run on python compiled with the GIL with zero modifications. Because of this I don’t expect there to be multiple versions of any library. Once a library does the (admittedly heavy) lift to no GIL it will just be the main ver…

Each library maintainer (probably mostly volunteers) has to decide whether to put effort into making their code thread safe. Clearly it won't be 100% of libraries that "upgrade". Then on top of that, they know their effort might be for nothing if the decision is made to keep Python GIL-only all along (one of the possible 3 outcomes at the end of the 5 years: ["gil", "nogil", "both supported").

That’s true. I was more thinking from the perspective of a library user not library dev. I suspect for some classes of problem going no GIL will be so tantalizing that the work will definitely be done. Either in the incumbent library or an upstart will come out and take over the community with no GIL support.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#155

Post by nine_k »

dathinab wrote: post #37
threatripper wrote: post #35

Earlier quoted context omitted.

You have a single big data structure that can't be shared easily between multiple processes. Can't you use multiprocessing with that? Maybe mapping the data structure to a file and mmapping that in multiple processes? Maybe wrapping the whole thing in database instead of just using one huge nested dictionary? To me multi-threading sounds so much less painful than all the alternatives that I could imagine. Just adding…

> You have a single big data structure that can't be shared easily between multiple processes. Can't you use multiprocessing with that? Maybe mapping the data structure to a file and mmapping that in multiple processes? Maybe wrapping the whole thing in database instead of just using one huge nested dictionary? ton of additional complexity, not worth it for many use-cases and anything on the line of "using multiple p…

How good is support for numpy / scipy / pandas or equivalents, if they exist, outside Python?

Actually the resulting structure should of course be dumped into an RDBMS or a graph DB and served from there more readily. Doing that takes skill and time though, which often are worth applying elsewhere.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#156

Post by throwaway2037 »

OkGoDoIt wrote: post #133
dekhn wrote: post #61

Earlier quoted context omitted.

Over quite some time I've become convinced multiprocessing module is better than an optional GIL removal. It may leave many useful bits on the table (compared to pure multithreaded coding, like C++/pthreads) but I've still been able to get it to scale my application performance (CPU-bound, large-memory) to the number of cores of even large boxes (96+ vCPUs). IIRC the future/concurrent library was key to being product…

Whatever happened to ironpython? I used to do a lot of C# development and remember dabbling with ironpython back in the day. It seemed like it was important to Microsoft, .Net added the whole concept of dynamic data types mostly to support ironpython and ironruby. But I never really used python much until recently, so of course when I finally needed to do python I looked for ironpython and it doesn’t appear to be a t…

It looks like Microsoft abandoned these dynamic language implementations in 2010. Maintaining parallel implementations of two complex, mature scripting languages is a huge feat. It would take some very expensive talent. That said, IronPython was loved by those who used it, which means it captured them in the DotNet ecosystem. Perhaps that win was not enough for Microsoft to continue the project. Ideally, Python foundation should "own" (and fund) Jython and IronPython development, but that takes (a lot of) money. (Sorry, I'm much less familiar with Ruby and IronRuby.)

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#157

Post by goatlover »

strictfp wrote: post #59
wmwmwm wrote: post #14

Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed somethin…

My tip for this is Node.js and some stream processing lib like Highland. You can get ridiculous IO parallelism with a very little code and a nice API. Python just scales terribly, no matter if you use multi-process or not. Java can get pretty good perf, but you'll need some libs or quite a bit of code to get nonblocking IO sending working well, or you're going to eat huge amounts of resources for moderate returns. No…

Wouldn't Elixir or Go be better for this use case? Node still blocks on compute heavy tasks.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#158

Post by dragonwriter »

wtetzner wrote: post #142
dragonwriter wrote: post #136

Earlier quoted context omitted.

Because at the time of the 2-3 migration, parallelism wasn’t viewed as being as important as it is today.

Is that really true? We already had multicore machines, and Herb Sutter's "The free lunch is over" article had been published for years by then.

> Is that really true?

For Python by Guido (this was still in the BDfL era)? Yes. For scripting languages generally? Also yes. For computing as a whole? While parallelism was more important than in either of the preceding contexts, it was still far less important than today, so, again, yes.

> We already had multicore machines, and Herb Sutter's "The free lunch is over" article had been published for years by then.

Barely, unless you are talking about multi-CPU SMP machines (which existed for PCs back to the 386 era); the first multicore x86 processors were released in May 2005 (And Sutter’s article was in March) about a year before the major decisions about Python 3.0 (published April 2006.)

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#159

Post by mattip »

metadat wrote: post #137
TillE wrote: post #123

Earlier quoted context omitted.

Well, we already have a mature, real-world Python JIT in PyPy, with impressive performance. I dunno if Python is ever gonna be as fast as Java or C#, but we know it can be much better.

I can't find any benchmarks of PyPy vs OpenJDK or GraalVM, but unless I'm mistaken it's still more than 100% difference, and maybe much, much more for pure-Python vs. Java.

Here ya go. On these sometimes one is faster, sometimes the other. https://github.com/kostya/jit-benchmarks/blob/master/README.... Personally i don’t like such comparisons. Benchmarking is hard and far from objective. Much of what makes python popular is the developer experience. Generic benchmarks will only give a rough guide about what to expect in your application. If you are in a niche like the OP, you will have to figure out how to handle your bottlenecks.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#160

Post by metadat »

mattip wrote: post #159
metadat wrote: post #137

Earlier quoted context omitted.

I can't find any benchmarks of PyPy vs OpenJDK or GraalVM, but unless I'm mistaken it's still more than 100% difference, and maybe much, much more for pure-Python vs. Java.

Here ya go. On these sometimes one is faster, sometimes the other. https://github.com/kostya/jit-benchmarks/blob/master/README.... Personally i don’t like such comparisons. Benchmarking is hard and far from objective. Much of what makes python popular is the developer experience. Generic benchmarks will only give a rough guide about what to expect in your application. If you are in a niche like the OP, you will have…

Eagerly awaiting no-Gil Flask vs. Dropwizard performance analysis.
Post reply on HN