Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's up, Python? The GIL removed, a new compiler, optparse deprecated

bitecode.dev

Post reply on HN
Print view View original

211–220 of 302 posts

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#211

Post by krylon »

pid-1 wrote: post #144
strictfp wrote: post #59

Earlier quoted context omitted.

My tip for this is Node.js and some stream processing lib like Highland. You can get ridiculous IO parallelism with a very little code and a nice API. Python just scales terribly, no matter if you use multi-process or not. Java can get pretty good perf, but you'll need some libs or quite a bit of code to get nonblocking IO sending working well, or you're going to eat huge amounts of resources for moderate returns. No…

Isn't Node single threaded, just like Python?

Python is technically multithreaded, but the GIL means only one thread can execute interpreter code at a time. If you use libraries written in C/C++, the library code can run in multiple threads simultaneously if they release the GIL.

I vaguely recall Node used to run multiple threads under the hood for disk I/O, but it might use kqueue/epoll these days.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#212

Post by pjmlp »

__d wrote: post #110
gpderetta wrote: post #105

Earlier quoted context omitted.

These days even elisp can be compiled. I think python need to be dragged kicking and screaming into cutting edge '80s dynamic compilation technology.

I'm sure skilled volunteers would be very welcome. There are numerous active, moderately serious efforts to both optimize and/or JIT Python bytecode. I think AOT compilation is mostly out-of-scope for 100% compatibility, but again, there's lots of different efforts to compile either subset languages or subsets of programs. "Kicking and screaming" suggests some reluctance to embrace this, but I think that's probably u…

It isn't as if PyPy doesn't exist. Embracing it during the 16 years of its existence is another matter.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#214

Post by krylon »

brightball wrote: post #66
antod wrote: post #46

Earlier quoted context omitted.

I wonder what lead to JRuby attracting support while Jython not? I know the Jython creator went on to other things (was it eg IronPython for dotnet?). I suppose it was the inverse with dotnet - eg IronPython surviving while IronRuby seems dead. Is it just down to corporate sponsorship?

JRuby has been pretty actively maintained for about 15 years and had a big release this year. It’s an impressive project.

I looked into it a long time ago (~10-12 years?), and was disappointed JRuby could not use extensions written in C. It's not surprising in retrospect, for obvious reasons, but has there been some progress in this area?

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#215

Post by crabbone »

doctoboggan wrote: post #31
fbdab103 wrote: post #11

Still not encouraged by the no-GIL, "We don't want another Python 2->3 situation", yet very little proffered on how to avoid that scenario. More documentation on writing thread-safe code, suggested tooling to lint for race conditions (for whatever it is worth), discussions with popular C libraries, dedicated support channels for top tier packages, what about the enormous long-tail of abandoned extensions which still…

The big and obvious difference is that all the GIL vs no-GIL stuff happens in the background and your average python dev can just ignore it if they want to. The interpreter will note if you have C extensions that don't opt in to no-GIL and then will give you the GIL version. This is _very_ different to the 2-to-3 transition where absolutely every single person, even those who couldn't care less, had to change their c…

> your average python dev can just ignore it if they want to.

Oh, so naive... All the mutation code in Python which "worked" because Python didn't really have any real concurrency. Add to it -- there's no real plan about what to do with Python concurrency. Removing GIL is only one "half" of the problem, you need to give developers some sort of a framework to use to deal with concurrency. Python's threads are extremely underdeveloped and dangerous to use. Python doesn't even have anything like "synchronized" from the Java world. So, all synchronization requires dealing with locks, mutexes, condition variables...

Most Python programs today didn't bother to deal with threads because they didn't confer enough benefits to be worth using. So, "automatically" parallelizing Python code, as in allowing it to run in actual threads is going to bring about lots and lots of bugs in trivial code written by people with no clue about concurrency.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#216

Post by dagw »

wiz21c wrote: post #182

Is LPython comparable to Nuitka ?

It seems to be closer to pythran than Nuitka. Mainly in that Lpython only supports a subset of python and focuses on performance of numeric code. Nuitka focuses primarily on being able to compile all of python, and has performance only as a secondary goal.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#217

Post by strictfp »

pid-1 wrote: post #144
strictfp wrote: post #59

Earlier quoted context omitted.

My tip for this is Node.js and some stream processing lib like Highland. You can get ridiculous IO parallelism with a very little code and a nice API. Python just scales terribly, no matter if you use multi-process or not. Java can get pretty good perf, but you'll need some libs or quite a bit of code to get nonblocking IO sending working well, or you're going to eat huge amounts of resources for moderate returns. No…

Isn't Node single threaded, just like Python?

Node is essentially a single-threaded API to a very capable multithreaded engine.

https://youtu.be/ztspvPYybIY

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#218

Post by ynik »

kzrdude wrote: post #209
doctoboggan wrote: post #148

Earlier quoted context omitted.

I don’t really understand this. Unless I am missing something you should always pick the “no GIL” version as that will work with or without a GIL. Thread safe No GIL code would be totally fine to run on python compiled with the GIL with zero modifications. Because of this I don’t expect there to be multiple versions of any library. Once a library does the (admittedly heavy) lift to no GIL it will just be the main ver…

Current plan says there has to be separate builds per module, as if it is an ABI break. Would be much better if it could be combined into one build. Hopefully necessity triggers some invention here.

There's no way to make it work with the old ABI. Because sizeof(PyObject) is fixed in the old ABI, there's simply no way to attach additional information (e.g. the new cross-thread ref count) to every Python object. The Python ABI (even the "limited" stable ABI) exposes too many implementation details, it's not really possible to make any fundamental changes to the Python interpreter without breaking that ABI.

You could have a single new ABI supporting both no-GIL and with-GIL, but it wouldn't be compatible with the existing stable ABI.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#219

Post by laichzeit0 »

keithalewis wrote: post #101

Sometimes there is no bandaid big enough to cover up a fundamental design decision in a language. Stick to solving problems it was designed for instead of crapping it up. Python isn't the only language , unless it's the only language you know.

The language part is the least of the problem. I would gladly develop in Turbo Pascal if it had the libraries I need. People use Python because of the ecosystem. PyTorch, Scikit learn, numpy, pandas, and many many other libraries built on top of those libraries.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#220

Post by crabbone »

masklinn wrote: post #8
andrewstuart wrote: post #3

From reading the thread on HN the other day, it sounds like removing the GIL isn't really of much value. Maybe for somewhat obscure multithreading cases. Is that right?

> Maybe for somewhat obscure multithreading cases. They're only "somewhat obscure" because currently you can't do it at all, so you don't do it and you do something else: it's of value for any case where you're multithreading for computational parallelism (as opposed to IO concurrency). The PEP also outlines a bunch of other situations where using process-based parallelism is problematic: https://peps.python.org/pep-…

> currently you can't do it at all

Is not true. First of all, Python threads are mapped to OS threads. So, you can do "something". Now, in CPython C API there are tools for releasing and acquiring the global lock. They aren't complicated, and I used them in my own extensions. Not sure how popular across other extensions is this practice, but, at least some do use it. Some Python native functions release and acquire this lock while running in non-main thread. For the most part, it's the functions that perform blocking I/O.

To sum this up and to make it easier to conceptualize, I describe this as Python can sleep concurrently, but can do no work concurrently.

As to "obscure multithreading cases"... well, ironically, some Python libraries use Python threads unironically... I believe Paramico uses them, but this is from memory, so please don't blame me if that's not the case. It's not very popular, but some have actually used threads. Typically it gives you no benefits when using Python, but on an odd day... There's also a thing about when Python threads can switch, which makes certain code impossible to race, but also makes some particular edge cases of errors harder to reproduce.

So, developers working on libraries that don't use any multithreading will probably not notice, but, these cases are rare because Python is on the path of dependency bloat. Which means that in a large enough project, you are bound to get a library that uses threads. And then you will be impacted by the bugs in multithreading even though you, personally, had nothing to do with it.

Post reply on HN