Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A viable solution for Python concurrency

lwn.net

Post reply on HN
Print view View original

281–290 of 366 posts

Re: A viable solution for Python concurrency

#281

Post by jillesvangurp »

huetius wrote: post #171

Many of the additions to Python in the past decade have been very impressive, but am I wrong in thinking that they suffer from a kind of diminishing marginal benefit? If I am building a project where concurrency or asynchrony are essential, am I going to choose Python? If I need to bolt these on to an existing project to meet a deadline, how much runway do I really get from these enhancements before I hit the limitat…

> If I am building a project where concurrency or asynchrony are essential,

Like many data processing & data science jobs. Also anything doing disk IO, network IO, etc. Or anything running on a machine with multiple cores (i.e. almost any machine these days). So, the sweet spot would be running data jobs running on multi core machines doing things with files and accessing things over a network. That sounds a lot like a core use case for Python.

Python works around this by using processes and outsourcing all the pesky difficult stuff to native components. It kind of works, hence the popularity but it does add a few layers of complexity and it is needlessly slow.

It seems the biggest blocker for removing the Gil has simply been the python community itself having talked themselves into this being hard/undesirable and resisting change.

This little quote in the article is a great example of this:

> as Guido van Rossum noted, the Python developers could always just take the performance improvements without the concurrency work and be even faster yet.

Re: A viable solution for Python concurrency

#282

Post by snovv_crash »

BeetleB wrote: post #258
throwaway81523 wrote: post #164

Earlier quoted context omitted.

The entire Python community was in pain over Python 3 for 10 years, even if migrating any particular program wasn't much trouble. If you want to contest the notion that there was pain, then fine: most of the community simply ignored Python 3 for 10 years, because there was no reason until quite late in the process to worry about it. I myself never bothered migrating any of my Python 2 stuff. It might not be difficult…

> The entire Python community was in pain over Python 3 for 10 years Sorry, but that's fiction. Yes, lots of projects (still the minority) simply didn't upgrade, but it's not because they tried and failed. It's because they never prioritized it. > Here's a pain story about a 2 to 3 migration though: Consistent with my claim: > Probably less than 1% of projects had serious issues with the migration. We just hear of a…

Lots of things didn't upgrade because their dependencies didn't. And the dependencies had no reason to, because 3.0 didn't offer anything they didn't have before.

Re: A viable solution for Python concurrency

#283

Post by ayush--s »

smetj wrote: post #277
btown wrote: post #97

Earlier quoted context omitted.

> instead we got a natively supported, robust, easy-to-use concurrency paradigm built around green/virtual threading that accommodates both IO and CPU bound work Minus the "natively supported" part, we have this today in http://www.gevent.org/ ! It's so, so empowering to be able to access the entire historical body of work of synchronous-I/O Python libraries, and with a single monkey patch cause every I/O operation,…

Gevent is the best thing which ever happened within the async world of Python. It's just great to work with. Rock solid. I have high IO production software running for years with gevent as the workhorse, copes with high load, no maintenance, brilliant. Then came Asyncio, which I personally disliked for the simple reason it became so popular that everybody thinks it's necessary to write an asyncio version of their lib…

It's really easy to shoot yourself in the foot by using global mutable variables. How do you guard against those when using gevent in production?

Re: A viable solution for Python concurrency

#284

Post by omnicognate »

sherjilozair wrote: post #214
omnicognate wrote: post #183

Earlier quoted context omitted.

I like the GIL and would prefer it not be removed, regardless of any impact on performance. It's a powerful assumption for both python code and extensions to be able to make that only one thread will be executing in the interpreter at a time. Knowing it, you can do a lot of things with a much lower cognitive burden. I tend to think through a problem initially in a non-concurrent way and then think "ok, but there's co…

You know it’s trivial to add a global lock to any concurrent program, right? What would you lose if other people started writing performant code?

This comment under the article gives an example: https://lwn.net/Articles/872961/. Several other comments there also discuss this problem.

Unless an extension explicitly releases the GIL it's not possible for the state of the interpreter to change during execution of its methods. That's an invariant that extensions rely on implicitly for safety in many ways and it's hard to imagine how one could make them safe without significant work on all of those extensions. I, for one, own extensions that would require complex, structural, performance-affecting changes.

And it's worth noting that "safety" here is not just safety from incorrect behaviour, it's safety from memory corruption, crashes and security issues.

Edit: Also, just to note - there is nowhere extension authors can add a global lock that would solve this problem. It would require top-level python programs to add the necessary locking, and the consequences of them not doing so would typically include crashes and severe security issues. The only place it's "trivial" to add a global lock to avoid these problems is the interpreter. A Global Interpreter Lock, if you will.

But, of course, if you think the GIL can be removed in such a way that these issues aren't a real problem, have at it. Plenty of people will thank you.

Edit 2: Also worth explicitly mentioning: when it comes to avoiding memory corruption, extension authors can't make any assumptions about what their python callers will do. I (and any responsible extension author) go to significant lengths to ensure my extensions can't crash regardless of how they're used from python.

One of my extensions, for example, is a (private, in-house) interop mechanism that allows python users to access an API developed in C#. If the GIL is removed and somebody goes and writes a bit of threaded python code that modifies the contents of some object while my extension is accessing it, without the necessary locking, and this results in memory corruption, the blame will rightly fall on my extension. Python isn't C, and the people writing it (unless they're using ctypes or whatever) don't expect to be able to cause memory corruption by making elementary programming errors.

Re: A viable solution for Python concurrency

#285

Post by smcl »

zinodaur wrote: post #145
efoto wrote: post #3

"The biggest source of problems might be multi-threaded programs with concurrency-related bugs that have been masked by the GIL until now."

> concurrency-related bugs that have been masked by the GIL Yeah... could phrase this as "All programs written with the assumption of a GIL are now broken" instead. Wish they had done this as part of the breaking changes for python 3, I guess they'll have to wait for Python 4 for this?

I think I read that there won’t be another big jump like there was for Python 2-3. If I understood correctly, there could be a Python 4, but it won’t indicate huge breaking changes, it’ll just be another release

Re: A viable solution for Python concurrency

#286

Post by nitred »

arc-in-space wrote: post #207
omnicognate wrote: post #183

Earlier quoted context omitted.

I like the GIL and would prefer it not be removed, regardless of any impact on performance. It's a powerful assumption for both python code and extensions to be able to make that only one thread will be executing in the interpreter at a time. Knowing it, you can do a lot of things with a much lower cognitive burden. I tend to think through a problem initially in a non-concurrent way and then think "ok, but there's co…

I'm confused. The article mentions that Python's data structures would be made thread-safe, which sounds like there is no intention to change concurrency semantics. How does the presence of the GIL help? Can you elaborate what would be lost by removing it?

Can't speak for the OP but here's what I think they may have meant and my intuitions agree with them as well.

In python I mostly use threads for non-CPU intensive tasks like IO or timers or event handling and I rarely implement these myself.

However for CPU intensive tasks, there's rarely any point to having 10 threads in Python because they all have to run on one core because of the GIL. Therefore when working with python the GIL indirectly lets me never have to think about writing concurrent CPU intensive code AND I know others wouldn't do it too because it wouldn't make sense most of the time. This is leads to a significant reduction in cognitive load when thinking about solving a problem in Python. And it isn't about thread-safety but that in concurrent code I have to worry how other threads might change the values of shared variables and global variables of which there are many.

Instead the GIL lets me think synchronously, write synchronous code and when I needed to parallelize my work I use multiprocessing where my synchronous code and assumptions work perfectly because each process has its own GIL and I explicitly pass shared variables to each of the processes.

I don't know enough to comment on the good or the bad of GIL. Just my two cents about how I think about code with Python in it's current state.

Re: A viable solution for Python concurrency

#287

Post by sillysaurusx »

ayush--s wrote: post #283
smetj wrote: post #277

Earlier quoted context omitted.

Gevent is the best thing which ever happened within the async world of Python. It's just great to work with. Rock solid. I have high IO production software running for years with gevent as the workhorse, copes with high load, no maintenance, brilliant. Then came Asyncio, which I personally disliked for the simple reason it became so popular that everybody thinks it's necessary to write an asyncio version of their lib…

It's really easy to shoot yourself in the foot by using global mutable variables. How do you guard against those when using gevent in production?

Global state isn't a problem for multithreaded code. State updates are the problem; it doesn't matter whether it's global or local.

If you can solve the local case, you can solve the global case. Pick your technique; any ol' technique is just as good as any other.

Re: A viable solution for Python concurrency

#288

Post by lucb1e »

globular-toast wrote: post #181
lucb1e wrote: post #147

Earlier quoted context omitted.

For a minute I thought I finally found someone else who likes the GIL, but then you said content about . Programs that just divide up work across processes are much easier to write without introducing obscure bugs due to the lack of atomicity. I'm definitely excited for a GIL-less python, even if it's a rare scenario where it makes sense to try to do performant code in python in the first place rather than offloading…

> Programs that just divide up work across processes are much easier to write without introducing obscure bugs due to the lack of atomicity. You often don't even need to do this yourself. GNU parallel is the way to go for dividing work up amongst CPU cores. Why reinvent the wheel? I agree with you that threads are talked about way more than they should be. It's like all programmers learn this one simple rule: to be f…

That's actually what I'm doing a lot of the time. Or even just bash: for i in {1..threadcount}; do pypy my.py $i/$threadcount & done;

Re: A viable solution for Python concurrency

#289

Post by quietbritishjim »

BiteCode_dev wrote: post #101
KaiserPro wrote: post #71

Earlier quoted context omitted.

> easy-to-use concurrency paradigm Well it has queues and threads already. Its just that asyncio for socket handling at least (in the testing that I did) is about 5% faster. (one asyncio socket "server" vs ten threads [with a number of ways to monitor for new connections]) I always assumed that people wanted asyncio because they look at javascript and thought "hey I want GOTOs cosplaying as a fun paradigm"

GOTO cosplaying should go away with structured concurrency (via TaskGroup) being adopted in 3.11, as pioneered by Trio. Check out anyio if you want to use them now.

TaskGroup in asyncio has been promised at least as far back as Python 3.8 [1]. They're still not in the draft "What's new in python 3.11" [2] and searching the web didn't return any official statements. I believe they're planned but don't believe they'll arrive any time soon.

If you want to use structured concurrency now, IMO the best bet is to use Trio directly. Reading posts by the author makes it clear that every detail of the library had been extremely scrutinised, not just the API (e.g. see this long post on ctrl-C handling [3], or any number of long technical discussions on the issue tracker), so I think it's a better choice in any case.

[1] https://twitter.com/1st1/status/1041855365745455104

[2] https://docs.python.org/3.11/whatsnew/3.11.html

[3] https://vorpus.org/blog/control-c-handling-in-python-and-tri...

Re: A viable solution for Python concurrency

#290

Post by orf »

sherjilozair wrote: post #216
zinodaur wrote: post #145

Earlier quoted context omitted.

> concurrency-related bugs that have been masked by the GIL Yeah... could phrase this as "All programs written with the assumption of a GIL are now broken" instead. Wish they had done this as part of the breaking changes for python 3, I guess they'll have to wait for Python 4 for this?

GIL can remain default on. Users can simply disable it for chosen parts of their program.

No they can’t, it would have to be disabled or enabled globally at the process and/or interpreter level.
Post reply on HN