Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A viable solution for Python concurrency

lwn.net

Post reply on HN
Print view View original

181–190 of 366 posts

Re: A viable solution for Python concurrency

#181

Post by globular-toast »

lucb1e wrote: post #147
jeffybefffy519 wrote: post #140

I dont know about others, but I really enjoy content about the Python GIL. Its a fascinatingly complex problem.

For a minute I thought I finally found someone else who likes the GIL, but then you said content about . Programs that just divide up work across processes are much easier to write without introducing obscure bugs due to the lack of atomicity. I'm definitely excited for a GIL-less python, even if it's a rare scenario where it makes sense to try to do performant code in python in the first place rather than offloading…

> Programs that just divide up work across processes are much easier to write without introducing obscure bugs due to the lack of atomicity.

You often don't even need to do this yourself. GNU parallel is the way to go for dividing work up amongst CPU cores. Why reinvent the wheel?

I agree with you that threads are talked about way more than they should be. It's like all programmers learn this one simple rule: to be fast you have to be multi-threaded. It's really not the case. There is also massive confusion amongst programmers on the difference between concurrency and parallelism. I sometimes ask applicants to describe the difference and few can. Python is fine at concurrency if that's all you want to do.

Re: A viable solution for Python concurrency

#182

Post by nerdponx »

miohtama wrote: post #172
quietbritishjim wrote: post #100

Earlier quoted context omitted.

Curio's spiritual successor is Trio [1], which was written by one of the main Curio contributors and is more actively maintained (and, at this point, much more widely used). Like Curio, it's much easier to use than asyncio, although ideas from it are gradually being incorporated back into asyncio e.g. asyncio.run() was inspired by curio.run()/trio.run(). I have used Trio in real projects and I thoroughly recommend it…

This is super interesting. How does async approaches work across different Python libraries? Are there any assumptions about usage of asyncio?

The "async/await" syntax is agnostic of the underlying async library, but Asyncio and Trio provide incompatible async "primitives". So yes, you need to write your code for one or the other, or use Anyio, which is a common layer over both.

Many more libraries use Asyncio than Trio, so I have come to recommend Anyio (which has a Trio-like API) but with Asyncio as the backend. That gives you the extensive Asyncio ecosystem but with the structured concurrency design of Trio.

Re: A viable solution for Python concurrency

#183

Post by omnicognate »

lucb1e wrote: post #147
jeffybefffy519 wrote: post #140

I dont know about others, but I really enjoy content about the Python GIL. Its a fascinatingly complex problem.

For a minute I thought I finally found someone else who likes the GIL, but then you said content about . Programs that just divide up work across processes are much easier to write without introducing obscure bugs due to the lack of atomicity. I'm definitely excited for a GIL-less python, even if it's a rare scenario where it makes sense to try to do performant code in python in the first place rather than offloading…

I like the GIL and would prefer it not be removed, regardless of any impact on performance.

It's a powerful assumption for both python code and extensions to be able to make that only one thread will be executing in the interpreter at a time. Knowing it, you can do a lot of things with a much lower cognitive burden. I tend to think through a problem initially in a non-concurrent way and then think "ok, but there's concurrency to think of too, so how is this all affected?". With the GIL you usually don't need to go far into that if at all, and that's useful.

That doesn't mean I don't approve of concurrency. I use lots of languages, and a need for proper concurrency is one of the major reasons I might avoid python as a tool for a particular task.

But there are lots of languages, and python became the popular tool it is with the GIL. If python were to embrace concurrency in the interpreter its nature would change, and I think my toolbox would lose something valuable.

That said, I'm not worried. CPython exists in a cloud of extensions developed against its C API, and these are heavily reliant on the GIL. Refcounting isn't even the start of the problems you would need to solve in order to remove it and not have everyone just move to a fork that still has it. I'll be astonished if anyone manages to pull it off.

Re: A viable solution for Python concurrency

#184

Post by sydthrowaway »

hatware wrote: post #179
sydthrowaway wrote: post #176

How does one get this good as a developer!

I always imagine these are the folks who started programming as teenagers or earlier. Or, incredibly seasoned vets into their 50's and beyond.

Or wealthy parents (feeds into 1)

Re: A viable solution for Python concurrency

#185

Post by dagenix »

throwaway81523 wrote: post #164
BeetleB wrote: post #143

Earlier quoted context omitted.

You are likely being downvoted because most claims about the pain of a Python 3 transition are inflated/hyperbole. It took less than a day to migrate all my code to Python 3. And by "less than a day" I mean "less than 2 hours". Granted, bigger projects would take longer, but saying stuff like "10+ years of pain" is ridiculous. Probably less than 1% of projects had serious issues with the migration. We just hear of a…

The entire Python community was in pain over Python 3 for 10 years, even if migrating any particular program wasn't much trouble. If you want to contest the notion that there was pain, then fine: most of the community simply ignored Python 3 for 10 years, because there was no reason until quite late in the process to worry about it. I myself never bothered migrating any of my Python 2 stuff. It might not be difficult…

IMO, it's a good thing that decode() doesn't on strings in Python 3.

Re: A viable solution for Python concurrency

#186

Post by huetius »

unityByFreedom wrote: post #174
huetius wrote: post #171

Many of the additions to Python in the past decade have been very impressive, but am I wrong in thinking that they suffer from a kind of diminishing marginal benefit? If I am building a project where concurrency or asynchrony are essential, am I going to choose Python? If I need to bolt these on to an existing project to meet a deadline, how much runway do I really get from these enhancements before I hit the limitat…

This is more about making parallel python easier to use. It's already fast if you know about the current GIL workarounds. It'd be nice to not have to monkey patch, for example.

Quite true. Definitely a benefit. I guess my point is that if you are leaning on Python for performant concurrent operations, you are likely to also be thinking about how to isolate that component for a rewrite, monkey-patching or no.

Re: A viable solution for Python concurrency

#188

Post by mrfusion »

I’d think there are a certain class of programs that could benefit from just turning off garbage collection and reference counting.

If you’re sure you won’t be needing a lot of memory and the program will be short lived that could be a large speed up and you could skip the Gil.

Re: A viable solution for Python concurrency

#189

Post by loeg »

rzimmerman wrote: post #177
pansa2 wrote: post #169

Is CPython the only widely-used language implementation that uses reference counting rather than tracing garbage collection? IIRC even PyPy and MicroPython use tracing GC.

Objective C does (explicit) reference counting, though I suppose that's a pretty limited use case these days.

Doesn't Swift (built on ObjC) also use reference counting?

Re: A viable solution for Python concurrency

#190

Post by dragonwriter »

otterley wrote: post #110

This may be a silly question, but if you really need concurrency, why not use a language that's built for concurrency from the ground up instead? Elixir is a great example.

> This may be a silly question, but if you really need concurrency, why not use a language that's built for concurrency from the ground up instead?

Because sometimes what you really need, or at least want, is an overlap of concurrency and other things, and Python is optimal for you for the other things.

If you just need concurrency, sure, but how often is that the only requirement?

Post reply on HN