Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A viable solution for Python concurrency

lwn.net

Post reply on HN
Print view View original

341–350 of 366 posts

Re: A viable solution for Python concurrency

#341

Post by kzrdude »

rich_sasha wrote: post #271
lucb1e wrote: post #160

Earlier quoted context omitted.

I actually have a great experience using that rather than dealing with concurrency hell. My use case is typically brute forcing this or that, for example a quick implementation to crack a key on some ctf challenge, or a proof of concept to crack a session token for a customer demo. Just spawn a few processes, each gets 1/nth of the work, not a big deal. But I could see how, if you want to have (e.g.) sound and UI ren…

For me the main thing is, I cannot parallelise easily a loop mid-function. I need to make a pool, separate out the loop body into a separate top-level function, and also deal with multiprocessing quirks (like processes dying semi-randomly). It feels quite heavy and clunky, is my issue.

Efficient threading is always like that anyway? You have to push the parallization a bit up in level.

By the way, concurrent.futures in Python provides identical (almost) features for a process and thread pool executor, so either choice there can be handled the same way. I'm most fond of the executor.map() method for very easy parallelization of work-loops.

Re: A viable solution for Python concurrency

#342

Post by Too »

fullstop wrote: post #322
Too wrote: post #312

Earlier quoted context omitted.

Let me guess. Using create_task() without await and assuming it will run to completion in the background to trigger some other event? This is almost always the case of async tasks disappearing into the void. After shooting myself in the foot with this one time to many I have a hard rule to never use create_task. Instead make sure every single task ends up in some kind of gather() or wait() awaited from the top level.…

Yes, exactly this scenario.

:)

Its not that strange actually. If you compare it to threads, where you shoot off a thread which you never join(). It’s exactly the same scenario.

Any thread which doesn’t have a join needs a watchdog. Or register the unhandled exception handler.

Can’t remember if threads do this by default. Asyncio tasks for sure don’t, and they can’t really do it either, since a task in error state could be saved for later use for legitimate reasons. Closest you get is the warning printed to log about tasks that got GCd without ever being awaited, if you enable ASYNCIO_DEBUG=1

Re: A viable solution for Python concurrency

#343

Post by rfoo »

modeless wrote: post #338
azinman2 wrote: post #330

Earlier quoted context omitted.

If the GIL is fixed, why would you want multiprocessing versus threads? Threads are cheaper to create, easier to communicate between (even if you need to be careful), and simply do different things than what multiprocessing intends (eg easier for blocking I/O on many threads, versus multiprocessing which is really more of a task queue)

Why don't we just run all code in different threads of the same process? Multiple processes are more robust to failure and easier to reason about because they are less tightly coupled and more explicit about sharing. You can do blocking I/O with multiprocessing, you just have to explicitly share buffers.

> Multiple processes are more robust to failure and easier to reason about

Yeah of course, except when you use multiprocessing to do some CPU-intensive work and one of the subprocess get OOMKilled. Now your main process just hang forever. This was reported on bpo roughly 10 years ago and the response is "we can't fix this".

Is this enough to convince you multiple processes sometimes lead to unnecessary complexity?

Re: A viable solution for Python concurrency

#344

Post by rfoo »

the__alchemist wrote: post #255

Maybe we just accept that Python isn't suitable for concurrency. There's a community of Python developers who don't want to branch out; write everything in Python and never learn or consider another language. Let them be. Let Python excel at its core competencies; use the right tool for the job.

> Let Python excel at its core competencies

Python is notably very popular in two communities: web developer and scientific computing.

The former usually yell loudly every time someone propose to remove GIL. Meanwhile everyone in the scientific computing community had to learn how to workaround GIL which absolutely sucks and sometimes just impossible. (e.g. I have a mostly memory-bandwidth-bound data loading pipeline but it sometimes need multiple cores for doing some trivial data transformation with numpy. This is impossible to do efficiently in Python right now, due to GIL)

Do you mean Python is not the right tool for my job and we should drop numpy/scipy/... and let Python be your shit tool for rendering web pages?

Re: A viable solution for Python concurrency

#345

Post by detaro »

rfoo wrote: post #344
the__alchemist wrote: post #255

Maybe we just accept that Python isn't suitable for concurrency. There's a community of Python developers who don't want to branch out; write everything in Python and never learn or consider another language. Let them be. Let Python excel at its core competencies; use the right tool for the job.

> Let Python excel at its core competencies Python is notably very popular in two communities: web developer and scientific computing. The former usually yell loudly every time someone propose to remove GIL. Meanwhile everyone in the scientific computing community had to learn how to workaround GIL which absolutely sucks and sometimes just impossible. (e.g. I have a mostly memory-bandwidth-bound data loading pipeline…

Why would webdevs care about work trying to remove the GIL?

Re: A viable solution for Python concurrency

#346

Post by rfoo »

detaro wrote: post #345
rfoo wrote: post #344

Earlier quoted context omitted.

> Let Python excel at its core competencies Python is notably very popular in two communities: web developer and scientific computing. The former usually yell loudly every time someone propose to remove GIL. Meanwhile everyone in the scientific computing community had to learn how to workaround GIL which absolutely sucks and sometimes just impossible. (e.g. I have a mostly memory-bandwidth-bound data loading pipeline…

Why would webdevs care about work trying to remove the GIL?

Previous GIL removal attempts hurts single thread performance and it isn't that scalable, so people are usually by default dismissive.

Most of Python codes depend on subtle details of CPython internal. For example sometimes it is just convenient to assume GIL exists (i.e. simplifies concurrency codes because "you know there are at most one thread running").

Re: A viable solution for Python concurrency

#347

Post by alfanerd »

ferdowsi wrote: post #11

If this effort succeeds (and I hope it does) now Python developers will need to contend with the event-loop albatross of asyncio and all of its weird complexity. In an alternate Python timeline, asyncio was not introduced into the Python standard library, and instead we got a natively supported, robust, easy-to-use concurrency paradigm built around green/virtual threading that accommodates both IO and CPU bound work.

Yes, yes, yes and I hope the alternate timeline will start now.

IMO the reason that we have so many ways of doing concurrency in Python (fra asyncio, curio, trio to gevent, multiprocessing etc.) is that it was never properly dealt with.

It has to be built into the language, a feature of the language, like fx. garbage collection. The model of Pony or Erland, where are execution thread is started per core could also have been used in python, instead we got the async/await mess, which almost created a whole new language, where every library has to be rewritten.

It saddens me to think that the ugly cludge that async/await is got added to Python almost without any discussion, whereas the insignificant walrus operator got so much heat that GVR quit.

Re: A viable solution for Python concurrency

#348

Post by BiteCode_dev »

quietbritishjim wrote: post #334
BiteCode_dev wrote: post #291

Earlier quoted context omitted.

Anyio is probably a best bet to use TaskGroup IMO. You can use the trio backend if you want, after all. But it's asyncio compatible, which makes it more future proof.

Hmm, we'll just have to disagree on that one. Trio has the better API, so you might as well use that directly rather some other library that attempts to match it, even if it can/does use it under the hood. It reminds me of that xkcd about standards. It's also going to have to settle for the least common denominator to some extent even if it reimplements some parts itself (e.g. if one library implements serial ports b…

The idea is that once task groups are implemented, you can switch back to asyncio, which as more chance than Trio to be supported.

Re: A viable solution for Python concurrency

#349

Post by BiteCode_dev »

masklinn wrote: post #296
BiteCode_dev wrote: post #76

Earlier quoted context omitted.

I used them both extensively, and here are the main reasons I can think of: - The event loop in JS is invisible and implicit. V8 proved it can be done without paying a cost for it, and in fact most real life python projects are using uvloop because it's faster than asyncio default loop. JS dev don't think of the loop at all, because it's always been there. They don't have to chose a loop, or thinking about its lifecy…

> Python is my favorite language, and I can live with the explicit loop, but explicit scheduling is ridiculous. Just run the damn coroutine You can't "just run the damn coroutine", that's not what coroutines are . But that does point to the mistake of Python, at least from a UX perspective: coroutines are more efficient but they're also a lot more prone to use errors, especially in dynamically typed languages. Async…

I know, but we could make coroutine creates with async have a special marker that is different to yield and auto schedule.

Re: A viable solution for Python concurrency

#350

Post by JanisL »

BeetleB wrote: post #329
snovv_crash wrote: post #282

Earlier quoted context omitted.

Lots of things didn't upgrade because their dependencies didn't. And the dependencies had no reason to, because 3.0 didn't offer anything they didn't have before.

Quite a bit before the deadline, pretty much all dependencies had upgraded. Those that hadn't were more or less abandoned projects. Sure, you could hunt and find some that were still maintained/developed with Python 2 in, say, 2019. But these again will be the tiny minority. I still stand by my 1% claim. The problems with upgrading have been vastly exaggerated.

The pain in the Python 2->3 conversion I found came more from in house packages being written in Python 2 rather than public packages.
Post reply on HN