Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A viable solution for Python concurrency

lwn.net

Post reply on HN
Print view View original

171–180 of 366 posts

Re: A viable solution for Python concurrency

#171

Post by huetius »

Many of the additions to Python in the past decade have been very impressive, but am I wrong in thinking that they suffer from a kind of diminishing marginal benefit?

If I am building a project where concurrency or asynchrony are essential, am I going to choose Python? If I need to bolt these on to an existing project to meet a deadline, how much runway do I really get from these enhancements before I hit the limitations of the language, and need a new tool anyway?

I love Python, and use it a great deal, but I’ve not seen anything in my experience to assuage these doubts. Happy to be convinced otherwise, though. I’m sure there are circumstances I haven’t thought of.

Re: A viable solution for Python concurrency

#172

Post by miohtama »

quietbritishjim wrote: post #100
harpiaharpyja wrote: post #84

Earlier quoted context omitted.

If you are ever considering making use of asyncio for your project, I would strongly recommend taking a look at curio [1] as an alternative. It's like asyncio but far, far easier to use. [1] https://curio.readthedocs.io/en/latest/index.html

Curio's spiritual successor is Trio [1], which was written by one of the main Curio contributors and is more actively maintained (and, at this point, much more widely used). Like Curio, it's much easier to use than asyncio, although ideas from it are gradually being incorporated back into asyncio e.g. asyncio.run() was inspired by curio.run()/trio.run(). I have used Trio in real projects and I thoroughly recommend it…

This is super interesting. How does async approaches work across different Python libraries? Are there any assumptions about usage of asyncio?

Re: A viable solution for Python concurrency

#173

Post by throwaway81523 »

heavyset_go wrote: post #162
throwaway81523 wrote: post #152

Earlier quoted context omitted.

There is at least some recognition in those cases that they introduced the new thing because they got it wrong in the old thing. That's different than saying they should co-exist on equal terms.

> That's different than saying they should co-exist on equal terms. I'm not sure who is claiming that. Here's the OP we're replying to: > They are techs with different trade off, and life is full of opportunities.

Yes, that says each has good and bad points and you should weigh them against each other in the context of your application, to figure out which one to use. I.e. equal terms.

Zen would be: pick one of the two approaches, keep its strengths while fixing it to get rid of its weaknesses, then declare the fixed version as the one obvious way to do it. You might still have to keep the other one around for legacy support, but that's similar to the situation with applying functions across iterables.

This is what Go did. Go has one way to do concurrency (goroutines) and they are superior to both of Python's current approaches. Erlang has of course been in the background all along, doing something similar.

Re: A viable solution for Python concurrency

#174

Post by unityByFreedom »

huetius wrote: post #171

Many of the additions to Python in the past decade have been very impressive, but am I wrong in thinking that they suffer from a kind of diminishing marginal benefit? If I am building a project where concurrency or asynchrony are essential, am I going to choose Python? If I need to bolt these on to an existing project to meet a deadline, how much runway do I really get from these enhancements before I hit the limitat…

This is more about making parallel python easier to use. It's already fast if you know about the current GIL workarounds. It'd be nice to not have to monkey patch, for example.

Re: A viable solution for Python concurrency

#175

Post by Buttons840 »

btown wrote: post #97
ferdowsi wrote: post #11

If this effort succeeds (and I hope it does) now Python developers will need to contend with the event-loop albatross of asyncio and all of its weird complexity. In an alternate Python timeline, asyncio was not introduced into the Python standard library, and instead we got a natively supported, robust, easy-to-use concurrency paradigm built around green/virtual threading that accommodates both IO and CPU bound work.

> instead we got a natively supported, robust, easy-to-use concurrency paradigm built around green/virtual threading that accommodates both IO and CPU bound work Minus the "natively supported" part, we have this today in http://www.gevent.org/ ! It's so, so empowering to be able to access the entire historical body of work of synchronous-I/O Python libraries, and with a single monkey patch cause every I/O operation,…

> our web servers and batch workers have throughput limited only by CPU and RAM

Are you able to fully utilize a multicore processor? I'm not familiar with gevent.

Re: A viable solution for Python concurrency

#177

Post by rzimmerman »

pansa2 wrote: post #169

Is CPython the only widely-used language implementation that uses reference counting rather than tracing garbage collection? IIRC even PyPy and MicroPython use tracing GC.

Objective C does (explicit) reference counting, though I suppose that's a pretty limited use case these days.

Re: A viable solution for Python concurrency

#178

Post by unityByFreedom »

btown wrote: post #97
ferdowsi wrote: post #11

If this effort succeeds (and I hope it does) now Python developers will need to contend with the event-loop albatross of asyncio and all of its weird complexity. In an alternate Python timeline, asyncio was not introduced into the Python standard library, and instead we got a natively supported, robust, easy-to-use concurrency paradigm built around green/virtual threading that accommodates both IO and CPU bound work.

> instead we got a natively supported, robust, easy-to-use concurrency paradigm built around green/virtual threading that accommodates both IO and CPU bound work Minus the "natively supported" part, we have this today in http://www.gevent.org/ ! It's so, so empowering to be able to access the entire historical body of work of synchronous-I/O Python libraries, and with a single monkey patch cause every I/O operation,…

Gevent is indeed amazing. I just started using it. Isn't the point of this post's effort to bake in its functionality? Handling exceptions with gevent doesn't seem trivial to me, for example, and it'd be nice to not have to monkey patch, as easy as it is.

Re: A viable solution for Python concurrency

#180

Post by nerdponx »

quietbritishjim wrote: post #100
harpiaharpyja wrote: post #84

Earlier quoted context omitted.

If you are ever considering making use of asyncio for your project, I would strongly recommend taking a look at curio [1] as an alternative. It's like asyncio but far, far easier to use. [1] https://curio.readthedocs.io/en/latest/index.html

Curio's spiritual successor is Trio [1], which was written by one of the main Curio contributors and is more actively maintained (and, at this point, much more widely used). Like Curio, it's much easier to use than asyncio, although ideas from it are gradually being incorporated back into asyncio e.g. asyncio.run() was inspired by curio.run()/trio.run(). I have used Trio in real projects and I thoroughly recommend it…

As yet another option, I really like Anyio, which is a "frontend" to either Asyncio or Trio, which provides a consistent and tidy API for what is known as "structured concurrency" (I think the name was popularized by the C library Dill).

https://anyio.readthedocs.io

Post reply on HN