Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Asyncio

superfastpython.com

Post reply on HN
Print view View original

151–160 of 188 posts

Re: Python Asyncio

#151

Post by throwaway81523 »

quietbritishjim wrote: post #10

Not complete - doesn't include Task Groups [1] In fairness they were only included in asyncio as of Python 3.11, which was released a couple of weeks ago. These were an idea originally from Trio [2] where they're called "nurseries" instead of "task groups". My view is that you're better off using Trio, or at least anyio [3] which gives a Trio-like interface to asyncio. One particularly nice thing about Trio (and anyi…

I have such a feeling of tragedy about Python. I wish it had migrated to BEAM or implemented something similar, instead of growing all this async stuff. Whenever I see anything about Python asyncio, I'm reminded of gar1t's hilarious but NSFW rant about node.js, https://www.youtube.com/watch?v=bzkRVzciAZg . Content warning: lots of swearing, mostly near the end.

Re: Python Asyncio

#152

Post by jerf »

number6 wrote: post #79
kissgyorgy wrote: post #12

After 2 years of using asyncio in production, I recommend to avoid it if you can. With async programming, you take the complexity of concurrent programming, which is way harder than you can imagine. Also nobody mentions this for some reason, but asyncio doesn't make your programs faster, in fact it makes everything 100x SLOWER (we measured it multiple times, compared the same thing to the sync version), but makes you…

Asyncio is not ment for CPU Bound Task but for IO Bound Tasks. You should have used multiprocessing. The problem you describe is exactly what asyncio is used for: saturate your bandwidth better. maybe the right aproach would have been a Threadpool? Plus you don't have to refractor the task. Just let it run sync, but you can make it also async

"Asyncio is not ment for CPU Bound Task but for IO Bound Tasks."

You can amplify this; pure Python is not meant for CPU-bound tasks. If you have a pure Python task, you've CPU optimized it somewhat, and it's still too slow, the answer is, get out of pure Python.

There's half-a-dozen options that still leave you essentially in Python land (just not pure Python anymore) like cython or NumPy, and of course dozens of other languages that leave Python entirely.

To accelerate a pure Python task to the speed that you can get out of a single thread of a more efficient language, you have to perfectly parallelize your task across upwards of 40 CPUs (conservatively!), because that's how slow pure Python is. asyncio isn't a solution to this, but neither is any sort of multiprocessing of the pure Python either.

This is not criticism. This is just engineering reality. Pure Python is a fun language, but a very slow one.

Re: Python Asyncio

#154

Post by dec0dedab0de »

kissgyorgy wrote: post #12

After 2 years of using asyncio in production, I recommend to avoid it if you can. With async programming, you take the complexity of concurrent programming, which is way harder than you can imagine. Also nobody mentions this for some reason, but asyncio doesn't make your programs faster, in fact it makes everything 100x SLOWER (we measured it multiple times, compared the same thing to the sync version), but makes you…

It used to get brought up every time asyncio was mentioned on here.

I either use multiprocessing, or celery. using async/await looks ugly, doesn't solve most problems I would want it to, and feels unstable.

Maybe once development around it settles down I'll revisit, but I can't imagine wanting to use it in it's current form.

Re: Python Asyncio

#155

Post by jacob019 »

kissgyorgy wrote: post #12

After 2 years of using asyncio in production, I recommend to avoid it if you can. With async programming, you take the complexity of concurrent programming, which is way harder than you can imagine. Also nobody mentions this for some reason, but asyncio doesn't make your programs faster, in fact it makes everything 100x SLOWER (we measured it multiple times, compared the same thing to the sync version), but makes you…

After using gevent for about a decade I have started using asyncio for new projects, just because it's in the standard library and has the official async blessing of the Python gods. Indeed it is way harder. I'm always coming up against little gotchas that take time to debug and fix. Part of me enjoys the challenge of learning something new and solving little puzzles. It's getting easier, especially as I build up a collection of in-house async libraries for various things. As for performance, it's not too bad for mostly io bound tasks, which is why one uses async in the first place. Some tight loop benchmarks for message passing with other processes show it to be about half the speed of gevent in my case, which is fine. It's nice to be able to deploy async microservices without installing gevent, and there's a certain value to the discipline that it imposes. I like how I am able to bring non-async code into the async world using threading. I imagine the performance would improve quite a bit with pypy, perhaps exceeding that of gevent. Gevent makes it so damn easy, I've been spoiled. I was disappointed when asyncio came out, as I would have preferred the ecosystem moved in the gevent direction instead; but I'm coming around. It's super annoying how the python ecosystem has been bifurcated with asyncio. You really have to choose one way or another at the beginning of a project and stick with it.

And yeah, async programming (in Python) isn't really for CPU bound stuff. You might benefit from multiprocessing and just use asyncio to coordinate, which is what it excels at. PyPy can really help with CPU bound stuff too, if the code is mostly pure Python.

Re: Python Asyncio

#156

Post by cturner »

amelius wrote: post #136

Async programming is Windows 3.1 style cooperative multitasking.

Different trade-offs. It is much easier to pursue cache-efficiency in a cooperative multitasking setting than with preempt threads.

Re: Python Asyncio

#157

Post by dekhn »

jborean93 wrote: post #150
throwawaymaths wrote: post #139

Earlier quoted context omitted.

The least they could have done is sugared it to something that isn't "def", like "gen" or "defgen"

Unless I'm misunderstanding your comment they do, it's `async def` to denote a coroutine.

They're referring to generator functions. A generator function typically open a resource and then use a for loop to read (for example) a line at a time and yield it. once you yield, control returns to the caller (like a return) but subsequent calls to next on the returned expression pick back up at the yield point. So function calls maintain state-behind-the-scenes. It's not just a simple call-and-return model anymore.

To me this violates a core principle: the principle of least surprise. It completely changes how I have to reason about function execution, especially if there are conditionals and different yields.

Re: Python Asyncio

#158

Post by dekhn »

phs2501 wrote: post #137
dekhn wrote: post #127

Earlier quoted context omitted.

The docs on that page explicitly say you just have to provide a different parameter (basically, to force it to use a thread-like instead of select-like approach). The docs on that page also point you to a different task library which does what I'd expect: "If the work is appropriate for concurrency and parallelism, also consider using the Task Parallel Library." I checked those docs and that is a framework that makes…

Not sure how that's different than Python asyncio's "run blocking stuff with `run_in_executor`, the result (future) of which integrates into the async event loop?

it's threaded with the option to use 1:M, N:N or N:M. It's a mental model that includes both asyncio and threads, while async will always only be for IO-blocking calls that can be multiplexed.

Re: Python Asyncio

#159

Post by jacob019 »

samwillis wrote: post #16

There is very little in everyday Python usage that benefits from Asyncio. Two in webdev, are long running request (Websockets, SSE, long polling), and processing multiple backend IO processes in parallel. However the later is very rare, you may think you have multiple DB request that could use asyncio, but most of the time they are dependent on each other. Almost all of the time a normal multithreaded Python server i…

I don't know what you use Python for every day. Sure for some utility scripts it doesn't matter. I use it to run my ecommerce business, and for a variety of other plumbing, mostly for passing messages around between users, APIs, databases, printers, etc. Async programming is a must for just about everything. I guess the alternative would be thread pools, or process pools, like back in the day; but that has a lot of downsides. It is slower, way more resource intensive, and state sharing/synchronization becomes a major issue, you can only handle thread number of tasks concurrently and you're going to use that memory all the time; not to mention all the code needs to be thread safe. Most of our systems use gevent, but we've starting using asyncio for new projects.

Re: Python Asyncio

#160

Post by rogers12 »

throwaway81523 wrote: post #151
quietbritishjim wrote: post #10

Not complete - doesn't include Task Groups [1] In fairness they were only included in asyncio as of Python 3.11, which was released a couple of weeks ago. These were an idea originally from Trio [2] where they're called "nurseries" instead of "task groups". My view is that you're better off using Trio, or at least anyio [3] which gives a Trio-like interface to asyncio. One particularly nice thing about Trio (and anyi…

I have such a feeling of tragedy about Python. I wish it had migrated to BEAM or implemented something similar, instead of growing all this async stuff. Whenever I see anything about Python asyncio, I'm reminded of gar1t's hilarious but NSFW rant about node.js, https://www.youtube.com/watch?v=bzkRVzciAZg . Content warning: lots of swearing, mostly near the end.

That's a blast from the past. Incredible that 10 years ago people thought using a thread per request is a good idea because anything else is too hard.
Post reply on HN