Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A conceptual overview of asyncio

github.com

Post reply on HN
Print view View original

21–30 of 32 posts

Re: A conceptual overview of asyncio

#21

Post by kevmo314 »

deathanatos wrote: post #19
omh1280 wrote: post #5

Great read! Python asyncio can really screw up your runtime performance if you use it poorly. And it's _really_ easy to use poorly. Consider a FastAPI server using asyncio instead of threading. _Any_ time you drop down into a synchrononous API, you better be sure that you're not doing anything slow. For example, encoding or decoding JSON in Python actually grabs the GIL depending on what library you're using, and the…

JSON encoding is, as someone else points out, a GIL problem, but I want to add that even if you do JSON encoding in an async context: async def foo(…): json.dumps(d) # you're blocking the event loop You're still going to block on it. def sync_foo(…): json.dumps(d) # you're holding the GIL … and so blocking here too Short of resolving the GIL somehow (either by getting ridding of it, which I think is still a WIP thoug…

This is more of a usability problem. In the second example, it's obvious that `json.dumps()` blocks everything else and it can be readily observed. It's not obvious that it blocks in the former and I've encountered many surprised coworkers despite it seeming obvious to me.

I think a lot of people assume you can slap `async` onto the function signature and it will not block anything anymore. I've had PRs come through that literally added `async` to a completely synchronous function with that misunderstanding.

Re: A conceptual overview of asyncio

#23

Post by boomer_joe »

bb88 wrote: post #13
kccqzy wrote: post #11

Earlier quoted context omitted.

That's a GIL problem not an async problem. Even if you choose to ditch asyncio and use threads, you still need to care about the GIL. And when I use asyncio I don't worry about CPU-bound tasks like encoding or decoding JSON; I worry about some library doing I/O synchronously regardless of whether such library releases the GIL or not.

This is spot on. GIL-less python will be a thing, and when it happens, there will still be no reason to combine asyncIO with thread primitives. Waiting for IO can be spun off into a new thread, and it will work as you expect it would. Trying to combine mental models of asyncio and threading is a model for pure insanity.

I fail to see why. You can have an event loop per thread, and a hypothetical requirement of wanting to make sure all compute in each thread is spent inside of its event loop (assuming OS level parallelism). Eg a latency-sensitive server in thread A and a logger in thread B (dont even need the event loop there for this example)

Re: A conceptual overview of asyncio

#25

Post by essnine »

This is a good read. I remember first using eventlet for writing concurrent code, and then having to do a bit of mental adjustment when moving to asyncio.

Another piece of writing I found useful for perspective at the time was What Color is Your Function?[1], which I bumped into after looking at the Node.js model of concurrency and being confused.

[1](https://journal.stuffwithstuff.com/2015/02/01/what-color-is-...)

Re: A conceptual overview of asyncio

#27

Post by jufter »

crvdgc wrote: post #26

asyncio by itself doesn't support asynchronous file I/O, see their wiki: https://github.com/python/asyncio/wiki/ThirdParty#filesystem You have to use something like aiofiles to do that.

Instead of adding another dependency you can just call `loop.run_in_executor` yourself: https://github.com/Tinche/aiofiles/blob/main/src/aiofiles/ba...

Re: A conceptual overview of asyncio

#28

Post by nromiun »

quotemstr wrote: post #20

Why would anyone want to use asyncio over trio. The latter is one of the few structured concurrency systems that doesn't make me want to pry my eyeballs out with a spoon.

Not every program needs thread/task cancellation. Somehow people have been convinced that threading is the same as goto and it is obviously the wrong thing to do. goto is goto, you can't take anything you dislike and say it will die like goto did.

Re: A conceptual overview of asyncio

#29

Post by Izkata »

anordin95 wrote: post #12
Izkata wrote: post #9

> Frankly, I'm not sure why that design decision was made and find it rather confuses the meaning of await: asynchronously wait. I've always understood it to mean "wait for asynchronous object", not that the wait itself is asynchronous. It's just an English word that roughly means "wait for", that was chosen for the nice "a" prefix for asynchronous stuff.

Mmm fair point! Though, coroutines aren't really asynchronous objects in that usage, right? Since `await coroutine` would run that coroutine synchronously.

But the coroutine object itself isn't synchronous, it represents suspended processing and by stuffing it into a task can be run asynchronously. If we don't try to consider it a separate third thing, I'd still put the coroutine object in the asynchronous bucket and say await is the thing synchronously waiting for it.

Re: A conceptual overview of asyncio

#30

Post by anordin95 »

crvdgc wrote: post #26

asyncio by itself doesn't support asynchronous file I/O, see their wiki: https://github.com/python/asyncio/wiki/ThirdParty#filesystem You have to use something like aiofiles to do that.

Mhm. You need another thread to accomplish async file reads, which is basically what aiofiles does. This isn't really to the fault of asyncio. The necessary OS primitive isn't available. See the Linux documentation for the O_NONBLOCK flag and note this part: "Note that this flag has no effect for regular files" [1]. I actually originally wrote the sockets example in this article as using file i/o until I came across this bump in the road.

[1] https://man7.org/linux/man-pages/man2/open.2.html

Post reply on HN