Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Asyncio

superfastpython.com

Post reply on HN
Print view View original

11–20 of 188 posts

Re: Python Asyncio

#11

Post by brrrrrm »

bjt2n3904 wrote: post #5
brrrrrm wrote: post #2

Important to note: > They are suited to non-blocking I/O with subprocesses and sockets, however, blocking I/O and CPU-bound tasks can be used in a simulated non-blocking manner using threads and processes under the covers. If you're using it for anything besides slow async I/O, you're going to have to do some heavy lifting. I've also found the actual asyncio implementation in CPython to be slow. Measuring purely even…

Ok. This is what has been a huge hangup for me. It really seems that if you're doing asyncio, you must do EVERYTHING async, it's like asyncio takes over (infects?) the entire program.

I feel the same way, it's a lot like `const` in C++. Basically it's enforcing usage for every upstream user to ensure there's no inefficiency (such as waiting for things like the network).

JavaScript has an extremely nice "out" (I'm sure other languages do too):

    async function() {
      await async_thing()
      do_sync_stuff()
    }
can be written

    function() {
      async_thing().then(do_sync_stuff)
    }
which is quite intuitive and helps glue code together. I think the callback-based thinking really clarifies user intent ("just tell me when it's done") without impacting performance.

Re: Python Asyncio

#12

Post by kissgyorgy »

After 2 years of using asyncio in production, I recommend to avoid it if you can. With async programming, you take the complexity of concurrent programming, which is way harder than you can imagine.

Also nobody mentions this for some reason, but asyncio doesn't make your programs faster, in fact it makes everything 100x SLOWER (we measured it multiple times, compared the same thing to the sync version), but makes your program concurrent (NOT parallel), where the tradeoff is you use more CPU, maybe less memory, and can saturate your bandwidth better. Never do anything CPU-bound in an async loop!

Re: Python Asyncio

#13

Post by quietbritishjim »

bjt2n3904 wrote: post #5
brrrrrm wrote: post #2

Important to note: > They are suited to non-blocking I/O with subprocesses and sockets, however, blocking I/O and CPU-bound tasks can be used in a simulated non-blocking manner using threads and processes under the covers. If you're using it for anything besides slow async I/O, you're going to have to do some heavy lifting. I've also found the actual asyncio implementation in CPython to be slow. Measuring purely even…

Ok. This is what has been a huge hangup for me. It really seems that if you're doing asyncio, you must do EVERYTHING async, it's like asyncio takes over (infects?) the entire program.

That's exactly the opposite of what it says really: if you need to do CPU stuff, then you can do that, it just won't be using asyncio. So it doesn't really infect your whole program.

You could easily have, say, a thread to do all your asyncio stuff, another to do some CPU intenstive stuff (so long as it blocks the GIL) and yet another to do some blocking I/O e.g. interacting with a database with its own blocking APIs. In that case, you could spawn separate threads and exchange messages between them like usual.

Where this falls down a little is if you want asyncio to infer your whole program - or more accurately, if you want to use it in all of the task-management code at the top level of your app. In that case, you would use asyncio's thread API [1] although it's a bit clunky for when you want to maintain state bound to a specific worker thread (like SQLite database objects).

[1] https://docs.python.org/3/library/asyncio-task.html#running-...

Re: Python Asyncio

#14

Post by bityard »

I bet the actual content here is great, but I find the page to be unreadable with the large text, huge amounts of whitespace, and quotes (with Amazon referral links) sprinkled in between every other sentence.

Re: Python Asyncio

#16

Post by samwillis »

There is very little in everyday Python usage that benefits from Asyncio. Two in webdev, are long running request (Websockets, SSE, long polling), and processing multiple backend IO processes in parallel. However the later is very rare, you may think you have multiple DB request that could use asyncio, but most of the time they are dependent on each other.

Almost all of the time a normal multithreaded Python server is perfectly sufficient and much easer to code.

My recommendation is only use it where it is really REALY required.

Re: Python Asyncio

#17

Post by brrrrrm »

quietbritishjim wrote: post #13
bjt2n3904 wrote: post #5

Earlier quoted context omitted.

Ok. This is what has been a huge hangup for me. It really seems that if you're doing asyncio, you must do EVERYTHING async, it's like asyncio takes over (infects?) the entire program.

That's exactly the opposite of what it says really: if you need to do CPU stuff, then you can do that, it just won't be using asyncio. So it doesn't really infect your whole program. You could easily have, say, a thread to do all your asyncio stuff, another to do some CPU intenstive stuff (so long as it blocks the GIL) and yet another to do some blocking I/O e.g. interacting with a database with its own blocking APIs…

I think the issue is more like this:

1. You jump into a huge codebase and find a very useful function you want to use in your code (it uses asyncio)

2. Your code doesn't use asyncio, so you start converting functions to be async (or else you can't use the `await` keyword)

3. It turns out your function is called by a bunch of different users, some of which do not use the asyncio runner

4. You end up having several meetings with upstream teams asking them to use the asyncio runner

5. A single team says "no"

6. Scrap the project and just rewrite that original function you found synchronously

Re: Python Asyncio

#18

Post by acedTrex »

kissgyorgy wrote: post #12

After 2 years of using asyncio in production, I recommend to avoid it if you can. With async programming, you take the complexity of concurrent programming, which is way harder than you can imagine. Also nobody mentions this for some reason, but asyncio doesn't make your programs faster, in fact it makes everything 100x SLOWER (we measured it multiple times, compared the same thing to the sync version), but makes you…

This is an insane take, if you are doing async you already HAVE a need for concurrent programming. Async just makes that simpler to read and write.

Re: Python Asyncio

#19

Post by acedTrex »

bjt2n3904 wrote: post #5
brrrrrm wrote: post #2

Important to note: > They are suited to non-blocking I/O with subprocesses and sockets, however, blocking I/O and CPU-bound tasks can be used in a simulated non-blocking manner using threads and processes under the covers. If you're using it for anything besides slow async I/O, you're going to have to do some heavy lifting. I've also found the actual asyncio implementation in CPython to be slow. Measuring purely even…

Ok. This is what has been a huge hangup for me. It really seems that if you're doing asyncio, you must do EVERYTHING async, it's like asyncio takes over (infects?) the entire program.

Yes async is viral, its all or nothing

Re: Python Asyncio

#20

Post by nicolaslem »

samwillis wrote: post #16

There is very little in everyday Python usage that benefits from Asyncio. Two in webdev, are long running request (Websockets, SSE, long polling), and processing multiple backend IO processes in parallel. However the later is very rare, you may think you have multiple DB request that could use asyncio, but most of the time they are dependent on each other. Almost all of the time a normal multithreaded Python server i…

Agreed, the SaaS Python application I maintain does SSE with threads. It is not the prettiest thing but it works and threads are cheaper than rewriting the whole thing to be async.
Post reply on HN