Just in time, I've been needing to implement s3 upload asynchronously.
Are you sure it's worth the complexity? Threads will drop the GIL for the duration of the network bound I/O .
Python Asyncio
41–50 of 188 posts
Re: Python Asyncio
#42After 2 years of using asyncio in production, I recommend to avoid it if you can. With async programming, you take the complexity of concurrent programming, which is way harder than you can imagine. Also nobody mentions this for some reason, but asyncio doesn't make your programs faster, in fact it makes everything 100x SLOWER (we measured it multiple times, compared the same thing to the sync version), but makes you…
asyncio gets a lot of hype, but I prefer concurrent.futures. It feels more simple and general since I don't have to make everything awaitable.
Re: Python Asyncio
#43Earlier quoted context omitted.
That's exactly the opposite of what it says really: if you need to do CPU stuff, then you can do that, it just won't be using asyncio. So it doesn't really infect your whole program. You could easily have, say, a thread to do all your asyncio stuff, another to do some CPU intenstive stuff (so long as it blocks the GIL) and yet another to do some blocking I/O e.g. interacting with a database with its own blocking APIs…
I think the issue is more like this: 1. You jump into a huge codebase and find a very useful function you want to use in your code (it uses asyncio) 2. Your code doesn't use asyncio, so you start converting functions to be async (or else you can't use the `await` keyword) 3. It turns out your function is called by a bunch of different users, some of which do not use the asyncio runner 4. You end up having several mee…
Depending on the details you can often call an async function from a sync function, by just running the event loop, it is just a few lines of code.
The problem is that the event loop is not reentrant [1], so if your sync function is being called from an async function, things do not work. Why would you ever do this you might think? Well, asyncio might be an implementation detail of whatever environment you are using. Jupyiter or ipython for example.
[1] there are... hacks to make this sort of work of course.
Re: Python Asyncio
#44Maybe off-topic, but my advice would be: if you need this guide, consider to switch to another language, if that's possible. In our company we switched to Go, and all those asyncio problems were magically solved.
Agreed. Moving my Python data analytics to a GRPC server that I call from a Go service has been _so much_ easier to manage and debug.
Re: Python Asyncio
#45Earlier quoted context omitted.
I think the issue is more like this: 1. You jump into a huge codebase and find a very useful function you want to use in your code (it uses asyncio) 2. Your code doesn't use asyncio, so you start converting functions to be async (or else you can't use the `await` keyword) 3. It turns out your function is called by a bunch of different users, some of which do not use the asyncio runner 4. You end up having several mee…
Depending on the details you can often call an async function from a sync function, by just running the event loop, it is just a few lines of code. The problem is that the event loop is not reentrant [1], so if your sync function is being called from an async function, things do not work. Why would you ever do this you might think? Well, asyncio might be an implementation detail of whatever environment you are using.…
> there are... hacks to make this sort of work of course.
The python motto.
Re: Python Asyncio
#46Not mentioned: The behavior is different between python 3.6 and 3.8. That was fun to debug. Python is simply not the right language for asynchronous programming. Things that are easy in many other languages are hard and don't work as you'd expect. If you are in a python-only shop, brush up on your presentation skills and see if you can convince them to branch out. Or write a shell script. You're better off with "./re…
All of distributed machine learning would benefit from async, but it's unfortunately a Python-only ecosystem for the time being. Probably gonna be a couple years before that has a chance of changing
Re: Python Asyncio
#47The only thing I ever use async in Python for is batch requests, like hundreds or thousands of little requests. I collect them up in a task list with context, run them with one of the “run all of these and return the results or errors” functions in the library, then process the results serially.
Anything I need to do that doesn’t use this simple IO pattern, like cpu bound workers, I prefer to use processes or multiple copies of the app synchronized with a task queue.
Re: Python Asyncio
#48Not mentioned: The behavior is different between python 3.6 and 3.8. That was fun to debug. Python is simply not the right language for asynchronous programming. Things that are easy in many other languages are hard and don't work as you'd expect. If you are in a python-only shop, brush up on your presentation skills and see if you can convince them to branch out. Or write a shell script. You're better off with "./re…
What changed between those two releases? 3.6 was the last time I worked with asyncio
Re: Python Asyncio
#49Just in time, I've been needing to implement s3 upload asynchronously.
Does aioboto3 help here?
Re: Python Asyncio
#50After 2 years of using asyncio in production, I recommend to avoid it if you can. With async programming, you take the complexity of concurrent programming, which is way harder than you can imagine. Also nobody mentions this for some reason, but asyncio doesn't make your programs faster, in fact it makes everything 100x SLOWER (we measured it multiple times, compared the same thing to the sync version), but makes you…
This is an insane take, if you are doing async you already HAVE a need for concurrent programming. Async just makes that simpler to read and write.
Unfortunately the majority of people who use asyncio in the real world do not have a need for concurrenct programming. They are drawn to it because they believe it is faster, which, generally, it is not. That isn't to say there aren't reasonable usecases - but most people using asyncio are writing webapps which spend a lot of time in the CPU generating or outputting JSON/GraphQL/HTML.