Missing yet very important topics: redis & db drivers in async. Or even async ORM.
[deleted]
Python Asyncio
81–90 of 188 posts
Re: Python Asyncio
#82Not mentioned: The behavior is different between python 3.6 and 3.8. That was fun to debug. Python is simply not the right language for asynchronous programming. Things that are easy in many other languages are hard and don't work as you'd expect. If you are in a python-only shop, brush up on your presentation skills and see if you can convince them to branch out. Or write a shell script. You're better off with "./re…
[deleted]
Re: Python Asyncio
#83In case the author reads this, there is an error in the "How to Execute a Blocking I/O or CPU-bound Function in Asyncio?" [0] It reads: > The asyncio.to_thread() function creates a ThreadPoolExecutor behind the scenes to execute blocking calls. > As such, the asyncio.to_thread() function is only appropriate for IO-bound tasks. It should say it's only appropriate for CPU-bound tasks. [0]: https://superfastpython.com/p…
I think the article is correct here, actually - if you need to run a CPU-bound task, you’ll need a ProcessPoolExecutor.
I second this. Python has a Global Interpreter Lock, so only one Python instruction executes at a time.
Async and threads only help you with I/O (where the OS affords waiting for multiple operations in parallel).
To execute CPU-bound code in parallel you need multiple processes.
I think this is a major disadvantage of Python, because processes are much costlier to spawn, and if you implement long-running workers to avoid frequent spawning, you have to incur serialization/deserialization costs, because shared memory support is very rudimentary (in essence, just fixed size numeric code).
Re: Python Asyncio
#84I've never bothered to learn Python asyncio. When Python 3.5 came out I just thought it looked overly complex. Coming from a C/C++ background on Linux I just use the select package for waiting on blocking I/O, mainly sockets. Do you think there is something to gain for me by learning asyncio?
Re: Python Asyncio
#85In case the author reads this, there is an error in the "How to Execute a Blocking I/O or CPU-bound Function in Asyncio?" [0] It reads: > The asyncio.to_thread() function creates a ThreadPoolExecutor behind the scenes to execute blocking calls. > As such, the asyncio.to_thread() function is only appropriate for IO-bound tasks. It should say it's only appropriate for CPU-bound tasks. [0]: https://superfastpython.com/p…
I think the article is correct here, actually - if you need to run a CPU-bound task, you’ll need a ProcessPoolExecutor.
Well, that section is talking about CPU-bound tasks, and both threads and processes are valid choices for CPU-bound tasks, with different tradeoffs.
I think there is often a lot of confusion around that because a lot of tutorials simply say to use threads for IO-bound tasks and processes for CPU-bound tasks, without going deeper into the differences. Threads can easily share memory which often increases complexity and requires locks to run safely, with processes you avoid locking (including the infamous Python GIL) but also have to pass around data which might hurt performance.
Regardless, it still doesn't make sense to say threads are only appropriate for IO-bound tasks, the official documentation [0] seems to lean towards preferring threads for mixing IO and CPU-bound tasks, and processes for purely CPU-bound tasks.
[0]: https://docs.python.org/3/library/concurrent.futures.html#th...
Re: Python Asyncio
#86I'm not in a much of a position to evaluate Python's asyncio since I really have not used it very much. However, over the last few days, I started to dig into it and tried to get some (what I think are) very basic examples working and really struggled. That alone is not fully dispositive because I've used many async implementations in various languages and each of them have a bit of a learning curve and their own wrinkles you have to ramp up on. That said, my limited experience at least tells me that Python's async has a FUX that leaves a lot to be desired. I've found other languages make it a lot easier to do concurrency, parallelism, async vs sync, blocking vs non-blocking and all that. I don't really know if the issue is poor documentation, the semantics of the APIs themselves...really not sure. I do know that I'm left with the feeling that "this seems like it's going to be a pretty big PITA".
Re: Python Asyncio
#87I'm not in a much of a position to evaluate Python's asyncio since I really have not used it very much. However, over the last few days, I started to dig into it and tried to get some (what I think are) very basic examples working and really struggled. That alone is not fully dispositive because I've used many async implementations in various languages and each of them have a bit of a learning curve and their own wri…
It IS easy.
The Python project just don’t make it obvious how to do it easy.
Re: Python Asyncio
#88Earlier quoted context omitted.
I think the article is correct here, actually - if you need to run a CPU-bound task, you’ll need a ProcessPoolExecutor.
I second this. Python has a Global Interpreter Lock, so only one Python instruction executes at a time. Async and threads only help you with I/O (where the OS affords waiting for multiple operations in parallel). To execute CPU-bound code in parallel you need multiple processes. I think this is a major disadvantage of Python, because processes are much costlier to spawn, and if you implement long-running workers to a…
It's just awkwardly worded, I think. It probably should include a mention of ProcessPoolExecutor in the following bit where it explains the run_in_executor function to make more sense. Especially as the whole section talks about passing CPU-bound task to a thread pool, but then recommends not to use a thread pool.
Though I think the GIL isn't really that big a deal for most CPU-bound code, with plenty of third party libraries and even within the python standard library a lot of the code is not native python, and therefore releases the GIL lock.
Re: Python Asyncio
#89Maybe off-topic, but my advice would be: if you need this guide, consider to switch to another language, if that's possible. In our company we switched to Go, and all those asyncio problems were magically solved.
I've had absolutely no problems using async programming in python. It's extremely easy to setup a script in 100-200 lines of code to do something like:
pull the rows from postgres that match query . Process the data and push each row as an event into rabbitmq. In <200 lines of code i was easily processing 25k/rows per second and it only took me a few minutes to figure out the script.
Re: Python Asyncio
#90I'm not in a much of a position to evaluate Python's asyncio since I really have not used it very much. However, over the last few days, I started to dig into it and tried to get some (what I think are) very basic examples working and really struggled. That alone is not fully dispositive because I've used many async implementations in various languages and each of them have a bit of a learning curve and their own wri…
It IS easy. The Python project just don’t make it obvious how to do it easy.
Right. So you think it's mainly an education/documentation issue? I'm totally open to that in fact being the case. One really simple use case I had was just annotating a method to run async and then calling that in a loop elsewhere which did speed up a bunch of work I was doing pretty dramatically.
import asyncio
def background(f):
def wrapped(*args, **kwargs):
return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs)
return wrapped
@background
def my_io_bound_function():...