61–70 of 188 posts
Missing yet very important topics: redis & db drivers in async. Or even async ORM.
Earlier quoted context omitted.
This is an insane take, if you are doing async you already HAVE a need for concurrent programming. Async just makes that simpler to read and write.
Async python by design is not concurrent, nothing is executing at the same time on different cores/threads. There's one worker loop and one task running on it at a time.
Not mentioned: The behavior is different between python 3.6 and 3.8. That was fun to debug. Python is simply not the right language for asynchronous programming. Things that are easy in many other languages are hard and don't work as you'd expect. If you are in a python-only shop, brush up on your presentation skills and see if you can convince them to branch out. Or write a shell script. You're better off with "./re…
What changed between those two releases? 3.6 was the last time I worked with asyncio
https://tryexceptpass.org/article/asyncio-in-37/
One of my ongoing frustrations is python isn't just the lack of backwards compatibility but that if you inherit a script there's no way to know which version of the language it was written for. If you're lucky you can track down the developers, but IME even they often say "I'm not sure, let me run python --version".
Earlier quoted context omitted.
> Async just makes that simpler to read and write. Really? I definitely had a huge struggle reading async python, because generators and yields break the concept of "what is a function" and struggled writing async python because of function coloring.
Day to day Python async should be using generators and yields.
There is very little in everyday Python usage that benefits from Asyncio. Two in webdev, are long running request (Websockets, SSE, long polling), and processing multiple backend IO processes in parallel. However the later is very rare, you may think you have multiple DB request that could use asyncio, but most of the time they are dependent on each other. Almost all of the time a normal multithreaded Python server i…
I have several python services running as glue between some of our components. They all use threading. I find it alot easier to reason about.
Not mentioned: The behavior is different between python 3.6 and 3.8. That was fun to debug. Python is simply not the right language for asynchronous programming. Things that are easy in many other languages are hard and don't work as you'd expect. If you are in a python-only shop, brush up on your presentation skills and see if you can convince them to branch out. Or write a shell script. You're better off with "./re…
I find Python async to be fun and exciting and interesting and powerful.
BUT it is a big power tool and there’s so much in it that it’s hard to work out how to drive it right.
I have pretty good experience with Python and javascript.
I prefer Python to javascript when writing async code.
Specific example I spent hours trying to drive some processes via stdin/stout/stderr with javascript and it kept failing for reasons I couldn’t determine.
Switched to Python async and it just worked.
The most frustrating thing about async Python is that it has been improving greatly. That means that it’s not obvious what “the right way” is, ie using the latest techniques. This is actually a really big problem for async Python. I’m fairly competent with it, but still have to spend ages working out if I’m doing it “the right way/the latest way”.
The Python project really owes it to its users to have a short cookbook that shows the easiest, most modern recommended way to do common tasks. Somehow this cookbook must give the reader instant 100% confidence that they are reading the very latest official recommendations and thinking on simple asyncio techniques.
Without such a “latest and greatest techniques of async Python cookbook” it’s too easy to get lost in years of refinement and improvement and lower and higher level techniques.
The Python project should address this, it’s a major ease of use problem.
Ironically, pythons years of async refinement mean there’s many many many ways to get the same things done, conflicting with pythons “one right way to do it” philosophy.
It can be solved with documentation that drives people to the simplest most modern approaches.
Not mentioned: The behavior is different between python 3.6 and 3.8. That was fun to debug. Python is simply not the right language for asynchronous programming. Things that are easy in many other languages are hard and don't work as you'd expect. If you are in a python-only shop, brush up on your presentation skills and see if you can convince them to branch out. Or write a shell script. You're better off with "./re…
I was tasked with improving a performance-critical object detection codebase for video that was written in Python. Before attempting a complete rewrite, I identified that outside of FFI calls related to PyTorch/OpenCV, some of the biggest latency overhead was due to the fact that the network I/O for fetching data to process was blocking the actual inference, so I tried to decouple the two so they could happen synchronously.
My attempt at doing this with asyncio went horrendously at best. There were no mature client libraries for the cloud services we used at the time. I was digging through PyTorch forums to figure out the relationship between the GIL and CUDA Streams. Boundless headaches.
Then I realized, I could just separate the code that loaded the data onto the machine from the inference code into two separate code bases and just have them communicate over really simple signals on a UNIX socket. Two different processes, sending a simple fixed-size string back and forth, and then I just used multiprocessing.Pool for parallelizing the downloads.
It reads:
> The asyncio.to_thread() function creates a ThreadPoolExecutor behind the scenes to execute blocking calls.
> As such, the asyncio.to_thread() function is only appropriate for IO-bound tasks.
It should say it's only appropriate for CPU-bound tasks.
[0]: https://superfastpython.com/python-asyncio/#How_to_Execute_a...