71–80 of 188 posts
Earlier quoted context omitted.
This is an insane take, if you are doing async you already HAVE a need for concurrent programming. Async just makes that simpler to read and write.
Async python by design is not concurrent, nothing is executing at the same time on different cores/threads. There's one worker loop and one task running on it at a time.
Earlier quoted context omitted.
Ok. This is what has been a huge hangup for me. It really seems that if you're doing asyncio, you must do EVERYTHING async, it's like asyncio takes over (infects?) the entire program.
Yes async is viral, its all or nothing
(If you really, really need to do this - triple check - use queues & worker threads to decouple the sync parts and keep them out of your event loop thread, as a sort of sync-async impedence match.)
In case the author reads this, there is an error in the "How to Execute a Blocking I/O or CPU-bound Function in Asyncio?" [0] It reads: > The asyncio.to_thread() function creates a ThreadPoolExecutor behind the scenes to execute blocking calls. > As such, the asyncio.to_thread() function is only appropriate for IO-bound tasks. It should say it's only appropriate for CPU-bound tasks. [0]: https://superfastpython.com/p…
Not mentioned: The behavior is different between python 3.6 and 3.8. That was fun to debug. Python is simply not the right language for asynchronous programming. Things that are easy in many other languages are hard and don't work as you'd expect. If you are in a python-only shop, brush up on your presentation skills and see if you can convince them to branch out. Or write a shell script. You're better off with "./re…
that is an absurd exaggeration, or you don't realize all the issues of low level sockets that are abstracted away
---
#!/bin/bash
python3.8 read_one_socket.py $1
---
Use the python socket libraries if you must, but don't use python asynchronously.
I have used asyncio through aiohttp, and I have been pretty happy with it, but I also started with it from the beginning, so that probably made things a little easier. My setup is a bunch of microservices that each run an aiohttp web server based api for calls from the browser where communications between services are done async using rabbitmq and a hand rolled pub/sub setup. Almost all calls are non-blocking, except…
How does this compare to doing the same with eg. Django Channels (or other ASGI-aware frameworks)?
I have yet to find a use case compelling enough to dive into async in Python (doesn't help that I also work in JS and Go so I just turn to them for in cases where I could maybe use asyncio). This is not to say it's useless, just that I'm still searching for a problem this is the best solution for.
* A Postfix TCP table.
* A milter.
* DNS request forwarding.
* Reading data from a Unix domain socket and firing off dynamic DNS updates.
* A DNS proxy for Redis.
* A netflow agent.
* A stream feeder for Redis.
https://github.com/search?q=user%3Am3047+asyncio&type=Reposi...
By the way you can't use it for disk I/O, but you can try to use it for e.g. STDOUT: https://github.com/m3047/shodohflo/blob/5a04f1df265d84e69f10...
class UniversalWriter(object):
"""Plastering over the differences between file descriptors and network sockets."""Earlier quoted context omitted.
All of distributed machine learning would benefit from async, but it's unfortunately a Python-only ecosystem for the time being. Probably gonna be a couple years before that has a chance of changing
How would ML benefit from async? I thought everything interesting in things like TF or PyTorch is done by native code which is free to use multithreading and anything at all. (I know little about ML.)
Right now the typical paradigm is this really clunky lock-step across nodes waiting for each other (everything is blocking). If your hardware is non-homogenous (both interconnects and accelerators) or your program needs to be run in a pipelined way, you're fighting an extremely uphill battle.
Go, JavaScript etc are the usual languages of choice for this type of stuff because it's easy to express non-blocking code.
Earlier quoted context omitted.
Depending on the details you can often call an async function from a sync function, by just running the event loop, it is just a few lines of code. The problem is that the event loop is not reentrant [1], so if your sync function is being called from an async function, things do not work. Why would you ever do this you might think? Well, asyncio might be an implementation detail of whatever environment you are using.…
> there are... hacks to make this sort of work of course. The python motto.
After 2 years of using asyncio in production, I recommend to avoid it if you can. With async programming, you take the complexity of concurrent programming, which is way harder than you can imagine. Also nobody mentions this for some reason, but asyncio doesn't make your programs faster, in fact it makes everything 100x SLOWER (we measured it multiple times, compared the same thing to the sync version), but makes you…
maybe the right aproach would have been a Threadpool? Plus you don't have to refractor the task. Just let it run sync, but you can make it also async
Earlier quoted context omitted.
Unfortunately the majority of people who use asyncio in the real world do not have a need for concurrenct programming. They are drawn to it because they believe it is faster, which, generally, it is not. That isn't to say there aren't reasonable usecases - but most people using asyncio are writing webapps which spend a lot of time in the CPU generating or outputting JSON/GraphQL/HTML.
I think this may be influenced by Node. With JS and Node, async is cheap and ergonomic, so it's very widely used, even if the latency gains are marginal. With Python and asyncio, neither assumption holds, but the idea of shaving off some latency persists: if we rewrite everything using asyncio, our DB accesses can be parallelized, yay! This may or may not be a net win in latency; reworking your DB access patterns may…
Nope: https://techspot.zzzeek.org/2015/02/15/asynchronous-python-a...