11–20 of 55 posts
[flagged]
the title seems unnecessarily clickbaity. rather than "What Python's asyncio primitives get wrong" this seems more like "why we chose one asyncio primitive (queue) instead of others (event and condition)" also, halfway through the post, the problem grows a new requirement: > Instead of waking consumers and asking "is the current state what you want?", buffer every transition into a per-consumer queue. Each consumer d…
I think it’s not so much that the asyncio primitives got wrong about shared state, as much as is what the authors got wrong about the usage of those primitives. They are classic concurrency primitives that’s been around for almost half a century. They work as designed, but require some care to use correctly.
[flagged]
I don't fully agree with this. Yes, surely shared mutable state can suffer from similar issues, however the cooperative nature of coroutines makes this much easier to handle. OS threads are preemptive and actually run in parallel, so you have to be aware of CPU concurrency and always be ready for a context switch.
The one thing I wish stock python queues had an option for (async or otherwise) was some kind of explicit termination. e.g. be split into producers and consumers, and have consumers indicate iteration complete when all producers have finished (and vice versa - signal producers that all consumers have gone away). You can kind of kludge around it in one direction with stop sentinals but it's a lot more awkward to deal…
https://docs.python.org/3/library/queue.html#queue.Queue.tas...
[flagged]
I don't fully agree with this. Yes, surely shared mutable state can suffer from similar issues, however the cooperative nature of coroutines makes this much easier to handle. OS threads are preemptive and actually run in parallel, so you have to be aware of CPU concurrency and always be ready for a context switch.
With traditional locking, the locked segment is usually very clear. It's possible to use race detectors to verify that objects are accessed with consistent locking. The yield points are also clear.
With async stuff, ANY await point can change ANY state. And await points are common, even sometimes for things like logging. There are also no tools to verify the consistent "locking".
So I often spend hours staring blankly at logs, trying to reconstruct a possible sequence of callbacks that could have led to a bug. E.g.: https://github.com/expo/expo/issues/39428
What about a more general message-passing mailbox approach? This works really well in the Erlang/gen_server/gen_fsm world. (and in plenty of other contexts, but Erlang's OTP is still some of the best, simplest incarnation of these things)
[flagged]
Being careful about what functions you call is quite fragile and tedious, and doesn't compose well: what if a library changes when it adds a yield point?
Overall, async/await is a result of people programming like it's 2003, when threads were still very expensive.
I'm sorry but how do you jump from 1. Polling to 2. Asyncio There's so many solutions in the middle, I have this theory that most people that get into async don't really know what threading is. Maybe they have a world vision where before 2023 python just could not do more than one thing at once, that's what the GIL was right? But now after 3.12 Guido really pulled himself by the bootstraps and removed the GIL and imp…
Python's asyncio library is single threaded, so I'm not sure why you are talking about threads and asyncio like they have anything to do with each other.
Python has been able to do more then one thing at a time for a long time. That's what the multiprocess library is for. It's not an ideal solution, but it does exist.