Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Python’s asyncio primitives get wrong about shared state

inngest.com

Post reply on HN
Print view View original

21–30 of 55 posts

Re: What Python’s asyncio primitives get wrong about shared state

#21

Post by itemize123 »

evil-olive wrote: post #8

the title seems unnecessarily clickbaity. rather than "What Python's asyncio primitives get wrong" this seems more like "why we chose one asyncio primitive (queue) instead of others (event and condition)" also, halfway through the post, the problem grows a new requirement: > Instead of waking consumers and asking "is the current state what you want?", buffer every transition into a per-consumer queue. Each consumer d…

it does seem the user wants a conditional variable.

For locking I am guessing they want multithreading, each with an event loop.

Re: What Python’s asyncio primitives get wrong about shared state

#22

Post by three14 »

The thing that burned me with asyncio primitives is that calling an async function doesn't even schedule it (by default). The pattern for fire-and-forget is impossible to guess when coming from any other language - although it's called out in the docs. You must also call create_task AND you must maintain a collection of tasks because otherwise the task might not run, because it can be garbage collected before running, AND you must clean up completed tasks from your collection.

Re: What Python’s asyncio primitives get wrong about shared state

#23

Post by hrmtst93837 »

TZubiri wrote: post #3

I'm sorry but how do you jump from 1. Polling to 2. Asyncio There's so many solutions in the middle, I have this theory that most people that get into async don't really know what threading is. Maybe they have a world vision where before 2023 python just could not do more than one thing at once, that's what the GIL was right? But now after 3.12 Guido really pulled himself by the bootstraps and removed the GIL and imp…

[flagged]

Re: What Python’s asyncio primitives get wrong about shared state

#25

Post by whilenot-dev »

three14 wrote: post #22

The thing that burned me with asyncio primitives is that calling an async function doesn't even schedule it (by default). The pattern for fire-and-forget is impossible to guess when coming from any other language - although it's called out in the docs. You must also call create_task AND you must maintain a collection of tasks because otherwise the task might not run, because it can be garbage collected before running…

> The pattern for fire-and-forget is impossible

Good, that's an antipattern in the coroutines concurrency model.

Re: What Python’s asyncio primitives get wrong about shared state

#26

Post by vlovich123 »

ori_b wrote: post #19
pothamk wrote: post #7

[flagged]

Async and await is manually scheduling threads. So, if you're quite careful about what functions you call, you can arrange things so that you don't get concurrency when you don't want it. Being careful about what functions you call is quite fragile and tedious, and doesn't compose well: what if a library changes when it adds a yield point? Overall, async/await is a result of people programming like it's 2003, when th…

Threads are still expensive in Python - can’t use them for concurrency really like you can with async io afaik.

Re: What Python’s asyncio primitives get wrong about shared state

#27

Post by josephg »

ori_b wrote: post #19
pothamk wrote: post #7

[flagged]

Async and await is manually scheduling threads. So, if you're quite careful about what functions you call, you can arrange things so that you don't get concurrency when you don't want it. Being careful about what functions you call is quite fragile and tedious, and doesn't compose well: what if a library changes when it adds a yield point? Overall, async/await is a result of people programming like it's 2003, when th…

This works a lot better in JavaScript, which is exactly this model - a single threaded executor with async await. The problem you talk about is solved with function colouring. Async functions are marked as such. In general, sync functions can’t call async functions. (Well, you can invoke them. You just can’t run them to completion before returning).

For all the complaints about function colouring, I’m glad JavaScript has them. A sync function becoming an async function is a breaking API change. This is much better than the situation in Python, where yield points are invisible.

Re: What Python’s asyncio primitives get wrong about shared state

#28

Post by mrkeen »

This is one of those cases where software transactional memory really shines.

You can often take the naive solution and it will be the correct one. Your code will looks like your intent.

TFA's first attempt:

  async def drain_requests():
      while state != "closing":
          await asyncio.sleep(0.1)
      print("draining pending requests")
Got it. Let's port it to STM:

  let drain_requests = do
          atomically (
              do s
Thread-safe and no busy-waiting. No mention of 'notify', 'sleep'. No attempt to evade the concurrency issues, as in the articles "The fix: per-consumer queues - Each consumer drains its own queue and checks each transition individually."

Re: What Python’s asyncio primitives get wrong about shared state

#29

Post by bootsmann »

pothamk wrote: post #7

[flagged]

This is why we moved a lot of our concurrent python project to golang. There were a couple of cases where some engineer built the system by implicitly relying on the assumption that some coroutine would run blocking until a certain point was reached (avoiding a potential data race) that was then later broken by another change. At least in go we know we cannot rely on this so the concurrency safety has to be considered at all times, leading to better code.

Re: What Python’s asyncio primitives get wrong about shared state

#30

Post by pocksuppet »

mrkeen wrote: post #28

This is one of those cases where software transactional memory really shines. You can often take the naive solution and it will be the correct one. Your code will looks like your intent. TFA's first attempt: async def drain_requests(): while state != "closing": await asyncio.sleep(0.1) print("draining pending requests") Got it. Let's port it to STM: let drain_requests = do atomically ( do s Thread-safe and no busy-wa…

In most STM models this is a busy-wait implemented with STM? Only Haskell blocks on `retry`
Post reply on HN