Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Python’s asyncio primitives get wrong about shared state

inngest.com

Post reply on HN
Print view View original

41–50 of 55 posts

Re: What Python’s asyncio primitives get wrong about shared state

#41

Post by josephg »

pocksuppet wrote: post #31
josephg wrote: post #27

Earlier quoted context omitted.

This works a lot better in JavaScript, which is exactly this model - a single threaded executor with async await. The problem you talk about is solved with function colouring. Async functions are marked as such. In general, sync functions can’t call async functions. (Well, you can invoke them. You just can’t run them to completion before returning). For all the complaints about function colouring, I’m glad JavaScript…

Until every function is async.

Except nobody writes code like that. It would be horrible.

Sync functions perform better than async functions, and they give more guarantees to the caller. They're strictly better, when you can use them.

If you find yourself making everything async, your design is bad and you should refactoring your code.

Re: What Python’s asyncio primitives get wrong about shared state

#42

Post by TZubiri »

scuff3d wrote: post #20
TZubiri wrote: post #3

I'm sorry but how do you jump from 1. Polling to 2. Asyncio There's so many solutions in the middle, I have this theory that most people that get into async don't really know what threading is. Maybe they have a world vision where before 2023 python just could not do more than one thing at once, that's what the GIL was right? But now after 3.12 Guido really pulled himself by the bootstraps and removed the GIL and imp…

As of 3.14 running without the GIL is optional, but the default still has the GIL in place. 3.13 had it as experimental, but not officially supported. 3.12 and back are all GIL all day. Python's asyncio library is single threaded, so I'm not sure why you are talking about threads and asyncio like they have anything to do with each other. Python has been able to do more then one thing at a time for a long time. That's…

> Python's asyncio library is single threaded, so I'm not sure why you are talking about threads and asyncio like they have anything to do with each other.

Ok, not OS threads, but it de facto creates application/green threads.

>That's what the multiprocess library is for. It's not an ideal solution, but it does exist.

Philosophical argument but, I'd say multiprocess is not python doing many things, there would be many python runtimes (each doing A Thing), and the OS would be the one doing multiple things / scheduling.

Re: What Python’s asyncio primitives get wrong about shared state

#43

Post by scuff3d »

TZubiri wrote: post #42
scuff3d wrote: post #20

Earlier quoted context omitted.

As of 3.14 running without the GIL is optional, but the default still has the GIL in place. 3.13 had it as experimental, but not officially supported. 3.12 and back are all GIL all day. Python's asyncio library is single threaded, so I'm not sure why you are talking about threads and asyncio like they have anything to do with each other. Python has been able to do more then one thing at a time for a long time. That's…

> Python's asyncio library is single threaded, so I'm not sure why you are talking about threads and asyncio like they have anything to do with each other. Ok, not OS threads, but it de facto creates application/green threads. >That's what the multiprocess library is for. It's not an ideal solution, but it does exist. Philosophical argument but, I'd say multiprocess is not python doing many things, there would be man…

It absolutely does not create green threads. Green threads can be preempted and switched by the runtime. Go does this for example.

Python's asyncio is tasked based, the event loop cannot switch out a task until it reaches a yield point.

Or in short, green threads like Go uses are preemptive multitasking, the task based model asyncio uses is cooperative. A CPU bound python task can block the event loop forever if it never yields, goroutines generally can't.

It's not a philosophical question at all. A single python program can use the multiprocessing library to run multiple chunks of work in parallel. It's heavier weight thanks to the need to basically run a full python interpreter in an os process, but it provides the functionality. And the fact that it's scheduled by the OS is irrelevant. Plenty of languages use the underlying OS threading capabilities to manage threads instead of their own runtime. Both Java and C# for example (though I think Java is adding green threads now).

You're conflating several different distinct ideas. It's a common mistake I see at work all the time. Took a good amount of reading for me to untangle them all.

Edit: I left out stackless coroutines vs stackful thread like runtime. That would be more accurate then the preemptive vs cooperative stuff, but either way the gist of my comment is correct.

Re: What Python’s asyncio primitives get wrong about shared state

#44

Post by bombolo »

ori_b wrote: post #19
pothamk wrote: post #7

[flagged]

Async and await is manually scheduling threads. So, if you're quite careful about what functions you call, you can arrange things so that you don't get concurrency when you don't want it. Being careful about what functions you call is quite fragile and tedious, and doesn't compose well: what if a library changes when it adds a yield point? Overall, async/await is a result of people programming like it's 2003, when th…

[dead]

Re: What Python’s asyncio primitives get wrong about shared state

#46

Post by cl3misch »

seanhunter wrote: post #34

A better title would be: “Person who doesn’t know how to write state machines struggles to write a state machine”. In attempt 2 the old school C way of writing the state machine would work just fine in python, avoid a bunch of the boilerplate and avoid the “state setter needs to know a bunch of stuff” problem. Basically you make the states as a table and put the methods you need in the table so in python a dictionary…

Is this being downvoted because of the tone, or because state machines are unpopular/inappropriate in this case?

Genuine question, because this feels like a sensible solution to the problem as stated in the article.

Re: What Python’s asyncio primitives get wrong about shared state

#47

Post by three14 »

whilenot-dev wrote: post #25
three14 wrote: post #22

The thing that burned me with asyncio primitives is that calling an async function doesn't even schedule it (by default). The pattern for fire-and-forget is impossible to guess when coming from any other language - although it's called out in the docs. You must also call create_task AND you must maintain a collection of tasks because otherwise the task might not run, because it can be garbage collected before running…

> The pattern for fire-and-forget is impossible Good, that's an antipattern in the coroutines concurrency model.

Then someone should really update the official python docs that explain the fire-and-forget pattern (https://docs.python.org/3/library/asyncio-task.html#asyncio....)! I had a FastAPI server, and calling a particular endpoint is supposed to kick off some work in the background. The background work does very little CPU work, but does often need to await more work for several minutes, so it's a good fit for asyncio. How do you want it to be structured? (In other words, on the level of human requirements, it IS fire and forget.)

Re: What Python’s asyncio primitives get wrong about shared state

#48

Post by mrkeen »

cl3misch wrote: post #46
seanhunter wrote: post #34

A better title would be: “Person who doesn’t know how to write state machines struggles to write a state machine”. In attempt 2 the old school C way of writing the state machine would work just fine in python, avoid a bunch of the boilerplate and avoid the “state setter needs to know a bunch of stuff” problem. Basically you make the states as a table and put the methods you need in the table so in python a dictionary…

Is this being downvoted because of the tone, or because state machines are unpopular/inappropriate in this case? Genuine question, because this feels like a sensible solution to the problem as stated in the article.

It made no reference to the 'shared' in 'shared state'.

No mention of asynchrony, multithreading, or the race condition that TFA encountered.

Re: What Python’s asyncio primitives get wrong about shared state

#50

Post by seanhunter »

mrkeen wrote: post #48
cl3misch wrote: post #46

Earlier quoted context omitted.

Is this being downvoted because of the tone, or because state machines are unpopular/inappropriate in this case? Genuine question, because this feels like a sensible solution to the problem as stated in the article.

It made no reference to the 'shared' in 'shared state'. No mention of asynchrony, multithreading, or the race condition that TFA encountered.

The “attempt 2” was literally a state machine implementation which the author rejected because they didn’t know how to do it properly and so did it badly using a bunch of if then else logic.
Post reply on HN