Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python has had async for 10 years – why isn't it more popular?

tonybaloney.github.io

Post reply on HN
Print view View original

261–270 of 305 posts

Re: Python has had async for 10 years – why isn't it more popular?

#261

Post by _dain_ »

harpiaharpyja wrote: post #237

I've been working quite heavily with async Python for five and a half years now. I've been the principal developer of a control system framework for laboratory automation, written pretty much entirely in async Python. I say framework because it's a reusable engine that has gone on to become the foundation for three projects so far. Our organization is primarily involved in materials research. At it's heart it's kind…

Oh hey, I'm the mirror universe version of you. I used to work in a semiconductor plant, writing Python code that controlled robot arms and electronic measurement instruments and so on. In my universe we used threads over blocking calls instead of async and it was exactly as bad as you might imagine.

>Having cancellation work smoothly is also pretty important to us

+10000. Threads don't have good cancellation semantics, so we never had a robust solution to the "emergency shutdown" problem where you need to tell all the running equipment to stop whatever they're doing and return to safe positions.

Every day I worked on that codebase I wished it had been async from the beginning, but I couldn't see a way to migrate gradually because function coloring makes it an all-or-nothing affair.

Re: Python has had async for 10 years – why isn't it more popular?

#262

Post by wink »

I think the 2/3 split had more impact than the author thinks.

Many people in my bubble (around 2013-2017) just never went with python 3, but chose other languages.

The company I was working for started important applications in python 2 as late as 2014 because the libraries we needed weren't ported yet. We never went python 3 later, but went to go instead, so we completely missed any python async thing.

Re: Python has had async for 10 years – why isn't it more popular?

#263

Post by pjmlp »

atomicnumber3 wrote: post #14

The author gets close to what I think the root problem is, but doesn't call it out. The truth is that in python, async was too little, too late. By the time it was introduced, most people who actually needed to do lots of io concurrently had their own workarounds (forking, etc) and people who didn't actually need it had found out how to get by without it (multiprocessing etc). Meanwhile, go showed us what good green…

Java has had green threads since day one, most vendors ended up going red threads full way, and now we're back into green and red world.

The main difference being that now both models are simultaneously supported instead of being an implementation detail of each JVM.

Re: Python has had async for 10 years – why isn't it more popular?

#264

Post by guappa »

tdumitrescu wrote: post #22
PaulHoule wrote: post #4

I went through a phase of writing asyncio servers for my side projects. Probably the most fun I had was writing things that were responsive in complex ways, such as a websockets server that was also listening on message queues or on a TCP connection to a Denon HEOS music player. Eventually I wrote an "image sorter" that I found was hanging up when the browser was trying to download images in parallel, the image servi…

That's the main problem with evented servers in general isn't it? If any one of your workloads is cpu-intensive, it has the potential to block the serving of everything else on the same thread, so requests that should always be snappy can end up taking randomly long times in practice. Basically if you have any cpu-heavy work, it shouldn't go in that same server.

Backend developers finding out why user interfaces have a thread for the GUI and a thread for doing work :D

Re: Python has had async for 10 years – why isn't it more popular?

#265

Post by guappa »

lyu07282 wrote: post #180
int_19h wrote: post #161

Earlier quoted context omitted.

FWIW Python got async/await before JavaScript did. I believe at the time the main inspiration was C#.

JavaScript was always single-threaded asynchronous, the added async/await keywords were just syntactic sugar. Node.js became popular before it as well, though I found at the time it was difficult to avoid callback hell similar to using libuv directly in C.

async await is syntactic sugar hiding calls to poll() and callbacks in every programming language.

Re: Python has had async for 10 years – why isn't it more popular?

#266

Post by guappa »

physicsguy wrote: post #251

One of the big reasons I'd say is that 90% of Python work doesn't actually benefit from async? Basically nothing in the data science / simulation / etc. world benefits at all. Web development it does, and Django has sprinkled it in (but not DRF), Flask has a fork that's async and FastAPI is async. At work people have suggested we should switch our Flask code to async to 'make it faster' but for us, we're largely usin…

You'd save a bit of memory by letting 1 thread handle multiple connections, but that's about it.

Re: Python has had async for 10 years – why isn't it more popular?

#267

Post by guappa »

seunosewa wrote: post #104
rsyring wrote: post #61

Not too long ago, I read a comment on HN that suggested, due to Python's support for free-threading, async in Python will no longer be needed and will lose out to free-threading due to it's use of "colored" functions. Which seems to align with where this author ends up: > Because parallelism in Python using threads has always been so limited, the APIs in the standard library are quite rudimentary. I think there is an…

async was the wrong solution to the right problem - improving general performance. Free threading is the prize in an increasingly multi-core CPU world.

Threads use a lot more memory than a single async thread, and if the load is IO, 1 thread is enough.

Speed might be similar but resource usage is not the same at all.

Re: Python has had async for 10 years – why isn't it more popular?

#268

Post by guappa »

BrenBarn wrote: post #244

The main reason I've encountered is that async is generally an all-or-nothing thing. It's generally not possible to take an existing code base and just "add a little async here and there". The entire thing has to be restructured from the ground up for async. The gain has to be really major for this to be worthwhile, more so the better your existing code already works. This is sort of like the article's Problem 3, but…

Before async they had asyncore, which they now entirely removed. So the real early adopters of async in python are punished by having to rewrite their software to run it with a current version of python.

Re: Python has had async for 10 years – why isn't it more popular?

#269

Post by guappa »

harpiaharpyja wrote: post #237

I've been working quite heavily with async Python for five and a half years now. I've been the principal developer of a control system framework for laboratory automation, written pretty much entirely in async Python. I say framework because it's a reusable engine that has gone on to become the foundation for three projects so far. Our organization is primarily involved in materials research. At it's heart it's kind…

I think most of the problems are due to people not understanding how async works (non blocking file descriptors and a call to poll).

Re: Python has had async for 10 years – why isn't it more popular?

#270

Post by notepad0x90 »

For me, when I use python, it's because I want faster dev time to prove a concept or that I expect others with little to no programming experience to maintain the code in the future. So, I rarely ever use async because I never seem to be in a position where the debugging complexity and how hard it is for others/newbies to read and be familiar with the code is worth the performance improvements.

Like others are saying, if I want it fast and efficient (processing), I'll just use Go. Python isn't like JS in browsers, you don't have to use it, you have to want to use it. and the same goes with its features. Maybe if python tutorials/books and "How do i ____ in python?" search results used async, map, filter, collections,etc.. these awesome python features would be more prevalent. But, I can see how mature projects should probably mandate their usage where it makes sense.

Post reply on HN