Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python has had async for 10 years – why isn't it more popular?

tonybaloney.github.io

Post reply on HN
Print view View original

131–140 of 305 posts

Re: Python has had async for 10 years – why isn't it more popular?

#131

Post by leecarraher »

atomicnumber3 wrote: post #14

The author gets close to what I think the root problem is, but doesn't call it out. The truth is that in python, async was too little, too late. By the time it was introduced, most people who actually needed to do lots of io concurrently had their own workarounds (forking, etc) and people who didn't actually need it had found out how to get by without it (multiprocessing etc). Meanwhile, go showed us what good green…

i agree, also add to that, that many python modules are foss projects that are maintained on a limited basis or budget. Refactoring code that may have some unsafe async routines would be costly for an org, and dreadful for recreation. So you can either have a rich library of modules, or go async and risk something you need not working then having to find a workaround. Personally, if parallelism is important enough, i use ctypes and openmp. If i need something more portable, i have a few multiprocessing wrappers that implement prange and a few other widgets for shared memory.

Re: Python has had async for 10 years – why isn't it more popular?

#132

Post by hinkley »

parhamn wrote: post #34
atomicnumber3 wrote: post #14

The author gets close to what I think the root problem is, but doesn't call it out. The truth is that in python, async was too little, too late. By the time it was introduced, most people who actually needed to do lots of io concurrently had their own workarounds (forking, etc) and people who didn't actually need it had found out how to get by without it (multiprocessing etc). Meanwhile, go showed us what good green…

pair this with needing async in depth and that's exactly it. The whole network stack needs to be async-first and all the popular networking libraries need to have been built on that. Many of those libraries are already C-extension based and don't jibe well with the newer python parts in any way.

Java is trying to do this but I haven’t read how that’s going.

Re: Python has had async for 10 years – why isn't it more popular?

#133

Post by jpgvm »

Because they chose the wrong API (well more correctly created a new, worse one).

In order to appease the various flavours they mixed and matched stuff from Tornado, gevent, etc.

They should have stuck with the most seamless of those (gevent) and instead of having it monkey-patch the runtime go the Java VirtualThread route and natively yield in all the I/O APIs.

This would have given a Go-esque ease of use and likely would have been immensely more popular.

Re: Python has had async for 10 years – why isn't it more popular?

#135

Post by AtlasBarfed »

dekhn wrote: post #38

It added intrusive- codebase-wide- functionality that more or less could have been done with other (thread-based) approaches. AWSCLI was broken for over a year- we had to do a ton of work to deal with the various packaging issues. Don't break userspace.

Why was awscli written in python? Bad decision to begin with.

Hey! We have a product that we clearly want to release worldwide. Let's build it on something that doesn't have Unicode. Or any real threading. And is slow as hell.

You picked a platform that was going to have to break user space.

At least it wasn't JavaScript

Re: Python has had async for 10 years – why isn't it more popular?

#136

Post by lstodd »

pclmulqdq wrote: post #97
Dylan16807 wrote: post #82

Earlier quoted context omitted.

Having to put "await" everywhere is very explicit. I'd even say it's equally explicit to a bunch of awkward closures. Why do you say it's less?

It's explicit that the code is async, but how the asynchrony happens is completely implicit with async/await, and is managed by a runtime of some kind. Kernel-style async code, where everything is explicit: * You write a poller that opens up queues and reads structs representing work * Your functions are not tagged as "async" but they do not block * When those functions finish, you explicitly put that struct in anoth…

> Green threads, where all asynchrony is implicit:

which are no different from app POV from kernel threads, or any threads for that matter.

the whole async stuff came up because context switch per event is way more expensive than just shoveling down a page of file descriptor state.

thus poll, kqueue, epoll, io_uring, whatever.

think of it as batch processing

Re: Python has had async for 10 years – why isn't it more popular?

#137

Post by lelanthran »

tripletpeaks wrote: post #9

It’s probably related to the fact that when they added “await” to JavaScript, it seemed to become the most popular keyword in the language overnight, just comical amounts of it in the average new JavaScript file in the wild.

Quite a lot of my js functions don't await; instead I simply return the promise and let the caller `await` or more often attach a `then` as they see fit.

The default linter in Vs Code keeps marking those functions with warnings though. Says I should mark them as async

Re: Python has had async for 10 years – why isn't it more popular?

#138

Post by topspin »

Analemma_ wrote: post #29
northisup wrote: post #17

because two colors of functions suuuuuuuuuuks to deal with

Doesn't really explain why async/await are hugely popular in C#, JavaScript, etc. but didn't take off in Python.

I'd explain your cases this way:

C# has a dictator with a budget: Microsoft integrated async into C# in a formal way, with 5.0, including standard libs, debugging, docs, samples, clear guidance going forward, etc. What holes there were were dealt with in an orderly and timely manner.

JavaScript actually had a pretty messy start with async, with divergent conventions and techniques. Ultimately this got smoothed out with language additions, but it wasn't all that wonderful in the early days. Also, JavaScript started from a simpler place (single-threaded event loop) that never had "fork" and threads and all that comes with those, so there was less legacy to accommodate and fewer problems to overcome.

Python had a vast base of existing non-async software chock full of blocking code, plus an incomplete and haphazard concurrency evolution. There are several legacy concurrency solutions in Python, most still in use today. Python async is still competing and conflicting with it all. Not unlike the Python 2->3 transition.

Re: Python has had async for 10 years – why isn't it more popular?

#139

Post by markandrewj »

throw-qqqqq wrote: post #30
atomicnumber3 wrote: post #14

The author gets close to what I think the root problem is, but doesn't call it out. The truth is that in python, async was too little, too late. By the time it was introduced, most people who actually needed to do lots of io concurrently had their own workarounds (forking, etc) and people who didn't actually need it had found out how to get by without it (multiprocessing etc). Meanwhile, go showed us what good green…

> But all it did was show us that async code just plain sucks compared to green thread code that can just block, instead of having to do the async dances. I take so much flak for this opinion at work, but I agree with you 100%. Code that looks synchronous, but is really async, has funny failure modes and idiosyncracies, and I generally see more bugs in the async parts of our code at work. Maybe I’m just old, but I do…

I can tell you guys work with languages like Go, so this isn't true for yourselves, but I usually find it is developers that only ever work with synchronous code who find async complicated. Which isn't surprising, if you don't understand something it can seem complicated. My views is almost that people should learn how to write async code by default now. Regardless of the language. Writing modern applications basically requires it, although not all the time obviously.

Re: Python has had async for 10 years – why isn't it more popular?

#140

Post by dapperdrake »

mixmastamyk wrote: post #41
dapperdrake wrote: post #7

Python's async is very difficult to use and debug. It seems to get stuck randomly, read like race conditions. And Python cannot work around this nicely with their lambdas only permitting a single expression in their body. Not worth the trouble. Shell pipelines are way easier to use. Or simply waiting —no pun intended— for the synchronous to finish.

> lambdas only permitting a single expression Use a tuple, maybe walrus, and return the last item[-1].

That idea sounds good.

How do I get variables for not redoing long-running computations that depend on one-another? So, what if the third tuple value depends on the second and the second in turn depends on the first?

Post reply on HN