Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Asyncio

superfastpython.com

Post reply on HN
Print view View original

51–60 of 188 posts

Re: Python Asyncio

#51

Post by packetlost »

nineplay wrote: post #34

Not mentioned: The behavior is different between python 3.6 and 3.8. That was fun to debug. Python is simply not the right language for asynchronous programming. Things that are easy in many other languages are hard and don't work as you'd expect. If you are in a python-only shop, brush up on your presentation skills and see if you can convince them to branch out. Or write a shell script. You're better off with "./re…

The asyncio module was heavily refactored in 3.7, which introduced the async/await keywords which was a *huge* improvement over the 3.6 and below implementation.

I don't agree with this, but I will concede that async Python is rarely the correct choice.

Re: Python Asyncio

#52

Post by leveraction »

I have used asyncio through aiohttp, and I have been pretty happy with it, but I also started with it from the beginning, so that probably made things a little easier.

My setup is a bunch of microservices that each run an aiohttp web server based api for calls from the browser where communications between services are done async using rabbitmq and a hand rolled pub/sub setup. Almost all calls are non-blocking, except for calls to Neo4j (sadly, they block, but Neo4j is fast, so its not really a problem.)

With an async api I like the fact that I can make very fast https replies to the browser while queing the resulting long running job and then responding back to the Vue based SPA client over a web socket connection. This gives the interface a really snappy feel.

But Complex? Oh yes.

But the upside is that it is also a very flexible architecture, and I like the code isolation that you get with microservices. Nevertheless, more than once I have thought about whether I would choose it all again knowing what I know now. Maybe a monolithic flask app would have been a lot easier if less sexy. But where's the fun in that?

Re: Python Asyncio

#53

Post by nine_k »

brrrrrm wrote: post #46
nineplay wrote: post #34

Not mentioned: The behavior is different between python 3.6 and 3.8. That was fun to debug. Python is simply not the right language for asynchronous programming. Things that are easy in many other languages are hard and don't work as you'd expect. If you are in a python-only shop, brush up on your presentation skills and see if you can convince them to branch out. Or write a shell script. You're better off with "./re…

All of distributed machine learning would benefit from async, but it's unfortunately a Python-only ecosystem for the time being. Probably gonna be a couple years before that has a chance of changing

How would ML benefit from async? I thought everything interesting in things like TF or PyTorch is done by native code which is free to use multithreading and anything at all.

(I know little about ML.)

Re: Python Asyncio

#54

Post by qbasic_forever »

kissgyorgy wrote: post #12

After 2 years of using asyncio in production, I recommend to avoid it if you can. With async programming, you take the complexity of concurrent programming, which is way harder than you can imagine. Also nobody mentions this for some reason, but asyncio doesn't make your programs faster, in fact it makes everything 100x SLOWER (we measured it multiple times, compared the same thing to the sync version), but makes you…

This isn't a problem specific to python though. You hint at it, async programming (single thread worker loop) is not the right way to deal with CPU-bound tasks. NodeJS or any other async language would have the same problems. You want multiprocessing or multithreading, where work is distributed to different CPU cores at the same time. Python gives you the ability to use any of those three paradigms. Choose wisely.

Re: Python Asyncio

#55

Post by andrewstuart »

throwawaymaths wrote: post #21
acedTrex wrote: post #18

Earlier quoted context omitted.

This is an insane take, if you are doing async you already HAVE a need for concurrent programming. Async just makes that simpler to read and write.

> Async just makes that simpler to read and write. Really? I definitely had a huge struggle reading async python, because generators and yields break the concept of "what is a function" and struggled writing async python because of function coloring.

Day to day Python async should be using generators and yields.

Re: Python Asyncio

#56

Post by [deleted] »

ollien wrote: post #48
nineplay wrote: post #34

Not mentioned: The behavior is different between python 3.6 and 3.8. That was fun to debug. Python is simply not the right language for asynchronous programming. Things that are easy in many other languages are hard and don't work as you'd expect. If you are in a python-only shop, brush up on your presentation skills and see if you can convince them to branch out. Or write a shell script. You're better off with "./re…

What changed between those two releases? 3.6 was the last time I worked with asyncio

[deleted]

Re: Python Asyncio

#57

Post by whakim »

kissgyorgy wrote: post #12

After 2 years of using asyncio in production, I recommend to avoid it if you can. With async programming, you take the complexity of concurrent programming, which is way harder than you can imagine. Also nobody mentions this for some reason, but asyncio doesn't make your programs faster, in fact it makes everything 100x SLOWER (we measured it multiple times, compared the same thing to the sync version), but makes you…

This just sounds like you fundamentally misunderstood how asyncio works and applied it to a use-case which it wasn't suited to. Why would you use async if you're doing a bunch of CPU-bound work?

Re: Python Asyncio

#58

Post by qbasic_forever »

acedTrex wrote: post #18
kissgyorgy wrote: post #12

After 2 years of using asyncio in production, I recommend to avoid it if you can. With async programming, you take the complexity of concurrent programming, which is way harder than you can imagine. Also nobody mentions this for some reason, but asyncio doesn't make your programs faster, in fact it makes everything 100x SLOWER (we measured it multiple times, compared the same thing to the sync version), but makes you…

This is an insane take, if you are doing async you already HAVE a need for concurrent programming. Async just makes that simpler to read and write.

Async python by design is not concurrent, nothing is executing at the same time on different cores/threads. There's one worker loop and one task running on it at a time.

Re: Python Asyncio

#59

Post by nine_k »

calpaterson wrote: post #50
acedTrex wrote: post #18

Earlier quoted context omitted.

This is an insane take, if you are doing async you already HAVE a need for concurrent programming. Async just makes that simpler to read and write.

Unfortunately the majority of people who use asyncio in the real world do not have a need for concurrenct programming. They are drawn to it because they believe it is faster, which, generally, it is not. That isn't to say there aren't reasonable usecases - but most people using asyncio are writing webapps which spend a lot of time in the CPU generating or outputting JSON/GraphQL/HTML.

I think this may be influenced by Node. With JS and Node, async is cheap and ergonomic, so it's very widely used, even if the latency gains are marginal.

With Python and asyncio, neither assumption holds, but the idea of shaving off some latency persists: if we rewrite everything using asyncio, our DB accesses can be parallelized, yay! This may or may not be a net win in latency; reworking your DB access patterns may gain you more.

Re: Python Asyncio

#60

Post by ollien »

grodes wrote: post #27

async is not comparable to threads: - async is concurrency - threads are parallelism They are not exclusives, the best approach is to be multi-threaded while each of the threads uses concurrency to prevent blocking the cpu.

> - async is concurrency - threads are parallelism

Even this isn't necessarily true, especially given you can limit a process to a single core (or, if you use a single core system, lest we forget those exist!), or, in Python's case, the GIL! You can still have parallelism with asynchronous programming, it's just not necessarily guaranteed. IIRC tokio lets you spin up runtimes on different threads which (putting the above caveat) _can_ run in parallel.

Post reply on HN