Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Asyncio

superfastpython.com

Post reply on HN
Print view View original

91–100 of 188 posts

Re: Python Asyncio

#91

Post by meitham »

leveraction wrote: post #52

I have used asyncio through aiohttp, and I have been pretty happy with it, but I also started with it from the beginning, so that probably made things a little easier. My setup is a bunch of microservices that each run an aiohttp web server based api for calls from the browser where communications between services are done async using rabbitmq and a hand rolled pub/sub setup. Almost all calls are non-blocking, except…

I share your sentiment and have been using aiohttp for five years and pretty happy with it. My current project is a web service with blocking SQL server backend, so I tend to do loop.run_in_executor for every DB.execute statement. But now I’m considering just running a set of light aio streams sub processes with a simple bespoke protocol that takes a SQL statement and returns the result json encoded to move away from threads.

Re: Python Asyncio

#93

Post by thunky »

kissgyorgy wrote: post #26
acedTrex wrote: post #18

Earlier quoted context omitted.

This is an insane take, if you are doing async you already HAVE a need for concurrent programming. Async just makes that simpler to read and write.

I'm pretty sure a lot of people don't understand the tradeoffs and don't really need concurrent code. There are really only a couple of use-cases where it's really handy: for example an API gateway, where you only get and send HTTP requests. Other than that, I don't see how it worth the insane complexity (compared to sync code). You can write web servers in a sync manner; Flask, Django is way better if you only need…

> FastAPI for a simple REST API is a huge mistake

FastAPI can be sync or async, although sync will have "a small performance penalty"

https://github.com/tiangolo/fastapi/issues/260#issuecomment-...

Re: Python Asyncio

#94

Post by wrigby »

mpeg wrote: post #85
wrigby wrote: post #73

Earlier quoted context omitted.

I think the article is correct here, actually - if you need to run a CPU-bound task, you’ll need a ProcessPoolExecutor.

Well, that section is talking about CPU-bound tasks, and both threads and processes are valid choices for CPU-bound tasks, with different tradeoffs. I think there is often a lot of confusion around that because a lot of tutorials simply say to use threads for IO-bound tasks and processes for CPU-bound tasks, without going deeper into the differences. Threads can easily share memory which often increases complexity an…

I think the section is addressing both CPU and IO-bound tasks:

> How to Execute a Blocking I/O or CPU-bound Function in Asyncio?

But to be precise, we should differentiate between Python’s interpreter threads and OS threads. In general, OS threads are a great way to parallelize CPU-bound tasks (with the issues of locking you mention), but Python’s interpreter threads are not (because of the GIL).

Re: Python Asyncio

#95

Post by m3047 »

xwowsersx wrote: post #90
andrewstuart wrote: post #87

Earlier quoted context omitted.

It IS easy. The Python project just don’t make it obvious how to do it easy.

Right. So you think it's mainly an education/documentation issue? I'm totally open to that in fact being the case. One really simple use case I had was just annotating a method to run async and then calling that in a loop elsewhere which did speed up a bunch of work I was doing pretty dramatically. import asyncio def background(f): def wrapped(*args, **kwargs): return asyncio.get_event_loop().run_in_executor(None, f,…

You need to run the executor somewhere ( loop.run_forever() or loop.run_until_complete() ), that can be in the current thread or a separate thread, keep in mind Python is still conceptually a single core.

Things I've found particularly useful:

* Optional / debug logging of coroutine start/exit.

* Stats logging, including count, runtime, request queue (multiple instances of the same function) depth.

* Printing tracebacks within the coroutine context.

Haven't expended any effort on decorators, good for you.

Re: Python Asyncio

#96

Post by btilly »

andrewstuart wrote: post #87
xwowsersx wrote: post #86

I'm not in a much of a position to evaluate Python's asyncio since I really have not used it very much. However, over the last few days, I started to dig into it and tried to get some (what I think are) very basic examples working and really struggled. That alone is not fully dispositive because I've used many async implementations in various languages and each of them have a bit of a learning curve and their own wri…

It IS easy. The Python project just don’t make it obvious how to do it easy.

It is easy if you know what to do. But you'll CONSTANTLY run into things where it is obvious that this is a toy that hasn't been thought through.

I've found random slowdowns of a factor of 10 for no reason.

A context manager needs to have __enter__ and __exit__ methods. An async context manager needs to have __aenter__ and __aexit__ methods. The error you get if you use a context manager as an async context manager or vice versa is big and scary and offers no clue to most programmers of what the problem is.

Using async with SQLAlchemy it isn't hard to wind up exiting the event loop before all cleanup has happened. They fixed the main cause of this, but it isn't hard to still trigger it.

And so on.

Re: Python Asyncio

#97

Post by xwowsersx »

m3047 wrote: post #95
xwowsersx wrote: post #90

Earlier quoted context omitted.

Right. So you think it's mainly an education/documentation issue? I'm totally open to that in fact being the case. One really simple use case I had was just annotating a method to run async and then calling that in a loop elsewhere which did speed up a bunch of work I was doing pretty dramatically. import asyncio def background(f): def wrapped(*args, **kwargs): return asyncio.get_event_loop().run_in_executor(None, f,…

You need to run the executor somewhere ( loop.run_forever() or loop.run_until_complete() ), that can be in the current thread or a separate thread, keep in mind Python is still conceptually a single core. Things I've found particularly useful: * Optional / debug logging of coroutine start/exit. * Stats logging, including count, runtime, request queue (multiple instances of the same function) depth. * Printing traceba…

Those do sound useful, thanks for this. Will keep in mind as I continue exploring.

Re: Python Asyncio

#98

Post by arecurrence »

andrewstuart wrote: post #67

I love Python async - it’s a complete game changer for certain types of applications. I find Python async to be fun and exciting and interesting and powerful. BUT it is a big power tool and there’s so much in it that it’s hard to work out how to drive it right. I have pretty good experience with Python and javascript. I prefer Python to javascript when writing async code. Specific example I spent hours trying to driv…

The way that this is formatted... I initially thought it was a Haiku :)

Re: Python Asyncio

#99

Post by matsemann »

xwowsersx wrote: post #86

I'm not in a much of a position to evaluate Python's asyncio since I really have not used it very much. However, over the last few days, I started to dig into it and tried to get some (what I think are) very basic examples working and really struggled. That alone is not fully dispositive because I've used many async implementations in various languages and each of them have a bit of a learning curve and their own wri…

Asyncio is annoying, and often unexpectedly slow. You think things are parallel, but then one misbehaving coroutine can hog your cpu bringing everything to a halt. GIL makes it useless for anything other than _heavily_ IO bound tasks.

And yeah, Python's documentation is useless. Never how to use stuff, only listing of everything that's possible to do / the API. Unfortunately that style is being mimicked by most other Python projects as well.

Post reply on HN