Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Asyncio

superfastpython.com

Post reply on HN
Print view View original

181–188 of 188 posts

Re: Python Asyncio

#181

Post by gpderetta »

quietbritishjim wrote: post #178
gpderetta wrote: post #43

Earlier quoted context omitted.

Depending on the details you can often call an async function from a sync function, by just running the event loop, it is just a few lines of code. The problem is that the event loop is not reentrant [1], so if your sync function is being called from an async function, things do not work. Why would you ever do this you might think? Well, asyncio might be an implementation detail of whatever environment you are using.…

Another possibility, if it's a matter of passing callback parameter that must be sync where you wish it could be an async function, is that the callback can be the `put_nowait()` method of an async queue. You can then have a background async task that pulls from that queue and does async stuff. This works well if the callback is just there to notify you about a result (which is the most common case in fairness) but d…

Yes, been there done that :). You can spawn a whole background thread running its own event loop, and post the async function with run-coroutine_threadsafe, the wait on the returned concurrent.future.

But it feels wrong having to spawn a thread and event loop just for that.

Re: Python Asyncio

#182

Post by gpderetta »

packetlost wrote: post #51
nineplay wrote: post #34

Not mentioned: The behavior is different between python 3.6 and 3.8. That was fun to debug. Python is simply not the right language for asynchronous programming. Things that are easy in many other languages are hard and don't work as you'd expect. If you are in a python-only shop, brush up on your presentation skills and see if you can convince them to branch out. Or write a shell script. You're better off with "./re…

The asyncio module was heavily refactored in 3.7, which introduced the async/await keywords which was a *huge* improvement over the 3.6 and below implementation. I don't agree with this, but I will concede that async Python is rarely the correct choice.

the async/await keyword predates 3.7. As far as I can tell it was introduced in 3.5.

Re: Python Asyncio

#183

Post by gpderetta »

nineplay wrote: post #63
ollien wrote: post #48

Earlier quoted context omitted.

What changed between those two releases? 3.6 was the last time I worked with asyncio

Some of it is here https://tryexceptpass.org/article/asyncio-in-37/ One of my ongoing frustrations is python isn't just the lack of backwards compatibility but that if you inherit a script there's no way to know which version of the language it was written for. If you're lucky you can track down the developers, but IME even they often say "I'm not sure, let me run python --version".

I have only a bit of experience with asyncio, but it seem that I have hit all issues that are described there and improved in 3.7. I still can't believe that wait_closed wasn't there from day 1.

Re: Python Asyncio

#184

Post by icedchai »

mrfox321 wrote: post #175
icedchai wrote: post #166

Earlier quoted context omitted.

I'd argue that thread-per-request is even a better idea now, given the massive number of cores and memory in modern servers.

Hybrid? Request passed to a thread pool. Each thread is juggling coroutines representing in flight requests

Is that worth the complexity? I like the way Go does it. The runtime is managing the actual threading and async IO as it schedules go routines, but to the programmer it all looks like normal synchronous code.

Re: Python Asyncio

#185

Post by samwillis »

fprct wrote: post #176
samwillis wrote: post #118

Earlier quoted context omitted.

Asyncio does not make your db calls or io faster, it does make your code more complex. The only thing async io gives you is scalability of many MANY concurrent io operations. You have to either be pushing seriously large traffic or doing a lot of long running websocket/sse/long polling type requests. The only other use case is lots of truly concurrent io within one request/response cycle. But again that is unusual, m…

> The only other use case is lots of truly concurrent io within one request/response cycle. But again that is unusual, most apis have low single digit db queries that are usually dependent on one another removing any advantage of async. Why just "within one cycle"? What about situation where you spend 1s in single db query and are 100ms cpu bound during request handling?

Transitional multithreaded Python web servers handle that very well and have for decades, you shouldn't need to manually handle yealding in simple cases in your application code.

The point is asyncio is to have find grade control of yealding.

Re: Python Asyncio

#186

Post by v3ss0n »

kissgyorgy wrote: post #12

After 2 years of using asyncio in production, I recommend to avoid it if you can. With async programming, you take the complexity of concurrent programming, which is way harder than you can imagine. Also nobody mentions this for some reason, but asyncio doesn't make your programs faster, in fact it makes everything 100x SLOWER (we measured it multiple times, compared the same thing to the sync version), but makes you…

You are doing it wrong then, asyncio is never designed for and not useful for CPU bound tasks. Use multiprocessing for CPU bound tasks. In our system before asyncio, sqlalchemy commits and httpx requests are blocking The API calls and it takes 21 seconds every time, now only 1.2 seconds. We had optimized by making every IO (db.commit(),httpx.post) to run inside asyncio.create_task.

So they are operating outside of the request without slowing it down.

We had used aync since 2014 ( twisted, tornado), and officially having async await in python starting 3.7 was the best thing happened to python.

Re: Python Asyncio

#187

Post by throwaway81523 »

number6 wrote: post #109
matsemann wrote: post #103

Earlier quoted context omitted.

In other languages the async paradigm work for multiple kind of workflows, not just heavily IO bound ones.

Can you give me an example which languages do this?

Look at seastar.io which is an async concurrency library for C++. It is painful in that it resembles node.js but it is very performant. I'd be interested in seeing a version adapted to use C++20 coroutines.

Re: Python Asyncio

#188

Post by BerislavLopac »

dekhn wrote: post #130
srik wrote: post #71

Earlier quoted context omitted.

Concurrency is not Parallelism. You should watch this quintessential talk about the difference - https://www.youtube.com/watch?v=oV9rvDllKEg

This is Rob's perspective, but it's not universally shared. In my mind, all concurrency is a form of parallelism (parallel tasks, but not parallel threads) while not all parallelism is concurrency. I have frequently solved concurrency problems with thread pools, rather than using non-blocking IO, because the programming paradigm is a lot simpler.

You got the the wrong way round: all parallelism is a form of concurrency.

In Python, thread pools and async work mostly the same in practice, except that threads have a bigger overhead. So they can be used to solve the same problems.

Post reply on HN