Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The magic of asyncio explained

hackernoon.com

Post reply on HN
Print view View original

71–80 of 95 posts

Re: The magic of asyncio explained

#71

Post by Sir_Cmpwn »

anilakar wrote: post #30

Yet another asyncio tutorial that shows you to run a few sleep tasks concurrently. Can we finally get one that shows how to do real stuff such like socket programming, wrapping non-async-compatible libraries and separating cpu-intensive blocking tasks to awaitable threads?

If anyone wants to see some small, practical asyncio code in action, here's a little LMTP daemon I wrote recently:

https://git.sr.ht/~sircmpwn/lists.sr.ht/tree/lists-srht-lmtp via https://git.sr.ht/~sircmpwn/lists.sr.ht

Or getting deeper, another project which implements Synapse's[0] RPC protocol and encapsulates high-level RPC actions in asyncio sugar:

https://git.sr.ht/~sircmpwn/broca/tree/broca/connection.py

Code which uses this code:

https://git.sr.ht/~sircmpwn/broca/tree/broca/rpc.py

[0] https://synapse-bt.org

Re: The magic of asyncio explained

#72

Post by chatmasta »

metalliqaz wrote: post #66
sametmax wrote: post #41

Earlier quoted context omitted.

That's because most of those tutorials have not been written by somebody actually putting something in production. I've been using asyncio for a while now, and you can't get away with a short introduction since: - it's very low level - it's full of design flaws and already has accumulated technical debt - it requires very specific best practices to be usable I'm not going to write a tutorial here, it would take me a…

I wish I could favorite comments on HN

You can. Click on the timestamp and then favorite it.

Re: The magic of asyncio explained

#73

Post by chatmasta »

shoo wrote: post #35

quoting the article: > Concurrency is like having two threads running on a single core CPU. > Parallelism is like having two threads running simultaneously on different cores > It is important to note that parallelism implies concurrency but not the other way round. Aurgh! I don't think this attempted definition-by-simile is helpful, or even somewhat correct. I much prefer yosefk's way of framing things: > > concurre…

Also a big fan of this "Visualizing Concurrency in Go": http://divan.github.io/posts/go_concurrency_visualize/

Re: The magic of asyncio explained

#74

Post by [deleted] »

heavenlyblue wrote: post #51
tlrobinson wrote: post #24

This is essentially how modern JavaScript works, in particular with the addition of async/await syntax [1] (which was originally from C#, I think), but it's been possible with libraries like task.js, co, and Bluebird [2] since generator functions were available (either natively or via transpiling). The main difference is in JavaScript the event loop is automatic and hidden, and asynchronous IO is the default, so it's…

Automatic and hidden? JS had never provided any proper tools for async debugging. Can I please access a list of all async threads running at this point of time? Surely you can shoot yourself in the foot, but you can also do many other things JS never even attempted to fix. Please stop this JS fanboyism. This is a python thread.

[deleted]

Re: The magic of asyncio explained

#75

Post by sametmax »

Sir_Cmpwn wrote: post #70
sametmax wrote: post #45

Earlier quoted context omitted.

> such like socket programming That's one of my biggest pet peeves (and if you see my other comments, you'll notice I have quite a few). To do socket programming in asyncio, you can either use: - protocols, with a nice reusable API and an interface that clearly tells you where to do what. But you can't use "await". You are back to creating futures and attaching callback like 10 years ago. - streams, where you can use…

Have you considered contributing improvements to the documentation, or even the API? Python is an open source project.

Yes. If you are not part of the club, it takes approximatly 18 months from a post on python-idea to an implementation, after so much debate it's madening. And most of the time gets rejected.

The process of contributing to python is more frustrating than writing for wikipedia.

Much easier to write something in pypi, then come back to python-idea once it gets popular.

Re: The magic of asyncio explained

#76

Post by Sir_Cmpwn »

sametmax wrote: post #75
Sir_Cmpwn wrote: post #70

Earlier quoted context omitted.

Have you considered contributing improvements to the documentation, or even the API? Python is an open source project.

Yes. If you are not part of the club, it takes approximatly 18 months from a post on python-idea to an implementation, after so much debate it's madening. And most of the time gets rejected. The process of contributing to python is more frustrating than writing for wikipedia. Much easier to write something in pypi, then come back to python-idea once it gets popular.

Well, with doc updates I think you'll probably find the process much smoother, and it's a good way of getting trust among the "club".

Re: The magic of asyncio explained

#77

Post by gpderetta »

pas wrote: post #37
toast0 wrote: post #12

Earlier quoted context omitted.

They're sort of similar, and you can probably get the same work done in either system, but I think real threading (green or otherwise), may leave you with less cognitive load. Spawning a thread may be complex, and thinking about how the threads are scheduled is often complex, but what each thread does can be very simple -- and you don't have to think about 'long running things need to be futured/awaited', you just do…

> Spawning a thread may be complex, and thinking about how the threads are scheduled is often complex, but what each thread does can be very simple And that's how it starts, and in the end it's New Year's Eve and you're somehow, again, debugging a deadlock. > green threads yes please

You can deadlock with futures as well. Except that with futures you do not get a (two actually) nice call stack pointing to the deadlocked resource.

Re: The magic of asyncio explained

#78

Post by cup-of-tea »

anilakar wrote: post #30

Yet another asyncio tutorial that shows you to run a few sleep tasks concurrently. Can we finally get one that shows how to do real stuff such like socket programming, wrapping non-async-compatible libraries and separating cpu-intensive blocking tasks to awaitable threads?

If you don't mind a video format, this talk by Robert Smallshire includes that: https://www.youtube.com/watch?v=M-UcUs7IMIM

Re: The magic of asyncio explained

#79

Post by AboutTheWhisles »

anilakar wrote: post #30

Yet another asyncio tutorial that shows you to run a few sleep tasks concurrently. Can we finally get one that shows how to do real stuff such like socket programming, wrapping non-async-compatible libraries and separating cpu-intensive blocking tasks to awaitable threads?

This architecture is made to match software threads to (logical) hardware threads, then have them loop over data separated into chunks that don't depend on each other.

If a function is blocking, needs the CPU and isn't thread safe, it can be wrapped in a message passing node that will get skipped over if a thread is already running it.

Every separate chunk of data that it creates will be dealt with concurrently and the high level structure can be put together in a graph that uses openGL.

https://github.com/LiveAsynchronousVisualizedArchitecture/la...

Re: The magic of asyncio explained

#80

Post by sametmax »

Sir_Cmpwn wrote: post #76
sametmax wrote: post #75

Earlier quoted context omitted.

Yes. If you are not part of the club, it takes approximatly 18 months from a post on python-idea to an implementation, after so much debate it's madening. And most of the time gets rejected. The process of contributing to python is more frustrating than writing for wikipedia. Much easier to write something in pypi, then come back to python-idea once it gets popular.

Well, with doc updates I think you'll probably find the process much smoother, and it's a good way of getting trust among the "club".

That's fair. I'll get in touch with stinner on the next pycon, I think he is the guy for that.
Post reply on HN