The magic of asyncio explained • Hacker News Board

Re: The magic of asyncio explained

#51

Post by heavenlyblue » Wed, Aug 08, 2018, 11:47 AM UTC

tlrobinson wrote: post #24

This is essentially how modern JavaScript works, in particular with the addition of async/await syntax [1] (which was originally from C#, I think), but it's been possible with libraries like task.js, co, and Bluebird [2] since generator functions were available (either natively or via transpiling). The main difference is in JavaScript the event loop is automatic and hidden, and asynchronous IO is the default, so it's…

Automatic and hidden? JS had never provided any proper tools for async debugging.

Can I please access a list of all async threads running at this point of time?

Surely you can shoot yourself in the foot, but you can also do many other things JS never even attempted to fix.

Please stop this JS fanboyism. This is a python thread.

Re: The magic of asyncio explained

#52

Post by _Codemonkeyism » Wed, Aug 08, 2018, 11:51 AM UTC

icebraining wrote: post #39

_Codemonkeyism wrote: post #19

Wasn't there a language where every call was async? Instead of async ... A/returing Future[A] it did/would return A from method calls. If it didn't exist, one can imagine one. A.x = 3 would be wrapped in A.map(_.x = 3) etc. So you write code that would be executed when you finally await a value. No more red/blue world. Would probably need coroutines instead of threads for executing.

Isn't Haskell somewhat like that, due to being lazy by default?

In Haskell you'd have the type signature everywhere I think, mostly as a monad transformer.

Re: The magic of asyncio explained

#53

Post by kasbah » Wed, Aug 08, 2018, 11:52 AM UTC

ioquatix wrote: post #2

Here is a comparison of `asyncio` (Python), `async` (Ruby) and Go: https://github.com/socketry/async-await/tree/master/examples... I wrote a similar article but for Ruby: https://www.codeotaku.com/journal/2018-06/asynchronous-ruby/... Yes, it's a good model for many use cases. One thing I wondered about Ruby, is it really necessary to have the `await` keyword?

`await` is useful since with the absence of it you can schedule multiple tasks concurrently. In JS:

    var task1 = someAsyncTask1()
    var task2 = someAsyncTask2()
 
    await Promise.all([task1, task2])

If `await` was implicit then task2 would wait for task1 to finish.

Re: The magic of asyncio explained

#54

Post by stinos » Wed, Aug 08, 2018, 12:01 PM UTC

sametmax wrote: post #41

greyman wrote: post #18

Python is my language of first choice, but I must say that I am not that thrilled how this multithreading ended up. There are many tutorials about the topic promising to explain how it works, usually in the form of "simple introduction". But when one tries to implement something production-ready, with correct error handling etc., things starts to complicate pretty quickly; at least that was my experience. I don't wan…

That's because most of those tutorials have not been written by somebody actually putting something in production. I've been using asyncio for a while now, and you can't get away with a short introduction since: - it's very low level - it's full of design flaws and already has accumulated technical debt - it requires very specific best practices to be usable I'm not going to write a tutorial here, it would take me a…

one problem only: when the bottleneck of your program is network IO

Do you mean literally what this says, or are you rather using 'network IO' as some (extremely) abstract term for any type of communication? Just checking because I haven't used asynchronous programming in Python but did so in other languages and we do things like await hardwareAxis.GoToTargetPosition(position=50, veolcity=100). Not what most people think of when reading network IO, that one.

Re: The magic of asyncio explained

#55

Post by sametmax » Wed, Aug 08, 2018, 12:03 PM UTC

windexh8er wrote: post #49

sametmax wrote: post #45

Earlier quoted context omitted.

> such like socket programming That's one of my biggest pet peeves (and if you see my other comments, you'll notice I have quite a few). To do socket programming in asyncio, you can either use: - protocols, with a nice reusable API and an interface that clearly tells you where to do what. But you can't use "await". You are back to creating futures and attaching callback like 10 years ago. - streams, where you can use…

I'm curious of your take on Trio or Curio. Do either address the peeves you've outlined? https://github.com/python-trio/trio https://github.com/dabeaz/curio

Trio is a better Curio, so you don't really need Curio anymore. It was what started everything and deserves credit for that though.

As for Trio, it's what asyncio should have been from the beginning, at least for the high level part (although not for the pet peeves of socket programming: it's too low level for Python IMO)

The problem with Trio is that it's incompatible with asyncio (minus some verbose quirky bridges), so you get yet another island, yet another ecosystem. So what, now we get twisted, tornado, gevent, qt, asyncio... and trio ?

The madness must stop.

And that's why I think there is a better way: creating a higher level API for asyncio, which enforces the best practices and make the common things easy, and the hard things decent.

A complete rewrite like Trio would be better (e.g: it famously handles Ctrl + C way better and has a smaller API). But this ship has sailed. We have asyncio now.

asyncio is pretty good honestly. But it needs some love.

So, considering asyncio is what we have to work with, and by experience, it's quite great if you know what you are doing, I advice people to actually write a wrapper around it.

If you don't feel like writing a wrapper, I'll plug in the one I'm working on in case people are curious: https://github.com/Tygs/ayo

It:

- is based on asyncio. Not a new framework. Integrated transparently with normal asyncio.

- implement some lessons leaned from trio (e.g: nurseries, cancellation, etc)

- expose a sweet API (running blocking code is run.aside(callback, args, *kwargs), and automate stuff it can (@ayo.run_as_main setup the event loop and run the function in one row)

- make hard things decent: timeout and concurrency limit are just a param away

I does need some serious doc, including a rich tutorial which features a pure asyncio part. Also it needs some mix between streams and protocols. I'm not going to skip that part, I think it's mandatory, but I'll need many months to make the whole thing.

Now, I am not Nathaniel or Yury, so my work is not nearly as bullet proof as theirs. I would not advice to install ayo in prod now, but I think it's a great proof of concept of how good asyncio can be.

And we most certainly can do even better.

Re: The magic of asyncio explained

#56

Post by diftraku » Wed, Aug 08, 2018, 12:13 PM UTC

sametmax wrote: post #45

anilakar wrote: post #30

Yet another asyncio tutorial that shows you to run a few sleep tasks concurrently. Can we finally get one that shows how to do real stuff such like socket programming, wrapping non-async-compatible libraries and separating cpu-intensive blocking tasks to awaitable threads?

> such like socket programming That's one of my biggest pet peeves (and if you see my other comments, you'll notice I have quite a few). To do socket programming in asyncio, you can either use: - protocols, with a nice reusable API and an interface that clearly tells you where to do what. But you can't use "await". You are back to creating futures and attaching callback like 10 years ago. - streams, where you can use…

It's hilarious when a single comment on HN opens asyncio more than the tutorial being discussed.

Re: The magic of asyncio explained

#57

Post by sametmax » Wed, Aug 08, 2018, 12:21 PM UTC

stinos wrote: post #54

sametmax wrote: post #41

Earlier quoted context omitted.

That's because most of those tutorials have not been written by somebody actually putting something in production. I've been using asyncio for a while now, and you can't get away with a short introduction since: - it's very low level - it's full of design flaws and already has accumulated technical debt - it requires very specific best practices to be usable I'm not going to write a tutorial here, it would take me a…

one problem only: when the bottleneck of your program is network IO Do you mean literally what this says, or are you rather using 'network IO' as some (extremely) abstract term for any type of communication? Just checking because I haven't used asynchronous programming in Python but did so in other languages and we do things like await hardwareAxis.GoToTargetPosition(position=50, veolcity=100). Not what most people t…

While async / await, futures, and the event loop are generic mechanisms, the asyncio module itself only implement selectors for sockets (ok, and subprocess pipes). You can't even do async file system operations with it: you need to call run_in_executor().

Now that doesn't mean you could not implement a selector that does asynchronous UI IO and plug it to the event loop. But the asyncio module doesn't provide it right now, and no lib that I know of does it either.

Re: The magic of asyncio explained

#58

Post by mehrdadn » Wed, Aug 08, 2018, 12:22 PM UTC

sametmax wrote: post #41

greyman wrote: post #18

Python is my language of first choice, but I must say that I am not that thrilled how this multithreading ended up. There are many tutorials about the topic promising to explain how it works, usually in the form of "simple introduction". But when one tries to implement something production-ready, with correct error handling etc., things starts to complicate pretty quickly; at least that was my experience. I don't wan…

That's because most of those tutorials have not been written by somebody actually putting something in production. I've been using asyncio for a while now, and you can't get away with a short introduction since: - it's very low level - it's full of design flaws and already has accumulated technical debt - it requires very specific best practices to be usable I'm not going to write a tutorial here, it would take me a…

Awesome comment. One thing I want to point out to those reading is that the nursery thing is an instantiation of the more general principle of, if you're finding your code is getting convoluted, it's likely that you're missing a noun. I can't explain this as well as others have, so see this comment: https://news.ycombinator.com/item?id=16468796

Re: The magic of asyncio explained

#59

Post by sametmax » Wed, Aug 08, 2018, 12:29 PM UTC

diftraku wrote: post #56

sametmax wrote: post #45

Earlier quoted context omitted.

> such like socket programming That's one of my biggest pet peeves (and if you see my other comments, you'll notice I have quite a few). To do socket programming in asyncio, you can either use: - protocols, with a nice reusable API and an interface that clearly tells you where to do what. But you can't use "await". You are back to creating futures and attaching callback like 10 years ago. - streams, where you can use…

It's hilarious when a single comment on HN opens asyncio more than the tutorial being discussed.

Well, honestly, I think most HN readers go to the comments before the actual article.

I know I do.

The whole value of this website is that we got 1000 of experts in their fields, ready to give you their insight.

Re: The magic of asyncio explained

#60

Post by wruza » Wed, Aug 08, 2018, 12:32 PM UTC

mtrovo wrote: post #50

sametmax wrote: post #41

Earlier quoted context omitted.

That's because most of those tutorials have not been written by somebody actually putting something in production. I've been using asyncio for a while now, and you can't get away with a short introduction since: - it's very low level - it's full of design flaws and already has accumulated technical debt - it requires very specific best practices to be usable I'm not going to write a tutorial here, it would take me a…

Wow, really nice list, I wish I knew it before I started to work with asyncio. > stay sane, you should never, ever, have an dangling awaitable anywhere. Always get a reference on all your awaitables. Decide where in the code you think their life should end. This is the most difficult part for me, it's not trivial to know if a function you're calling is async or not without looking at the function source, specially wh…

And still people vote for async-await because “true light threads are hard to implement at low level”. This generator-based madness has to end, but few seem to understand what hassle it brings to their coding and what an alternative could be. I don’t get it.