The magic of asyncio explained • Hacker News Board

Re: The magic of asyncio explained

#91

Post by marcosdumay » Thu, Aug 09, 2018, 9:32 PM UTC

_Codemonkeyism wrote: post #89

marcosdumay wrote: post #82

Earlier quoted context omitted.

Just to post it as an answer instead of a question. That's Haskell's IO. It is just one of the lots of concurrency behaviors available in libraries. Also, parallelism is "free" on pure code.

From my understanding, this is not Haskells IO - though my time with Haskell is limited. 1. Haskell uses special notation 'do' to handle access to IO wrapped values, e.g. (contrieved example, one would not use do for such simple cases) y = do x instead of y = x + 1 2. Haskell method signatures do include IO, e.g. doSomething :: Int -> IO Int instead of def doSomething(i:Int):Int 3. Because IO is usually not the only…

Well, it does not have the exact same syntax of your example. Even more because your example was pure. Haskell does that automatically for pure code too (`y = x + 1` would do exactly what you described) but it's not really relevant.

IO code always returns a promise, and the next statement on a `do` block may await the previous promise and yield the execution to whatever other piece of code can run, based on some rules on the compiler, based in large part on data dependency. If I'm reading your comment correctly, that is what you are asking for.

Re: The magic of asyncio explained

#92

Post by _Codemonkeyism » Fri, Aug 10, 2018, 6:14 AM UTC

marcosdumay wrote: post #91

_Codemonkeyism wrote: post #89

Earlier quoted context omitted.

From my understanding, this is not Haskells IO - though my time with Haskell is limited. 1. Haskell uses special notation 'do' to handle access to IO wrapped values, e.g. (contrieved example, one would not use do for such simple cases) y = do x instead of y = x + 1 2. Haskell method signatures do include IO, e.g. doSomething :: Int -> IO Int instead of def doSomething(i:Int):Int 3. Because IO is usually not the only…

Well, it does not have the exact same syntax of your example. Even more because your example was pure. Haskell does that automatically for pure code too (`y = x + 1` would do exactly what you described) but it's not really relevant. IO code always returns a promise, and the next statement on a `do` block may await the previous promise and yield the execution to whatever other piece of code can run, based on some rule…

Sorry for being so confusing

"Even more because your example was pure."

No it wasn't which is the whole point.

    y = x + 1

means

    y = x.map(_ + 1)

by default with async effects.

"same syntax"

Which is also the point, that there is syntax for the effects. When everything is async, there should be no special syntax as you have that syntax all over your code and it's redundant.

Re: The magic of asyncio explained

#93

Post by sachin18590 » Fri, Aug 10, 2018, 9:05 AM UTC

sandGorgon wrote: post #34

sachin18590 wrote: post #23

Earlier quoted context omitted.

I am curious what considerations your company had before switching some projects to Go. Python multithreading has been an issue for us as well. While asyncio looked good in the tutorials, gevent was much easier to work with. However, we still face multiple issues moving our celery workers to gevent and I am not sure if there is a better production friendly alternative for celery-gevent in python.

we love gevent as well. we use rq instead of celery . Try that instead ( http://python-rq.org/docs/workers/ )

We tried rq but wanted something more configurable/extensible. Also, worker level concurrencies were easier to manage with celery. But that said, rq definitely is a much easier and lightweight alternative to celery and is my goto option whenever I need to spin up something quick without much overhead.

Re: The magic of asyncio explained

#94

Post by smittywerben » Fri, Aug 10, 2018, 7:09 PM UTC

nemothekid wrote: post #10

manaskarekar wrote: post #5

Earlier quoted context omitted.

This might be interesting reading. Comments on rust's approach to async from green thread. https://aturon.github.io/blog/2016/08/11/futures/

One of the key reasons Rust took this approach is because Rust's implementation is zero cost - it doesn't require a runtime to implement. It is potentially very efficient, and thanks to Rust's other guarantees its very safe to use. It's as close to bare metal as you can get for a async framework and as a result its incredibly efficient. However, to me, the implementation is a lot complex that green threads. The Futur…

> Python is single threaded.

That's not true, there are multiple threads in Python i.e. zlib from the standard library.

People want unsafe code to communicate concurrently through the Python thread state. I'll let someone else tackle that one.

Re: The magic of asyncio explained

#95

Post by nerdwaller » Sun, Aug 12, 2018, 2:13 AM UTC

BlackFly wrote: post #25

nerdwaller wrote: post #15

Earlier quoted context omitted.

Having worked in asyncio for a bit I don’t entirely follow this (it truly could just be familiarity), very little of asyncio (especially post `async/await` were introduced in the language) is callback based and reads more procedural. Regarding generator coroutines it feels like a natural evolution of the language. Given that yield previously suspended the current function’s state providing value(s) to the closure, it…

The cooperating part of this concurrency model is the complicated part. Consider how you would go about making an orm like sqlalchemy cooperate. Now you have to access properties like this: name = await account.user.name since a lookup may have to occur. This is extremely unnatural and would be better if you could just avoid writing await yet still depend on it being concurrent without blocking your event loop. The f…

This is certainly one "drawback", depending on your perspective, and certainly a cost more of an ORM and how they tend to work than a runtime environment. You'd have a similar issue, for example, in Go if you want to lazy load a property (however there you can't await a goroutine).

Long story short, this drawback tends to be primarily based on experience of the overall system. Coming from traditional rails/django/etc will make these constructs seem awkward.