Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Async I/O for Python 3

dropbox.com

Post reply on HN
Print view View original

51–60 of 92 posts

Re: Async I/O for Python 3

#51

Post by visitorx »

rdtsc wrote: post #10

Sigh, another Async framework. Yes it has nice features such can replace the reactor/hub thing. Has futures/promises/deferreds. That has all been done before in Twisted. Yields are cute and there was monocle, I wouldn't say it exactly took off : https://github.com/saucelabs/monocle Twisted has inlineCallbacks that use yields as well. Just import Twisted into stdlib then and use that. I am surprised that gevent was di…

seconded, I doubt many a sane person would use Python for IO bound concurrency, they should definitely be looking at Rust (in the future slightly) and Erlang (currently the defacto standard for easy-of-use concurrency).

Re: Async I/O for Python 3

#52

Post by visitorx »

Ixiaus wrote: post #18
hosay123 wrote: post #12

Earlier quoted context omitted.

Sane is in the eye of the beholder.. gevent looks nice, but I'd be very diffident when it comes to actually supporting it in production. It monkey patches the standard library and messes with CPython internals to achieve what it does, infinitely increasing the chance it will conflict with some other piece of code (for example, that bizarre ancient internal propriety library you're using that started life in Fortran,…

I used gevent at first for a project that needed Async I/O and it worked really well, but then I switched to Erlang and I realized how poor a choice Python is for such tasks. The language really needs to be designed from the start for it (like Go, Rust, Erlang &c... Haskell wasn't designed from start for it but because of it's functional purity, bolting it on was "natural" but - it isn't so for Python IMHO).

Go was designed for concurrency from the start and even has null pointers and lots of mutation, this makes the concurrency problem easier to solve because you have more control over state.

Re: Async I/O for Python 3

#53

Post by rdtsc »

akuchling wrote: post #37
rdtsc wrote: post #32

Earlier quoted context omitted.

I think there is another issue here. Python world has watched as Node.js has been eating its lunch on the server side and they decided, ah, surely that is because Node.js has async, if we add that too, everyone will love Python again and come crawling back. They are not saying, I think that is written between the lines. Except one thing, as you pointed out, people use Node.js -- 1) it is JS 2) V8 is fast.

People have been writing async apps in Python since 1995 (Medusa); Twisted was first published around 2001. It's not like async programming is new to Python.

Sorry I didn't say it correctly, I meant that it seems the renewed interest in async is stemming from watching Node.js get all the attention.

I have been using Twisted for 5 years full time and have also used eventlet and gevent. From talking to others, I have found few who enjoyed or loved Twisted. It was pretty much the only sane way to do concurrent, performant IO for a while. But then when green thread approach came about, I had never looked back.

All was well. Then one day Node.js showed up, and it seems it has started to eat Python's lunch -- fast, scripted development on the server side, with some reasonable concurrency. And it was faster too.

Python devs looked at it and couldn't believe their eyes. And I speculate many have concluded it was because everyone was in love with a callback based async IO paradigm. So that's my guess why we are seeing this proposal.

Re: Async I/O for Python 3

#54

Post by rdtsc »

nashequilibrium wrote: post #44
rdtsc wrote: post #32

Earlier quoted context omitted.

I think there is another issue here. Python world has watched as Node.js has been eating its lunch on the server side and they decided, ah, surely that is because Node.js has async, if we add that too, everyone will love Python again and come crawling back. They are not saying, I think that is written between the lines. Except one thing, as you pointed out, people use Node.js -- 1) it is JS 2) V8 is fast.

The reason why you getting so many downvotes is because your comment is not just silly but flat wrong. Twisted, Tornado were around before nodejs. There are also async frameworks that make writing async code the same as synchronous code. I like tornado but I am using nodejs for my current app because of the libraries. This is where nodejs really shines, the community and libraries are awesome. Twisted has a lot of li…

Sorry I didn't express myself correctly (see my reply to akuchling below).

Basically yes, Python had Twisted for years, it had Diesel, Monocle, Tornado, and some other ones. I am aware of those and as you've read my comment you saw that I used Twisted enough to know its ins and outs (5 years).

> There are also async frameworks that make writing async code the same as synchronous code.

Yes there is inlineCallbacks and I used. Node.js also has async (https://github.com/caolan/async). But you don't address the main problem that I raised -- fragmentation of libraries. Python is great because it comes with batteries, and then you can find even more batteries everywhere, _except_ if you use an async framework like Twisted, which, percolates all the way through you API code. Once your socket.recv() returns a Deferred(), that deferred will bubble up all the way to the user interface. So you now you end up searching or recreating a parallel set of libraries.

> Twisted has a lot of libraries but it has so much going on that many developers find it too complex.

It is too complex with too many libraries for those who want to take it up but it is not complex and doesn't have enough libraries if you are in it already -- every library you use has to be Twisted now. That's the danger of inventing a new framework.

Yes it will be standard, but there is already a practical standard -- eventlet and gevent. This is somethin Node.js doesn't have. I will personally take monkey-patching and the danger that my function will context switch inside while doing IO over using Twisted. I saw a practical benefit from it at least.

Re: Async I/O for Python 3

#55

Post by rdtsc »

pekk wrote: post #22
hosay123 wrote: post #12

Earlier quoted context omitted.

Sane is in the eye of the beholder.. gevent looks nice, but I'd be very diffident when it comes to actually supporting it in production. It monkey patches the standard library and messes with CPython internals to achieve what it does, infinitely increasing the chance it will conflict with some other piece of code (for example, that bizarre ancient internal propriety library you're using that started life in Fortran,…

Have you actually got any war stories about gevent, or is this all guesswork about how you think it stands to reason that it would be bad? Do you think this would be relevant if greenlets were adopted as a part of Python?

I have -- we used Twisted (about 10k lines of code). Then switched to eventlet -- and the code reduced to about 5 or 6K. There were some issues with monkey-patching, but tests showed them, and we fixed them.

As someone mentioned, if they instead standardized on greenlet then monkeypatching talk wouldn't make sense.

Re: Async I/O for Python 3

#57

Post by johnsoft »

Locke1689 wrote: post #33

As Guido mentions, @coroutine/yield from is very similar to C#'s async implementation (with some differences like type safety). Since Guido has the barest of descriptions on how this works, you may find the C# async description useful. [1] [1] http://msdn.microsoft.com/en-us/library/vstudio/hh191443.asp...

Just to check if I'm understanding the presentation right, will the implementation involve compiler magic to turn this:

    @coroutine
    def getresp():
        s = socket()
        yield from loop.sock_connect(s, host, port)
        yield from loop.sock_sendall(s, b'xyzzy')
        data = yield from loop.sock_recv(s, 100)
        # ...
into this, similar to how C# does it? (let's pretend multi-line lambdas exist for a minute)

    def getresp():
        s = socket()
        loop.sock_connect(s, host, port).add_done_callback(lambda:
            loop.sock_sendall(s, b'xyzzy').add_done_callback(lambda:
                data = loop.sock_recv(s, 100).add_done_callback(lambda:
                    # ...
                )
            )
        )
Or will the `yield from`s bubble up all the way to the event loop and avoid the need for that?

Re: Async I/O for Python 3

#58

Post by csears »

opminion wrote: post #56

Wait, Guido is proposing implementing INTERCAL'S COME FROM? (as yield from)?

For anyone curious, INTERCAL was originally a joke language which included a COMEFROM instruction that acted like GOTO in reverse: http://en.wikipedia.org/wiki/COMEFROM

Python's "yield from" hands off execution to a sub-generator: http://www.python.org/dev/peps/pep-0380/

Re: Async I/O for Python 3

#59

Post by jlgreco »

ceronman wrote: post #48
pekk wrote: post #3

Still puzzled why the primary inspiration for this is Twisted, which has some really ugly APIs.

Twisted with inlineCallbacks is actually quite nice IMO. It's very similar to C# async/await.

Twisted's inlineCallbacks singlehandedly turn Twisted in my mind from an abomination into something that is a joy to work with. In lei of a more hands off Erlang/Go approach, I am convinced that style is the only way to go.

Re: Async I/O for Python 3

#60

Post by eurleif »

rdtsc wrote: post #10

Sigh, another Async framework. Yes it has nice features such can replace the reactor/hub thing. Has futures/promises/deferreds. That has all been done before in Twisted. Yields are cute and there was monocle, I wouldn't say it exactly took off : https://github.com/saucelabs/monocle Twisted has inlineCallbacks that use yields as well. Just import Twisted into stdlib then and use that. I am surprised that gevent was di…

One thing I love about gevent is that you can share code between async and non-async. Most of my project benefits from async IO, but there's one part that needs to use a lot of CPU within a single process. So that part uses multithreaded Jython, the rest uses gevent, the common code is shared, and it all just works.
Post reply on HN