Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Async I/O for Python 3

dropbox.com

Post reply on HN
Print view View original

81–90 of 92 posts

Re: Async I/O for Python 3

#81

Post by rdtsc »

codygman wrote: post #68
jessaustin wrote: post #61

Earlier quoted context omitted.

This is really a matter of taste. You should at least be aware that you're monkeypatching, and code and test accordingly. Many people have good results from monkeypatching, and even more have good results from calling well-written-and-tested libraries that monkeypatch.

But monkey patching turns python from explicit to implicit. It just doesn't feel pythonic to me, and I don't think I'm alone in this. A big reason I use (and enjoy using) python is because it doesn't feel like a "bolted on" solution. All current concurrency options for python feel bolted on to me personally. The python internals weren't designed for this, which is the reason they have to use monkey patching. It doesn…

> It just doesn't feel pythonic to me, and I don't think I'm alone in this.

But how does a tangled mess of callback1 callback2 callback3 feel when all you want to do is do a couple of db reads and writes while processing a simple shopping cart. Is that Pythonic?

> The python internals weren't designed for this, which is the reason they have to use monkey patching.

So fix the internals. Here is a practical way people use Python every day, make that the default, don't revert to some academic or callback mechanism.

> I just couldn't write something which depends on speed and concurrency in python right now, knowing there are solutions much better designed for the problem.

See that is what saddens me. gevent and eventlet do let you write reasonably good and concise IO concurrent code. Some have run large sites and deployments with it. I haven't found any major slowdowns or downsides to switch yet. Because I it is easy and simple to experiment, I'll always try Python first, even though later I might switch to Go or Erlang.

Re: Async I/O for Python 3

#82

Post by rdtsc »

eurleif wrote: post #60
rdtsc wrote: post #10

Sigh, another Async framework. Yes it has nice features such can replace the reactor/hub thing. Has futures/promises/deferreds. That has all been done before in Twisted. Yields are cute and there was monocle, I wouldn't say it exactly took off : https://github.com/saucelabs/monocle Twisted has inlineCallbacks that use yields as well. Just import Twisted into stdlib then and use that. I am surprised that gevent was di…

One thing I love about gevent is that you can share code between async and non-async. Most of my project benefits from async IO, but there's one part that needs to use a lot of CPU within a single process. So that part uses multithreaded Jython, the rest uses gevent, the common code is shared, and it all just works.

Exactly, I was surprised how in the whole "ideas" mailing list discussion Guido had and in other forums that is dismissed as "meh" not even mentioned.

Discussions quickly turn theoretical and academic. "But you don't know when your green threads will switch, man, so I'll add yields in there for you". Yes, and then also make sure there is a complete universe of libraries.

Python is awesome not just because it is fun to write little hello world examples in (so is Logo), it is awesome because it is easy to GetShitDone(TM) quickly. The big part of GetShitDone(TM) quickly is reusing libraries not rewriting everything from scratch.

Using an exotic database for some reason -- great. Found a Python library to interact with it -- great. Oh but my framework is based on Deferreds and this one was written without Deferreds or this one returns Futures. Sorry, go write your own from scratch.

This has been the story of my life for 5+ years search or re-writing Twisted version of already existing libraries.

Now at least just adopt Twisted and go with it if they are going this route. But now, they are 'standardizing' on something new. I think had they done this in 2007, yeah rock on, that would have made sense. They didn't. What saved and kept Python on the back-end during the past 5 or so years was greenlet (eventlet and gevent). Guido is kicking all those people in nuts and saying, "no", we'll do Twisted now (with some changes).

Re: Async I/O for Python 3

#83

Post by rdtsc »

johnsoft wrote: post #57
Locke1689 wrote: post #33

As Guido mentions, @coroutine/yield from is very similar to C#'s async implementation (with some differences like type safety). Since Guido has the barest of descriptions on how this works, you may find the C# async description useful. [1] [1] http://msdn.microsoft.com/en-us/library/vstudio/hh191443.asp...

Just to check if I'm understanding the presentation right, will the implementation involve compiler magic to turn this: @coroutine def getresp(): s = socket() yield from loop.sock_connect(s, host, port) yield from loop.sock_sendall(s, b'xyzzy') data = yield from loop.sock_recv(s, 100) # ... into this, similar to how C# does it? (let's pretend multi-line lambdas exist for a minute) def getresp(): s = socket() loop.soc…

No magic there.

It is Eventlet and Gevent have that magic. Here is how that looks:

    def getresp():
        s = socket()
        s.connect((host,port))
        s.sendall(s,b'xyzzy')
        data = s.recv(s,100)
Compare that to any of the above. This is what is thrown away in favor of 'yield from' and @coroutine mess coupled with a completely parallel set of IO libraries.

Re: Async I/O for Python 3

#84

Post by fzzzy »

rdtsc wrote: post #80
fzzzy wrote: post #77

Earlier quoted context omitted.

Guido has been resisting the stackless stack slicing assembly technique since I first learned about Python and Stackless Python in 1999. That's obviously never going to change.

That reminds me of one of those famous Roman Emperors that all is well and good as well as they make rational decisions, then eventually they turn senile or mad, and everyone realizes how dictatorship is not that much fun sometimes.

From a certain perspective it is a rational decision. Because the CPython API relies so heavily on the C stack, either some platform-specific assembly is required to slice up the C stack to implement green threads, or the entire CPython API would have to be redesigned to not keep the Python stack state on the C stack.

Way back in the day [1] the proposal for merging Stackless into mainline Python involved removing Python's stack state from the C stack. However there are complications with calling from C extensions back into Python that ultimately killed this approach.

After this Stackless evolved to be a much less modified fork of the Python codebase with a bit of platform specific assembly that performed "stack slicing". Basically when a coro starts, the contents of the stack pointer register are recorded, and when a coro wishes to switch, the slice of the stack from the recorded stack pointer value to the current stack pointer value is copied off onto the heap. The stack pointer is then adjusted back down to the saved value and another task can run in that same stack space, or a stack slice that was stored on the heap previously can be copied back onto the stack and the stack pointer adjusted so that the task resumes where it left off.

Then around 2005 the Stackless stack slicing assembly was ported into a CPython extension as part of py.lib. This was known as greenlet. Unfortunately all the original codespeak.net py.lib pages are 404 now, but here's a blog post from around that time that talks about it [2].

Finally the relevant parts of greenlet were extracted from py.lib into a standalone greenlet module, and eventlet, gevent, et cetera grew up around this packaging of the Stackless stack slicing code.

So you see, using the Stackless strategy in mainline python would have either required breaking a bunch of existing C extensions and placing limitations on how C extensions could call back into Python, or custom low level stack slicing assembly that has to be maintained for each processor architecture. CPython does not contain any assembly, only portable C, so using greenlet in core would mean that CPython itself would become less portable.

Generators, on the other hand, get around the issue of CPython's dependence on the C stack by unwinding both the C and Python stack on yield. The C and Python stack state is lost, but a program counter state is kept so that the next time the generator is called, execution resumes in the middle of the function instead of the beginning.

There are problems with this approach; the previous stack state is lost, so stack traces have less information in them; the entire call stack must be unwound back up to the main loop instead of a deeply nested call being able to switch without the callers being aware that the switch is happening; and special syntax (yield or yield from) must be explicitly used to call out a switch.

But at least generators don't require breaking changes to the CPython API or non-portable stack slicing assembly. So maybe now you can see why Guido prefers it.

Myself, I decided that the advantages of transparent stack switching and interoperability outweighed the disadvantages of relying on non-portable stack slicing assembly. However Guido just sees things in a different light, and I understand his perspective.

  [1] http://www.python.org/dev/peps/pep-0219/
  [2] http://agiletesting.blogspot.com/2005/07/py-lib-gems-greenlets-and-pyxml.html

Re: Async I/O for Python 3

#85

Post by fzzzy »

rdtsc wrote: post #83
johnsoft wrote: post #57

Earlier quoted context omitted.

Just to check if I'm understanding the presentation right, will the implementation involve compiler magic to turn this: @coroutine def getresp(): s = socket() yield from loop.sock_connect(s, host, port) yield from loop.sock_sendall(s, b'xyzzy') data = yield from loop.sock_recv(s, 100) # ... into this, similar to how C# does it? (let's pretend multi-line lambdas exist for a minute) def getresp(): s = socket() loop.soc…

No magic there. It is Eventlet and Gevent have that magic. Here is how that looks: def getresp(): s = socket() s.connect((host,port)) s.sendall(s,b'xyzzy') data = s.recv(s,100) Compare that to any of the above. This is what is thrown away in favor of 'yield from' and @coroutine mess coupled with a completely parallel set of IO libraries.

Well... There actually are a completely parallel set of IO libraries, it just happens that the interface can be identical to the existing blocking interfaces because of the greenlet stack slicing magic... So it only appears like there are not a completely parallel set of IO libraries.

But that's just a nitpick.

Re: Async I/O for Python 3

#86

Post by fzzzy »

johnsoft wrote: post #57
Locke1689 wrote: post #33

As Guido mentions, @coroutine/yield from is very similar to C#'s async implementation (with some differences like type safety). Since Guido has the barest of descriptions on how this works, you may find the C# async description useful. [1] [1] http://msdn.microsoft.com/en-us/library/vstudio/hh191443.asp...

Just to check if I'm understanding the presentation right, will the implementation involve compiler magic to turn this: @coroutine def getresp(): s = socket() yield from loop.sock_connect(s, host, port) yield from loop.sock_sendall(s, b'xyzzy') data = yield from loop.sock_recv(s, 100) # ... into this, similar to how C# does it? (let's pretend multi-line lambdas exist for a minute) def getresp(): s = socket() loop.soc…

The yield from has to be explicitly bubbled all the way up the call chain to the main loop.

Re: Async I/O for Python 3

#87

Post by lucian1900 »

ekimekim wrote: post #49
stuaxo wrote: post #21

Earlier quoted context omitted.

Eh, the whole point is that this is at a lower level to all of those. All the above frameworks will be ported to this and become interoperable - in a similar way to how wsgi works for the web.

It won't be useful if the standard interface is, basically, Twisted. Stack-based solutions like gevent, eventlet get screwed over by this.

No, they don't. Gevent and eventlet can simply use this instead of their own event loops. In addition, they could install their own event loops.

Tulip separates API from event loop mechanism.

Re: Async I/O for Python 3

#89

Post by rdtsc »

fzzzy wrote: post #85
rdtsc wrote: post #83

Earlier quoted context omitted.

No magic there. It is Eventlet and Gevent have that magic. Here is how that looks: def getresp(): s = socket() s.connect((host,port)) s.sendall(s,b'xyzzy') data = s.recv(s,100) Compare that to any of the above. This is what is thrown away in favor of 'yield from' and @coroutine mess coupled with a completely parallel set of IO libraries.

Well... There actually are a completely parallel set of IO libraries, it just happens that the interface can be identical to the existing blocking interfaces because of the greenlet stack slicing magic... So it only appears like there are not a completely parallel set of IO libraries. But that's just a nitpick.

> So it only appears like there are not a completely parallel set of IO libraries.

Right on. That's the great part -- both a simple way to program and re usability of libraries.

So far, I see library ecosystem fragmentation as the biggest issue of all and nobody seems to want to talk to it.

Academically all the yields and co-routines look so cool, in practice when you need 5 libraries to help with some task, and now you have to re-write them -- not so cool.

Re: Async I/O for Python 3

#90

Post by codygman »

rdtsc wrote: post #81
codygman wrote: post #68

Earlier quoted context omitted.

But monkey patching turns python from explicit to implicit. It just doesn't feel pythonic to me, and I don't think I'm alone in this. A big reason I use (and enjoy using) python is because it doesn't feel like a "bolted on" solution. All current concurrency options for python feel bolted on to me personally. The python internals weren't designed for this, which is the reason they have to use monkey patching. It doesn…

> It just doesn't feel pythonic to me, and I don't think I'm alone in this. But how does a tangled mess of callback1 callback2 callback3 feel when all you want to do is do a couple of db reads and writes while processing a simple shopping cart. Is that Pythonic? > The python internals weren't designed for this, which is the reason they have to use monkey patching. So fix the internals. Here is a practical way people…

I agree, we should fix the internals. Haskell didn't have concurrency by default and things turned out pretty well there.

I actually did try out a solution based on gevent and eventlet before switching to Go.

Post reply on HN