Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Async I/O for Python 3

dropbox.com

Post reply on HN
Print view View original

71–80 of 92 posts

Re: Async I/O for Python 3

#71

Post by coldtea »

benatkin wrote: post #25

Why does Guido think this is general purpose enough to add to Python but that the scientific features to make it competitive with R aren't? Is he envious of node.js?

The scientific features are only of interest to the (drum roll) scientific community.

Async interests potentially everybody, including the server guys, the backend guys AND the scientific community.

Not to mention that this is a few contained classes, whereas the scientific stuff is tons and tons of code to be included into Python, including lots of Fortran and C, that would more than triple the size of the standard library.

Lastly, node.js? Lots of languages have a good story for async, from C# and Scala, to Go and Rust...

Re: Async I/O for Python 3

#72

Post by nixarn »

I think it's a great idea. I haven't tried Twisted and having to install some 3rd party component to get it working doesn't sound tempting, however being supported by default, does.

Re: Async I/O for Python 3

#73

Post by masklinn »

johnsoft wrote: post #57
Locke1689 wrote: post #33

As Guido mentions, @coroutine/yield from is very similar to C#'s async implementation (with some differences like type safety). Since Guido has the barest of descriptions on how this works, you may find the C# async description useful. [1] [1] http://msdn.microsoft.com/en-us/library/vstudio/hh191443.asp...

Just to check if I'm understanding the presentation right, will the implementation involve compiler magic to turn this: @coroutine def getresp(): s = socket() yield from loop.sock_connect(s, host, port) yield from loop.sock_sendall(s, b'xyzzy') data = yield from loop.sock_recv(s, 100) # ... into this, similar to how C# does it? (let's pretend multi-line lambdas exist for a minute) def getresp(): s = socket() loop.soc…

Python does not do AST-rewriting at compilation, `yield` and `yield from` will handle stack reification for freezing and thawing of coroutines.

So chains of `yield`s and `yield from`s will bubble to the event loop.

Re: Async I/O for Python 3

#74

Post by masklinn »

ekimekim wrote: post #47

A minor gripe (seperate from all my other gripes, which other people have already talked about): "...run code in another thread - sometimes there is no alternative - eg. getaddrinfo(), database connections" Just thought I'd mention that async-supporting DNS libs do exist (eg. gevent ships with C-ares), and in particular I've used async postgres database connections in both C and gevent. The code to gevent-ise psycopg…

> The code to gevent-ise psycopg2 connections is about 10 or 15 lines, iirc.

Because psycopg2 has supported async OOTB since 2.2 by exposing a pollable socket: http://initd.org/psycopg/docs/advanced.html#asynchronous-sup...

There are limitations though, as noted by the docs: COPY and LOs don't work.

Re: Async I/O for Python 3

#75

Post by fzzzy »

cpprototypes wrote: post #64
rdtsc wrote: post #54

Earlier quoted context omitted.

Sorry I didn't express myself correctly (see my reply to akuchling below). Basically yes, Python had Twisted for years, it had Diesel, Monocle, Tornado, and some other ones. I am aware of those and as you've read my comment you saw that I used Twisted enough to know its ins and outs (5 years). > There are also async frameworks that make writing async code the same as synchronous code. Yes there is inlineCallbacks and…

I have a question for you since you have a lot of experience with async. Eventually node.js will have generators (when V8 implements ECMAScript 6) which should allow node.js to have something like gevent. What kind of effect do you think this will have on the node.js world?

I'm really looking forward to that future. See for example task.js:

  http://taskjs.org/
Also, if you want to see generators in v8 (spidermonkey has had them for years now) watch the issue:

  https://code.google.com/p/v8/issues/detail?id=2355

Re: Async I/O for Python 3

#76

Post by fzzzy »

rdw wrote: post #43
hosay123 wrote: post #12

Earlier quoted context omitted.

Sane is in the eye of the beholder.. gevent looks nice, but I'd be very diffident when it comes to actually supporting it in production. It monkey patches the standard library and messes with CPython internals to achieve what it does, infinitely increasing the chance it will conflict with some other piece of code (for example, that bizarre ancient internal propriety library you're using that started life in Fortran,…

As someone who's contributed to Eventlet, I've always felt that the only way that it could get over this scary hurdle (and it is legitimately scary) is for it to be integrated within Python itself. Almost all of the weird problems come from fighting with the baked-in assumptions of the Python runtime. Eventlet does try to alleviate the scariness a little bit by allowing you to import "greened" modules rather than cha…

You da man, rdw.

Re: Async I/O for Python 3

#77

Post by fzzzy »

ekimekim wrote: post #50
CoffeeDregs wrote: post #15

Perhaps the video makes more clear the rationale. E.g. Possible solution: "Standardizing gevent solves all its problems". One of the responses: "I like to write clean code from scratch". Another: "I really like clean interfaces". So I'd prefer that the BDFL work with the gevent folks to get it cleaned up and integrated while adjusting it to expose a "clean interface". Perhaps the whole thing will make more sense once…

I find that unlikely: Guido doesn't like gevent. Though yes, it's a nice thought.

Guido has been resisting the stackless stack slicing assembly technique since I first learned about Python and Stackless Python in 1999. That's obviously never going to change.

Re: Async I/O for Python 3

#78

Post by fzzzy »

simonw wrote: post #40
rdtsc wrote: post #10

Sigh, another Async framework. Yes it has nice features such can replace the reactor/hub thing. Has futures/promises/deferreds. That has all been done before in Twisted. Yields are cute and there was monocle, I wouldn't say it exactly took off : https://github.com/saucelabs/monocle Twisted has inlineCallbacks that use yields as well. Just import Twisted into stdlib then and use that. I am surprised that gevent was di…

If you're using monkey patching in the standard library for a language, something has gone very wrong.

But it's just the tiniest bit of monkey patching! A wafer thin monkey patch!

Re: Async I/O for Python 3

#79

Post by rdtsc »

cpprototypes wrote: post #64
rdtsc wrote: post #54

Earlier quoted context omitted.

Sorry I didn't express myself correctly (see my reply to akuchling below). Basically yes, Python had Twisted for years, it had Diesel, Monocle, Tornado, and some other ones. I am aware of those and as you've read my comment you saw that I used Twisted enough to know its ins and outs (5 years). > There are also async frameworks that make writing async code the same as synchronous code. Yes there is inlineCallbacks and…

I have a question for you since you have a lot of experience with async. Eventually node.js will have generators (when V8 implements ECMAScript 6) which should allow node.js to have something like gevent. What kind of effect do you think this will have on the node.js world?

If it has generators it will make it easier. I remember being happy about finding out Twisted has inlineCallbacks. I basically let you not have to split your logical function into multiple functions simply because it has to to do some IO.

Before it used to be code like:

    def processShoppingCart(...):
       d.addCallback(_cb1)
       return d # (d is a Deferred)

   def _cb1(...):
      d2.addCallback(_cb2)
      return d2 # d2 is a another Deferred
etc.

Which with generators and inlineCallbacks turns into

    @someDecoratorThatEnableUsingInlineCallbacks
    def processShoppingCart(...):
       .. do some work ..
       yield 
       .. do some more work ..
       yield 
       ....
You get the idea. I am not too familiar with Node.js but I imagine it will help quite a bit.

Re: Async I/O for Python 3

#80

Post by rdtsc »

fzzzy wrote: post #77
ekimekim wrote: post #50

Earlier quoted context omitted.

I find that unlikely: Guido doesn't like gevent. Though yes, it's a nice thought.

Guido has been resisting the stackless stack slicing assembly technique since I first learned about Python and Stackless Python in 1999. That's obviously never going to change.

That reminds me of one of those famous Roman Emperors that all is well and good as well as they make rational decisions, then eventually they turn senile or mad, and everyone realizes how dictatorship is not that much fun sometimes.
Post reply on HN