Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python for Humans

python-for-humans.heroku.com

Post reply on HN
Print view View original

91–100 of 108 posts

Re: Python for Humans

#91

Post by Silhouette »

loevborg wrote: post #85
Silhouette wrote: post #77

Earlier quoted context omitted.

> subprocess.call() accepts an array in the style of ["ls", "-l", "/mnt/My SD card"]. This has obvious advantages over having to deal with escaping shell characters. Unless you're running on Windows, in which case IME it will corrupt your carefully constructed parameters in completely inappropriate ways that can be debugged only at the cost of (a) changing the call() to execute a script that dumps the actual paramete…

On Unix, a new process is supplied argv[], an array that contains the executable name and invidual arguments. Clearly, supplying call() a list of arguments is the right thing to do. I seem to remember that on Win32, all you get is an argument string, and the process is required to do the parsing itself. This is simply a different model, and the Unix way is cleaner and easier to work with. It seems to me reasonable to…

The big difference is that the UNIX shell does all kinds of expansion/interpolation for you, while Windows basically leaves things alone. You get an argv[] array in both cases if you're writing in C.

Neither of these approaches is inherently superior, they're just placing responsibility for certain operations in different places. But the fact is that because Windows programs don't have to assume a certain set of conventions for their command line, many do not, and if you have the misfortune to want to automate those using subprocess, you're in for a world of pain (until you just give up and use the single-string version instead of the list of arguments, having realised that this is enough to stop it messing around with your carefully crafted strings and just pass them through verbatim).

Re: Python for Humans

#92

Post by cturner »

teyc wrote: post #57

I blame GOF for making Python Standard Libs hard. The patterns described were for an OO system where functions were not first class. Python didn't need to be complicated. If you have a look at the older libraries, most of them were written in a procedural style. Not only that, it is very amenable to testing in the REPL. import smtplib s=smtplib.SMTP("localhost") s.sendmail("me@my.org",tolist,msg) note the absence of…

    > If anyone remembers, Java had to do OO in a
    > big-style with OO everywhere -- there were no
    > alternatives.
You can write Java that isn't heavily OO, but you have to implement alternatives to sections of the stdlib that most people assume or take for granted.

Related to what you're saying about the GUI, I'd be interested to see a detailed summary of what the Lighthouse people did, and how it was different to Java. I've found that NeXT tradition stuff - despite claims that it's heavily OO - in facat tends to err away from subclassing towards composition. I suspect the Lighthouse interface patterns did too.

Re: Python for Humans

#93

Post by obtu »

pnathan wrote: post #58
socratic wrote: post #31

This presentation brings up a tangential point that has always confused me: how error-prone is starting a subprocess, really? I agree with the author's goals of making common tasks easier and more obvious. urllib2 is an easy target, as it was added to the standard library over a decade ago, long before REST was something people talked about. The best tools for packaging, versioning, and testing have always been a bit…

I have never been able to figure out how - in Python - to be able to stream asynchronously both stdout and stderr from the subprocess, both printing both of them as well as writing the data to a file.

You're listening for two file descriptor events, so you need some sort of event loop. select can do it but it's low-level; and since there can be only one event loop per program, your choices are frameworks and not simply libraries.

Here's a way to do it with Twisted (docs here: http://twistedmatrix.com/documents/current/core/howto/proces... ):

  from twisted.internet import reactor, protocol

  class PrintAndLogProtocol(protocol.ProcessProtocol):
      def outReceived(self, data):
          # print and log
      errReceived = outReceived

  reactor.spawnProcess(PrintAndLogProtocol(),
       '/path/to/exe', ['exe', 'arg1', 'arg2'])
  reactor.run()

Re: Python for Humans

#94

Post by cturner »

bretthoerner wrote: post #35
socratic wrote: post #31

This presentation brings up a tangential point that has always confused me: how error-prone is starting a subprocess, really? I agree with the author's goals of making common tasks easier and more obvious. urllib2 is an easy target, as it was added to the standard library over a decade ago, long before REST was something people talked about. The best tools for packaging, versioning, and testing have always been a bit…

It's funny you mention that. The author/speaker wrote a "Subprocesses for humans" module, too: https://github.com/kennethreitz/envoy There's no fundamental problem that's stopped Python from doing this before. For some reason, all of the ways to spawn a subprocess in Python have tried to map almost directly to the underlying C API... which is pretty awful.

    > For some reason, all of the ways to spawn a subprocess
    > in Python have tried to map almost directly to the
    > underlying C API
I think both are good and necessary. One of the strengthes of python is that if you have a copy of Stevens you can usually work out how to do something in Python. And this is awesome. I've written things on top of unix that in times past would have been written in C. However, that mechanism is usually not very "pythonic".

In the early days python had a principle that there should be one way to do things. You don't hear this so much any more: we're long past that now, with some things different between 2.6 and 2.7 (arg handling), and with multiple broken libraries in the stdlib. When you're working on your own computer and your own time with root access you can always hand-roll outcmes. But it's common to have to deal with a spread of python's and cater to the most obsolete version. Yet I suspect some people still aspire to the one-way-to-do-it, and pretend it's true.

I think we should dump the principle.

A good example of why compromise is not the right outcome is the curses library - it's not quite Stevens, but it's not friendly either. It's hard to do good work with the curses library. We'd be better off if there was (1) a close curses mapping to the C ncurses mechanisms and (2) a nice-to-use abstraction layer that hid far more away from you.

Re: Python for Humans

#95

Post by teyc »

cturner wrote: post #92
teyc wrote: post #57

I blame GOF for making Python Standard Libs hard. The patterns described were for an OO system where functions were not first class. Python didn't need to be complicated. If you have a look at the older libraries, most of them were written in a procedural style. Not only that, it is very amenable to testing in the REPL. import smtplib s=smtplib.SMTP("localhost") s.sendmail("me@my.org",tolist,msg) note the absence of…

> If anyone remembers, Java had to do OO in a > big-style with OO everywhere -- there were no > alternatives. You can write Java that isn't heavily OO, but you have to implement alternatives to sections of the stdlib that most people assume or take for granted. Related to what you're saying about the GUI, I'd be interested to see a detailed summary of what the Lighthouse people did, and how it was different to Java.…

I've struggled through a few Cocoa tutorials. I think the programmer creates a delegate object which handles events. These delegates are assigned to UI elements, similar to how one would addMouseListener to a Java UI element. I may be entirely wrong here. Perhaps someone who knows can put the facts straight.

Re: Python for Humans

#96

Post by adeelk »

mixmastamyk wrote: post #53
adeelk wrote: post #51

Earlier quoted context omitted.

Wrappers are handy, but as soon as you need something beyond the basic use case they become useless. What’s great about Requests is that it seems to have minimal leakiness as an abstraction over HTTP.

My wrapper is quite robust after almost five years. e.g. It can save headers to alternate data streams (on NTFS) for proper 304 handling. If there is anything left to implement it could be done pretty quickly. Still I like these new projects; it's a shame they missed the python 3.x boat by only a year or two. That would have been a great time to include them in the stdlib.

It sounds like you should release this wrapper :)

Re: Python for Humans

#97

Post by fdkz »

pnathan wrote: post #58
socratic wrote: post #31

This presentation brings up a tangential point that has always confused me: how error-prone is starting a subprocess, really? I agree with the author's goals of making common tasks easier and more obvious. urllib2 is an easy target, as it was added to the standard library over a decade ago, long before REST was something people talked about. The best tools for packaging, versioning, and testing have always been a bit…

I have never been able to figure out how - in Python - to be able to stream asynchronously both stdout and stderr from the subprocess, both printing both of them as well as writing the data to a file.

I'm using the mkfifo method on linux/macosx:

    import os
    import sys
    import time
    import subprocess

    # turn off stdout buffering. otherwise we won't see things like wget progress-bars that update without newlines.
    sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

    pipename = "tempfile"

    if os.path.exists(pipename):
        os.remove(pipename)

    # create a pipe. one side is connected to the ping process, other side is connected to python.
    os.mkfifo(pipename)
    read_fd = os.open(pipename, os.O_RDONLY|os.O_NONBLOCK)
    writer = open(pipename, "w+")

    proc = subprocess.Popen("ping www.google.com", cwd=sys.path[0], stdout=writer, stderr=writer, shell=True)

    while 1:
        try:
            # nonblocking poll data from the external process.
            s = os.read(read_fd, 1024)
            if s:
                sys.stdout.write(s)
        except OSError:
            pass
        # sidenote: minimum sleep time is 1/64 seconds on many windows pc-s.
        time.sleep(0.1)

    # remember to remove the pipe "tempfile"

Re: Python for Humans

#98

Post by dhalexander »

The Python standard library has gotten worse over time, as it got loaded up with more and more features, obfuscating the common use cases. The irony now is that to do simple, everyday things (like http requests) you are now better off installing a third party package like "requests" than using the standard library. So much for "batteries included."

The standard library needs a reboot. Why not do it in Python 3? Nobody's using it yet anyway ;-)

Re: Python for Humans

#99

Post by mixmastamyk »

adeelk wrote: post #96
mixmastamyk wrote: post #53

Earlier quoted context omitted.

My wrapper is quite robust after almost five years. e.g. It can save headers to alternate data streams (on NTFS) for proper 304 handling. If there is anything left to implement it could be done pretty quickly. Still I like these new projects; it's a shame they missed the python 3.x boat by only a year or two. That would have been a great time to include them in the stdlib.

It sounds like you should release this wrapper :)

It is online, but embedded in my employers software. I could probably do some extraction of it, but not sure if it is worth the effort.

Re: Python for Humans

#100

Post by fdkz »

fdkz wrote: post #97
pnathan wrote: post #58

Earlier quoted context omitted.

I have never been able to figure out how - in Python - to be able to stream asynchronously both stdout and stderr from the subprocess, both printing both of them as well as writing the data to a file.

I'm using the mkfifo method on linux/macosx: import os import sys import time import subprocess # turn off stdout buffering. otherwise we won't see things like wget progress-bars that update without newlines. sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0) pipename = "tempfile" if os.path.exists(pipename): os.remove(pipename) # create a pipe. one side is connected to the ping process, other side is connected to p…

Replying to myself. Using mkfifo is not necessary:

    import os, sys, time, subprocess, fcntl
    sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
    read_fd, write_fd = os.pipe()
    fcntl.fcntl(read_fd, fcntl.F_SETFL, os.O_NONBLOCK) # don't know of any windows equivalent for this line
    proc = subprocess.Popen("ping www.google.com", cwd=sys.path[0], stdout=write_fd, stderr=write_fd, shell=True)
    while 1:
        try:
            s = os.read(read_fd, 1024)
            if s:
                sys.stdout.write(s)
        except OSError:
            pass
        time.sleep(0.1)
Post reply on HN