Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fixing the Python subprocess interface

amoffat.github.com

Post reply on HN
Print view View original

41–50 of 65 posts

Re: Fixing the Python subprocess interface

#41

Post by mLewisLogic »

If you don't like the import *, don't use it. Python supports it, so why shouldn't this library? (not that it's a great idea)

If you'd like to handle missing system executables, catch the exception. You should be writing in that style anyways.

Honestly, this cleans up a ton of system scripting code, making it way more readable/maintainable.

Maybe the code could be cleaned up, but this is the direction that Python should be heading. Abstract away the complications when possible, keep low-level stuff around for when it's absolutely needed.

Beautiful is better than ugly.

Re: Fixing the Python subprocess interface

#42

Post by logic »

petsos wrote: post #6

Why not use pipes for piping? e.g. print du("*", "-sb") | sort("-rn")

Order of evaluation; du() will be instantiated, sort() will be instantiated, and then du().__or__() is evaluated, which means you won't have du's stdout to attach to stdin on sort when it's instantiated.

Figuring out a way around this (mocking up an fd to pass to sort at instantiation time that blocks until stdout on du is available) is left as an exercise to the reader. I've already said too much. ;)

Re: Fixing the Python subprocess interface

#43

Post by ulope »

scott_s wrote: post #40

I use Python for shell scripting a lot. Ignoring all of the issues people have brought up here, I really like how function composition is piping: # sort this directory by biggest file print sort(du("*", "-sb"), "-rn") # print the number of folders and files in /etc print wc(ls("/etc", "-1"), "-l") The reason that I like that method over, say, envoy's [1] method is that envoy.run('uptime | pbcopy') has what I consider…

As long as the package you need doesn't include a C extension (which most don't) you can just ship it with your code (license permitting ofc.) - just add the path to the libary to sys.path. It's not a very clean solution but can be a real life saver when you have to work on "broken" systems.

Re: Fixing the Python subprocess interface

#44

Post by jamesaguilar »

tswicegood wrote: post #17

FWIW, re-replaying the stack trace to figure out what was imported and re-implementing it is a horrible idea. There are much better ways to do this type of import voodoo, specifically the import hooks that Python ships with. Here's an example of their use inside a small side project of mine: https://github.com/tswicegood/maxixe/blob/master/maxixe/__in... All that said, this is horribly un-Pythonic. A much better rout…

envoy looks a lot more verbose than this though . . . if you're really trying to replace shell scripting, having envoy.run(foo) on every line is going to get annoying.

Also, it's not clear on what basis you make this assertion:

> FWIW, re-replaying the stack trace to figure out what was imported and re-implementing it is a horrible idea.

Does it not work?

Re: Fixing the Python subprocess interface

#45

Post by akx »

The import hack is the only thing I don't really like (I'd prefer the `from pbs import sh` syntax proposed elsewhere) -- and without reading the code... is piping really piping, or will the function composition example actually consume the first command result fully first?

Anyway, this is a great idea. :)

Re: Fixing the Python subprocess interface

#47

Post by scott_s »

ulope wrote: post #43
scott_s wrote: post #40

I use Python for shell scripting a lot. Ignoring all of the issues people have brought up here, I really like how function composition is piping: # sort this directory by biggest file print sort(du("*", "-sb"), "-rn") # print the number of folders and files in /etc print wc(ls("/etc", "-1"), "-l") The reason that I like that method over, say, envoy's [1] method is that envoy.run('uptime | pbcopy') has what I consider…

As long as the package you need doesn't include a C extension (which most don't) you can just ship it with your code (license permitting ofc.) - just add the path to the libary to sys.path. It's not a very clean solution but can be a real life saver when you have to work on "broken" systems.

Sorry, I wasn't clear. This is a system I log into every day, and I don't want to maintain my own install of Python and related packages on it. It's too much overhead. I'd rather just use the default Python, even though it's old.

Re: Fixing the Python subprocess interface

#48

Post by bryze »

Can we please stop titling postings like this? How about "An Alternative to the Python Subprocess Interface". Fixing something implies that it's broken or inadequate, and from my limited experience, Subprocess is already a major improvement over os.system. I'm not saying this idea doesn't have value, I'm saying that the way it's framed lacks the requisite humility it ought to.

Re: Fixing the Python subprocess interface

#49

Post by mgedmin »

cobbal wrote: post #36

It states that the lines curl("http://duckduckgo.com/", "-o page.html", "--silent") curl("http://duckduckgo.com/", "-o", "page.html", "--silent") are equivalent. This worries me, I would much rather always have it be one argument corresponding to exactly one shell argument. Here it looks like in some (maybe all?) cases arguments are split on spaces, which means always having to be extremely cautious about escaping, s…

Exactly. If you look at the sources, you'll see that the arguments are all joined into a single string with spaces, then split back into separate words using shlex.split().

So cat("filename with spaces in it") will fail, but cat("'filename with spaces in it') ought to succeed.

It's a neat experiment, but using this module in production would not be a great idea.

Re: Fixing the Python subprocess interface

#50

Post by daenz »

samps wrote: post #8

It's odd that the import mechanism is abused here to make objects "out of thin air". The fact that "from pbs import ffmpeg" works only if ffmpeg is actually on the path is somewhat surprising. I think the more comfortable (and Pythonic?) way to do this would be to explicitly create these command objects: >>> import pbs >>> ffmpeg = pbs.Command('ffmpeg') # or '/usr/bin/ffmpeg', perhaps >>> result = ffmpeg(...) [Edit:…

Hi! Author here, there are a few ways to use it, including your suggestion (did you make your suggestion up or were you pulling from the docs?):

    # magical, designed only for single shell scripts
    from pbs import *
    ffmpeg()

    # less magical
    from pbs import ffmpeg
    ffmpeg()

    # or
    import pbs
    pbs.ffmpeg()

    # no magic
    import pbs
    ffmpeg = pbs.Command(pbs.which("ffmpeg")) # command takes full path
    ffmpeg()
I tried to cover the main use cases adequately. My goal was to ease a pain point myself and others have experienced, that other popular packages don't address well unfortunately.

If it's helped anyone like it's helped me, I'm happy and glad to share!

Post reply on HN