Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using Python for Scripting

hypirion.com

Post reply on HN
Print view View original

71–80 of 112 posts

Re: Using Python for Scripting

#71

Post by lgas »

_jzlw wrote: post #3

How do you handle packages? I want scripts to a be a single file with a shebang, not a repo with a requirements.txt that I need to run in a venv. To me, this is the biggest blocker to using Python for any non-trivial scripting (which is precisely the kind where I wouldn't want to use bash), but I'd like to know how others deal with it. C# scripts let you reference packages in a comment at the top of the file, for exa…

I don't have much need of this personally, but I was playing around with an example from earlier in the thread and ended up with this:

    #!/usr/bin/env -S uv run --with sh --script
    from sh import ifconfig
    print(ifconfig("en0"))
which is a pretty nice experience assuming you already have `uv` in the target environment.

Re: Using Python for Scripting

#72

Post by lgas »

kstrauser wrote: post #44
tayo42 wrote: post #18

Pretty much anything longer then a throwaway one liner I write in python. Would be cool if python had a pipe operator though. The back ticks in ruby is pretty ergonomic too. Wish python had a simpler way to run commands. Kind of tedious to look up subprocess run arguments and also break things up into arrays.

You can always set shell=True and pass in an entire command line as a string, but… don’t do that. It seems really nice until the first time you get the shell escaping wrong, and then it’s something you tend never to do again. For example, subprocess.run(“rm -rf ~/ some file”, shell=True) and subprocess.run([“rm”, “-rf”, “~/ some file”]) have significant different behavior.

What is the difference in behavior? They both look like they would delete the user's home directory. I assume the latter would try to delete a directory literally named with a tilde instead?

Re: Using Python for Scripting

#73

Post by btown »

theomega wrote: post #52
rjmill wrote: post #17

Odd, I don't see any mention of subprocess.run, the workhorse of python scripting. Quick rundown for the unfamiliar: Give it a command as a list of strings (e.g., subprocess.run(["echo", "foo"]).) It takes a bunch of flags, but the most useful (but not immediately obvious) ones are: check=True: Raise an error if the command fails capture_output=True: Captures stdout/stderr on the CompletedProcess text=True: Automatic…

One thing I can recommend that makes scripting in python with external commands a lot easier is the `sh` module: https://pypi.org/project/sh/ Basically you can just `from sh import [command]` and then have an installed binary command available as function from sh import ifconfig print(ifconfig("eth0"))

I love how this import trick shows how hackable Python is - and it’s this very hackability that has led to so many of the advances we see in AI. Arguably without operator overloads we’d be 5 or more years behind.

https://github.com/amoffat/sh/blob/2a90b1f87a877e5e09da32fd4...

Re: Using Python for Scripting

#74

Post by kstrauser »

lgas wrote: post #72
kstrauser wrote: post #44

Earlier quoted context omitted.

You can always set shell=True and pass in an entire command line as a string, but… don’t do that. It seems really nice until the first time you get the shell escaping wrong, and then it’s something you tend never to do again. For example, subprocess.run(“rm -rf ~/ some file”, shell=True) and subprocess.run([“rm”, “-rf”, “~/ some file”]) have significant different behavior.

What is the difference in behavior? They both look like they would delete the user's home directory. I assume the latter would try to delete a directory literally named with a tilde instead?

The latter passes each item in the list into the child processes’s argv, as-is, without the shell parsing them. That means this would delete a single item named “~/ some file”, spaces and all, instead of three items named “~/“, “some”, and “file”.

Edit: I’m typing this on my phone, so brevity won over explicitness. The latter probably wouldn’t expand ~. Imagine a file named “/home/me/ some file” for a better example.

Re: Using Python for Scripting

#75

Post by theamk »

theomega wrote: post #52
rjmill wrote: post #17

Odd, I don't see any mention of subprocess.run, the workhorse of python scripting. Quick rundown for the unfamiliar: Give it a command as a list of strings (e.g., subprocess.run(["echo", "foo"]).) It takes a bunch of flags, but the most useful (but not immediately obvious) ones are: check=True: Raise an error if the command fails capture_output=True: Captures stdout/stderr on the CompletedProcess text=True: Automatic…

One thing I can recommend that makes scripting in python with external commands a lot easier is the `sh` module: https://pypi.org/project/sh/ Basically you can just `from sh import [command]` and then have an installed binary command available as function from sh import ifconfig print(ifconfig("eth0"))

Please don't use "sh" python library!

By default (1) captures stdout and stderr of all processes and (2) create tty for processs stdout.

Those are really bad defaults. The tty on stdout means many programs run in "interactive" rather then "batch" mode: programs which use pager get output truncated, auto-colors may get enabled and emit ESC controls into output streams (or not, depending on user's distro... fun!). And captured stderr means warnings and progress messages just disappear.

For example, this hangs forever without any output, at least if executed from interactive terminal:

    from sh import man
    print(man("tty"))
Compare to "subprocess" which does the right thing and returns manpage as a string:

    import subprocess
    subprocess.check_output(["man", "tty"], text=True)
Can you fix "sh"? sure, you need to bake in option to disable tty. But you've got to do it in _every_ script, or you'll see failure sooner or later. So it's much easier, not to mention safer, to simply use "subprocess". And as a bonus, one less dependency!

(Fun fact: back when "sh" first appeared, everyone was using "git log" as an example of why tty was bad (it was silently truncating data). They fixed it.. by disabling tty only for "git" command. So my example uses "man" :) )

Re: Using Python for Scripting

#76

Post by Lvl999Noob »

theomega wrote: post #52
rjmill wrote: post #17

Odd, I don't see any mention of subprocess.run, the workhorse of python scripting. Quick rundown for the unfamiliar: Give it a command as a list of strings (e.g., subprocess.run(["echo", "foo"]).) It takes a bunch of flags, but the most useful (but not immediately obvious) ones are: check=True: Raise an error if the command fails capture_output=True: Captures stdout/stderr on the CompletedProcess text=True: Automatic…

One thing I can recommend that makes scripting in python with external commands a lot easier is the `sh` module: https://pypi.org/project/sh/ Basically you can just `from sh import [command]` and then have an installed binary command available as function from sh import ifconfig print(ifconfig("eth0"))

And for the opposite, where you keep your main pipeline in shell but want to use python for some parts of it, there is pypyp.

https://pypi.org/project/pypyp/

It takes cares of the input and output boilerplate so you can focus on the actual code that you wanted python for.

    > seq 1 5 | pyp 'sum(map(int, lines))'
    > ls | pyp 'Path(x).suffix'

Re: Using Python for Scripting

#78

Post by zelphirkalt »

theomega wrote: post #52
rjmill wrote: post #17

Odd, I don't see any mention of subprocess.run, the workhorse of python scripting. Quick rundown for the unfamiliar: Give it a command as a list of strings (e.g., subprocess.run(["echo", "foo"]).) It takes a bunch of flags, but the most useful (but not immediately obvious) ones are: check=True: Raise an error if the command fails capture_output=True: Captures stdout/stderr on the CompletedProcess text=True: Automatic…

One thing I can recommend that makes scripting in python with external commands a lot easier is the `sh` module: https://pypi.org/project/sh/ Basically you can just `from sh import [command]` and then have an installed binary command available as function from sh import ifconfig print(ifconfig("eth0"))

Does this give live output (meaning before completion) of processes run?

Re: Using Python for Scripting

#79

Post by zelphirkalt »

sevensor wrote: post #16

The Python stdlib does not get enough credit. People complain about things like how its http client is dated and slow, but it’s pretty amazing that it’s just right there if you need it, no external dependencies needed. And it’s sitting right next to difflib, graphlib, pathlib, struct, glob, tkinter, and dozens of others. Sure, every one of these is limited individually, but those limitations are stable and well under…

Currently developing a tkinter app, and loving, that I don't have to install an additional GUI framework. This will be much easier to package than with GTK or QT or Pyside or something. I am sure people have figured out all of that, but my project has so minimal dependencies, and yet offers a full GUI.

Re: Using Python for Scripting

#80

Post by zelphirkalt »

Alifatisk wrote: post #61

I like the message the article is trying to convey, Python is good alternative to complicated shell scripts in my opinion. I do wonder, let's say the scripting file is using lots of libraries, do you have to include some kind of requirements.txt file with it aswell when you want to share it with other people? In Ruby, there is inline bundler which makes sharing a single Ruby script very portable. https://bundler.io/g…

One would be well-advised to provide not only a requirements.txt file, but also a lock file, if the use case is important enough.
Post reply on HN