Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hints for writing Unix tools

monkey.org

Post reply on HN
Print view View original

31–40 of 131 posts

Re: Hints for writing Unix tools

#31

Post by dap »

burke wrote: post #28
dap wrote: post #27

Earlier quoted context omitted.

IMO, this is an anti-pattern. It's violates the principle of least surprise. (How come I see X when I run the command, but I can't grep for X in its output? How come it works when I run it from my interactive shell, but it's broken when I run it from a script? And things like that.)

I think it depends what sort of things you use it for. I often use it to switch on or off ANSI colourization, which doesn't really violate the principle of least surprise. When used sparingly and thoughtfully, I've never personally had an issue with it.

Good point, and it makes me even more convinced that all rules of thumb have important exceptions. In fact, I've also used tools that use ANSI colorization, and disable that when not talking to a tty, and wished that they wouldn't because I was piping them to "less -R" :) (At least that's a graceful failure mode, and it's pretty clear what probably happened even if one doesn't exactly understand the details.)

Re: Hints for writing Unix tools

#32

Post by grosskur »

seanp2k2 wrote: post #8
voltagex_ wrote: post #2

I'm not sure I agree with the "no JSON, please" remark. If I'm parsing normal *nix output I'm going to have to use sed, grep, awk, cut or whatever and the invocation is probably going to be different for each tool. If it's JSON and I know what object I want, I just have to pipe to something like jq [1]. PowerShell takes this further and uses the concept of passing objects around - so I can do things like ls | $_.Name…

+1 for jq. A lot of my work these days involves using web APIs in addition to "local" ones from CLI tools. xpath was good for dealing with XML stuff in a similar fashion, and HTML-XML-utils is an awesome suite of CLI things for slicing and dicing, if you're into that sort of thing: http://www.maketecheasier.com/manipulate-html-and-xml-files-...

xmlstarlet is also handy for command-line XML parsing:

http://xmlstar.sourceforge.net/

Re: Hints for writing Unix tools

#33

Post by pimlottc »

userbinator wrote: post #16
acabal wrote: post #3

Great article. The other thing I've always wished for command-line tools is some kind of consistency for flags and arguments. Kind of like a HIG for the command line. I know some distros have something like this, and that it's not practical to do as many common commands evolved decades ago and changing the interface would break pretty much everything. But things like `grep -E,--extended-regexp` vs `sed -r,--regexp-ex…

Actually most of them are quite consistent since POSIX published guidelines for it - and the only inconsistencies are historical exceptions: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_... (I'm not so convinced that long options are a good thing, as evidenced by the --extended-regexp/--regexp-extended and other little "was it spelt this way or that?" type of confusions. It's not hard to remember singl…

Long options are very nice to use in scripts as they are somewhat self-documenting. Compare:

    curl -kLIiso example.org www.example.org
versus:

    curl --insecure --location --head --include --silent --output example.org www.example.rog
And of course as a practical matter, with short opts you'll run out of characters eventually, and meaningful mnemonics before that.

Re: Hints for writing Unix tools

#34

Post by grosskur »

Animats wrote: post #14

1978 called. It wants its pipes back. That approach dates from the days when you got multi-column directory listings with ls | mc Putting multi-column output code in "ls" wasn't consistent with the UNIX philosophy. There's a property of UNIX program interconnection that almost nobody thinks about. You can feed named environment variables into a program, but you can't get them back out when the program exits. This is…

You can simulate this with so-called "Bernstein chaining". Basically, each program takes another program as an argument, and finishes by calling exec() on it rather than exit(), which preserves the environment. See:

http://www.catb.org/~esr/writings/taoup/html/ch06s06.html

Or write environment variables to stdout in Bourne shell syntax so the caller call run "eval" on it. Like ssh-agent, for example.

Re: Hints for writing Unix tools

#35

Post by 4ad »

Animats wrote: post #14

1978 called. It wants its pipes back. That approach dates from the days when you got multi-column directory listings with ls | mc Putting multi-column output code in "ls" wasn't consistent with the UNIX philosophy. There's a property of UNIX program interconnection that almost nobody thinks about. You can feed named environment variables into a program, but you can't get them back out when the program exits. This is…

In Plan 9 programs return strings instead of numeric codes.

Re: Hints for writing Unix tools

#36

Post by pimlottc »

osandov wrote: post #25

A nitpicky tip: --help is normal execution, not an error, so the usage information should be printed to stdout, not stderr (and it should exit with a successful status). Nothing is more annoying than trying to use a convoluted program with a million flags (which should have a man page in the first place) and piping --help into less with no success.

This annoys me to no end. Of course, you can work around it:

    annoying_program 2>&1 | less
but it is very unfriendly to stymie a user's attempt to get help when they're already probably confused.

Re: Hints for writing Unix tools

#37

Post by gohrt »

dap wrote: post #27
jzwinck wrote: post #12

Here's one more tip: did you ever notice that "ls" displays multiple columns, but "ls | cat" prints only one filename per line? Or how "ps -f" truncates long lines instead of wrapping, while "ps -f | cat" lets the long lines live? You can do it too, and if you're serious about writing Unix-style filter programs, you will someday need to. How do you know which format to write? Call "isatty(STDOUT_FILENO)" in C or C++,…

IMO, this is an anti-pattern. It's violates the principle of least surprise. (How come I see X when I run the command, but I can't grep for X in its output? How come it works when I run it from my interactive shell, but it's broken when I run it from a script? And things like that.)

This behavior of Unix programs is basically the same concept as Perl's "context" (list vs scalar), but even moreso.

Re: Hints for writing Unix tools

#38

Post by agumonkey »

grosskur wrote: post #34
Animats wrote: post #14

1978 called. It wants its pipes back. That approach dates from the days when you got multi-column directory listings with ls | mc Putting multi-column output code in "ls" wasn't consistent with the UNIX philosophy. There's a property of UNIX program interconnection that almost nobody thinks about. You can feed named environment variables into a program, but you can't get them back out when the program exits. This is…

You can simulate this with so-called "Bernstein chaining". Basically, each program takes another program as an argument, and finishes by calling exec() on it rather than exit(), which preserves the environment. See: http://www.catb.org/~esr/writings/taoup/html/ch06s06.html Or write environment variables to stdout in Bourne shell syntax so the caller call run "eval" on it. Like ssh-agent, for example.

Oh wow, unix continuation passing style. Never heard of that o_o;

Re: Hints for writing Unix tools

#39

Post by gohrt »

grosskur wrote: post #34
Animats wrote: post #14

1978 called. It wants its pipes back. That approach dates from the days when you got multi-column directory listings with ls | mc Putting multi-column output code in "ls" wasn't consistent with the UNIX philosophy. There's a property of UNIX program interconnection that almost nobody thinks about. You can feed named environment variables into a program, but you can't get them back out when the program exits. This is…

You can simulate this with so-called "Bernstein chaining". Basically, each program takes another program as an argument, and finishes by calling exec() on it rather than exit(), which preserves the environment. See: http://www.catb.org/~esr/writings/taoup/html/ch06s06.html Or write environment variables to stdout in Bourne shell syntax so the caller call run "eval" on it. Like ssh-agent, for example.

Continuation Passing Style! http://en.wikipedia.org/wiki/Continuation-passing_style

Re: Hints for writing Unix tools

#40

Post by burke »

dap wrote: post #31
burke wrote: post #28

Earlier quoted context omitted.

I think it depends what sort of things you use it for. I often use it to switch on or off ANSI colourization, which doesn't really violate the principle of least surprise. When used sparingly and thoughtfully, I've never personally had an issue with it.

Good point, and it makes me even more convinced that all rules of thumb have important exceptions. In fact, I've also used tools that use ANSI colorization, and disable that when not talking to a tty, and wished that they wouldn't because I was piping them to "less -R" :) (At least that's a graceful failure mode, and it's pretty clear what probably happened even if one doesn't exactly understand the details.)

I've always thought it would be nice to have a utility like cat that I could pipe commands to, which would trick them into thinking all their streams were attached to a tty, so you could do "uses_colours | ttycat | less -R".

I'm sure it's possible, but you'd have to acquire a new pty and decide what termios settings you want. It's a nontrivial hack, I think.

I'm actually kind of surprised it's not in moreutils[1].

[1]: https://joeyh.name/code/moreutils/

EDIT: Hmm, maybe it's not possible. I can't figure out exactly how to do it, anyway.

EDIT again: Apparently `script` does this on Linux: http://monosnap.com/image/Qlig4CHmQgV9pxvSmndVUgMTU88Adz

EDIT again: Expect's `unbuffer` also does it: http://expect.sourceforge.net/example/unbuffer.man.html#toc

Post reply on HN