Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hints for writing Unix tools

monkey.org

Post reply on HN
Print view View original

11–20 of 131 posts

Re: Hints for writing Unix tools

#11

Post by jzwinck »

acabal wrote: post #3

Great article. The other thing I've always wished for command-line tools is some kind of consistency for flags and arguments. Kind of like a HIG for the command line. I know some distros have something like this, and that it's not practical to do as many common commands evolved decades ago and changing the interface would break pretty much everything. But things like `grep -E,--extended-regexp` vs `sed -r,--regexp-ex…

You're right, myriad popular tools are not totally consistent (ls -h and du -h are similar but grep -h is very different). There is a bit of hope however--the GNU folks have documented lots of the options currently in use so you can try to find one that fits when you build new tools: https://www.gnu.org/prep/standards/html_node/Option-Table.ht...

Re: Hints for writing Unix tools

#12

Post by jzwinck »

Here's one more tip: did you ever notice that "ls" displays multiple columns, but "ls | cat" prints only one filename per line? Or how "ps -f" truncates long lines instead of wrapping, while "ps -f | cat" lets the long lines live?

You can do it too, and if you're serious about writing Unix-style filter programs, you will someday need to. How do you know which format to write? Call "isatty(STDOUT_FILENO)" in C or C++, "sys.stdout.isatty()" in Python, etc. This returns true if stdout is a terminal, in which case you can provide pretty output for humans and machine-readable output for programs, automatically.

Re: Hints for writing Unix tools

#13

Post by daxelrod »

acabal wrote: post #3

Great article. The other thing I've always wished for command-line tools is some kind of consistency for flags and arguments. Kind of like a HIG for the command line. I know some distros have something like this, and that it's not practical to do as many common commands evolved decades ago and changing the interface would break pretty much everything. But things like `grep -E,--extended-regexp` vs `sed -r,--regexp-ex…

One sort-of attempt at this is GNU's Coding Standards.

Long and Short Options: https://www.gnu.org/prep/standards/html_node/Option-Table.ht...

General Interfaces: https://www.gnu.org/prep/standards/html_node/User-Interfaces...

Command Line Interfaces: https://www.gnu.org/prep/standards/html_node/Command_002dLin...

Program Argument Syntax: http://www.gnu.org/software/libc/manual/html_node/Argument-S...

Re: Hints for writing Unix tools

#14

Post by Animats »

1978 called. It wants its pipes back.

That approach dates from the days when you got multi-column directory listings with

  ls | mc
Putting multi-column output code in "ls" wasn't consistent with the UNIX philosophy.

There's a property of UNIX program interconnection that almost nobody thinks about. You can feed named environment variables into a program, but you can't get them back out when the program exits. This is a lack. "exit()" should have taken an optional list of name/value pairs as an argument, and the calling program (probably a shell) should have been able to use them. With that, calling programs would be more like calling subroutines.

PowerShell does something like that.

Re: Hints for writing Unix tools

#15

Post by pstuart »

fragmede wrote: post #10
to3m wrote: post #7

Additional tip: if writing a tool that prints a list of file names, provide a -0 option that prints them separated by '\x0' rather than white space. Then the output can be piped through xargs -0 and it won't go wrong if there are files with spaces in their paths. I suggest -0 for symmetry with xargs. find calls it -print0, I think. (In my view, this is poor design on xargs's part; it should be reading a newline-separ…

> newline-separated list of unescaped file names That breaks when you have newlines in filenames, no?

> That breaks when you have newlines in filenames, no?

That seems like an extremely pathological case.

Re: Hints for writing Unix tools

#16

Post by userbinator »

acabal wrote: post #3

Great article. The other thing I've always wished for command-line tools is some kind of consistency for flags and arguments. Kind of like a HIG for the command line. I know some distros have something like this, and that it's not practical to do as many common commands evolved decades ago and changing the interface would break pretty much everything. But things like `grep -E,--extended-regexp` vs `sed -r,--regexp-ex…

Actually most of them are quite consistent since POSIX published guidelines for it - and the only inconsistencies are historical exceptions:

http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_...

(I'm not so convinced that long options are a good thing, as evidenced by the --extended-regexp/--regexp-extended and other little "was it spelt this way or that?" type of confusions. It's not hard to remember single letters, especially if they're mnemonic.)

Re: Hints for writing Unix tools

#17

Post by userbinator »

pstuart wrote: post #15
fragmede wrote: post #10

Earlier quoted context omitted.

> newline-separated list of unescaped file names That breaks when you have newlines in filenames, no?

> That breaks when you have newlines in filenames, no? That seems like an extremely pathological case.

Pathological or not, ensuring that pathnames can essentially contain any byte value except the 0 terminator, and it will still work, is important to prevent surprising behaviour which often has security implications.

Re: Hints for writing Unix tools

#18

Post by lstamour »

voltagex_ wrote: post #2

I'm not sure I agree with the "no JSON, please" remark. If I'm parsing normal *nix output I'm going to have to use sed, grep, awk, cut or whatever and the invocation is probably going to be different for each tool. If it's JSON and I know what object I want, I just have to pipe to something like jq [1]. PowerShell takes this further and uses the concept of passing objects around - so I can do things like ls | $_.Name…

On Mac (OS NeXT, perhaps?), the convention seems to be that most commands produce human readable output by default, but you can pass a parameter like -x or -xml to get (usually) XML, machine-readable output, and with some tools, -j or -json will give you that format.

But then you've oddities like plutil behaving like gzip by modifying the file you specify rather than printing to stdout. You have to pass -o and a dash to get it to leave the file alone and instead reformat it to stdout. That one gets me every time. And I'm not alone: https://twitter.com/mavcunha/status/417823730505895936

But other parts are nice. For instance, "system_profiler -xml > MyReport.spx" generates XML that will open in the System Profiler GUI app. The XML generated is usually a Plist, since that's as native to the platform as the Registry might be to Windows...

Let me know when PowerShell gets tabs though. Maybe there's a Terminal.app port running in Mono somewhere? Seriously, I wish somebody would build a better terminal, maybe get creative with scrollback and chaining commands, and ship it in an OS... with tabs. ;-)

Re: Hints for writing Unix tools

#19

Post by wahern »

to3m wrote: post #7

Additional tip: if writing a tool that prints a list of file names, provide a -0 option that prints them separated by '\x0' rather than white space. Then the output can be piped through xargs -0 and it won't go wrong if there are files with spaces in their paths. I suggest -0 for symmetry with xargs. find calls it -print0, I think. (In my view, this is poor design on xargs's part; it should be reading a newline-separ…

xargs assumes the input is composed of quoted and escaped atoms. Compare

$ printf '"foo bar"' | xargs -n1

and

$ printf '"foo" "bar"' | xargs -n1

and

$ printf "%s" '\\"foo bar\\"' | xargs -n1

Re: Hints for writing Unix tools

#20

Post by myhf »

userbinator wrote: post #17
pstuart wrote: post #15

Earlier quoted context omitted.

> That breaks when you have newlines in filenames, no? That seems like an extremely pathological case.

Pathological or not, ensuring that pathnames can essentially contain any byte value except the 0 terminator, and it will still work, is important to prevent surprising behaviour which often has security implications.

too pathological; didn't implement
Post reply on HN