Live data from Hacker News

Hints for writing Unix tools

monkey.org

21–30 of 131 posts

Re: Hints for writing Unix tools

#21

I'm not sure I agree with the "no JSON, please" remark. If I'm parsing normal *nix output I'm going to have to use sed, grep, awk, cut or whatever and the invocation is probably going to be different for each tool. If it's JSON and I know what object I want, I just have to pipe to something like jq [1]. PowerShell takes this further and uses the concept of passing objects around - so I can do things like ls | $_.Name…

jq looks nice, I use another similar tool quite a lot [1].

[1]: https://github.com/trentm/json

Re: Hints for writing Unix tools

#22
post #12

Here's one more tip: did you ever notice that "ls" displays multiple columns, but "ls | cat" prints only one filename per line? Or how "ps -f" truncates long lines instead of wrapping, while "ps -f | cat" lets the long lines live? You can do it too, and if you're serious about writing Unix-style filter programs, you will someday need to. How do you know which format to write? Call "isatty(STDOUT_FILENO)" in C or C++,…

Or, execute "/bin/[ -t 1" (or "test -t 1", or "[[ -t 1 ]]", or ...). This is handy in shellscripts (obviously), but also in languages like Go, which lack a builtin way to test whether stdout is a TTY. e.g.:

    cmd := exec.Command("/bin/[", "-t", "1")
    cmd.Stdout = os.Stdout
    isatty := nil == cmd.Run()

Re: Hints for writing Unix tools

#23
post #15

Earlier quoted context omitted.

> newline-separated list of unescaped file names That breaks when you have newlines in filenames, no?

> That breaks when you have newlines in filenames, no? That seems like an extremely pathological case.

> That seems like an extremely pathological case.

When a human is creating files by hand, I almost certainly agree. When a program is creating files, however, it's only a matter of time before weird characters wind their way in there.

I really wish newlines had been disallowed. (There's UI implications, in addition to the parsing ones — how do you do a list view with newlines in the filename?; I also wish filenames had a reliable character set and weren't just bytes.)

Re: Hints for writing Unix tools

#24
post #7

Additional tip: if writing a tool that prints a list of file names, provide a -0 option that prints them separated by '\x0' rather than white space. Then the output can be piped through xargs -0 and it won't go wrong if there are files with spaces in their paths. I suggest -0 for symmetry with xargs. find calls it -print0, I think. (In my view, this is poor design on xargs's part; it should be reading a newline-separ…

> newline-separated list of unescaped file names That breaks when you have newlines in filenames, no?

And \x0 separator breaks when you have \x0 in filenames. Pragmatically it's a question of rarity, but ultimately the shell should support something like prepared queries in SQL.

Re: Hints for writing Unix tools

#25
A nitpicky tip: --help is normal execution, not an error, so the usage information should be printed to stdout, not stderr (and it should exit with a successful status). Nothing is more annoying than trying to use a convoluted program with a million flags (which should have a man page in the first place) and piping --help into less with no success.

Re: Hints for writing Unix tools

#26
post #24

Earlier quoted context omitted.

> newline-separated list of unescaped file names That breaks when you have newlines in filenames, no?

And \x0 separator breaks when you have \x0 in filenames. Pragmatically it's a question of rarity, but ultimately the shell should support something like prepared queries in SQL.

Filenames do not contain nulls: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_...

Re: Hints for writing Unix tools

#27
post #12

Here's one more tip: did you ever notice that "ls" displays multiple columns, but "ls | cat" prints only one filename per line? Or how "ps -f" truncates long lines instead of wrapping, while "ps -f | cat" lets the long lines live? You can do it too, and if you're serious about writing Unix-style filter programs, you will someday need to. How do you know which format to write? Call "isatty(STDOUT_FILENO)" in C or C++,…

IMO, this is an anti-pattern. It's violates the principle of least surprise. (How come I see X when I run the command, but I can't grep for X in its output? How come it works when I run it from my interactive shell, but it's broken when I run it from a script? And things like that.)

Re: Hints for writing Unix tools

#28
post #27
post #12

Here's one more tip: did you ever notice that "ls" displays multiple columns, but "ls | cat" prints only one filename per line? Or how "ps -f" truncates long lines instead of wrapping, while "ps -f | cat" lets the long lines live? You can do it too, and if you're serious about writing Unix-style filter programs, you will someday need to. How do you know which format to write? Call "isatty(STDOUT_FILENO)" in C or C++,…

IMO, this is an anti-pattern. It's violates the principle of least surprise. (How come I see X when I run the command, but I can't grep for X in its output? How come it works when I run it from my interactive shell, but it's broken when I run it from a script? And things like that.)

I think it depends what sort of things you use it for. I often use it to switch on or off ANSI colourization, which doesn't really violate the principle of least surprise.

When used sparingly and thoughtfully, I've never personally had an issue with it.

Re: Hints for writing Unix tools

#29
Lots of great points here, but as always, these can be taken too far. Header lines are really useful for human-readable output, and can be easily skipped with an optional flag. (-H is common for this).

The "portable output" thing is especially subjective. I buy that it probably makes sense for compilers to print full paths. But it's nice that tools like ls(1) and find(1) use paths in the same form you gave them on the command-line (i.e., absolute pathnames in output if given absolute paths, but relative pathnames if given relative paths). For one, it means that when you provide instructions to someone (e.g., a command to run on a cloned git repo), and you want to include sample output, the output matches exactly what they'd see. Similarly, it makes it easier to write test suites that check for expected stdout contents. And if you want absolute paths in the output, you can specify the input that way.

Re: Hints for writing Unix tools

#30

I'm not sure I agree with the "no JSON, please" remark. If I'm parsing normal *nix output I'm going to have to use sed, grep, awk, cut or whatever and the invocation is probably going to be different for each tool. If it's JSON and I know what object I want, I just have to pipe to something like jq [1]. PowerShell takes this further and uses the concept of passing objects around - so I can do things like ls | $_.Name…

On Mac (OS NeXT, perhaps?), the convention seems to be that most commands produce human readable output by default, but you can pass a parameter like -x or -xml to get (usually) XML, machine-readable output, and with some tools, -j or -json will give you that format. But then you've oddities like plutil behaving like gzip by modifying the file you specify rather than printing to stdout. You have to pass -o and a dash…

Not sure if its what you had in mind for Windows and tabs but I've found ConsoleZ [1] quite nice and allows powershell, cmd and others to have tabs.

[1] https://github.com/cbucher/console

Post reply on HN