Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hints for writing Unix tools

monkey.org

Post reply on HN
Print view View original

41–50 of 131 posts

Re: Hints for writing Unix tools

#41

Post by dnlrn »

burke wrote: post #40
dap wrote: post #31

Earlier quoted context omitted.

Good point, and it makes me even more convinced that all rules of thumb have important exceptions. In fact, I've also used tools that use ANSI colorization, and disable that when not talking to a tty, and wished that they wouldn't because I was piping them to "less -R" :) (At least that's a graceful failure mode, and it's pretty clear what probably happened even if one doesn't exactly understand the details.)

I've always thought it would be nice to have a utility like cat that I could pipe commands to, which would trick them into thinking all their streams were attached to a tty, so you could do "uses_colours | ttycat | less -R". I'm sure it's possible, but you'd have to acquire a new pty and decide what termios settings you want. It's a nontrivial hack, I think. I'm actually kind of surprised it's not in moreutils[1]. [1…

It’s not possible in the way you think, because the two programs don’t know about each other. All the pipe character does is put the stdout into the stdin of the other program.

So this feature must actually be present in the shell and maybe it is. I’m no expert but maybe zsh already offers something like this?

Re: Hints for writing Unix tools

#42

Post by tracker1 »

lstamour wrote: post #18
voltagex_ wrote: post #2

I'm not sure I agree with the "no JSON, please" remark. If I'm parsing normal *nix output I'm going to have to use sed, grep, awk, cut or whatever and the invocation is probably going to be different for each tool. If it's JSON and I know what object I want, I just have to pipe to something like jq [1]. PowerShell takes this further and uses the concept of passing objects around - so I can do things like ls | $_.Name…

On Mac (OS NeXT, perhaps?), the convention seems to be that most commands produce human readable output by default, but you can pass a parameter like -x or -xml to get (usually) XML, machine-readable output, and with some tools, -j or -json will give you that format. But then you've oddities like plutil behaving like gzip by modifying the file you specify rather than printing to stdout. You have to pass -o and a dash…

I've been digging conemu[1]

[1]: https://code.google.com/p/conemu-maximus5/

Re: Hints for writing Unix tools

#43

Post by userbinator »

dap wrote: post #27
jzwinck wrote: post #12

Here's one more tip: did you ever notice that "ls" displays multiple columns, but "ls | cat" prints only one filename per line? Or how "ps -f" truncates long lines instead of wrapping, while "ps -f | cat" lets the long lines live? You can do it too, and if you're serious about writing Unix-style filter programs, you will someday need to. How do you know which format to write? Call "isatty(STDOUT_FILENO)" in C or C++,…

IMO, this is an anti-pattern. It's violates the principle of least surprise. (How come I see X when I run the command, but I can't grep for X in its output? How come it works when I run it from my interactive shell, but it's broken when I run it from a script? And things like that.)

I agree, especially for the behaviour from the parent:

"ps -f" truncates long lines instead of wrapping, while "ps -f | cat" lets the long lines live

How people usually discover what these commands do is by running them interactively, and if that results in some output being hidden vs being run noninteractively, then they have little reason to believe that it could yield more output than what they're used to seeing. I think a certain number of "ps" users don't know it can display full paths and commands, if they've only ever used it interactively.

Re: Hints for writing Unix tools

#44

Post by vog »

dap wrote: post #27
jzwinck wrote: post #12

Here's one more tip: did you ever notice that "ls" displays multiple columns, but "ls | cat" prints only one filename per line? Or how "ps -f" truncates long lines instead of wrapping, while "ps -f | cat" lets the long lines live? You can do it too, and if you're serious about writing Unix-style filter programs, you will someday need to. How do you know which format to write? Call "isatty(STDOUT_FILENO)" in C or C++,…

IMO, this is an anti-pattern. It's violates the principle of least surprise. (How come I see X when I run the command, but I can't grep for X in its output? How come it works when I run it from my interactive shell, but it's broken when I run it from a script? And things like that.)

Ideed. I've seen people over and over stumbling over this weird behaviour.

It may have some merits, but as a general advice this is definitely an anti-pattern.

Another example is "curl", where "curl URL >outfile" is chatty on stderr, while "curl URL" is quiet on stderr. That's very annoying for scripting, you easily forget to set "-s" in your scripts due to that behaviour.

Re: Hints for writing Unix tools

#45

Post by zaptheimpaler »

dap wrote: post #29

Lots of great points here, but as always, these can be taken too far. Header lines are really useful for human-readable output, and can be easily skipped with an optional flag. (-H is common for this). The "portable output" thing is especially subjective. I buy that it probably makes sense for compilers to print full paths. But it's nice that tools like ls(1) and find(1) use paths in the same form you gave them on th…

I also think headers should be included. Its really annoying to go pore through a man page just to see what the columns mean. You could use flags, or maybe send headers to STDERR.

Re: Hints for writing Unix tools

#46

Post by hoggle »

“One thing well” misses the point: it should be “One thing well AND COMPOSES WELL”

If the implementation isn't respecting The Rule of Composition it's actually not adhering to the Unix philosophy in the first place. The tweet is referring to one of Doug McIlroy's (one of the Unix founders, inventor of the Unix pipe) famous quotes:

"This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface."

Pure beauty, but it's almost too concise a definition if you haven't experienced the culture of Unix (many years of usage / reading code / writing code / communication with other followers). ESR's exhaustive list of Unix rules in plain English might be a better start for the uninitiated (among which one will find the aforementioned Rule of Composition).

For all those seeking enlightenment, go forth and read the The Art of Unix Programming:

https://en.wikipedia.org/wiki/The_Art_of_Unix_Programming

17 Unix Rules:

https://en.wikipedia.org/wiki/Unix_philosophy#Eric_Raymond.E...

Re: Hints for writing Unix tools

#48

Post by reirob »

grosskur wrote: post #32
seanp2k2 wrote: post #8

Earlier quoted context omitted.

+1 for jq. A lot of my work these days involves using web APIs in addition to "local" ones from CLI tools. xpath was good for dealing with XML stuff in a similar fashion, and HTML-XML-utils is an awesome suite of CLI things for slicing and dicing, if you're into that sort of thing: http://www.maketecheasier.com/manipulate-html-and-xml-files-...

xmlstarlet is also handy for command-line XML parsing: http://xmlstar.sourceforge.net/

Agree, had to use it extensively several years ago, it saved me so much time. But even then the development seemed to have stopped on this tool.

Re: Hints for writing Unix tools

#49

Post by teddyh »

dap wrote: post #27
jzwinck wrote: post #12

Here's one more tip: did you ever notice that "ls" displays multiple columns, but "ls | cat" prints only one filename per line? Or how "ps -f" truncates long lines instead of wrapping, while "ps -f | cat" lets the long lines live? You can do it too, and if you're serious about writing Unix-style filter programs, you will someday need to. How do you know which format to write? Call "isatty(STDOUT_FILENO)" in C or C++,…

IMO, this is an anti-pattern. It's violates the principle of least surprise. (How come I see X when I run the command, but I can't grep for X in its output? How come it works when I run it from my interactive shell, but it's broken when I run it from a script? And things like that.)

Indeed, the GNU Coding Standards explicitly argues against doing that:

“please don’t make the behavior of a command-line program depend on the type of output device it gets as standard output or standard input.”¹

① https://www.gnu.org/prep/standards/standards.html#User-Inter...

Re: Hints for writing Unix tools

#50

Post by oakwhiz »

dap wrote: post #27
jzwinck wrote: post #12

Here's one more tip: did you ever notice that "ls" displays multiple columns, but "ls | cat" prints only one filename per line? Or how "ps -f" truncates long lines instead of wrapping, while "ps -f | cat" lets the long lines live? You can do it too, and if you're serious about writing Unix-style filter programs, you will someday need to. How do you know which format to write? Call "isatty(STDOUT_FILENO)" in C or C++,…

IMO, this is an anti-pattern. It's violates the principle of least surprise. (How come I see X when I run the command, but I can't grep for X in its output? How come it works when I run it from my interactive shell, but it's broken when I run it from a script? And things like that.)

I have run into this problem when trying to automate certain tasks on UNIX boxes.

Dealing with programs that act differently depending on their output device is very annoying.

Post reply on HN